1
00:00:09,240 --> 00:00:11,400
Hi, so my name is Tim Yardley.

2
00:00:11,400 --> 00:00:17,600
I'm here to talk about some work that I've done under DARPA RADIX program.

3
00:00:20,550 --> 00:00:22,510
Whoa, I'm getting audio from somewhere.

4
00:00:22,510 --> 00:00:23,210
Hold on.

5
00:00:35,120 --> 00:00:36,480
All right, sorry about that.

6
00:00:36,480 --> 00:00:38,600
I was getting feedback from the live channel.

7
00:00:39,420 --> 00:00:49,320
So I am here to talk about building a cyber physical testbed and how we've used that to support black start restoration under cyber fire.

8
00:00:49,320 --> 00:00:52,640
So this has been part of a DARPA RADIX program.

9
00:00:52,640 --> 00:00:57,240
The RADIX program is Rapid Attack Detection Isolation and Characterization Systems.

10
00:00:57,240 --> 00:00:59,760
And I'll talk a little bit about what that is here in a minute.

11
00:01:01,300 --> 00:01:15,080
So just for sort of coverage and statement, the testbed work that I'm going to talk about was funded under the DARPA's Rapid Attack Detection Isolation and Characterization Systems program.

12
00:01:15,080 --> 00:01:16,960
I work for the University of Illinois.

13
00:01:16,960 --> 00:01:22,680
I'm the principal investigator of the University of Illinois' effort under there.

14
00:01:22,680 --> 00:01:30,060
We also had some government SME support, subject matter expert support from Idaho National Lab.

15
00:01:30,060 --> 00:01:35,240
And then the program evaluator is a company called Probatech.

16
00:01:35,240 --> 00:01:46,900
And all three of those organizations have been critical to the work that I'm talking about in terms of enabling the black star restoration efforts and the validation of the work itself.

17
00:01:47,020 --> 00:01:50,760
So let's talk a little bit, give you a little background on me.

18
00:01:50,760 --> 00:01:52,800
So my name is Tim Yardley.

19
00:01:52,800 --> 00:01:58,120
I'm a principal research scientist at the Information Trust Institute at the University of Illinois.

20
00:01:58,220 --> 00:02:04,160
And I am also a father, a husband, and broadly a researcher.

21
00:02:04,160 --> 00:02:10,420
I've been doing work in industrial control systems for about 14 years now.

22
00:02:10,420 --> 00:02:19,180
And I've been doing work in security for, geez, probably almost 30 years now across the board.

23
00:02:19,280 --> 00:02:38,020
My background came from sort of the think tank IRC eras back in the early days of FNET groups that that, let's say, like to explore and like to figure out things, but also cause some havoc in the process.

24
00:02:38,020 --> 00:02:44,620
I'm one of the original members of WooWoo and a variety of other security think tanks over the years.

25
00:02:45,000 --> 00:02:52,160
And I've sort of been in the realm of computer security for quite some time.

26
00:02:52,160 --> 00:02:55,800
I've been working in academia for about the past 12 years.

27
00:02:55,800 --> 00:03:00,620
And prior to that, I worked in industry supporting a variety of different efforts.

28
00:03:00,860 --> 00:03:05,780
So the DARPA RADIX effort itself, let's talk a little bit about what it is.

29
00:03:05,780 --> 00:03:12,180
So the program is designed to build technologies that fill a gap.

30
00:03:12,180 --> 00:03:28,600
And that gap really is the notion that if we are attacked by nation states, we are ill-prepared to be able to determine exactly what happened, where it happened, and how it happened, and get rid of it as fast as possible.

31
00:03:28,980 --> 00:03:42,840
So DARPA stood up this program to build technology that advanced that state of, let's say, preparedness, and then to evaluate that technology on a realistic facility.

32
00:03:42,840 --> 00:03:47,640
And so our role in that program was to build that realistic facility.

33
00:03:48,220 --> 00:03:52,480
We'll talk in more detail about the objectives of the program itself.

34
00:03:52,680 --> 00:03:58,220
So these are all public slides from DARPA that are here.

35
00:03:58,220 --> 00:04:07,120
But the key objective of the program itself is to enable black start recovery of the power grid amidst a cyber attack on the energy sector's critical infrastructure.

36
00:04:07,120 --> 00:04:12,020
In this particular case, the critical infrastructure is the electric power grid.

37
00:04:12,020 --> 00:04:21,280
So in a prevention a sort of state, you would say, okay, well, let's defend and detect what's going on.

38
00:04:21,280 --> 00:04:25,200
But this program starts from the adversary has been successful.

39
00:04:25,200 --> 00:04:32,200
So you have a complete blackout, no power anywhere, and you have to figure out exactly what happened, how it happened, etc.

40
00:04:33,640 --> 00:04:47,820
So the devices that are involved, you have to figure out what devices can be trusted, because you don't know what's compromised and what is safe or what is operating per norm.

41
00:04:47,880 --> 00:04:55,760
And many of the physical infrastructures are effectively controlled by intelligent devices.

42
00:04:55,760 --> 00:05:05,160
So these are the ICS devices that you have to explore on, let's say, the cyber side of the equation that are controlling the physical aspects of the grid.

43
00:05:06,320 --> 00:05:10,000
And these assets are spread throughout the country.

44
00:05:10,000 --> 00:05:12,380
So how can you do this in a scalable way?

45
00:05:12,380 --> 00:05:17,020
How can you deeply and forensically poke at these embedded devices?

46
00:05:17,160 --> 00:05:26,240
How can you figure out exactly what is trustable, what is not trustable, what was attacked, how it was attacked, and then get rid of it as fast as possible.

47
00:05:26,240 --> 00:05:31,620
The goal is to do so across the entire United States within seven days.

48
00:05:31,620 --> 00:05:40,480
And that's to isolate, characterize, and restore any crank paths necessary to bringing the grid for the United States back online.

49
00:05:42,360 --> 00:05:45,820
So let's talk about the manifestation of this.

50
00:05:45,820 --> 00:06:04,300
So in the first year or so, year and a half of the program, we built exercise environments that were run at the University of Illinois that were getting people's feet wet and getting the technology ready to explore, let's say, the beginning edges of the problem space.

51
00:06:04,460 --> 00:06:10,980
And as we evolved both the technology and the program, we had to turn the corner.

52
00:06:10,980 --> 00:06:19,440
So people will only believe what happens in a lab to the extent that it's, oh, that was in a lab environment.

53
00:06:19,440 --> 00:06:22,560
Oh, that's fine if you do it just on science.

54
00:06:22,560 --> 00:06:24,540
But that's not the real world.

55
00:06:24,540 --> 00:06:29,420
So we took it to a federal island, which is called Plum Island.

56
00:06:29,420 --> 00:06:42,960
It's the home of the Animal Disease Center, controlled and owned by DHS, S&T, and is a former siting of Fort Terry in the Spanish-American War and a bunch of other stuff.

57
00:06:42,960 --> 00:06:48,200
The size of the space that we're controlling there is roughly the size of Central Park.

58
00:06:48,220 --> 00:06:51,640
And we have built electrical infrastructure on that island.

59
00:06:51,640 --> 00:06:58,540
And so I'll talk about the most recent incarnation of it that was completed in November of 2019.

60
00:06:58,540 --> 00:07:01,400
That was the sixth exercise of the program.

61
00:07:01,400 --> 00:07:03,920
We have one more exercise still to do.

62
00:07:03,920 --> 00:07:08,100
COVID has postponed that a little bit, but still pending.

63
00:07:08,100 --> 00:07:10,000
So far has not been fully canceled.

64
00:07:10,440 --> 00:07:19,960
And we look to sort of do that last hurrah of the program and validate the advancements from the last exercise till now.

65
00:07:19,980 --> 00:07:27,040
So in the island infrastructure, we basically have built three full utilities.

66
00:07:27,040 --> 00:07:32,260
And those utilities are represented by gear that we'll talk about here in a moment.

67
00:07:32,360 --> 00:07:35,700
They have an emergency operations center for each utility.

68
00:07:35,700 --> 00:07:42,120
And then there's a regional coordinator that's trying to coordinate the actions amongst the different utilities as it goes through.

69
00:07:42,120 --> 00:07:45,040
We have the National Guard involved.

70
00:07:45,040 --> 00:08:05,520
We have the RADIX performers involved and lots of other entities that help establish communications in a variety of way and really set up what is a fairly austere environment from the beginning to enable the capture and execution of these exercises.

71
00:08:05,660 --> 00:08:09,360
So utility A has five low voltage substations.

72
00:08:09,360 --> 00:08:11,920
And I'll talk about what low voltage means to us.

73
00:08:11,920 --> 00:08:13,860
And one high voltage substation.

74
00:08:13,860 --> 00:08:20,520
It also had one generator, which is the crank path, in essence, that's getting to the high value assets.

75
00:08:20,520 --> 00:08:32,660
Utility B had seven low voltage substations along with three high voltage substations and that crank path had a critical national asset on it that needed to be up and maintained no matter what.

76
00:08:32,900 --> 00:08:36,320
Utility C was similar to utility A.

77
00:08:36,320 --> 00:08:41,380
It had five low voltage substations and one high voltage substation as well as one generator.

78
00:08:41,700 --> 00:08:47,760
And so each of these utilities differed in the physical topology and layout of the substations.

79
00:08:47,760 --> 00:08:52,300
They also differed in the equipment that was deployed across each of those substations.

80
00:08:53,700 --> 00:09:00,180
So the program is really broken into multiple different technical areas.

81
00:09:00,180 --> 00:09:04,540
You could look at it as four or five depending on how you want to count.

82
00:09:04,540 --> 00:09:06,940
The first area is situational awareness.

83
00:09:06,940 --> 00:09:18,260
And so when they deploy on this infrastructure that we've built, they are trying to figure out what happened or what is currently happening on the infrastructure to get as much situational awareness as you can provide.

84
00:09:18,260 --> 00:09:24,540
And the reason for that is you can't really trust the devices once they've been attacked to be telling you the right things.

85
00:09:24,540 --> 00:09:32,400
And you are in a blackout scenario, so you may not have visibility in much of the grid environment.

86
00:09:32,620 --> 00:09:35,440
Network isolation is technical area two.

87
00:09:35,440 --> 00:09:49,600
And technical area two is focused on taking the communications that are no longer necessarily trustworthy and expanding those into a realm of trustable or not.

88
00:09:49,600 --> 00:09:54,980
And so by that I mean point A to point B may have talked to each other before.

89
00:09:55,040 --> 00:09:59,580
And it used to be a dedicated private link or whatever it may be.

90
00:09:59,580 --> 00:10:06,360
But now traffic that's going across there is not reflecting what it did previously.

91
00:10:06,360 --> 00:10:08,040
So maybe somebody's man-in-the-middle-ing it.

92
00:10:08,040 --> 00:10:09,760
Maybe somebody is manipulating it.

93
00:10:09,760 --> 00:10:12,260
Maybe it's getting black-holed in some other way.

94
00:10:12,500 --> 00:10:14,220
Unknown operation, right?

95
00:10:14,220 --> 00:10:28,300
So how do you take the outputs from A and get it to B in a way that's trustworthy and secure and such that it cannot be modified in the middle or that it is evident when it is modified in the middle?

96
00:10:28,300 --> 00:10:33,040
So that's the area of research for technical area two.

97
00:10:33,040 --> 00:10:35,640
Technical area three is threat analysis.

98
00:10:35,640 --> 00:10:42,340
And they really are intended to do the forensic response per se on the devices itself.

99
00:10:42,340 --> 00:10:58,920
So how do you diagnose and remove cyber threats from the embedded devices, from the different pieces that are involved, et cetera, as you go through the investigation part of the environment?

100
00:10:58,940 --> 00:11:02,580
So the environment itself is technical area four.

101
00:11:02,580 --> 00:11:05,740
And that's conducted by us at the University of Illinois.

102
00:11:05,740 --> 00:11:29,680
And that is the environment by which the exercise happens, which is on Plum Island, but also the environment at the University of Illinois' campus that the performers remote into to build out their technology, to extend their capabilities, and to investigate the edge cases of how the grid operates when certain things happen to these devices.

103
00:11:30,700 --> 00:11:34,140
So that's part of our central facility, and I'll talk a little bit about that.

104
00:11:34,140 --> 00:11:41,140
And then the island is basically a distributed manifestation of that central facility at Illinois.

105
00:11:41,140 --> 00:11:44,380
The last technical area is technical area five.

106
00:11:44,380 --> 00:11:47,940
And effectively, that is the evaluators of the program.

107
00:11:47,940 --> 00:12:09,160
And they build out how to run an exercise in this space and how to determine whether or not progress is being made on the technology, what the appropriate challenge levels are, the, let's say, pain points, or how deep or how hard the red team pushes, et cetera,

108
00:12:09,160 --> 00:12:11,240
as we go through the environment.

109
00:12:11,240 --> 00:12:18,480
And they technically grade technical areas one, two, three, and four as they go through that.

110
00:12:19,980 --> 00:12:23,000
So on to the next slide.

111
00:12:23,320 --> 00:12:39,460
RADIX exercise six was really a move from strategic to the strategic notion early on in the program to operational notion with the utilities involved, et cetera, to a tactical deployment on top of that.

112
00:12:39,460 --> 00:12:49,680
So if you look up in the upper left-hand corner, the strategic notion is the concept that we talked about of you have a large-scale blackout and what you need to accomplish.

113
00:12:49,680 --> 00:12:55,060
The operational notion is the building and establishment of these crank paths.

114
00:12:55,060 --> 00:13:01,680
And then the tactical is let's execute on those crank paths and figure out exactly what happened and how it happened.

115
00:13:01,760 --> 00:13:12,160
So the picture on the bottom right at the moment is a drone footage sort of zoomed in on one of the substations.

116
00:13:12,160 --> 00:13:15,320
And you can see that they're built in shipping containers.

117
00:13:15,320 --> 00:13:20,520
And those shipping containers have gear inside them that I'll show you here in a moment.

118
00:13:20,540 --> 00:13:25,600
And then these containers are arranged and linked together to build the crank path.

119
00:13:25,600 --> 00:13:38,860
And we do that with basically above ground number two SO cord that has connectors on the end that we plug into to the individual gear that we have inside the boxes.

120
00:13:38,860 --> 00:13:40,680
And I'll show you what that looks like here in a moment.

121
00:13:42,460 --> 00:13:48,080
So you can see sort of a little bit of an edge here of what is inside a container.

122
00:13:48,080 --> 00:13:53,280
So this is standing inside a container looking out at the moment with the door open.

123
00:13:53,280 --> 00:13:57,060
So on the left-hand side there, you'll see what we call the relay box.

124
00:13:57,060 --> 00:14:06,440
On the right-hand side, you'll see what we call the power box, which is a skid-mounted Hoffman enclosure that controls the flow in essence of the substation.

125
00:14:06,440 --> 00:14:09,120
So that's the power grid aspect.

126
00:14:09,200 --> 00:14:16,840
And the controls of those grid components are in the relay box on the left-hand side.

127
00:14:16,840 --> 00:14:27,760
There's also some sensors that are up on top that are providing or that are performer technology, providing some of the situational awareness and attempt to determine ground truth as we go through the environment.

128
00:14:27,760 --> 00:14:30,500
And I'll zoom in on a lot of this as we talk.

129
00:14:30,740 --> 00:14:34,180
So I'm going to only talk about the testbed itself.

130
00:14:34,180 --> 00:14:44,260
I'm not going to talk about any of the performer technology that's specifically built, but I'll dig quite deeply into the testbed, which is my area of responsibility.

131
00:14:44,460 --> 00:15:00,280
So the mission that we set out to do is to provide realistic environments that enable this cutting-edge R&D that's not yet done by any commercial available product or any existing research off the shelf.

132
00:15:00,280 --> 00:15:10,500
And then we take and create this environment in a way that allows us to validate the effectiveness and frankly, the efficiency of those tools as we go through.

133
00:15:10,700 --> 00:15:18,540
So the goal of the program itself is really to take a generational leap forward in the capabilities of testbeds.

134
00:15:18,540 --> 00:15:28,480
So we've been building cyber-physical testbeds and leveraging the cyber-physical testbeds for about 13, 14 years now at the University of Illinois.

135
00:15:28,480 --> 00:15:38,100
By many accounts, we're sort of the gold standard in terms of capabilities across the nation and arguably the world.

136
00:15:38,100 --> 00:15:54,400
But even so, when we pitched our capabilities for this program, our proposal and the going in salve of our proposal was effectively, we have assembled the right team to solve this problem.

137
00:15:54,400 --> 00:16:09,620
But what technology exists today and where testbeds are today across all of them that you will encounter and anyone that bids, all of them are woefully inadequate to be able to actually go to the level of realism that will be necessary to validate these tools.

138
00:16:10,080 --> 00:16:22,960
And even with us, it is an extreme long shot as to whether or not this will be achievable in the time frame and advancing fast enough to be able to support this post-attack analysis.

139
00:16:23,040 --> 00:16:25,100
And so let me riff on that for a second.

140
00:16:25,160 --> 00:16:44,420
Cyber-physical testbeds before this program started were primarily focused on let's either build an environment where we're looking purely at a physical phenomenon, or let's build an environment that's proving out a hypothesis, physical or cyber, and look at it from that particular angle,

141
00:16:44,420 --> 00:16:46,740
sort of ignoring all of the other details.

142
00:16:46,800 --> 00:16:51,200
But in this program, everything is unknown coming in.

143
00:16:51,200 --> 00:16:53,160
You don't know what the attackers did to you.

144
00:16:53,160 --> 00:16:57,280
You don't know even what the attackers want to do to you on the environment.

145
00:16:57,280 --> 00:17:00,580
So you have to have every piece of it as real as possible.

146
00:17:00,580 --> 00:17:14,400
But you can't possibly go build, you know, three real crank paths and 24, 27 real substations out there because it just costs way too much money to do.

147
00:17:14,400 --> 00:17:16,520
And it's a dangerous environment to be in.

148
00:17:16,520 --> 00:17:31,740
So how can you minimize the environment, maximize the safety, and also maximize the realism without running into problems of naysayers with simulation being involved or emulation of devices, etc.

149
00:17:31,740 --> 00:17:46,840
So they need to be able to touch it, they need to be able to feel it, they need to be able to see it, and they need to be able to trust that what it does and how it works is going to be reflected or reflective of what happens in the real world.

150
00:17:47,240 --> 00:18:04,080
So the outcomes of the testbed work itself, obviously we've created a lot of tools, we've pioneered some new techniques and methodologies, we've combined existing solutions, both that we've had and that others have had, to build an environment together.

151
00:18:04,080 --> 00:18:28,300
And we combined that, not just the academic knowledge that we had at the University of Illinois, but in partnerships with key vendors and also with asset owners and operators, and to build really an environment that reflected not only the real world, but that took a whole leap forward on its ability to evaluate research.

152
00:18:28,480 --> 00:18:30,860
So what is a testbed?

153
00:18:30,860 --> 00:18:39,780
So a testbed really is somebody has a need, let's call them the customer, and that need is to evaluate something.

154
00:18:39,780 --> 00:18:44,920
And so a testbed is assets, the thing maybe that they want to evaluate on.

155
00:18:44,920 --> 00:18:52,460
It's the people with the knowledge on how to build that environment in the way that represents the scenario they need to look at, etc.

156
00:18:52,500 --> 00:18:55,840
What to capture, how to capture, where to capture it, etc.

157
00:18:55,840 --> 00:19:05,620
It's the science of how to do so in a realistic way, while still enabling the necessary data capture that sometimes these systems inherently don't support.

158
00:19:05,720 --> 00:19:07,720
And then it's that data itself.

159
00:19:07,720 --> 00:19:12,740
And that data itself is what is captured from the devices, either willingly or not.

160
00:19:12,740 --> 00:19:26,880
How the system was operating, packet captures as an example of the communications that are going across, ground truth as to the physical telemetry of what was actually going across, not just simply what the devices are reporting is happening.

161
00:19:26,880 --> 00:19:37,280
And then a manifestation of or configuration of all of those things together that is provisioned out into an environment that you then do the work on.

162
00:19:37,280 --> 00:19:42,920
Our capability, we can provision locally the assets in our central environment.

163
00:19:42,920 --> 00:19:48,920
We can provision portable environments like what we've built on Plum Island and deploy on Plum Island.

164
00:19:48,920 --> 00:19:52,740
And we can also provision into the cloud in a variety of different ways.

165
00:19:53,600 --> 00:19:56,140
So why a testbed?

166
00:19:56,160 --> 00:19:58,500
What's the value of a testbed?

167
00:19:58,500 --> 00:20:02,560
You've seen, obviously, the ICS Village and Capture the Flag stuff.

168
00:20:02,560 --> 00:20:04,340
But this is a little bit different.

169
00:20:04,560 --> 00:20:12,900
And the reason for a testbed, the reason for the work that we do, is that this mission-critical technology, and why do I call this mission-critical technology?

170
00:20:12,900 --> 00:20:26,400
So the technology being built under RADIX is intended to be, let's say, have its glass broken when we're in a blackout scenario, effectively, after we've been hit.

171
00:20:26,400 --> 00:20:29,700
That's where the real value of this technology is.

172
00:20:29,720 --> 00:20:36,760
And there's arguments, and I am one of the people that will argue this, that that technology needs to be used even before we're hit.

173
00:20:36,980 --> 00:20:40,260
But in the end, our grid is down.

174
00:20:40,400 --> 00:20:43,000
That's what this technology is built to solve.

175
00:20:43,160 --> 00:20:55,900
And so it is absolutely essential, if this technology is called to practice, that it works, and that it resolves the issue, or that it can figure out what's going on in the issues, etc., before we need it.

176
00:20:55,900 --> 00:21:12,440
Because if we don't, and we're in a national blackout, national disaster sort of scenario, attacked by an enemy or otherwise, how do we come back if we break the glass on this technology, and it's not been proven to actually work?

177
00:21:12,440 --> 00:21:15,360
And so you run it, and it's like, I can't figure out what's going on.

178
00:21:15,360 --> 00:21:16,620
I see nothing wrong.

179
00:21:16,620 --> 00:21:20,560
There's no problems here whatsoever, but the devices still won't turn on.

180
00:21:20,560 --> 00:21:22,820
The grid still is down.

181
00:21:22,820 --> 00:21:25,480
These devices aren't operating correctly.

182
00:21:25,580 --> 00:21:27,600
And that's a bad scenario to be in.

183
00:21:27,600 --> 00:21:33,240
So this is truly mission-critical technology, and we have to prove that it's effective before we need it.

184
00:21:33,280 --> 00:21:35,600
But we have to go beyond the theoretical testing of it.

185
00:21:35,600 --> 00:21:51,420
We have to put it in all sorts of scenarios across all sorts of different platforms to verify that it works, even in edge cases that it wasn't expecting, in circumstances where it's missing data, in circumstances where it's even being directly attacked or attempted to be,

186
00:21:51,420 --> 00:21:54,580
let's say, misled on what is going on.

187
00:21:55,160 --> 00:22:09,960
So our solution for that is a realistic, recomposable, and well-instrumented testbed is essential to being able to prove that out, because even the real grid environment cannot be manipulated in the way that we can with a testbed environment.

188
00:22:09,960 --> 00:22:13,200
And I'll talk a little bit about some of the innovation in that space.

189
00:22:13,820 --> 00:22:23,780
And frankly, everything, as I started with my opening salvo and the proposal, that existed, including the Illinois capabilities, wasn't good enough before this program started.

190
00:22:24,560 --> 00:22:29,200
So our approach across it is to build real systems.

191
00:22:29,200 --> 00:22:50,400
We also build models, looking at models on the cyber side and physical side, that adapt to the exercise needs, that help us build out behaviors and changes in the flows and the communications of systems, to operate like the real world or to operate in a way that an adversary may be able to manipulate.

192
00:22:50,860 --> 00:22:53,660
Everything is built in this modular way.

193
00:22:53,660 --> 00:22:58,240
It's adaptable and, let's say, recomposable in a variety of different ways.

194
00:22:58,240 --> 00:23:09,880
So we can take a piece and take a substation and how it's physically wired and physically set up at the moment, press a couple buttons, push a different configuration, and now it's a different substation.

195
00:23:09,880 --> 00:23:14,940
Same devices, different configs, different network layout, etc.

196
00:23:14,980 --> 00:23:25,600
And all of that adaptability or modularity allows us to recompose the system in any way necessary to present different challenges, etc., as we go through.

197
00:23:25,680 --> 00:23:27,580
There's also instrumentation.

198
00:23:27,580 --> 00:23:35,180
And so the instrumentation is key in that many of these systems, let's say, will cooperatively give you a certain amount of data.

199
00:23:35,180 --> 00:23:40,840
But sometimes when you're doing forensic analysis, you need to be able to gather things that are deeper.

200
00:23:40,960 --> 00:23:48,540
Or if you're trying to do experimental validation, you need to be able to look at things that the system inherently won't tell you.

201
00:23:48,540 --> 00:23:53,100
Or you need to look at it with much more scrutiny than what you would typically look at in the real world.

202
00:23:53,100 --> 00:24:03,940
So how do you do that and turn on an appropriate level of data output, data capture, etc., but that doesn't actually affect the behavior of the systems?

203
00:24:03,940 --> 00:24:07,680
Because sadly, some of these systems, as you may know, are underpowered.

204
00:24:07,680 --> 00:24:15,440
And if you turn on, let's say, full complete logging of the system or other things, it can bog down the operation of the system.

205
00:24:15,440 --> 00:24:19,620
And then it no longer participates or acts as it would in the real world.

206
00:24:20,200 --> 00:24:23,600
So the last part is really knowledge.

207
00:24:23,600 --> 00:24:30,960
And by that, I mean, we don't just say, look, as academics, we're bright people, trust us, this works.

208
00:24:30,960 --> 00:24:33,000
We had to bring in real operators.

209
00:24:33,000 --> 00:24:54,960
We had to bring in the manufacturers, the vendors across many different platforms and talk through with them their best practices, their common misconfigurations that they see when integrators are building their platforms, the common ways that they configure their substations in the real world for the asset owners,

210
00:24:54,960 --> 00:25:08,180
etc., to both cause, let's say, human error to happen in ways that people accidentally misconfigure things, but also to mimic as closely as possible how people are actually configuring these in practice.

211
00:25:08,200 --> 00:25:22,720
And that is to get the right level of protection, the right level of output of data, and even notions of like, okay, what does a SIP-compliant substation look like in terms of what it is logging or not, versus one that's not.

212
00:25:23,320 --> 00:25:26,340
So that all comes together in that knowledge area.

213
00:25:27,220 --> 00:25:32,400
On the innovation side, we had to innovate quite a bit.

214
00:25:32,500 --> 00:25:45,440
And so the, let's say, orchestration or automation stuff that we had previously was good enough for research, but it made a lot of assumptions.

215
00:25:45,440 --> 00:26:02,900
And so by that, I mean, it would operate in a centralized environment, but when it tells something to be reconfigured or tries to control something, it expects that, A, the device is reachable, B, that it has access to that device, that it has the credentials to get on that device.

216
00:26:02,900 --> 00:26:10,580
It assumes, I guess, C, that it knows what the state of that device is.

217
00:26:10,580 --> 00:26:19,660
And then lastly, everything that it did before, it also assumed that the device was trustworthy and in a known sort of condition.

218
00:26:19,980 --> 00:26:32,240
So we had to, let's say, break down all of those assumptions and operate in a way that, let's say, didn't rely on any of those existing substations.

219
00:26:32,240 --> 00:26:35,380
We applied a bunch of research as well.

220
00:26:35,380 --> 00:26:47,460
Obviously, we had over a decade of work in the prevention space and in the detection space and remediation space at the University of Illinois.

221
00:26:47,920 --> 00:26:50,640
And we had used that in the testbed in a variety of ways.

222
00:26:50,640 --> 00:26:58,940
We had used it or proved it out in the testbed, some of which is even in formal companies now transitioned either to big vendors or as startups.

223
00:26:59,020 --> 00:27:07,680
And we had to apply that in sort of a different way as part of the testbed, not to just say, okay, look, here's the testbed environment.

224
00:27:07,720 --> 00:27:24,560
But for instance, if we could reach deeper into a device, then we used some of that research that we had to dig deeper into those devices and pull out and extract information that supports the validation of the technology without affecting the performance of the device itself.

225
00:27:25,080 --> 00:27:34,600
Our team, in particular, myself and a few others have went really deep on some of these platforms over the years.

226
00:27:34,600 --> 00:27:42,980
And so we brought a wide variety of devices to the table that we already knew quite a bit about on the inside.

227
00:27:43,120 --> 00:27:48,740
Sometimes even, let's say, one could argue more than what the vendors know about their own devices.

228
00:27:49,600 --> 00:27:54,900
In terms of the knowledge and ways that we could poke around inside of these platforms.

229
00:27:55,440 --> 00:28:05,800
We had to also build, because we needed to deploy on an austere environment, if you don't have it, then you better bring it type notion.

230
00:28:05,800 --> 00:28:08,980
And so we had to build these boxes in a way that were field serviceable.

231
00:28:08,980 --> 00:28:20,540
We had to be able to quickly replace components of the system if it were to break or if the intent of the cyber attack against it was literally to brick it so that it was no longer functional.

232
00:28:20,540 --> 00:28:31,100
How did we restore that or replace that in as fast of a situation as possible to move on and not basically stop the whole exercise if something were to break?

233
00:28:31,300 --> 00:28:45,340
We had to advance our automated configuration, data extraction, and also the notion of the system and its observation when even the network links were no longer trustworthy or reliable to be up or down.

234
00:28:45,340 --> 00:28:53,620
Remember, we're in a black start scenario, so we're not even guaranteed that we'll have power on each of the substations to be able to communicate to them.

235
00:28:53,620 --> 00:29:08,560
And when they're brought up, we need to maintain the state of everything we captured as they go up and down like a seesaw as they're being attacked and brought up and brought back down, etc.

236
00:29:08,640 --> 00:29:24,120
And then we also needed to have the environment in a way that could be recomposable, change the structure of the crank paths, change the behavior of the substation itself, without going through and recabling or rewiring everything that's in there on a hands-on nature.

237
00:29:24,980 --> 00:29:27,400
So what are these environments?

238
00:29:27,700 --> 00:29:32,140
So combined, I call them the substations in a box.

239
00:29:32,140 --> 00:29:33,680
It's two components.

240
00:29:33,680 --> 00:29:40,100
This has been built on the extensive facilities we have at the University of Illinois that I've alluded to.

241
00:29:40,100 --> 00:29:50,840
We have roughly $100 million worth of hardware and software at the University of Illinois that have been built up over the past decade plus, much of which by donation.

242
00:29:51,120 --> 00:30:03,580
That's enabled all sorts of research that we've done in the past with Trustworthy Cyber Infrastructure for Power, which was an NSF effort, a DOE DHS effort called Trustworthy Cyber Infrastructure for the Power Grid.

243
00:30:03,580 --> 00:30:05,280
It added grid to the end.

244
00:30:05,280 --> 00:30:16,160
Our most recent center that's wrapping up in the next year or two called the Cyber Resilient Energy Delivery Consortium, which is also DOE and DHS funded.

245
00:30:16,160 --> 00:30:25,720
Our Critical Infrastructure Resiliency Institute, and a variety of other things that have and leverage the testbed resources at Illinois.

246
00:30:26,520 --> 00:30:36,360
So the substations in a box, as I mentioned, they're designed to support this Blackstart crank path analysis and deployed in the field, real grid environments built, etc.

247
00:30:36,360 --> 00:30:42,460
They're built in pelican style cases, so they're literally shippable and deployable anywhere we need to stand them up.

248
00:30:42,460 --> 00:30:52,000
They're generally mostly IP55 watertight when they're shipped and moved around.

249
00:30:52,000 --> 00:30:57,340
When you physically deploy them, we take the case lids off and put them in enclosures.

250
00:30:57,340 --> 00:31:08,720
The reason for that is literally so the devices inside don't overheat, but also because you do need some physical access to the devices to control breaker operations and other aspects.

251
00:31:09,120 --> 00:31:24,780
We built an environment on an island, so the power infrastructure of what is in the overhead and underground stays, but basically everything else gets torn down and built back up every six months in a different way.

252
00:31:24,900 --> 00:31:30,860
There are currently 26 variants of substations deployed across that infrastructure.

253
00:31:30,860 --> 00:31:47,920
Those substations have relays, RTUs, substation network switches, routers, as well as an experimental fabric underneath that's controlled by SDN that allows us to do a lot of the capture and, let's say, dynamic changes of the substation itself.

254
00:31:47,920 --> 00:31:49,860
All sorts of protocols are deployed.

255
00:31:49,860 --> 00:31:53,000
You can see a list of them up on the screen.

256
00:31:53,000 --> 00:31:55,860
There's both serial and Ethernet communications.

257
00:31:55,920 --> 00:32:03,660
We have custom power connections on the power boxes that allow us to link these systems together in a safe way.

258
00:32:03,660 --> 00:32:07,860
And then we have high-voltage infrastructure that I'll talk about as well.

259
00:32:08,600 --> 00:32:12,000
So what's a power box and what's in a power box?

260
00:32:12,000 --> 00:32:21,140
So power boxes basically think of it like the physical infrastructure of the island or of a real substation.

261
00:32:21,140 --> 00:32:25,920
So that's the breakers, the bus bars, the incoming and outgoing feeders on the system.

262
00:32:25,920 --> 00:32:28,460
We have a local load feeder as well.

263
00:32:28,460 --> 00:32:35,020
We have signalization lights that indicate what the status is of energization, what the status is of breakers.

264
00:32:35,140 --> 00:32:40,080
We have a dead bus sync light that's provided.

265
00:32:40,080 --> 00:32:52,700
We have analog sync check relays, contactors, auxiliary contacts on the systems, CTs and PTs, control circuitry behind that allows us to operate breakers and various other stuff.

266
00:32:52,700 --> 00:33:06,800
We have different modes of operation, sort of a safe mode where we can walk away from the system and the system can't possibly change, which is a sort of a unique scenario differing from the real world.

267
00:33:06,800 --> 00:33:11,760
And then we obviously have the ability to locally control breakers as well.

268
00:33:11,760 --> 00:33:13,280
So what does that look like?

269
00:33:13,280 --> 00:33:19,060
So there are effectively two types of power boxes inherently that we've built.

270
00:33:19,060 --> 00:33:22,360
One is a 208 volt three-phase system.

271
00:33:22,360 --> 00:33:25,280
One is a 480 volt three-phase system.

272
00:33:25,280 --> 00:33:30,820
They look basically identical from the front, except for the size of them is a little bit different.

273
00:33:30,820 --> 00:33:53,220
We also have these high voltage systems, which really are Hoffman enclosures that are wall mounted and act like, let's say, semi-intelligent breakout boards for providing telemetry from the high voltage gear to the corresponding devices that are then operating and controlling that high voltage gear.

274
00:33:53,540 --> 00:33:58,440
And so think of it sort of like a mapping board in a way.

275
00:33:58,840 --> 00:34:09,640
And each power box generally has an incoming circuit, a load circuit, and then two outgoing circuits as it's built out.

276
00:34:09,640 --> 00:34:16,120
So basic electrical diagrams in the middle, but nothing, let's say, shocking about that.

277
00:34:16,120 --> 00:34:17,900
And then there's the other side of it.

278
00:34:17,900 --> 00:34:24,580
And so each of these devices have the number two SO cord coming into these Hubble connectors on the edge.

279
00:34:24,580 --> 00:34:39,860
But they also have umbilical cords, which are amphenol connectors, mil-spec amphenol connectors, that basically take all of the telemetry of what is happening on inside the box and provide that telemetry to the devices that need to control it.

280
00:34:39,860 --> 00:34:56,100
So that includes the analog and digital signals that need to be sent back and forth between the devices to control them, but also the CTs and PT outputs, etc., from the system itself so that all of the sensing is detectable by the relay boxes.

281
00:34:56,180 --> 00:34:58,220
And so what are the relay boxes?

282
00:34:58,220 --> 00:35:01,060
Well, the relay boxes are really the brains of the substation.

283
00:35:01,060 --> 00:35:06,620
So here are a couple examples showing some of the different technology that's in play.

284
00:35:06,860 --> 00:35:12,740
Up in the upper left, those are ABB relays along with an ABB RTU.

285
00:35:12,740 --> 00:35:22,340
This is sort of a legacy RTU platform that ABB leverages or uses and has deployed around the world called the RTU-560.

286
00:35:22,440 --> 00:35:27,120
In the middle, you'll see some more ABB relays, middle-top.

287
00:35:27,220 --> 00:35:31,640
And above that, instead of an RTU-560, you see a Motorola device.

288
00:35:31,640 --> 00:35:34,080
This is a Motorola ACE-3680.

289
00:35:34,160 --> 00:35:47,760
If you move to the next image, upper right-hand corner, that is an ABB COM-600 rack mount or a COM-600R that is acting as the RTU over those ABB relays that are there.

290
00:35:47,760 --> 00:35:52,640
Bottom left-hand corner, you'll see some touchscreen SEL-751 relays.

291
00:35:52,640 --> 00:36:00,080
Those are controlled... the RTU in that particular case is an SEL-RTAC, a 3505.

292
00:36:00,480 --> 00:36:03,280
In the middle, you'll see touchscreen relays.

293
00:36:03,280 --> 00:36:08,200
In that one, there's an SEL-RTAC as well, but that's an SEL-RTAC.

294
00:36:08,200 --> 00:36:12,040
It's a SEL-3530 instead of a 3505.

295
00:36:12,780 --> 00:36:16,860
And then the far right, you'll see more SEL relays.

296
00:36:16,860 --> 00:36:18,200
These ones are not touchscreen.

297
00:36:18,200 --> 00:36:21,040
These are another variant of the SEL-751.

298
00:36:21,040 --> 00:36:26,920
And those ones are being controlled from an RTU perspective by a Novatek Orion LX.

299
00:36:27,600 --> 00:36:32,600
So this shows just some of the diversity of platforms that are there.

300
00:36:32,600 --> 00:36:37,040
There are much more, obviously, across 26 substations.

301
00:36:37,400 --> 00:36:41,120
Every single substation is unique in some way, shape, or form.

302
00:36:41,120 --> 00:36:48,800
So we have a lot of diversity across the environment in terms of platforms, technologies, and configurations.

303
00:36:48,800 --> 00:36:52,100
And so diversity could be purely on the configuration side.

304
00:36:52,100 --> 00:37:04,260
For instance, different protocols being communicated, different topologies being set up between Utility A, Utility C, Utility B, etc.

305
00:37:05,420 --> 00:37:08,640
And lots of other variation on top of that.

306
00:37:08,700 --> 00:37:18,820
So let's talk about some, let's say, lessons learned from the program or challenges that we had to tackle, now that you understand some of the gear.

307
00:37:18,820 --> 00:37:20,640
First off is safety.

308
00:37:21,640 --> 00:37:40,940
When you can't trust anything in the system whatsoever because it is compromised and it's compromised in a way that you may or may not know, may or may not be able to determine, and you don't know what is trustable or not, all of the systems that are there are effectively designed to protect you.

309
00:37:41,120 --> 00:37:46,980
But if you can't trust the digital systems to protect you anymore, then you need additional layers of protection.

310
00:37:46,980 --> 00:37:51,200
So we had to layer protection throughout the system in both physical and cyber form.

311
00:37:51,200 --> 00:38:04,440
And that included things like analog time, analog protections in the system, like analog sync check relays, time over current protections, thermal protections.

312
00:38:04,500 --> 00:38:15,140
On the digital side or on the cyber side, the protective relays were configured with safe and sane settings for overvoltage and undervoltage conditions, etc.

313
00:38:15,140 --> 00:38:23,880
The boxes had arc flash analysis done on them to determine potential exposure or safety measures from PPE perspective that needed to be done.

314
00:38:23,880 --> 00:38:27,820
The cabinets were all either pad lockable or direct key lockable.

315
00:38:27,980 --> 00:38:35,060
We had, on the high voltage side, interrupters that were acting as fail safes if there's anything that flows through.

316
00:38:35,060 --> 00:38:50,500
The low voltage to the high voltage side, an out of sync close or something like that that happened to happen, or a surge somewhere, the interrupters or a fault on the line, the interrupters were there to protect the system at the high voltage side.

317
00:38:50,780 --> 00:38:55,220
We also purposefully did not target the high voltage control.

318
00:38:55,300 --> 00:39:00,040
That way we didn't run into, let's say, big issues.

319
00:39:00,040 --> 00:39:07,120
Low voltage was something we could cause a problem on and be okay with, but on the high voltage side people could get hurt.

320
00:39:07,500 --> 00:39:12,020
All of the connectors we used were screw-in locking style connectors.

321
00:39:12,160 --> 00:39:15,160
We had all sorts of internal wiring protection, etc.

322
00:39:15,160 --> 00:39:21,400
So the key is that the environment itself was designed to be safe no matter what.

323
00:39:21,520 --> 00:39:31,240
So people that knew nothing about power systems could still safely operate in this environment and not run a risk of being electrocuted or whatever.

324
00:39:31,240 --> 00:39:45,420
We always had power engineers and safety officers effectively on site that were making sure that people did safe operations and maintained the necessary perimeters, even at a noise level from the generators and various other aspects.

325
00:39:45,420 --> 00:40:08,520
But really the system protected itself in every way, so let's talk about some operational lessons that we learned in executing exercises for blackstart restoration, but also in austere environments under conditions of blackouts, etc.

326
00:40:08,640 --> 00:40:13,640
And so let me talk a little bit about the mode of execution as we go through this.

327
00:40:13,640 --> 00:40:19,920
So when I say that we're operating utility environments, we are actually operating the utility environments.

328
00:40:19,920 --> 00:40:29,560
And by that I mean utility operators from real utilities come to the island and they run the infrastructure.

329
00:40:29,780 --> 00:40:37,580
And they basically take control of it, we hand it over to them, and then they tell everyone else what to do, how to do it, when to do it, etc.

330
00:40:37,920 --> 00:40:39,300
on the system.

331
00:40:39,300 --> 00:40:46,620
Now obviously we have some exercise control over that, but the intent is to really make this as real as possible.

332
00:40:46,620 --> 00:40:48,740
So let's talk about that realism.

333
00:40:49,240 --> 00:40:55,580
Many people don't believe what is possible until it really is, let's say, slapping them in the face.

334
00:40:55,580 --> 00:41:12,540
And by that I mean a common view is, look, relays, they're embedded devices, you can't make them do things, you can't disable their protection, you can't change their mode of operation beyond their config.

335
00:41:13,220 --> 00:41:17,340
And until they saw us do that, they didn't really believe it.

336
00:41:17,340 --> 00:41:34,780
Even when it was happening right in front of them, they still didn't believe it until they dug deeper and started to look deeper at what was happening and how it was happening to realize that, look, bad things really are possible, that right now your mindset is that this isn't possible at all,

337
00:41:34,780 --> 00:41:35,960
that no one can do that.

338
00:41:35,960 --> 00:41:39,980
You had to physically modify the device in order to do that.

339
00:41:39,980 --> 00:41:43,220
And it's like, no, we can actually do that via cyber means.

340
00:41:43,260 --> 00:41:46,360
So let's talk about the people, right?

341
00:41:46,380 --> 00:42:10,400
One of the things that was an operational lesson is academia is really good about thinking outside the box, but we needed to be extremely agile and think even further outside of the box and push through and build stuff that, frankly, was, let's say, impossible to build at any given moment.

342
00:42:10,400 --> 00:42:17,280
We took on building one utility, then two, then three, all in six-month iterations.

343
00:42:17,280 --> 00:42:29,160
Completely different architectures, completely different devices, building the physical boxes, and standing it up in a new realistic configuration at each sort of interval.

344
00:42:29,160 --> 00:42:31,080
That's pretty difficult to do.

345
00:42:31,080 --> 00:42:34,040
It's multiple weeks to build the environment each time.

346
00:42:34,040 --> 00:42:35,880
It's multiple weeks of testing.

347
00:42:35,880 --> 00:42:53,580
It's multiple weeks of evaluation to make sure that it is as real as possible, that it doesn't have inherent artifacts itself that people may view as compromises, etc., in the system, that it is pristine, blue sky, trustable, etc., and built the way it should be built.

348
00:42:53,860 --> 00:43:07,480
And that results in, let's say, extreme levels of stress at times, but as a team, not just the University of Illinois, but broadly the entire program, we all pulled together and found success in every exercise that we had.

349
00:43:07,480 --> 00:43:13,260
There was no exercise that failed because the infrastructure or the people failed to deliver.

350
00:43:14,700 --> 00:43:16,860
And so that brings us to pace.

351
00:43:16,860 --> 00:43:19,720
So I mentioned every six months.

352
00:43:19,720 --> 00:43:28,620
So DARPA programs move very fast, and the expectations are very high of the technology, of the evaluation, of the test bed, etc.

353
00:43:28,620 --> 00:43:33,780
And so there were many times that we were facing failure.

354
00:43:33,780 --> 00:43:54,540
But let's say by pure blunt force, and by pure blunt force, I mean number of hours and long days and leveraging and leaning on each other throughout the program, we were able to pull through and pull off what seemed to be impossible when we started.

355
00:43:54,880 --> 00:44:01,920
So those are some just operational aspects that we learned as we built the environment, especially out on an island.

356
00:44:01,920 --> 00:44:05,260
And so when I say, you know, what are some challenges, right?

357
00:44:05,260 --> 00:44:24,820
So there were times when we were, you know, we took a ferry every day from the mainland to the island, and there were times when, you know, nor'easters were blowing in and other aspects where the waves were 12, 14 feet high, and the, you know, conditions were such that if you go to the island,

358
00:44:24,820 --> 00:44:28,440
you're going to be sleeping there because you're probably not going to make it off.

359
00:44:28,760 --> 00:44:36,360
And that's kind of, you know, let's say a bit much, but there were many days that happened like that.

360
00:44:36,360 --> 00:44:52,680
There were days we were out there executing the environment, trying to restore after the cyber attack, and, you know, there was nor'easters blowing through with 40 plus mile an hour winds and downpours and inches of rain falling an hour, etc.

361
00:44:52,680 --> 00:45:10,220
So not just hard from a technical perspective, but even harsh environments that we were out in as we were trying to restore these devices, and as the teams and the utility operators were operating these devices faced with those types of environmental conditions.

362
00:45:10,600 --> 00:45:13,940
So we learned a whole bunch of other lessons too.

363
00:45:13,940 --> 00:45:35,120
So one of them that was interesting to us was that when the systems are intended to break, when you know that you can't trust anything anymore, you have to think differently about what you can build and how you build it so that it works consistently and reliably even when everything is intended to be broken.

364
00:45:35,120 --> 00:45:40,720
So that goes back to some of the stuff we did with safety, but also on the cyber side.

365
00:45:40,720 --> 00:45:47,420
So as an example, we had no guarantee of power, reliably or not, at any point in time.

366
00:45:47,420 --> 00:46:09,320
So how do we guarantee we don't lose data, as an example, as we're going through, or that we get, let's say, eventual consistency where if part of the network is up and being controlled by our orchestration and another part of it is down, how do we get it to catch back up and get in the right configuration when it does come back up if it needed to be changed,

367
00:46:09,320 --> 00:46:19,720
or to collect all of the data that it had when it wasn't centrally reachable, when it was isolated and only, let's say, on backup power on its own.

368
00:46:20,220 --> 00:46:28,160
So that presented some interesting challenges that we had to tackle, which is basically fault-tolerant and distributed computing in a nutshell.

369
00:46:29,200 --> 00:46:37,460
The other issue that was somewhat interesting that we had to innovate on was the gear in the field typically isn't hot swappable.

370
00:46:37,460 --> 00:46:52,740
You can't just go grab a relay that's in a real substation and pull it out and pop another one in, in most of the substations, without having to do some rewiring, without having to take circuits out of commission, de-energize them, lock out, tag out procedures,

371
00:46:52,740 --> 00:46:53,460
etc.

372
00:46:53,700 --> 00:47:12,200
But if you're in a fast-paced exercise environment, we have seven days, and if people get stuck and we need to reverse out basically what the bad people did, how do we reverse that out in as fast a way as possible without bringing the whole exercise down or the whole exercise to a halt for hours on end?

373
00:47:12,200 --> 00:47:18,420
So we had to create sort of quick connects and all sorts of other things that allowed us to swap gear out very, very quickly.

374
00:47:19,240 --> 00:47:30,980
Further, when we're testing bleeding edge technology, things fail because sometimes that technology doesn't work, or the research doesn't do what it's supposed to do.

375
00:47:30,980 --> 00:47:35,300
And so that's your primary plan is, okay, look, it's going to try this, and if it works, great.

376
00:47:35,300 --> 00:47:36,900
Well, you have to have a backup plan, right?

377
00:47:36,900 --> 00:47:45,080
And in many cases, we had to have another backup to the backup plan because we couldn't fail and make everything stop working.

378
00:47:45,360 --> 00:47:55,620
So if anyone in the program failed, we had to have a backup plan for what would happen if they failed, and then a backup plan for if our backup plan failed.

379
00:47:56,440 --> 00:48:05,240
And then the last time, the last one, which is a lesson learned is we thought we were prepared in many, many, many occasions.

380
00:48:05,240 --> 00:48:26,760
But let's say, you know, in between days of the exercise, when we would pause and go back to our hotels to sleep, etc., I would often make runs to Home Depot or Lowe's or electrician stores or whatever, because no matter how many spare parts we had, no matter what tools we had on us or available to us,

381
00:48:26,760 --> 00:48:31,940
anything that can break will break at some point when you're running these environments.

382
00:48:31,960 --> 00:48:37,120
And it's almost always in the way that you never thought of or that you couldn't have anticipated.

383
00:48:37,120 --> 00:48:46,580
Something that has, you know, a 10,000-hour mean time between failure or 100,000-hour mean time between failure fails in, you know, 10 hours instead.

384
00:48:46,860 --> 00:48:56,440
So lots of interesting challenges on just keeping things up and operational and making sure that stuff was readily available if anything did break.

385
00:48:56,940 --> 00:49:01,580
So let's talk a little bit about some personal takeaways, and I'll wrap these up really quickly here.

386
00:49:02,020 --> 00:49:10,180
But let's talk, let's say, not as the University of Illinois, but let's talk about this from my perspective.

387
00:49:10,180 --> 00:49:25,160
So vendors, be this ICS cyber solutions or the vendors themselves, they often claim capabilities that, let's say, are much more limited in the real world application than what most people realize.

388
00:49:25,160 --> 00:49:32,820
So a utility may say, I go buy this platform and it's got me covered and it can do all of these things and I can check that box and I'm good.

389
00:49:32,940 --> 00:49:40,960
The reality is, no matter what vendors claim, there's often still very large gaps there.

390
00:49:41,300 --> 00:49:51,220
And so even the commercial off-the-shelf ICS cyber solutions that are out there, they're missing huge amounts of surface area on what adversaries can do against these boxes.

391
00:49:51,360 --> 00:49:56,160
Radix technology was designed to close some of those gaps, but not all of them, right?

392
00:49:56,160 --> 00:50:01,640
So even as great as the Radix technology is, we still have a long way to go, right?

393
00:50:01,640 --> 00:50:22,280
In building the environments that we built and helping to construct the exercise evaluations and the test effective payloads and all that, in essence, is how do you cause a condition from a cyber means to happen on these devices, such that there are artifacts or implementations that then people can forensically find,

394
00:50:22,280 --> 00:50:23,000
right?

395
00:50:23,520 --> 00:50:28,380
So all of those, in doing that, we found hundreds of issues on these devices.

396
00:50:28,380 --> 00:50:30,980
And it's not like we've never looked at these devices before.

397
00:50:30,980 --> 00:50:36,600
And by issues, I don't necessarily mean security vulnerabilities, but in some cases it was security vulnerabilities.

398
00:50:36,640 --> 00:50:43,580
But merely other items like, look, compatibility between host A and host B, it doesn't work.

399
00:50:43,620 --> 00:50:46,520
Even though they're supposed to interoperate, it doesn't.

400
00:50:46,520 --> 00:50:55,280
There's nuances, differences, there's differences in documentation on what is communicated on the actual wire versus what it says it's going to communicate as.

401
00:50:56,200 --> 00:51:02,400
Another thing, which was a personal takeaway, is being the adversary is fun, right?

402
00:51:03,240 --> 00:51:09,100
It's nice to be on the red team and hack these devices and break them in a variety of ways.

403
00:51:09,360 --> 00:51:27,380
But in the program, the defense and recovery technology, including the RADIX technology and commercial off-the-shelf stuff that I've personally seen, it's still way behind what I was capable of or what others on the red team were capable of doing to these systems.

404
00:51:27,440 --> 00:51:34,940
So if we're so far ahead, we could just obliterate any of that tech, then what fun is there in that?

405
00:51:34,940 --> 00:51:36,900
It's not even a fair fight at that point.

406
00:51:36,900 --> 00:51:43,800
So a lot of times we limited the activities that we were doing to poking the bear rather than destroying it.

407
00:51:43,800 --> 00:51:46,820
So we didn't go out for the throat kill.

408
00:51:46,820 --> 00:51:56,280
We kept a pace with what the technology was capable of and designed something that was just a little bit ahead of that, pushing them each iteration to improve.

409
00:51:56,940 --> 00:52:14,000
And so one sort of broad takeaway is, and I say this not just of technology in North America or whatever, but the whole world, what they have in this space in industrial control systems, detecting cyber attacks on these devices, that detection, that mitigation,

410
00:52:14,000 --> 00:52:18,020
and that remediation technology for the electric power grid.

411
00:52:18,040 --> 00:52:26,020
And I'll stretch it a bit and say actually all critical infrastructure still really has a long way to go.

412
00:52:26,060 --> 00:52:39,500
There's still so much work that needs to be put on those platforms to really protect the systems from somebody who is truly focused and determined and understands these systems at a deep and inherent level.

413
00:52:40,060 --> 00:52:43,240
So that's all of my technical content.

414
00:52:43,280 --> 00:52:54,440
I will just flash a slide real quick, which is the testbed at Illinois and much of what we've built in the RADIX program has been enabled by lots of companies.

415
00:52:54,440 --> 00:53:04,340
These are some of the the companies that have donated gear to us, software to us, et cetera, that have helped enable the things that we've done.

416
00:53:04,340 --> 00:53:13,260
But without them and without the commercial support and the vendor support and the utility support broadly across this program, we wouldn't be successful.

417
00:53:13,260 --> 00:53:15,380
The DARPA RADIX program wouldn't have been successful.

418
00:53:15,380 --> 00:53:19,280
So thank you to all of those companies and what they did.

419
00:53:19,320 --> 00:53:23,340
And then I think we still have a couple minutes for questions.

420
00:53:23,560 --> 00:53:28,260
And so I'll also leave a sort of bonus link down there at the bottom.

421
00:53:28,260 --> 00:53:38,960
There's a GitHub repo that I created a number of years ago at the S4 conference, and I've been maintaining it sort of ad hoc ever since.

422
00:53:39,020 --> 00:53:52,200
And that's a bunch of ICS security tools in a variety of different forms that are aggregated and categorized in various ways and mirrored when their original location is no longer available.

423
00:53:52,300 --> 00:53:53,360
So do check that out.

424
00:53:53,360 --> 00:53:55,640
It has a whole bunch of useful things in it.

425
00:53:55,640 --> 00:54:01,980
And with that, I will stop talking and I think we might have a little bit of time for question and answer.

426
00:54:06,070 --> 00:54:07,250
Let's see.

427
00:54:07,250 --> 00:54:08,310
Hey, Tim, this is Bryson.

428
00:54:08,310 --> 00:54:09,210
How are you doing?

429
00:54:09,350 --> 00:54:10,010
Wonderful.

430
00:54:10,010 --> 00:54:10,910
Hi, Bryson.

431
00:54:11,070 --> 00:54:11,530
Thank you.

432
00:54:11,530 --> 00:54:12,990
Thank you for the talk.

433
00:54:12,990 --> 00:54:19,670
I noticed that you are on our Discord and I actually promoted you to a speaker during your talk.

434
00:54:19,970 --> 00:54:22,570
So you should now have that badge tied to you.

435
00:54:22,790 --> 00:54:32,390
And what we recommend is if you could post that GitHub link in the speaker Q&A section of Discord.

436
00:54:32,530 --> 00:54:32,890
Okay.

437
00:54:32,890 --> 00:54:38,630
And then reach out for folks to engage you there and for questions.

438
00:54:38,930 --> 00:54:39,310
Okay.

439
00:54:39,310 --> 00:54:40,550
Sounds good.

440
00:54:40,970 --> 00:54:44,710
So really appreciate you jumping on and giving this talk.

441
00:54:45,370 --> 00:54:50,190
Having been and seen this for myself, it is really impressive.

442
00:54:50,230 --> 00:54:57,850
And my favorite part for the sensors was of course the inflatable guys like you see next to the used car sales.

443
00:54:58,050 --> 00:55:02,690
I thought that was a really interesting way to show whether something is up or down at a physical distance.

444
00:55:02,950 --> 00:55:05,510
Yeah, we affectionately call them the dancing men.

445
00:55:07,770 --> 00:55:17,670
And sort of a funny aside, in the Nor'easters, when you have torrential downpour, those things get wet and then they turn into sort of like whiplashes.

446
00:55:17,670 --> 00:55:27,490
So I was up there, you know, untangling them on many occasion and getting sort of smacked by the dancing men as I was trying to untangle them so that they could fly.

447
00:55:27,970 --> 00:55:30,130
Well, that's how you know it works.

448
00:55:32,720 --> 00:55:41,120
Well, anyway, Tim, I appreciate you joining us and supporting the village and look forward to the commentary and the Q&A on Discord.

449
00:55:41,620 --> 00:55:42,160
Awesome.

450
00:55:42,160 --> 00:55:42,360
Yeah.

451
00:55:42,360 --> 00:55:47,980
And everyone do check out what they've set up for, you know, the CTFs and other things in the village.

452
00:55:47,980 --> 00:55:52,540
You know, they do an awesome job of creating environments that you can play with.

453
00:55:52,660 --> 00:55:58,160
Sadly, I can't offer my environment out to the world in such an easy and accessible way.

454
00:55:58,200 --> 00:56:07,680
But hopefully in the future, I'll get deeper engaged with the ICS village and bring some of this tech and some of this capability to the village so you guys can all play with it too.

455
00:56:08,100 --> 00:56:09,560
Yeah, we look forward to that.

456
00:56:09,560 --> 00:56:13,580
That's probably been the biggest innovation we've had is because of the pandemic.

457
00:56:13,920 --> 00:56:22,600
The amount of effort we've had to spend on figuring out how to make these things virtually accessible, which is the typical limitation for concurrent access.

458
00:56:22,820 --> 00:56:24,500
We kind of have solved it.

459
00:56:24,500 --> 00:56:27,740
So we'd love to touch base with you afterward and talk about it.

460
00:56:27,800 --> 00:56:32,100
Yeah, it's, you know, great to hear that you guys have had to tackle that.

461
00:56:32,100 --> 00:56:36,280
We're currently tackling that for the DARPA Radix effort as well.

462
00:56:36,520 --> 00:56:40,560
With the last exercise, it will be predominantly remote.

463
00:56:40,680 --> 00:56:59,400
And we went from the prior exercise being, you know, network isolated, completely trusted, everyone in a specific zone to, you know, part of this is deployed in the cloud, and everyone is distributed around the country, and all accessing this in a controlled and crazy way,

464
00:56:59,400 --> 00:57:03,120
including like streaming body cams that we'll have and all sorts of stuff.

465
00:57:03,120 --> 00:57:07,160
So we're on that same roller coaster due to COVID at the moment.

466
00:57:09,160 --> 00:57:11,540
Well, we look forward to collaborating.

467
00:57:11,540 --> 00:57:14,800
So again, thank you very much, and we'll see you on Discord.

468
00:57:15,340 --> 00:57:16,800
Yeah, you guys have a great day.

469
00:57:16,960 --> 00:57:17,600
All right, take care.