WEBVTT 00:09.240 --> 00:11.400 Hi, so my name is Tim Yardley. 00:11.400 --> 00:17.600 I'm here to talk about some work that I've done under DARPA RADIX program. 00:20.550 --> 00:22.510 Whoa, I'm getting audio from somewhere. 00:22.510 --> 00:23.210 Hold on. 00:35.120 --> 00:36.480 All right, sorry about that. 00:36.480 --> 00:38.600 I was getting feedback from the live channel. 00:39.420 --> 00:49.320 So I am here to talk about building a cyber physical testbed and how we've used that to support black start restoration under cyber fire. 00:49.320 --> 00:52.640 So this has been part of a DARPA RADIX program. 00:52.640 --> 00:57.240 The RADIX program is Rapid Attack Detection Isolation and Characterization Systems. 00:57.240 --> 00:59.760 And I'll talk a little bit about what that is here in a minute. 01:01.300 --> 01:15.080 So just for sort of coverage and statement, the testbed work that I'm going to talk about was funded under the DARPA's Rapid Attack Detection Isolation and Characterization Systems program. 01:15.080 --> 01:16.960 I work for the University of Illinois. 01:16.960 --> 01:22.680 I'm the principal investigator of the University of Illinois' effort under there. 01:22.680 --> 01:30.060 We also had some government SME support, subject matter expert support from Idaho National Lab. 01:30.060 --> 01:35.240 And then the program evaluator is a company called Probatech. 01:35.240 --> 01:46.900 And all three of those organizations have been critical to the work that I'm talking about in terms of enabling the black star restoration efforts and the validation of the work itself. 01:47.020 --> 01:50.760 So let's talk a little bit, give you a little background on me. 01:50.760 --> 01:52.800 So my name is Tim Yardley. 01:52.800 --> 01:58.120 I'm a principal research scientist at the Information Trust Institute at the University of Illinois. 01:58.220 --> 02:04.160 And I am also a father, a husband, and broadly a researcher. 02:04.160 --> 02:10.420 I've been doing work in industrial control systems for about 14 years now. 02:10.420 --> 02:19.180 And I've been doing work in security for, geez, probably almost 30 years now across the board. 02:19.280 --> 02:38.020 My background came from sort of the think tank IRC eras back in the early days of FNET groups that that, let's say, like to explore and like to figure out things, but also cause some havoc in the process. 02:38.020 --> 02:44.620 I'm one of the original members of WooWoo and a variety of other security think tanks over the years. 02:45.000 --> 02:52.160 And I've sort of been in the realm of computer security for quite some time. 02:52.160 --> 02:55.800 I've been working in academia for about the past 12 years. 02:55.800 --> 03:00.620 And prior to that, I worked in industry supporting a variety of different efforts. 03:00.860 --> 03:05.780 So the DARPA RADIX effort itself, let's talk a little bit about what it is. 03:05.780 --> 03:12.180 So the program is designed to build technologies that fill a gap. 03:12.180 --> 03:28.600 And that gap really is the notion that if we are attacked by nation states, we are ill-prepared to be able to determine exactly what happened, where it happened, and how it happened, and get rid of it as fast as possible. 03:28.980 --> 03:42.840 So DARPA stood up this program to build technology that advanced that state of, let's say, preparedness, and then to evaluate that technology on a realistic facility. 03:42.840 --> 03:47.640 And so our role in that program was to build that realistic facility. 03:48.220 --> 03:52.480 We'll talk in more detail about the objectives of the program itself. 03:52.680 --> 03:58.220 So these are all public slides from DARPA that are here. 03:58.220 --> 04:07.120 But the key objective of the program itself is to enable black start recovery of the power grid amidst a cyber attack on the energy sector's critical infrastructure. 04:07.120 --> 04:12.020 In this particular case, the critical infrastructure is the electric power grid. 04:12.020 --> 04:21.280 So in a prevention a sort of state, you would say, okay, well, let's defend and detect what's going on. 04:21.280 --> 04:25.200 But this program starts from the adversary has been successful. 04:25.200 --> 04:32.200 So you have a complete blackout, no power anywhere, and you have to figure out exactly what happened, how it happened, etc. 04:33.640 --> 04:47.820 So the devices that are involved, you have to figure out what devices can be trusted, because you don't know what's compromised and what is safe or what is operating per norm. 04:47.880 --> 04:55.760 And many of the physical infrastructures are effectively controlled by intelligent devices. 04:55.760 --> 05:05.160 So these are the ICS devices that you have to explore on, let's say, the cyber side of the equation that are controlling the physical aspects of the grid. 05:06.320 --> 05:10.000 And these assets are spread throughout the country. 05:10.000 --> 05:12.380 So how can you do this in a scalable way? 05:12.380 --> 05:17.020 How can you deeply and forensically poke at these embedded devices? 05:17.160 --> 05:26.240 How can you figure out exactly what is trustable, what is not trustable, what was attacked, how it was attacked, and then get rid of it as fast as possible. 05:26.240 --> 05:31.620 The goal is to do so across the entire United States within seven days. 05:31.620 --> 05:40.480 And that's to isolate, characterize, and restore any crank paths necessary to bringing the grid for the United States back online. 05:42.360 --> 05:45.820 So let's talk about the manifestation of this. 05:45.820 --> 06:04.300 So in the first year or so, year and a half of the program, we built exercise environments that were run at the University of Illinois that were getting people's feet wet and getting the technology ready to explore, let's say, the beginning edges of the problem space. 06:04.460 --> 06:10.980 And as we evolved both the technology and the program, we had to turn the corner. 06:10.980 --> 06:19.440 So people will only believe what happens in a lab to the extent that it's, oh, that was in a lab environment. 06:19.440 --> 06:22.560 Oh, that's fine if you do it just on science. 06:22.560 --> 06:24.540 But that's not the real world. 06:24.540 --> 06:29.420 So we took it to a federal island, which is called Plum Island. 06:29.420 --> 06:42.960 It's the home of the Animal Disease Center, controlled and owned by DHS, S&T, and is a former siting of Fort Terry in the Spanish-American War and a bunch of other stuff. 06:42.960 --> 06:48.200 The size of the space that we're controlling there is roughly the size of Central Park. 06:48.220 --> 06:51.640 And we have built electrical infrastructure on that island. 06:51.640 --> 06:58.540 And so I'll talk about the most recent incarnation of it that was completed in November of 2019. 06:58.540 --> 07:01.400 That was the sixth exercise of the program. 07:01.400 --> 07:03.920 We have one more exercise still to do. 07:03.920 --> 07:08.100 COVID has postponed that a little bit, but still pending. 07:08.100 --> 07:10.000 So far has not been fully canceled. 07:10.440 --> 07:19.960 And we look to sort of do that last hurrah of the program and validate the advancements from the last exercise till now. 07:19.980 --> 07:27.040 So in the island infrastructure, we basically have built three full utilities. 07:27.040 --> 07:32.260 And those utilities are represented by gear that we'll talk about here in a moment. 07:32.360 --> 07:35.700 They have an emergency operations center for each utility. 07:35.700 --> 07:42.120 And then there's a regional coordinator that's trying to coordinate the actions amongst the different utilities as it goes through. 07:42.120 --> 07:45.040 We have the National Guard involved. 07:45.040 --> 08:05.520 We have the RADIX performers involved and lots of other entities that help establish communications in a variety of way and really set up what is a fairly austere environment from the beginning to enable the capture and execution of these exercises. 08:05.660 --> 08:09.360 So utility A has five low voltage substations. 08:09.360 --> 08:11.920 And I'll talk about what low voltage means to us. 08:11.920 --> 08:13.860 And one high voltage substation. 08:13.860 --> 08:20.520 It also had one generator, which is the crank path, in essence, that's getting to the high value assets. 08:20.520 --> 08:32.660 Utility B had seven low voltage substations along with three high voltage substations and that crank path had a critical national asset on it that needed to be up and maintained no matter what. 08:32.900 --> 08:36.320 Utility C was similar to utility A. 08:36.320 --> 08:41.380 It had five low voltage substations and one high voltage substation as well as one generator. 08:41.700 --> 08:47.760 And so each of these utilities differed in the physical topology and layout of the substations. 08:47.760 --> 08:52.300 They also differed in the equipment that was deployed across each of those substations. 08:53.700 --> 09:00.180 So the program is really broken into multiple different technical areas. 09:00.180 --> 09:04.540 You could look at it as four or five depending on how you want to count. 09:04.540 --> 09:06.940 The first area is situational awareness. 09:06.940 --> 09:18.260 And so when they deploy on this infrastructure that we've built, they are trying to figure out what happened or what is currently happening on the infrastructure to get as much situational awareness as you can provide. 09:18.260 --> 09:24.540 And the reason for that is you can't really trust the devices once they've been attacked to be telling you the right things. 09:24.540 --> 09:32.400 And you are in a blackout scenario, so you may not have visibility in much of the grid environment. 09:32.620 --> 09:35.440 Network isolation is technical area two. 09:35.440 --> 09:49.600 And technical area two is focused on taking the communications that are no longer necessarily trustworthy and expanding those into a realm of trustable or not. 09:49.600 --> 09:54.980 And so by that I mean point A to point B may have talked to each other before. 09:55.040 --> 09:59.580 And it used to be a dedicated private link or whatever it may be. 09:59.580 --> 10:06.360 But now traffic that's going across there is not reflecting what it did previously. 10:06.360 --> 10:08.040 So maybe somebody's man-in-the-middle-ing it. 10:08.040 --> 10:09.760 Maybe somebody is manipulating it. 10:09.760 --> 10:12.260 Maybe it's getting black-holed in some other way. 10:12.500 --> 10:14.220 Unknown operation, right? 10:14.220 --> 10:28.300 So how do you take the outputs from A and get it to B in a way that's trustworthy and secure and such that it cannot be modified in the middle or that it is evident when it is modified in the middle? 10:28.300 --> 10:33.040 So that's the area of research for technical area two. 10:33.040 --> 10:35.640 Technical area three is threat analysis. 10:35.640 --> 10:42.340 And they really are intended to do the forensic response per se on the devices itself. 10:42.340 --> 10:58.920 So how do you diagnose and remove cyber threats from the embedded devices, from the different pieces that are involved, et cetera, as you go through the investigation part of the environment? 10:58.940 --> 11:02.580 So the environment itself is technical area four. 11:02.580 --> 11:05.740 And that's conducted by us at the University of Illinois. 11:05.740 --> 11:29.680 And that is the environment by which the exercise happens, which is on Plum Island, but also the environment at the University of Illinois' campus that the performers remote into to build out their technology, to extend their capabilities, and to investigate the edge cases of how the grid operates when certain things happen to these devices. 11:30.700 --> 11:34.140 So that's part of our central facility, and I'll talk a little bit about that. 11:34.140 --> 11:41.140 And then the island is basically a distributed manifestation of that central facility at Illinois. 11:41.140 --> 11:44.380 The last technical area is technical area five. 11:44.380 --> 11:47.940 And effectively, that is the evaluators of the program. 11:47.940 --> 12:09.160 And they build out how to run an exercise in this space and how to determine whether or not progress is being made on the technology, what the appropriate challenge levels are, the, let's say, pain points, or how deep or how hard the red team pushes, et cetera, 12:09.160 --> 12:11.240 as we go through the environment. 12:11.240 --> 12:18.480 And they technically grade technical areas one, two, three, and four as they go through that. 12:19.980 --> 12:23.000 So on to the next slide. 12:23.320 --> 12:39.460 RADIX exercise six was really a move from strategic to the strategic notion early on in the program to operational notion with the utilities involved, et cetera, to a tactical deployment on top of that. 12:39.460 --> 12:49.680 So if you look up in the upper left-hand corner, the strategic notion is the concept that we talked about of you have a large-scale blackout and what you need to accomplish. 12:49.680 --> 12:55.060 The operational notion is the building and establishment of these crank paths. 12:55.060 --> 13:01.680 And then the tactical is let's execute on those crank paths and figure out exactly what happened and how it happened. 13:01.760 --> 13:12.160 So the picture on the bottom right at the moment is a drone footage sort of zoomed in on one of the substations. 13:12.160 --> 13:15.320 And you can see that they're built in shipping containers. 13:15.320 --> 13:20.520 And those shipping containers have gear inside them that I'll show you here in a moment. 13:20.540 --> 13:25.600 And then these containers are arranged and linked together to build the crank path. 13:25.600 --> 13:38.860 And we do that with basically above ground number two SO cord that has connectors on the end that we plug into to the individual gear that we have inside the boxes. 13:38.860 --> 13:40.680 And I'll show you what that looks like here in a moment. 13:42.460 --> 13:48.080 So you can see sort of a little bit of an edge here of what is inside a container. 13:48.080 --> 13:53.280 So this is standing inside a container looking out at the moment with the door open. 13:53.280 --> 13:57.060 So on the left-hand side there, you'll see what we call the relay box. 13:57.060 --> 14:06.440 On the right-hand side, you'll see what we call the power box, which is a skid-mounted Hoffman enclosure that controls the flow in essence of the substation. 14:06.440 --> 14:09.120 So that's the power grid aspect. 14:09.200 --> 14:16.840 And the controls of those grid components are in the relay box on the left-hand side. 14:16.840 --> 14:27.760 There's also some sensors that are up on top that are providing or that are performer technology, providing some of the situational awareness and attempt to determine ground truth as we go through the environment. 14:27.760 --> 14:30.500 And I'll zoom in on a lot of this as we talk. 14:30.740 --> 14:34.180 So I'm going to only talk about the testbed itself. 14:34.180 --> 14:44.260 I'm not going to talk about any of the performer technology that's specifically built, but I'll dig quite deeply into the testbed, which is my area of responsibility. 14:44.460 --> 15:00.280 So the mission that we set out to do is to provide realistic environments that enable this cutting-edge R&D that's not yet done by any commercial available product or any existing research off the shelf. 15:00.280 --> 15:10.500 And then we take and create this environment in a way that allows us to validate the effectiveness and frankly, the efficiency of those tools as we go through. 15:10.700 --> 15:18.540 So the goal of the program itself is really to take a generational leap forward in the capabilities of testbeds. 15:18.540 --> 15:28.480 So we've been building cyber-physical testbeds and leveraging the cyber-physical testbeds for about 13, 14 years now at the University of Illinois. 15:28.480 --> 15:38.100 By many accounts, we're sort of the gold standard in terms of capabilities across the nation and arguably the world. 15:38.100 --> 15:54.400 But even so, when we pitched our capabilities for this program, our proposal and the going in salve of our proposal was effectively, we have assembled the right team to solve this problem. 15:54.400 --> 16:09.620 But what technology exists today and where testbeds are today across all of them that you will encounter and anyone that bids, all of them are woefully inadequate to be able to actually go to the level of realism that will be necessary to validate these tools. 16:10.080 --> 16:22.960 And even with us, it is an extreme long shot as to whether or not this will be achievable in the time frame and advancing fast enough to be able to support this post-attack analysis. 16:23.040 --> 16:25.100 And so let me riff on that for a second. 16:25.160 --> 16:44.420 Cyber-physical testbeds before this program started were primarily focused on let's either build an environment where we're looking purely at a physical phenomenon, or let's build an environment that's proving out a hypothesis, physical or cyber, and look at it from that particular angle, 16:44.420 --> 16:46.740 sort of ignoring all of the other details. 16:46.800 --> 16:51.200 But in this program, everything is unknown coming in. 16:51.200 --> 16:53.160 You don't know what the attackers did to you. 16:53.160 --> 16:57.280 You don't know even what the attackers want to do to you on the environment. 16:57.280 --> 17:00.580 So you have to have every piece of it as real as possible. 17:00.580 --> 17:14.400 But you can't possibly go build, you know, three real crank paths and 24, 27 real substations out there because it just costs way too much money to do. 17:14.400 --> 17:16.520 And it's a dangerous environment to be in. 17:16.520 --> 17:31.740 So how can you minimize the environment, maximize the safety, and also maximize the realism without running into problems of naysayers with simulation being involved or emulation of devices, etc. 17:31.740 --> 17:46.840 So they need to be able to touch it, they need to be able to feel it, they need to be able to see it, and they need to be able to trust that what it does and how it works is going to be reflected or reflective of what happens in the real world. 17:47.240 --> 18:04.080 So the outcomes of the testbed work itself, obviously we've created a lot of tools, we've pioneered some new techniques and methodologies, we've combined existing solutions, both that we've had and that others have had, to build an environment together. 18:04.080 --> 18:28.300 And we combined that, not just the academic knowledge that we had at the University of Illinois, but in partnerships with key vendors and also with asset owners and operators, and to build really an environment that reflected not only the real world, but that took a whole leap forward on its ability to evaluate research. 18:28.480 --> 18:30.860 So what is a testbed? 18:30.860 --> 18:39.780 So a testbed really is somebody has a need, let's call them the customer, and that need is to evaluate something. 18:39.780 --> 18:44.920 And so a testbed is assets, the thing maybe that they want to evaluate on. 18:44.920 --> 18:52.460 It's the people with the knowledge on how to build that environment in the way that represents the scenario they need to look at, etc. 18:52.500 --> 18:55.840 What to capture, how to capture, where to capture it, etc. 18:55.840 --> 19:05.620 It's the science of how to do so in a realistic way, while still enabling the necessary data capture that sometimes these systems inherently don't support. 19:05.720 --> 19:07.720 And then it's that data itself. 19:07.720 --> 19:12.740 And that data itself is what is captured from the devices, either willingly or not. 19:12.740 --> 19:26.880 How the system was operating, packet captures as an example of the communications that are going across, ground truth as to the physical telemetry of what was actually going across, not just simply what the devices are reporting is happening. 19:26.880 --> 19:37.280 And then a manifestation of or configuration of all of those things together that is provisioned out into an environment that you then do the work on. 19:37.280 --> 19:42.920 Our capability, we can provision locally the assets in our central environment. 19:42.920 --> 19:48.920 We can provision portable environments like what we've built on Plum Island and deploy on Plum Island. 19:48.920 --> 19:52.740 And we can also provision into the cloud in a variety of different ways. 19:53.600 --> 19:56.140 So why a testbed? 19:56.160 --> 19:58.500 What's the value of a testbed? 19:58.500 --> 20:02.560 You've seen, obviously, the ICS Village and Capture the Flag stuff. 20:02.560 --> 20:04.340 But this is a little bit different. 20:04.560 --> 20:12.900 And the reason for a testbed, the reason for the work that we do, is that this mission-critical technology, and why do I call this mission-critical technology? 20:12.900 --> 20:26.400 So the technology being built under RADIX is intended to be, let's say, have its glass broken when we're in a blackout scenario, effectively, after we've been hit. 20:26.400 --> 20:29.700 That's where the real value of this technology is. 20:29.720 --> 20:36.760 And there's arguments, and I am one of the people that will argue this, that that technology needs to be used even before we're hit. 20:36.980 --> 20:40.260 But in the end, our grid is down. 20:40.400 --> 20:43.000 That's what this technology is built to solve. 20:43.160 --> 20:55.900 And so it is absolutely essential, if this technology is called to practice, that it works, and that it resolves the issue, or that it can figure out what's going on in the issues, etc., before we need it. 20:55.900 --> 21:12.440 Because if we don't, and we're in a national blackout, national disaster sort of scenario, attacked by an enemy or otherwise, how do we come back if we break the glass on this technology, and it's not been proven to actually work? 21:12.440 --> 21:15.360 And so you run it, and it's like, I can't figure out what's going on. 21:15.360 --> 21:16.620 I see nothing wrong. 21:16.620 --> 21:20.560 There's no problems here whatsoever, but the devices still won't turn on. 21:20.560 --> 21:22.820 The grid still is down. 21:22.820 --> 21:25.480 These devices aren't operating correctly. 21:25.580 --> 21:27.600 And that's a bad scenario to be in. 21:27.600 --> 21:33.240 So this is truly mission-critical technology, and we have to prove that it's effective before we need it. 21:33.280 --> 21:35.600 But we have to go beyond the theoretical testing of it. 21:35.600 --> 21:51.420 We have to put it in all sorts of scenarios across all sorts of different platforms to verify that it works, even in edge cases that it wasn't expecting, in circumstances where it's missing data, in circumstances where it's even being directly attacked or attempted to be, 21:51.420 --> 21:54.580 let's say, misled on what is going on. 21:55.160 --> 22:09.960 So our solution for that is a realistic, recomposable, and well-instrumented testbed is essential to being able to prove that out, because even the real grid environment cannot be manipulated in the way that we can with a testbed environment. 22:09.960 --> 22:13.200 And I'll talk a little bit about some of the innovation in that space. 22:13.820 --> 22:23.780 And frankly, everything, as I started with my opening salvo and the proposal, that existed, including the Illinois capabilities, wasn't good enough before this program started. 22:24.560 --> 22:29.200 So our approach across it is to build real systems. 22:29.200 --> 22:50.400 We also build models, looking at models on the cyber side and physical side, that adapt to the exercise needs, that help us build out behaviors and changes in the flows and the communications of systems, to operate like the real world or to operate in a way that an adversary may be able to manipulate. 22:50.860 --> 22:53.660 Everything is built in this modular way. 22:53.660 --> 22:58.240 It's adaptable and, let's say, recomposable in a variety of different ways. 22:58.240 --> 23:09.880 So we can take a piece and take a substation and how it's physically wired and physically set up at the moment, press a couple buttons, push a different configuration, and now it's a different substation. 23:09.880 --> 23:14.940 Same devices, different configs, different network layout, etc. 23:14.980 --> 23:25.600 And all of that adaptability or modularity allows us to recompose the system in any way necessary to present different challenges, etc., as we go through. 23:25.680 --> 23:27.580 There's also instrumentation. 23:27.580 --> 23:35.180 And so the instrumentation is key in that many of these systems, let's say, will cooperatively give you a certain amount of data. 23:35.180 --> 23:40.840 But sometimes when you're doing forensic analysis, you need to be able to gather things that are deeper. 23:40.960 --> 23:48.540 Or if you're trying to do experimental validation, you need to be able to look at things that the system inherently won't tell you. 23:48.540 --> 23:53.100 Or you need to look at it with much more scrutiny than what you would typically look at in the real world. 23:53.100 --> 24:03.940 So how do you do that and turn on an appropriate level of data output, data capture, etc., but that doesn't actually affect the behavior of the systems? 24:03.940 --> 24:07.680 Because sadly, some of these systems, as you may know, are underpowered. 24:07.680 --> 24:15.440 And if you turn on, let's say, full complete logging of the system or other things, it can bog down the operation of the system. 24:15.440 --> 24:19.620 And then it no longer participates or acts as it would in the real world. 24:20.200 --> 24:23.600 So the last part is really knowledge. 24:23.600 --> 24:30.960 And by that, I mean, we don't just say, look, as academics, we're bright people, trust us, this works. 24:30.960 --> 24:33.000 We had to bring in real operators. 24:33.000 --> 24:54.960 We had to bring in the manufacturers, the vendors across many different platforms and talk through with them their best practices, their common misconfigurations that they see when integrators are building their platforms, the common ways that they configure their substations in the real world for the asset owners, 24:54.960 --> 25:08.180 etc., to both cause, let's say, human error to happen in ways that people accidentally misconfigure things, but also to mimic as closely as possible how people are actually configuring these in practice. 25:08.200 --> 25:22.720 And that is to get the right level of protection, the right level of output of data, and even notions of like, okay, what does a SIP-compliant substation look like in terms of what it is logging or not, versus one that's not. 25:23.320 --> 25:26.340 So that all comes together in that knowledge area. 25:27.220 --> 25:32.400 On the innovation side, we had to innovate quite a bit. 25:32.500 --> 25:45.440 And so the, let's say, orchestration or automation stuff that we had previously was good enough for research, but it made a lot of assumptions. 25:45.440 --> 26:02.900 And so by that, I mean, it would operate in a centralized environment, but when it tells something to be reconfigured or tries to control something, it expects that, A, the device is reachable, B, that it has access to that device, that it has the credentials to get on that device. 26:02.900 --> 26:10.580 It assumes, I guess, C, that it knows what the state of that device is. 26:10.580 --> 26:19.660 And then lastly, everything that it did before, it also assumed that the device was trustworthy and in a known sort of condition. 26:19.980 --> 26:32.240 So we had to, let's say, break down all of those assumptions and operate in a way that, let's say, didn't rely on any of those existing substations. 26:32.240 --> 26:35.380 We applied a bunch of research as well. 26:35.380 --> 26:47.460 Obviously, we had over a decade of work in the prevention space and in the detection space and remediation space at the University of Illinois. 26:47.920 --> 26:50.640 And we had used that in the testbed in a variety of ways. 26:50.640 --> 26:58.940 We had used it or proved it out in the testbed, some of which is even in formal companies now transitioned either to big vendors or as startups. 26:59.020 --> 27:07.680 And we had to apply that in sort of a different way as part of the testbed, not to just say, okay, look, here's the testbed environment. 27:07.720 --> 27:24.560 But for instance, if we could reach deeper into a device, then we used some of that research that we had to dig deeper into those devices and pull out and extract information that supports the validation of the technology without affecting the performance of the device itself. 27:25.080 --> 27:34.600 Our team, in particular, myself and a few others have went really deep on some of these platforms over the years. 27:34.600 --> 27:42.980 And so we brought a wide variety of devices to the table that we already knew quite a bit about on the inside. 27:43.120 --> 27:48.740 Sometimes even, let's say, one could argue more than what the vendors know about their own devices. 27:49.600 --> 27:54.900 In terms of the knowledge and ways that we could poke around inside of these platforms. 27:55.440 --> 28:05.800 We had to also build, because we needed to deploy on an austere environment, if you don't have it, then you better bring it type notion. 28:05.800 --> 28:08.980 And so we had to build these boxes in a way that were field serviceable. 28:08.980 --> 28:20.540 We had to be able to quickly replace components of the system if it were to break or if the intent of the cyber attack against it was literally to brick it so that it was no longer functional. 28:20.540 --> 28:31.100 How did we restore that or replace that in as fast of a situation as possible to move on and not basically stop the whole exercise if something were to break? 28:31.300 --> 28:45.340 We had to advance our automated configuration, data extraction, and also the notion of the system and its observation when even the network links were no longer trustworthy or reliable to be up or down. 28:45.340 --> 28:53.620 Remember, we're in a black start scenario, so we're not even guaranteed that we'll have power on each of the substations to be able to communicate to them. 28:53.620 --> 29:08.560 And when they're brought up, we need to maintain the state of everything we captured as they go up and down like a seesaw as they're being attacked and brought up and brought back down, etc. 29:08.640 --> 29:24.120 And then we also needed to have the environment in a way that could be recomposable, change the structure of the crank paths, change the behavior of the substation itself, without going through and recabling or rewiring everything that's in there on a hands-on nature. 29:24.980 --> 29:27.400 So what are these environments? 29:27.700 --> 29:32.140 So combined, I call them the substations in a box. 29:32.140 --> 29:33.680 It's two components. 29:33.680 --> 29:40.100 This has been built on the extensive facilities we have at the University of Illinois that I've alluded to. 29:40.100 --> 29:50.840 We have roughly $100 million worth of hardware and software at the University of Illinois that have been built up over the past decade plus, much of which by donation. 29:51.120 --> 30:03.580 That's enabled all sorts of research that we've done in the past with Trustworthy Cyber Infrastructure for Power, which was an NSF effort, a DOE DHS effort called Trustworthy Cyber Infrastructure for the Power Grid. 30:03.580 --> 30:05.280 It added grid to the end. 30:05.280 --> 30:16.160 Our most recent center that's wrapping up in the next year or two called the Cyber Resilient Energy Delivery Consortium, which is also DOE and DHS funded. 30:16.160 --> 30:25.720 Our Critical Infrastructure Resiliency Institute, and a variety of other things that have and leverage the testbed resources at Illinois. 30:26.520 --> 30:36.360 So the substations in a box, as I mentioned, they're designed to support this Blackstart crank path analysis and deployed in the field, real grid environments built, etc. 30:36.360 --> 30:42.460 They're built in pelican style cases, so they're literally shippable and deployable anywhere we need to stand them up. 30:42.460 --> 30:52.000 They're generally mostly IP55 watertight when they're shipped and moved around. 30:52.000 --> 30:57.340 When you physically deploy them, we take the case lids off and put them in enclosures. 30:57.340 --> 31:08.720 The reason for that is literally so the devices inside don't overheat, but also because you do need some physical access to the devices to control breaker operations and other aspects. 31:09.120 --> 31:24.780 We built an environment on an island, so the power infrastructure of what is in the overhead and underground stays, but basically everything else gets torn down and built back up every six months in a different way. 31:24.900 --> 31:30.860 There are currently 26 variants of substations deployed across that infrastructure. 31:30.860 --> 31:47.920 Those substations have relays, RTUs, substation network switches, routers, as well as an experimental fabric underneath that's controlled by SDN that allows us to do a lot of the capture and, let's say, dynamic changes of the substation itself. 31:47.920 --> 31:49.860 All sorts of protocols are deployed. 31:49.860 --> 31:53.000 You can see a list of them up on the screen. 31:53.000 --> 31:55.860 There's both serial and Ethernet communications. 31:55.920 --> 32:03.660 We have custom power connections on the power boxes that allow us to link these systems together in a safe way. 32:03.660 --> 32:07.860 And then we have high-voltage infrastructure that I'll talk about as well. 32:08.600 --> 32:12.000 So what's a power box and what's in a power box? 32:12.000 --> 32:21.140 So power boxes basically think of it like the physical infrastructure of the island or of a real substation. 32:21.140 --> 32:25.920 So that's the breakers, the bus bars, the incoming and outgoing feeders on the system. 32:25.920 --> 32:28.460 We have a local load feeder as well. 32:28.460 --> 32:35.020 We have signalization lights that indicate what the status is of energization, what the status is of breakers. 32:35.140 --> 32:40.080 We have a dead bus sync light that's provided. 32:40.080 --> 32:52.700 We have analog sync check relays, contactors, auxiliary contacts on the systems, CTs and PTs, control circuitry behind that allows us to operate breakers and various other stuff. 32:52.700 --> 33:06.800 We have different modes of operation, sort of a safe mode where we can walk away from the system and the system can't possibly change, which is a sort of a unique scenario differing from the real world. 33:06.800 --> 33:11.760 And then we obviously have the ability to locally control breakers as well. 33:11.760 --> 33:13.280 So what does that look like? 33:13.280 --> 33:19.060 So there are effectively two types of power boxes inherently that we've built. 33:19.060 --> 33:22.360 One is a 208 volt three-phase system. 33:22.360 --> 33:25.280 One is a 480 volt three-phase system. 33:25.280 --> 33:30.820 They look basically identical from the front, except for the size of them is a little bit different. 33:30.820 --> 33:53.220 We also have these high voltage systems, which really are Hoffman enclosures that are wall mounted and act like, let's say, semi-intelligent breakout boards for providing telemetry from the high voltage gear to the corresponding devices that are then operating and controlling that high voltage gear. 33:53.540 --> 33:58.440 And so think of it sort of like a mapping board in a way. 33:58.840 --> 34:09.640 And each power box generally has an incoming circuit, a load circuit, and then two outgoing circuits as it's built out. 34:09.640 --> 34:16.120 So basic electrical diagrams in the middle, but nothing, let's say, shocking about that. 34:16.120 --> 34:17.900 And then there's the other side of it. 34:17.900 --> 34:24.580 And so each of these devices have the number two SO cord coming into these Hubble connectors on the edge. 34:24.580 --> 34:39.860 But they also have umbilical cords, which are amphenol connectors, mil-spec amphenol connectors, that basically take all of the telemetry of what is happening on inside the box and provide that telemetry to the devices that need to control it. 34:39.860 --> 34:56.100 So that includes the analog and digital signals that need to be sent back and forth between the devices to control them, but also the CTs and PT outputs, etc., from the system itself so that all of the sensing is detectable by the relay boxes. 34:56.180 --> 34:58.220 And so what are the relay boxes? 34:58.220 --> 35:01.060 Well, the relay boxes are really the brains of the substation. 35:01.060 --> 35:06.620 So here are a couple examples showing some of the different technology that's in play. 35:06.860 --> 35:12.740 Up in the upper left, those are ABB relays along with an ABB RTU. 35:12.740 --> 35:22.340 This is sort of a legacy RTU platform that ABB leverages or uses and has deployed around the world called the RTU-560. 35:22.440 --> 35:27.120 In the middle, you'll see some more ABB relays, middle-top. 35:27.220 --> 35:31.640 And above that, instead of an RTU-560, you see a Motorola device. 35:31.640 --> 35:34.080 This is a Motorola ACE-3680. 35:34.160 --> 35:47.760 If you move to the next image, upper right-hand corner, that is an ABB COM-600 rack mount or a COM-600R that is acting as the RTU over those ABB relays that are there. 35:47.760 --> 35:52.640 Bottom left-hand corner, you'll see some touchscreen SEL-751 relays. 35:52.640 --> 36:00.080 Those are controlled... the RTU in that particular case is an SEL-RTAC, a 3505. 36:00.480 --> 36:03.280 In the middle, you'll see touchscreen relays. 36:03.280 --> 36:08.200 In that one, there's an SEL-RTAC as well, but that's an SEL-RTAC. 36:08.200 --> 36:12.040 It's a SEL-3530 instead of a 3505. 36:12.780 --> 36:16.860 And then the far right, you'll see more SEL relays. 36:16.860 --> 36:18.200 These ones are not touchscreen. 36:18.200 --> 36:21.040 These are another variant of the SEL-751. 36:21.040 --> 36:26.920 And those ones are being controlled from an RTU perspective by a Novatek Orion LX. 36:27.600 --> 36:32.600 So this shows just some of the diversity of platforms that are there. 36:32.600 --> 36:37.040 There are much more, obviously, across 26 substations. 36:37.400 --> 36:41.120 Every single substation is unique in some way, shape, or form. 36:41.120 --> 36:48.800 So we have a lot of diversity across the environment in terms of platforms, technologies, and configurations. 36:48.800 --> 36:52.100 And so diversity could be purely on the configuration side. 36:52.100 --> 37:04.260 For instance, different protocols being communicated, different topologies being set up between Utility A, Utility C, Utility B, etc. 37:05.420 --> 37:08.640 And lots of other variation on top of that. 37:08.700 --> 37:18.820 So let's talk about some, let's say, lessons learned from the program or challenges that we had to tackle, now that you understand some of the gear. 37:18.820 --> 37:20.640 First off is safety. 37:21.640 --> 37:40.940 When you can't trust anything in the system whatsoever because it is compromised and it's compromised in a way that you may or may not know, may or may not be able to determine, and you don't know what is trustable or not, all of the systems that are there are effectively designed to protect you. 37:41.120 --> 37:46.980 But if you can't trust the digital systems to protect you anymore, then you need additional layers of protection. 37:46.980 --> 37:51.200 So we had to layer protection throughout the system in both physical and cyber form. 37:51.200 --> 38:04.440 And that included things like analog time, analog protections in the system, like analog sync check relays, time over current protections, thermal protections. 38:04.500 --> 38:15.140 On the digital side or on the cyber side, the protective relays were configured with safe and sane settings for overvoltage and undervoltage conditions, etc. 38:15.140 --> 38:23.880 The boxes had arc flash analysis done on them to determine potential exposure or safety measures from PPE perspective that needed to be done. 38:23.880 --> 38:27.820 The cabinets were all either pad lockable or direct key lockable. 38:27.980 --> 38:35.060 We had, on the high voltage side, interrupters that were acting as fail safes if there's anything that flows through. 38:35.060 --> 38:50.500 The low voltage to the high voltage side, an out of sync close or something like that that happened to happen, or a surge somewhere, the interrupters or a fault on the line, the interrupters were there to protect the system at the high voltage side. 38:50.780 --> 38:55.220 We also purposefully did not target the high voltage control. 38:55.300 --> 39:00.040 That way we didn't run into, let's say, big issues. 39:00.040 --> 39:07.120 Low voltage was something we could cause a problem on and be okay with, but on the high voltage side people could get hurt. 39:07.500 --> 39:12.020 All of the connectors we used were screw-in locking style connectors. 39:12.160 --> 39:15.160 We had all sorts of internal wiring protection, etc. 39:15.160 --> 39:21.400 So the key is that the environment itself was designed to be safe no matter what. 39:21.520 --> 39:31.240 So people that knew nothing about power systems could still safely operate in this environment and not run a risk of being electrocuted or whatever. 39:31.240 --> 39:45.420 We always had power engineers and safety officers effectively on site that were making sure that people did safe operations and maintained the necessary perimeters, even at a noise level from the generators and various other aspects. 39:45.420 --> 40:08.520 But really the system protected itself in every way, so let's talk about some operational lessons that we learned in executing exercises for blackstart restoration, but also in austere environments under conditions of blackouts, etc. 40:08.640 --> 40:13.640 And so let me talk a little bit about the mode of execution as we go through this. 40:13.640 --> 40:19.920 So when I say that we're operating utility environments, we are actually operating the utility environments. 40:19.920 --> 40:29.560 And by that I mean utility operators from real utilities come to the island and they run the infrastructure. 40:29.780 --> 40:37.580 And they basically take control of it, we hand it over to them, and then they tell everyone else what to do, how to do it, when to do it, etc. 40:37.920 --> 40:39.300 on the system. 40:39.300 --> 40:46.620 Now obviously we have some exercise control over that, but the intent is to really make this as real as possible. 40:46.620 --> 40:48.740 So let's talk about that realism. 40:49.240 --> 40:55.580 Many people don't believe what is possible until it really is, let's say, slapping them in the face. 40:55.580 --> 41:12.540 And by that I mean a common view is, look, relays, they're embedded devices, you can't make them do things, you can't disable their protection, you can't change their mode of operation beyond their config. 41:13.220 --> 41:17.340 And until they saw us do that, they didn't really believe it. 41:17.340 --> 41:34.780 Even when it was happening right in front of them, they still didn't believe it until they dug deeper and started to look deeper at what was happening and how it was happening to realize that, look, bad things really are possible, that right now your mindset is that this isn't possible at all, 41:34.780 --> 41:35.960 that no one can do that. 41:35.960 --> 41:39.980 You had to physically modify the device in order to do that. 41:39.980 --> 41:43.220 And it's like, no, we can actually do that via cyber means. 41:43.260 --> 41:46.360 So let's talk about the people, right? 41:46.380 --> 42:10.400 One of the things that was an operational lesson is academia is really good about thinking outside the box, but we needed to be extremely agile and think even further outside of the box and push through and build stuff that, frankly, was, let's say, impossible to build at any given moment. 42:10.400 --> 42:17.280 We took on building one utility, then two, then three, all in six-month iterations. 42:17.280 --> 42:29.160 Completely different architectures, completely different devices, building the physical boxes, and standing it up in a new realistic configuration at each sort of interval. 42:29.160 --> 42:31.080 That's pretty difficult to do. 42:31.080 --> 42:34.040 It's multiple weeks to build the environment each time. 42:34.040 --> 42:35.880 It's multiple weeks of testing. 42:35.880 --> 42:53.580 It's multiple weeks of evaluation to make sure that it is as real as possible, that it doesn't have inherent artifacts itself that people may view as compromises, etc., in the system, that it is pristine, blue sky, trustable, etc., and built the way it should be built. 42:53.860 --> 43:07.480 And that results in, let's say, extreme levels of stress at times, but as a team, not just the University of Illinois, but broadly the entire program, we all pulled together and found success in every exercise that we had. 43:07.480 --> 43:13.260 There was no exercise that failed because the infrastructure or the people failed to deliver. 43:14.700 --> 43:16.860 And so that brings us to pace. 43:16.860 --> 43:19.720 So I mentioned every six months. 43:19.720 --> 43:28.620 So DARPA programs move very fast, and the expectations are very high of the technology, of the evaluation, of the test bed, etc. 43:28.620 --> 43:33.780 And so there were many times that we were facing failure. 43:33.780 --> 43:54.540 But let's say by pure blunt force, and by pure blunt force, I mean number of hours and long days and leveraging and leaning on each other throughout the program, we were able to pull through and pull off what seemed to be impossible when we started. 43:54.880 --> 44:01.920 So those are some just operational aspects that we learned as we built the environment, especially out on an island. 44:01.920 --> 44:05.260 And so when I say, you know, what are some challenges, right? 44:05.260 --> 44:24.820 So there were times when we were, you know, we took a ferry every day from the mainland to the island, and there were times when, you know, nor'easters were blowing in and other aspects where the waves were 12, 14 feet high, and the, you know, conditions were such that if you go to the island, 44:24.820 --> 44:28.440 you're going to be sleeping there because you're probably not going to make it off. 44:28.760 --> 44:36.360 And that's kind of, you know, let's say a bit much, but there were many days that happened like that. 44:36.360 --> 44:52.680 There were days we were out there executing the environment, trying to restore after the cyber attack, and, you know, there was nor'easters blowing through with 40 plus mile an hour winds and downpours and inches of rain falling an hour, etc. 44:52.680 --> 45:10.220 So not just hard from a technical perspective, but even harsh environments that we were out in as we were trying to restore these devices, and as the teams and the utility operators were operating these devices faced with those types of environmental conditions. 45:10.600 --> 45:13.940 So we learned a whole bunch of other lessons too. 45:13.940 --> 45:35.120 So one of them that was interesting to us was that when the systems are intended to break, when you know that you can't trust anything anymore, you have to think differently about what you can build and how you build it so that it works consistently and reliably even when everything is intended to be broken. 45:35.120 --> 45:40.720 So that goes back to some of the stuff we did with safety, but also on the cyber side. 45:40.720 --> 45:47.420 So as an example, we had no guarantee of power, reliably or not, at any point in time. 45:47.420 --> 46:09.320 So how do we guarantee we don't lose data, as an example, as we're going through, or that we get, let's say, eventual consistency where if part of the network is up and being controlled by our orchestration and another part of it is down, how do we get it to catch back up and get in the right configuration when it does come back up if it needed to be changed, 46:09.320 --> 46:19.720 or to collect all of the data that it had when it wasn't centrally reachable, when it was isolated and only, let's say, on backup power on its own. 46:20.220 --> 46:28.160 So that presented some interesting challenges that we had to tackle, which is basically fault-tolerant and distributed computing in a nutshell. 46:29.200 --> 46:37.460 The other issue that was somewhat interesting that we had to innovate on was the gear in the field typically isn't hot swappable. 46:37.460 --> 46:52.740 You can't just go grab a relay that's in a real substation and pull it out and pop another one in, in most of the substations, without having to do some rewiring, without having to take circuits out of commission, de-energize them, lock out, tag out procedures, 46:52.740 --> 46:53.460 etc. 46:53.700 --> 47:12.200 But if you're in a fast-paced exercise environment, we have seven days, and if people get stuck and we need to reverse out basically what the bad people did, how do we reverse that out in as fast a way as possible without bringing the whole exercise down or the whole exercise to a halt for hours on end? 47:12.200 --> 47:18.420 So we had to create sort of quick connects and all sorts of other things that allowed us to swap gear out very, very quickly. 47:19.240 --> 47:30.980 Further, when we're testing bleeding edge technology, things fail because sometimes that technology doesn't work, or the research doesn't do what it's supposed to do. 47:30.980 --> 47:35.300 And so that's your primary plan is, okay, look, it's going to try this, and if it works, great. 47:35.300 --> 47:36.900 Well, you have to have a backup plan, right? 47:36.900 --> 47:45.080 And in many cases, we had to have another backup to the backup plan because we couldn't fail and make everything stop working. 47:45.360 --> 47:55.620 So if anyone in the program failed, we had to have a backup plan for what would happen if they failed, and then a backup plan for if our backup plan failed. 47:56.440 --> 48:05.240 And then the last time, the last one, which is a lesson learned is we thought we were prepared in many, many, many occasions. 48:05.240 --> 48:26.760 But let's say, you know, in between days of the exercise, when we would pause and go back to our hotels to sleep, etc., I would often make runs to Home Depot or Lowe's or electrician stores or whatever, because no matter how many spare parts we had, no matter what tools we had on us or available to us, 48:26.760 --> 48:31.940 anything that can break will break at some point when you're running these environments. 48:31.960 --> 48:37.120 And it's almost always in the way that you never thought of or that you couldn't have anticipated. 48:37.120 --> 48:46.580 Something that has, you know, a 10,000-hour mean time between failure or 100,000-hour mean time between failure fails in, you know, 10 hours instead. 48:46.860 --> 48:56.440 So lots of interesting challenges on just keeping things up and operational and making sure that stuff was readily available if anything did break. 48:56.940 --> 49:01.580 So let's talk a little bit about some personal takeaways, and I'll wrap these up really quickly here. 49:02.020 --> 49:10.180 But let's talk, let's say, not as the University of Illinois, but let's talk about this from my perspective. 49:10.180 --> 49:25.160 So vendors, be this ICS cyber solutions or the vendors themselves, they often claim capabilities that, let's say, are much more limited in the real world application than what most people realize. 49:25.160 --> 49:32.820 So a utility may say, I go buy this platform and it's got me covered and it can do all of these things and I can check that box and I'm good. 49:32.940 --> 49:40.960 The reality is, no matter what vendors claim, there's often still very large gaps there. 49:41.300 --> 49:51.220 And so even the commercial off-the-shelf ICS cyber solutions that are out there, they're missing huge amounts of surface area on what adversaries can do against these boxes. 49:51.360 --> 49:56.160 Radix technology was designed to close some of those gaps, but not all of them, right? 49:56.160 --> 50:01.640 So even as great as the Radix technology is, we still have a long way to go, right? 50:01.640 --> 50:22.280 In building the environments that we built and helping to construct the exercise evaluations and the test effective payloads and all that, in essence, is how do you cause a condition from a cyber means to happen on these devices, such that there are artifacts or implementations that then people can forensically find, 50:22.280 --> 50:23.000 right? 50:23.520 --> 50:28.380 So all of those, in doing that, we found hundreds of issues on these devices. 50:28.380 --> 50:30.980 And it's not like we've never looked at these devices before. 50:30.980 --> 50:36.600 And by issues, I don't necessarily mean security vulnerabilities, but in some cases it was security vulnerabilities. 50:36.640 --> 50:43.580 But merely other items like, look, compatibility between host A and host B, it doesn't work. 50:43.620 --> 50:46.520 Even though they're supposed to interoperate, it doesn't. 50:46.520 --> 50:55.280 There's nuances, differences, there's differences in documentation on what is communicated on the actual wire versus what it says it's going to communicate as. 50:56.200 --> 51:02.400 Another thing, which was a personal takeaway, is being the adversary is fun, right? 51:03.240 --> 51:09.100 It's nice to be on the red team and hack these devices and break them in a variety of ways. 51:09.360 --> 51:27.380 But in the program, the defense and recovery technology, including the RADIX technology and commercial off-the-shelf stuff that I've personally seen, it's still way behind what I was capable of or what others on the red team were capable of doing to these systems. 51:27.440 --> 51:34.940 So if we're so far ahead, we could just obliterate any of that tech, then what fun is there in that? 51:34.940 --> 51:36.900 It's not even a fair fight at that point. 51:36.900 --> 51:43.800 So a lot of times we limited the activities that we were doing to poking the bear rather than destroying it. 51:43.800 --> 51:46.820 So we didn't go out for the throat kill. 51:46.820 --> 51:56.280 We kept a pace with what the technology was capable of and designed something that was just a little bit ahead of that, pushing them each iteration to improve. 51:56.940 --> 52:14.000 And so one sort of broad takeaway is, and I say this not just of technology in North America or whatever, but the whole world, what they have in this space in industrial control systems, detecting cyber attacks on these devices, that detection, that mitigation, 52:14.000 --> 52:18.020 and that remediation technology for the electric power grid. 52:18.040 --> 52:26.020 And I'll stretch it a bit and say actually all critical infrastructure still really has a long way to go. 52:26.060 --> 52:39.500 There's still so much work that needs to be put on those platforms to really protect the systems from somebody who is truly focused and determined and understands these systems at a deep and inherent level. 52:40.060 --> 52:43.240 So that's all of my technical content. 52:43.280 --> 52:54.440 I will just flash a slide real quick, which is the testbed at Illinois and much of what we've built in the RADIX program has been enabled by lots of companies. 52:54.440 --> 53:04.340 These are some of the the companies that have donated gear to us, software to us, et cetera, that have helped enable the things that we've done. 53:04.340 --> 53:13.260 But without them and without the commercial support and the vendor support and the utility support broadly across this program, we wouldn't be successful. 53:13.260 --> 53:15.380 The DARPA RADIX program wouldn't have been successful. 53:15.380 --> 53:19.280 So thank you to all of those companies and what they did. 53:19.320 --> 53:23.340 And then I think we still have a couple minutes for questions. 53:23.560 --> 53:28.260 And so I'll also leave a sort of bonus link down there at the bottom. 53:28.260 --> 53:38.960 There's a GitHub repo that I created a number of years ago at the S4 conference, and I've been maintaining it sort of ad hoc ever since. 53:39.020 --> 53:52.200 And that's a bunch of ICS security tools in a variety of different forms that are aggregated and categorized in various ways and mirrored when their original location is no longer available. 53:52.300 --> 53:53.360 So do check that out. 53:53.360 --> 53:55.640 It has a whole bunch of useful things in it. 53:55.640 --> 54:01.980 And with that, I will stop talking and I think we might have a little bit of time for question and answer. 54:06.070 --> 54:07.250 Let's see. 54:07.250 --> 54:08.310 Hey, Tim, this is Bryson. 54:08.310 --> 54:09.210 How are you doing? 54:09.350 --> 54:10.010 Wonderful. 54:10.010 --> 54:10.910 Hi, Bryson. 54:11.070 --> 54:11.530 Thank you. 54:11.530 --> 54:12.990 Thank you for the talk. 54:12.990 --> 54:19.670 I noticed that you are on our Discord and I actually promoted you to a speaker during your talk. 54:19.970 --> 54:22.570 So you should now have that badge tied to you. 54:22.790 --> 54:32.390 And what we recommend is if you could post that GitHub link in the speaker Q&A section of Discord. 54:32.530 --> 54:32.890 Okay. 54:32.890 --> 54:38.630 And then reach out for folks to engage you there and for questions. 54:38.930 --> 54:39.310 Okay. 54:39.310 --> 54:40.550 Sounds good. 54:40.970 --> 54:44.710 So really appreciate you jumping on and giving this talk. 54:45.370 --> 54:50.190 Having been and seen this for myself, it is really impressive. 54:50.230 --> 54:57.850 And my favorite part for the sensors was of course the inflatable guys like you see next to the used car sales. 54:58.050 --> 55:02.690 I thought that was a really interesting way to show whether something is up or down at a physical distance. 55:02.950 --> 55:05.510 Yeah, we affectionately call them the dancing men. 55:07.770 --> 55:17.670 And sort of a funny aside, in the Nor'easters, when you have torrential downpour, those things get wet and then they turn into sort of like whiplashes. 55:17.670 --> 55:27.490 So I was up there, you know, untangling them on many occasion and getting sort of smacked by the dancing men as I was trying to untangle them so that they could fly. 55:27.970 --> 55:30.130 Well, that's how you know it works. 55:32.720 --> 55:41.120 Well, anyway, Tim, I appreciate you joining us and supporting the village and look forward to the commentary and the Q&A on Discord. 55:41.620 --> 55:42.160 Awesome. 55:42.160 --> 55:42.360 Yeah. 55:42.360 --> 55:47.980 And everyone do check out what they've set up for, you know, the CTFs and other things in the village. 55:47.980 --> 55:52.540 You know, they do an awesome job of creating environments that you can play with. 55:52.660 --> 55:58.160 Sadly, I can't offer my environment out to the world in such an easy and accessible way. 55:58.200 --> 56:07.680 But hopefully in the future, I'll get deeper engaged with the ICS village and bring some of this tech and some of this capability to the village so you guys can all play with it too. 56:08.100 --> 56:09.560 Yeah, we look forward to that. 56:09.560 --> 56:13.580 That's probably been the biggest innovation we've had is because of the pandemic. 56:13.920 --> 56:22.600 The amount of effort we've had to spend on figuring out how to make these things virtually accessible, which is the typical limitation for concurrent access. 56:22.820 --> 56:24.500 We kind of have solved it. 56:24.500 --> 56:27.740 So we'd love to touch base with you afterward and talk about it. 56:27.800 --> 56:32.100 Yeah, it's, you know, great to hear that you guys have had to tackle that. 56:32.100 --> 56:36.280 We're currently tackling that for the DARPA Radix effort as well. 56:36.520 --> 56:40.560 With the last exercise, it will be predominantly remote. 56:40.680 --> 56:59.400 And we went from the prior exercise being, you know, network isolated, completely trusted, everyone in a specific zone to, you know, part of this is deployed in the cloud, and everyone is distributed around the country, and all accessing this in a controlled and crazy way, 56:59.400 --> 57:03.120 including like streaming body cams that we'll have and all sorts of stuff. 57:03.120 --> 57:07.160 So we're on that same roller coaster due to COVID at the moment. 57:09.160 --> 57:11.540 Well, we look forward to collaborating. 57:11.540 --> 57:14.800 So again, thank you very much, and we'll see you on Discord. 57:15.340 --> 57:16.800 Yeah, you guys have a great day. 57:16.960 --> 57:17.600 All right, take care.