1 00:00:00,000 --> 00:00:05,480 Good afternoon, everybody. Next up is Chris Sistrunk with what's the difference for DFIR 2 00:00:05,840 --> 00:00:16,260 ins for ICS. So give it over to Chris. Thanks for coming. Can everybody hear me? Okay, 3 00:00:16,280 --> 00:00:22,700 great. So hopefully you've been able to go see the ICS village and that's what we're here 4 00:00:22,700 --> 00:00:29,620 representing. If you get a chance to, oh, thanks, nice. If you get a chance to go over there and 5 00:00:29,620 --> 00:00:34,960 play around, actually try to plug in and see things because a lot of people don't get a chance 6 00:00:34,960 --> 00:00:40,300 to do that and a lot of the equipment is cost prohibitive for most researchers. So go get a 7 00:00:40,300 --> 00:00:46,440 chance to see that. So my name is Chris Sistrunk. I'm a professional engineer. I can talk 8 00:00:46,440 --> 00:00:54,340 more about myself later. This is a lot of information to cram into a talk. But I'm going to 9 00:00:54,340 --> 00:00:58,900 cover digital forensics and incident response like an overview. How many of you have ever done 10 00:00:58,900 --> 00:00:59,600 digital forensics and incident response? I don't know if you've ever done digital forensics. 11 00:01:02,960 --> 00:01:10,260 Okay. How many of you have done it for control systems? Yeah, very few people. When I gave this 12 00:01:10,260 --> 00:01:16,860 talk at Black Hat last year, it was like three people raised their hand that had done for 13 00:01:16,860 --> 00:01:23,700 embedded systems. So we're going to talk about DFIR for embedded systems and ICS, embedded 14 00:01:23,700 --> 00:01:28,840 devices, what to collect, what to analyze and do some. Since I'm an electrical engineer, I work 15 00:01:28,840 --> 00:01:29,580 for a power company. I'm going to talk a little bit more about that. I'm going to talk about the 16 00:01:29,580 --> 00:01:30,580 power of the power. I'm going to talk about the power of the power of the power. And we're going to 17 00:01:30,580 --> 00:01:34,200 talk about what we want to do. What we want to do is cover some examples for substation RTUs and 18 00:01:34,200 --> 00:01:40,020 how to do digital forensics on those. And it covers a couple of operating systems there. So 19 00:01:40,020 --> 00:01:45,060 what we want to do for incident response. We have find evil, assess the situation, define 20 00:01:45,060 --> 00:01:52,360 objectives, collect evidence. This is from the book, the incident response book. And this is the 21 00:01:52,360 --> 00:01:59,560 same framework that we want to do for any incident response. So if it's for IT or for the 22 00:01:59,580 --> 00:02:08,920 or anything. It's almost akin to doing root cause analysis in engineering. So we do an 23 00:02:08,920 --> 00:02:14,840 overview where we're collecting files, operating system, volatile and nonvolatile things, 24 00:02:15,080 --> 00:02:21,300 network traffic, application data. We do an examination in digital forensics. We do an 25 00:02:21,300 --> 00:02:26,560 analysis and reporting. So this is from the NIST standards in the incident response and 26 00:02:26,560 --> 00:02:33,400 computer forensics book. Those are some really great resources. If you get a chance to go get 27 00:02:33,400 --> 00:02:42,500 those books or get that standard, it's a free standard, NIST SP 800-86. So there's a lot of 28 00:02:42,500 --> 00:02:47,980 traditional DFIR tools that are mature, red line and volatility. Anybody ever use those 29 00:02:47,980 --> 00:02:56,380 before? I mean, I think it's great that the SANS ICS team has done some challenges where 30 00:02:56,560 --> 00:03:01,760 they actually wanted you to use red line and volatility for an ICS type challenge. That's 31 00:03:01,760 --> 00:03:07,780 great. And there's websites like virus total malware and a couple of others. We have cheat 32 00:03:07,780 --> 00:03:12,480 sheets and books. But what do we have for control systems? We don't have any of those kind of 33 00:03:12,480 --> 00:03:18,220 tools. So what is the difference for ICS? Hopefully that's the pun of the day. So for 34 00:03:18,220 --> 00:03:24,360 assess the situation, you know, when, where, how is ICS affected? Define objectives, return 35 00:03:24,360 --> 00:03:26,540 the control system to normal. So that's a great tool. And then there's a lot of other 36 00:03:26,560 --> 00:03:31,160 things that you can do quickly and safely. Collect evidence. You know, ICS devices have real 37 00:03:31,160 --> 00:03:38,060 time operating systems. I won't read the rest. But here's the difference. You have a similar 38 00:03:39,800 --> 00:03:47,380 situation between IT, DFIR and control system DFIR for assess the situation. But for define the 39 00:03:47,380 --> 00:03:53,580 objectives, it's physical processes. It's not just network. It's not just a data center. You have 40 00:03:53,580 --> 00:03:56,540 to deal with, you know, a lot of different things. You have to deal with a lot of different things. 41 00:03:56,560 --> 00:04:04,240 You have to deal with oil moving in a pipeline or, you know, power moving through power lines or 42 00:04:04,240 --> 00:04:11,960 water through pipes. Manufacturing facilities where you have a chain of, you know, things 43 00:04:11,960 --> 00:04:18,520 happening. Those are the things that are going to be important when you do DFIR. In collecting 44 00:04:18,520 --> 00:04:22,680 evidence, there's a difference there. It must be collected manually most of the time because we 45 00:04:22,680 --> 00:04:26,540 don't have any tools. And a lot of times you'll have to use the tools. You'll have to use the tools. 46 00:04:26,560 --> 00:04:30,700 You'll have to use the tools that are in the engineering workstation like a software from a 47 00:04:30,700 --> 00:04:37,220 vendor or something like that. When we do analysis, same thing. There's no ICS specific DFIR 48 00:04:37,220 --> 00:04:44,520 tools. You can't use a red line or a volatility. You can if it's a Windows based HMI. You can do 49 00:04:44,520 --> 00:04:49,300 that. But that's not the focus of this talk. We want to focus on the embedded systems and the 50 00:04:49,300 --> 00:04:56,540 custom operating systems. Communicate the same. Develop a remediation plan. You know, we're 51 00:04:56,560 --> 00:05:00,200 talking about how and when to regain control of the control system. ICS devices have 52 00:05:00,200 --> 00:05:08,200 constraints. So you can't just go reimage a PLC like you can a laptop. Sometimes you have to load 53 00:05:08,200 --> 00:05:15,460 firmware and the configuration and make sure you have, you know, a lot more things. So it's not 54 00:05:15,460 --> 00:05:22,280 just willy‑nilly go just like we do in IT. Reimage the laptop, no big deal. So with control 55 00:05:22,280 --> 00:05:26,540 systems, you have to really think about how you're going to do that. You have to really think about 56 00:05:26,540 --> 00:05:33,800 how we do the remediation plan. Kick the bad guys out or if they're still attacking. Find out what 57 00:05:33,800 --> 00:05:42,020 to do and do it safely. So let's talk about ICS anomalies. I know some of you in here know about 58 00:05:42,020 --> 00:05:48,040 anomalies on control systems. But we want to say an anomaly of some kind has occurred. Increased 59 00:05:48,040 --> 00:05:53,600 network activity, strange behavior, some kind of a failure. Now we need to investigate it. Is it 60 00:05:53,600 --> 00:05:56,520 known bad? Is it unknown bad? Do we escalate this to a failure? Is it known bad? Is it unknown bad? 61 00:05:56,540 --> 00:06:03,620 As a security incident, do we have to call in our security teams or notify EISAC or any, you know, 62 00:06:03,640 --> 00:06:11,060 ICS cert or whoever? And who do we call? Do we call engineers, admins, PR, safety vendors, legal? 63 00:06:12,560 --> 00:06:21,100 So here's an example of an incident. Might be an anomaly. So someone made a mistake in an example 64 00:06:21,100 --> 00:06:26,520 configuration. So instead of 2400 ball, they put 2600 ball. So instead of 2400 ball, they put 2600 65 00:06:26,540 --> 00:06:32,240 ball. And so how many times if you're an engineer, you've been called at 5 o'clock on Friday because 66 00:06:32,240 --> 00:06:39,340 someone missed a configuration. And that's happened to me a lot. So then you have, you know, 67 00:06:39,380 --> 00:06:46,620 maybe squirrels. That caused an incident. Or you may have some kind of unknown thing doing 68 00:06:46,620 --> 00:06:55,380 something weird that you've never seen before. So don't reboot it. Don't reboot it. 69 00:06:56,540 --> 00:07:03,240 A lot of times, unless it's a safety issue. Or there's some other issue that you can do to fix it. 70 00:07:03,300 --> 00:07:11,380 But normally we want to preserve the forensics. And if you reboot the system, you lose the memory 71 00:07:11,380 --> 00:07:17,960 and a lot of other things. So have you tried turning off it again? Don't do that. ICS collection 72 00:07:17,960 --> 00:07:25,620 tools. So we don't have any ICS specific DFR tools, especially for embedded devices. But we do 73 00:07:25,620 --> 00:07:26,520 have, you know, we have, you know, we have, you know, we have, you know, we have a lot of 74 00:07:26,540 --> 00:07:33,320 other ways to collect the data manually. We can use TerraTerm or some terminal program to connect, 75 00:07:33,420 --> 00:07:41,400 you know, Telnet or serial or whatever. Serial cable, right? Connect to the console. You can use 76 00:07:41,400 --> 00:07:47,300 the software like from GE or Schweitzer or whoever, Siemens, to collect information about the 77 00:07:47,300 --> 00:07:53,160 PLCs, the configurations, all that stuff. So you have to use the tools that are designed to go 78 00:07:53,160 --> 00:07:53,840 with that system. 79 00:07:55,500 --> 00:08:02,000 So what can we collect? Two types. Physical data and digital data. So whenever you have a response 80 00:08:02,000 --> 00:08:07,160 that you need to do something, you need to figure out exactly what device it is and where it is and 81 00:08:07,160 --> 00:08:14,760 what rack it's in or what panel or what platform, whatever it is. Any identifying information, 82 00:08:15,240 --> 00:08:20,940 manufacturer, serial number, part number, name. Because most control systems, you try to have a 83 00:08:20,940 --> 00:08:23,820 good asset inventory. But we all know that there's a lot of data in there. So you have to figure out 84 00:08:23,820 --> 00:08:28,880 what's in there. There may not be an up-to-date asset inventory. And you've got to make sure that you 85 00:08:28,880 --> 00:08:35,020 collect that right there and then. What kind of connections does it have? What are the LEDs on the 86 00:08:35,020 --> 00:08:41,460 front and the back? They have diagnostic LEDs that are maybe green or red or, you know. With RTs 87 00:08:41,460 --> 00:08:47,620 down, well, what are the lights? Are they green? Well, yeah. Well, it's good. There's something else 88 00:08:47,620 --> 00:08:53,700 that's wrong. Power consumption, is it using more power than normal if you're monitoring that? 89 00:08:53,820 --> 00:08:59,700 Evidence of tampering. So temperature running high, that's an interesting one. Mark Fabro 90 00:08:59,700 --> 00:09:10,500 found a HMI that was running hot by using a FLIR camera. And it was hotter than the other HMI, and 91 00:09:10,500 --> 00:09:18,420 they found out it had a trojan on it, and it was using more than normal CPU. So it made it run 92 00:09:18,420 --> 00:09:23,800 hotter. So that's how they detected it, with the FLIR camera. Digital data. You want to have data, 93 00:09:23,820 --> 00:09:27,920 to be able to get the running configuration, whatever user account sits in there, last 94 00:09:27,920 --> 00:09:34,460 known good configuration. So if it's in a repository or if you have a CD or a thumb 95 00:09:34,460 --> 00:09:39,800 drive, however they store it, if it's on the engineering workstation, try to find the 96 00:09:39,800 --> 00:09:46,000 last known good configuration. What is the running firmware, approved firmware? Because 97 00:09:46,000 --> 00:09:53,140 we've seen in the real world someone put in non-approved firmware. CPU usage percentage, 98 00:09:53,140 --> 00:09:58,540 memory usage, RAM, storage, all that stuff. What running processes, just like you do with 99 00:09:58,540 --> 00:10:04,680 Windows, you know, net stat and all these things, you can find out what running processes, 100 00:10:04,880 --> 00:10:12,420 what running ports are active. Logs, if you have any, you might be able to do, you might not 101 00:10:12,420 --> 00:10:17,880 have syslog but you can log in and look at the console logs. You may have to use terminal 102 00:10:17,880 --> 00:10:23,120 server with a serial port. And then if you, if you have, if you have, if you have a 103 00:10:24,240 --> 00:10:29,240 system that may be running VXWorks, you can do a memory dump. And I'll show you how we did 104 00:10:29,240 --> 00:10:38,460 that. So what to analyze? Find evil or ways for evil to do evil. So we have a time base. So 105 00:10:38,460 --> 00:10:43,620 things, you know, things you want to do fast and first and things that take more analysis. So, 106 00:10:45,160 --> 00:10:53,060 you know, first responders, what do the user and event logs say? Sometimes an RTU may only have 107 00:10:53,140 --> 00:11:01,880 255 spots in its buffer before it rolls over. So you've got to go grab that quick. What is the 108 00:11:01,880 --> 00:11:07,320 configuration? Does it match the firmware? Is it firmware approved? Same thing. A lot of these 109 00:11:07,320 --> 00:11:12,140 things that you got to go send a technician out there to collect or an engineer or whoever. 110 00:11:12,380 --> 00:11:17,780 Is the configuration and logic correct for the process? So an engineer can look at the 111 00:11:17,780 --> 00:11:23,120 configuration and say, oh, someone made a mistake. So that is not as fast as grabbing the configuration. 112 00:11:23,120 --> 00:11:28,340 configuration. But you've got to do a little analysis there. Are the communications normal? 113 00:11:28,640 --> 00:11:33,340 So you're going to have to look at a serial packet capture or Ethernet packet capture. And 114 00:11:33,340 --> 00:11:39,480 that takes a little bit more time. And what takes even more slower, I mean, is things where 115 00:11:39,480 --> 00:11:44,740 we're having to do getting a vendor, a digital forensic specialist, an embedded systems 116 00:11:44,740 --> 00:11:50,820 analyst. So analyze embedded operating system files, capture data at rest, capture data in 117 00:11:50,820 --> 00:11:58,820 transit. And, you know, in the case of, like, in destroyer, crash override or any, like, 118 00:11:58,860 --> 00:12:03,680 Stuxnet, you know, this takes time. It takes a lot of time to analyze it and reproduce it. 119 00:12:04,140 --> 00:12:10,740 Volatile memory. And to look for code injection or potential root kits. So keep these things in 120 00:12:10,740 --> 00:12:16,440 mind when you have a DFR process for control systems. So let's do DFR and two substation 121 00:12:16,440 --> 00:12:20,620 RTUs. The one on the left is a D20 from GE. The one on the right is a Schweitzer. 122 00:12:21,840 --> 00:12:28,000 And if you are working in the electrical sector, you've seen these. So time to read the manual. 123 00:12:28,620 --> 00:12:35,760 So that's the Apollo manual. And you have to have good manuals. So let's look at the GE D20 124 00:12:35,760 --> 00:12:44,060 MX manual. Anybody here ever seen a D20 before? Okay. I didn't bring it. But this is just an 125 00:12:44,060 --> 00:12:49,660 example. If you have different PLCs, you don't have to know this stuff. But just remember, go 126 00:12:49,660 --> 00:12:55,600 look in the manuals that you have when you get back home. Because you may have, when you get back 127 00:12:55,600 --> 00:13:00,240 to work, when you look at the manuals, you may have all the same information. So this is going to 128 00:13:00,240 --> 00:13:06,900 show you. So we found out the specs. It has an embedded power quick pro. It tells you how 129 00:13:06,900 --> 00:13:13,900 much memory it has. It's running VX works. And then, you know, we've got the tools there. We've 130 00:13:13,900 --> 00:13:19,640 got the software, SG config. I can use terminal. I can use WinSCP. And I can look at the, you know, 131 00:13:19,660 --> 00:13:25,980 the manual that came with it. And then in chapter 11 in there, it actually shows how to 132 00:13:25,980 --> 00:13:31,580 troubleshoot serial communications, firmware version, mismatches, shell commands and logs. So 133 00:13:31,580 --> 00:13:37,140 in this manual, it actually shows you how to do some digital forensics. So here's some examples 134 00:13:37,480 --> 00:13:48,680 that it has some shells that belongs to GE. And then there's the VX works shells. So let's look 135 00:13:48,680 --> 00:13:49,640 at some of the error logs. 136 00:13:49,660 --> 00:13:58,660 Whenever an RTU technician first logs in, they want to look at the error logs. So you see, you 137 00:13:58,660 --> 00:14:02,940 know, some diagnostic information. If there's any errors in the configuration, they're going to show 138 00:14:02,940 --> 00:14:09,680 up right here. I don't know if you can see my laser pointer. But you can see the top part, it 139 00:14:09,680 --> 00:14:15,380 tells some information about the Ethernet ports. And radius is not configured. So there's some 140 00:14:15,380 --> 00:14:19,580 errors. But if there's other errors, they would show up in that screen. Now the user 141 00:14:19,660 --> 00:14:28,100 log is really nice. You can export it to a CSV file. And it also shows every command. So if there's, 142 00:14:28,100 --> 00:14:32,340 you know, what they type and all that stuff. So it's very, very important for digital forensics. 143 00:14:33,500 --> 00:14:40,840 So the power of the three shells, there's a D20 shell and two VX work shells. And a lot of these 144 00:14:40,840 --> 00:14:48,280 shells, sometimes GE, the factory, may want to use these commands with a technician on site. So just 145 00:14:48,280 --> 00:14:56,480 be careful when you're in these things. Other, like Ovation and other GE devices, they use VX 146 00:14:56,480 --> 00:15:03,580 works. So some of these same things apply. So anybody remember this movie? He doesn't know how to use 147 00:15:03,580 --> 00:15:11,080 the three C shells. So you got to know how to use the shells. So the main shell shows you all these 148 00:15:11,080 --> 00:15:18,260 different commands. So you can just type in help and it tells you in the D20. And then what you can do 149 00:15:18,280 --> 00:15:25,280 is look in the user manual. And if you type IFINF, it will tell you what the Ethernet port is. And if 150 00:15:25,280 --> 00:15:32,360 you look in the manual, it tells you what that command does. So read your manuals. And this is great 151 00:15:32,360 --> 00:15:43,540 for an attacker, too, because if you see, you can also do things like restart, inject memory. You 152 00:15:43,540 --> 00:15:48,260 can dump the memory. You can do a lot of things from this. So it's very juicy for you. And you can 153 00:15:48,280 --> 00:15:54,000 use this for an attacker's perspective. So let's do data collection. We can get the running 154 00:15:54,000 --> 00:16:00,000 configuration. It's a very common task. Last known good configuration. Like I said before, you can 155 00:16:00,000 --> 00:16:05,960 type in IMG to get the running firmware. You can query the processes. You can query the 156 00:16:05,960 --> 00:16:14,760 performance. And get the CPU usage. All these things that we would do in Windows forensics. You 157 00:16:14,760 --> 00:16:18,260 just have to type these in manually. So let's go ahead and get started. So let's get started. 158 00:16:18,280 --> 00:16:23,680 You can do a serial analyzer. Remember what we had before, Wireshark? You have to look in 159 00:16:23,680 --> 00:16:31,480 serial analyzer. So we can look in the RTU and say it can tell us this is an example DMP3 string. 160 00:16:31,740 --> 00:16:39,400 So to and from. And we get this and send it off for troubleshooting if we were in the field, if 161 00:16:39,400 --> 00:16:46,240 there was a communication problem. Now, this is really great. VxWorks allows you to dump the 162 00:16:46,240 --> 00:16:48,260 memory. So you can do a serial analyzer. You can do a serial analyzer. You can do a serial analyzer. 163 00:16:48,260 --> 00:16:53,660 You can do a system information dump and it tells you the start of where the memory is in the 164 00:16:53,660 --> 00:17:00,140 addresses. So you need to know that information when you get ready to dump the information. And 165 00:17:00,140 --> 00:17:07,580 you type in D and you can see I put a D, a bunch of zeros, and then FF at the end. So I just 166 00:17:07,580 --> 00:17:18,240 grabbed the first 255 bytes of the memory. But in the D20, you can only do a serial analyzer. 167 00:17:18,260 --> 00:17:24,580 You can't do it over serial. They turned off doing it over SSH. So there's a gig of RAM and it 168 00:17:24,580 --> 00:17:29,600 takes a while to dump all the memory. But we're going to show you some more commands in VxWorks. 169 00:17:29,720 --> 00:17:38,360 Because, like, the Mars Rover uses VxWorks. A lot of these, it's the most popular embedded 170 00:17:38,360 --> 00:17:43,460 operating system outside of like Android. So VxWorks is everywhere and you can use these same 171 00:17:43,460 --> 00:17:48,240 commands. You can, in the, in the, in the, in the, in the, in the, in the, in the, in the 172 00:17:48,260 --> 00:17:53,960 D20 that runs VxWorks, but anything that runs VxWorks, if they have it enabled in the firmware, 173 00:17:54,180 --> 00:18:00,260 you can run a PCAP straight from the command line, which is really great for forensics. 174 00:18:03,820 --> 00:18:11,100 Okay. Now we're going to put on our evil hat for a minute. And we're going to do a live memory 175 00:18:11,100 --> 00:18:18,240 code injection and memory dump on the D20. So we injected M and then we're going to run the 176 00:18:18,260 --> 00:18:25,300 memory to, to do a memory edit. And then we put in dead beef. And you can kind of see it there. 177 00:18:26,760 --> 00:18:35,840 And then we can do D and dump the memory and we see dead beef when we dumped it. So there's no 178 00:18:35,840 --> 00:18:41,200 live demo here. We just took screenshots. So we show that if you are an attacker and you have 179 00:18:41,200 --> 00:18:48,200 console access to this RTU, you can inject memory just like you can do with Windows or Linux or 180 00:18:48,260 --> 00:18:57,340 whatever. Okay. Now we need tools to enable us to perform DFR for ICS and embedded devices. So 181 00:18:57,340 --> 00:19:04,460 with FRI, we got, I got with my good friend at, on the Flare team, they are reverse engineers, and 182 00:19:04,460 --> 00:19:12,040 I said, can you write us a tool that will dump memory out of VxWorks and then do a differential 183 00:19:12,040 --> 00:19:16,580 analysis on that. So we've written it and I'm going to show you the links to it and show you 184 00:19:16,580 --> 00:19:17,940 some things about it. 185 00:19:18,000 --> 00:19:18,240 Okay. 186 00:19:18,260 --> 00:19:28,020 So we wrote a VxWorks DFR tool. It has a bunch of, it's all in Python and it's up on our 187 00:19:28,020 --> 00:19:36,160 GitHub. So there's the address ICSmemcollect on the Fri GitHub. What it can do, it will read and 188 00:19:36,160 --> 00:19:42,220 write to memory of the device. It can cache the live memory locally. And it can compare the 189 00:19:42,220 --> 00:19:48,240 system image. So from GE, you get a firmware set when you download it. And then when you download 190 00:19:48,240 --> 00:19:50,220 it, you get a firmware that is there to compare the memory. And what we do is we compare the 191 00:19:50,220 --> 00:19:58,300 memory that we don't against with the firmware image. And then it does that comparison. It can 192 00:19:59,220 --> 00:20:12,000 work over serial, TCP IP and it supports caching. So if you're say on a D20, you have to do it 193 00:20:12,000 --> 00:20:18,220 over serial. And it takes a long time. So we had some problems dumping it. And it takes like a 194 00:20:18,220 --> 00:20:25,700 six days. Because the fastest you can go is 115, 200 baud. If you could do it over SSH, it would be 195 00:20:25,700 --> 00:20:31,320 a lot faster to dump the memory. So we allowed to resume if connectivity is lost. And it works on 196 00:20:31,320 --> 00:20:36,680 anything that looks like a Python file object. So cache files, memory dumps, sparse memory maps 197 00:20:37,820 --> 00:20:48,200 and other special objects. So here's an example. Josh Triplett explains this a little bit better. 198 00:20:48,200 --> 00:20:53,840 Than I can. If you go back and watch the black hat presentation from last year, you can hear his 199 00:20:53,840 --> 00:21:05,180 explanation. But here's a, it says Python validate image, disk image, VX works. And it looks at 200 00:21:05,180 --> 00:21:13,020 the cache. And you can see a section of the memory says IP firewall start. So in VX works, that 201 00:21:13,020 --> 00:21:18,180 is where it starts the firewall. So what we did is we started the firewall. So what we did is we 202 00:21:20,160 --> 00:21:29,200 injected something to make the firewall not start. So we turned the firewall off in memory. And 203 00:21:29,820 --> 00:21:35,500 so here's some more about the, I'll show you, we're going to show you the outputs from the tool. 204 00:21:35,640 --> 00:21:44,820 But it uses, it loads everything. It works with the architecture of that device, which is really 205 00:21:44,820 --> 00:21:48,180 cool. I don't know how all that works. But it works with the architecture of that device, which is 206 00:21:48,180 --> 00:21:54,440 really cool. It basically pulls from these two things. It loads everything in capstone. 207 00:21:56,100 --> 00:22:03,220 Basically it assembles and disassembles all of the things that go into the firmware for that 208 00:22:03,840 --> 00:22:12,300 instruction set for that processor. Now the plans for the future, we want to do better 209 00:22:12,300 --> 00:22:18,160 documentation. Testing with other embedded devices using VX works like Ovation controller. Or, you 210 00:22:18,160 --> 00:22:25,080 know, for the GE digital relay. Expand the tool to work on other devices and other embedded 211 00:22:25,080 --> 00:22:30,300 operating systems. And refine the scripts in the easy to use modules. Maybe you can have a 212 00:22:30,300 --> 00:22:38,540 volatility plug-in of some kind. And allow for feedback. So if you are interested in using this, 213 00:22:39,700 --> 00:22:44,960 go to that FireEye GitHub and you can download it and you'll have to compile it and do a lot of 214 00:22:44,960 --> 00:22:48,140 things. But if you have any questions, feel free to reach out to me. I'll be happy to answer any of 215 00:22:48,160 --> 00:22:55,000 your questions about it. Just send us an e-mail and we'll help. Okay. Let's go to the Switzer 216 00:22:55,000 --> 00:23:01,880 R-TAC. So this is we wanted to cover a Linux embedded device. And this one doesn't give us any 217 00:23:01,880 --> 00:23:10,500 of the shells like the VX works does. They protected the command line. But it runs on a power PC. 218 00:23:10,740 --> 00:23:17,860 Has a gig of RAM and embedded Linux. And, you know, it has the instruction manual that you can get. 219 00:23:18,160 --> 00:23:28,080 And the software for it. And you can use the web browser and terminal for SSH. So let's do some 220 00:23:28,080 --> 00:23:33,340 collection here. We have to look at the digital data so we can get the running configuration. User 221 00:23:33,340 --> 00:23:38,320 counts, running firmware. All that stuff is on the screen when you log in to it on the 222 00:23:48,300 --> 00:23:55,040 computer. And then you can reset the device so you can log in. And I've highlighted over, there's a 223 00:23:55,040 --> 00:24:00,120 button at the top. If you highlight over it, it says if the password jumper is in or not. So it's 224 00:24:00,120 --> 00:24:03,560 just something to keep track of. If an attacker was physically there, they could put that 225 00:24:03,560 --> 00:24:10,280 password jumper in. And then get into your system. But then it's a physical security problem. 226 00:24:11,620 --> 00:24:18,140 So here's an example of the screens that we did from whenever we did an attack with DMP3. So back in, 227 00:24:18,140 --> 00:24:31,020 2003, 2013, we, Adam Crane and I were doing DMP3 fuzzing. And we had a Switzer 3530. And it was 228 00:24:31,020 --> 00:24:40,760 vulnerable to DMP3. It basically didn't validate the incoming DMP3 messages. So this is a 229 00:24:40,760 --> 00:24:48,120 screenshot, just a little snippet of the web page. And when we sent a DMP3 packet, unsolicited, it was 230 00:24:49,560 --> 00:24:59,980 made to CPU to 100%, which is not normal. Then you can see the memory usage was very large. And 231 00:24:59,980 --> 00:25:06,200 then overtime the RAM crept up to a 100%. And then what happened was, the hardware watchdog 232 00:25:06,200 --> 00:25:11,660 rebooted, you can see the middle screen there, this application has failed, continue anyway. 233 00:25:11,840 --> 00:25:18,100 When it rebooted, the configuration was gone. So you can see that the configuration on the right side, this screen, the 234 00:25:18,100 --> 00:25:26,160 scratched through and it's got a failed project error message. And you can see that the memory 235 00:25:26,160 --> 00:25:33,320 went back to nothing. And then the storage usage was zero kilobytes. So it was really 236 00:25:33,320 --> 00:25:38,840 strange that this message caused this thing to happen. But we could monitor that. If we 237 00:25:38,840 --> 00:25:42,860 were monitoring these things, we could have some really good digital forensics and incident 238 00:25:42,860 --> 00:25:52,400 response capabilities. So data collection, you can get several different reports and logs and 239 00:25:52,400 --> 00:26:04,300 especially syslog. So a good friend of mine, Nathan, he, what he did was he took syslog, but in 240 00:26:04,300 --> 00:26:10,260 the Swyter device you can make anything go into syslog. So he did all kinds of cool things to 241 00:26:10,260 --> 00:26:12,840 custom make syslog. So if you have a device and you want to make it go into syslog, you can 242 00:26:12,840 --> 00:26:16,480 do that. Someone plugged in the Ethernet cable, he could write that to syslog. If you unplug 243 00:26:16,480 --> 00:26:22,980 it, it writes it to syslog. If you, if a port is active and then he tied in a door switch to the 244 00:26:22,980 --> 00:26:28,820 cabinet. So he tied that into syslog. So all these really cool things that you can do with this 245 00:26:28,820 --> 00:26:37,760 device. It talks about all these different reports on the devices that's connected to it. And 246 00:26:37,760 --> 00:26:42,820 you can access it, Microsoft access. So you can run some database type things. It does a lot of 247 00:26:42,820 --> 00:26:48,020 things. There's no Linux shell. So that's, that's sad. But there's pros and cons to that. It 248 00:26:48,020 --> 00:26:54,960 means that an attacker can't write memory to it like they could with a D20 or any VxWorks device. 249 00:26:57,880 --> 00:27:04,080 We can also capture packets just like what the D20 could. And you can, it ties right into 250 00:27:04,080 --> 00:27:12,240 Wireshark. So it's really great. And this is an example message on the right. And it can do, they 251 00:27:12,820 --> 00:27:18,780 wrote serial Wireshark detectors for cell fast messaging, IntelliGear 8979. Those are two 252 00:27:18,780 --> 00:27:24,280 serial protocols that they wrote Wireshark parsers for. Which is really great. Because we can 253 00:27:24,280 --> 00:27:33,240 tunnel those over TCPIP now to make things work. As we upgrade our systems to newer things. So 254 00:27:33,240 --> 00:27:39,260 we're getting close to the end of the talk. For further reading about digital forensics and VxWorks 255 00:27:39,260 --> 00:27:42,800 and other embedded systems, you can go to the link in the show notes. You can download the 256 00:27:42,800 --> 00:27:44,540 files. They can get to the end of the talk. So you can see more. Had a blog post about VxWorks 257 00:27:44,540 --> 00:27:52,080 seven years ago. They wrote a Medusploit module for VxWorks remote memory dump. It does the 258 00:27:52,080 --> 00:28:00,920 same thing that our tool does. But it's you can look it up. WBRPC memory dump is the module 259 00:28:00,920 --> 00:28:10,480 name. David O'Dell wrote a blog post about QNX. And that's a really good blog post to check out. 260 00:28:10,480 --> 00:28:17,040 out. And then ICS has recommended practices for ICS forensics. If you need to take a picture of 261 00:28:17,040 --> 00:28:24,560 that. Go to the next one. Travis Goodspeed, hopefully you guys know Travis. He wrote some 262 00:28:24,560 --> 00:28:34,360 very good information about MSP 430. So he did some digital forensics on that software family 263 00:28:34,360 --> 00:28:41,180 and the hardware family there with that specific chip. And Ralph Langner did a really great 264 00:28:41,180 --> 00:28:48,340 paper, especially in forensics data about Stuxnet. So if you read his paper to kill the 265 00:28:48,340 --> 00:28:54,300 centrifuge, it's really great. And then if you read Sands' Duck 5 about the Ukrainian power grid 266 00:28:54,300 --> 00:29:03,640 attack, included parts of writing firmware of the embedded serial devices. So just more things 267 00:29:03,640 --> 00:29:08,120 that's embedded. That's what we've got to get to. Does anybody have any questions? 268 00:29:14,820 --> 00:29:20,460 Yeah. So I've presented this at Black Hat. So you can download the slides from Black Hat. If you 269 00:29:20,460 --> 00:29:27,480 want these with the ICS Village logo, that's fine. But, yeah, the video from Black Hat is up as 270 00:29:27,480 --> 00:29:33,620 well. You can see Josh's explanation is a lot better about the IP firewalls and the IP 271 00:29:33,620 --> 00:29:39,600 firewall start and how he injected that and doing the Python memory differential analysis, 272 00:29:40,000 --> 00:29:45,360 which is really great. Hopefully this will inspire people to write more tools for embedded 273 00:29:45,360 --> 00:29:53,820 systems. And we need a community around that. So please try to install some of these things. And 274 00:29:53,820 --> 00:30:00,220 if you have that ability, and then maybe we can work on making more modules for this type of 275 00:30:00,220 --> 00:30:01,760 tool. Any other questions? 276 00:30:02,240 --> 00:30:02,800 Okay. 277 00:30:05,620 --> 00:30:10,640 Thank you so much for coming. And come see the ICS Village. We appreciate it.