1 00:00:00,450 --> 00:00:01,770 - So I'm Jimmy Wylie. 2 00:00:01,770 --> 00:00:04,380 I'm a principal malware analyst at Dragos 3 00:00:04,380 --> 00:00:06,600 and primarily, my job is to search for 4 00:00:06,600 --> 00:00:09,330 and analyze malware that targets critical 5 00:00:09,330 --> 00:00:11,550 infrastructure companies and equipment. 6 00:00:11,550 --> 00:00:14,040 So this could be lame shit like ransomware 7 00:00:14,040 --> 00:00:16,530 or it could be the dangerous stuff like Trisis 8 00:00:16,530 --> 00:00:19,560 or CrashOverride and now, Pipedream, 9 00:00:19,560 --> 00:00:21,840 which is what I'm gonna talk to you about today. 10 00:00:21,840 --> 00:00:25,320 And mostly the sort of problems and the results 11 00:00:25,320 --> 00:00:27,633 from testing this toolkit in our lab. 12 00:00:29,130 --> 00:00:30,900 So first we're gonna start with a little bit of background. 13 00:00:30,900 --> 00:00:33,153 Talk about what Pipedream's capabilities are. 14 00:00:34,230 --> 00:00:35,970 Talk about the hardware that we acquired, 15 00:00:35,970 --> 00:00:37,920 the lab set up and then the fun stuff, 16 00:00:37,920 --> 00:00:41,400 which is the actual results from testing in our lab 17 00:00:41,400 --> 00:00:42,240 and then talk a little bit 18 00:00:42,240 --> 00:00:44,820 about real world usage of this toolkit. 19 00:00:44,820 --> 00:00:46,020 Real quick before I go forward, 20 00:00:46,020 --> 00:00:47,130 I just wanna acknowledge all the people 21 00:00:47,130 --> 00:00:48,120 that helped me with this. 22 00:00:48,120 --> 00:00:49,860 It'd be disingenuous for me to represent 23 00:00:49,860 --> 00:00:51,390 all of this as my own work. 24 00:00:51,390 --> 00:00:52,830 There was a whole team behind me helping 25 00:00:52,830 --> 00:00:54,030 with the analysis and the writing 26 00:00:54,030 --> 00:00:56,160 and just fielding my stupid questions 27 00:00:56,160 --> 00:00:58,683 about certain critical infrastructure things. 28 00:00:59,790 --> 00:01:01,920 And one more caveat, you can read that yourselves, 29 00:01:01,920 --> 00:01:06,090 but really, the short summary here is I'm under an NDA 30 00:01:06,090 --> 00:01:08,670 and I can only say certain things about the malware. 31 00:01:08,670 --> 00:01:11,580 So in particular, I can't tell you where it came from. 32 00:01:11,580 --> 00:01:13,530 I can't show you screenshots of the malware. 33 00:01:13,530 --> 00:01:16,440 What I can do is show you what happens when we run it. 34 00:01:16,440 --> 00:01:18,930 The important thing to take away from this, 35 00:01:18,930 --> 00:01:20,310 what I'm allowed to say is that we found 36 00:01:20,310 --> 00:01:22,680 the malware before it was used. 37 00:01:22,680 --> 00:01:24,660 So in the ICS community, that does provide us 38 00:01:24,660 --> 00:01:27,510 with a unique opportunity to start building defenses 39 00:01:27,510 --> 00:01:28,710 for this kinds of stuff. 40 00:01:30,030 --> 00:01:31,273 So what is Pipedream? 41 00:01:31,273 --> 00:01:34,770 It's the seventh ICS impacting malware 42 00:01:34,770 --> 00:01:37,333 after the S-word and Havex and BlackEnergy, 43 00:01:38,741 --> 00:01:41,480 CrashOverride, Trisis, Industoryer 2. 44 00:01:43,230 --> 00:01:45,450 What distinguishes Pipedream from all these 45 00:01:45,450 --> 00:01:49,200 other sample sets, it's the first cross-industry, 46 00:01:49,200 --> 00:01:53,460 scalable, flexible attack framework for ICS equipment. 47 00:01:56,040 --> 00:01:58,710 So there's five components in total. 48 00:01:58,710 --> 00:02:01,980 We're only gonna focus on the three on the left, 49 00:02:01,980 --> 00:02:05,490 which are EvilScholar, BadOmen and MouseHole. 50 00:02:05,490 --> 00:02:08,070 All three of these are remotely interactive 51 00:02:08,070 --> 00:02:11,100 command line utilities for manipulating equipment. 52 00:02:11,100 --> 00:02:15,120 EvilScholar is a CODESYS version 3 attack tool 53 00:02:15,120 --> 00:02:18,480 with an initial targeting of Schneider Electric controllers. 54 00:02:18,480 --> 00:02:20,370 BadOmen is a similar utility command line, 55 00:02:20,370 --> 00:02:22,260 but it targets the OMRON controllers 56 00:02:22,260 --> 00:02:23,730 and you have MouseHole, which is a tool 57 00:02:23,730 --> 00:02:26,073 for interacting with OPC UA servers. 58 00:02:28,571 --> 00:02:29,820 Then we talk about our lab setup 59 00:02:29,820 --> 00:02:31,530 and some of the equipment. 60 00:02:31,530 --> 00:02:34,200 These are all the impacted devices of the initial set, 61 00:02:34,200 --> 00:02:35,703 what we know right now. 62 00:02:37,830 --> 00:02:39,420 But the thing to take away from this slide 63 00:02:39,420 --> 00:02:42,090 is that column on the left, CODESYS, 64 00:02:42,090 --> 00:02:44,250 Modbus and OPC UA are everywhere. 65 00:02:44,250 --> 00:02:45,450 If you go to CODESYS' website, 66 00:02:45,450 --> 00:02:48,000 there's hundreds of vendors that use CODESYS 67 00:02:48,000 --> 00:02:51,030 and they have multiple types of equipment that they sell. 68 00:02:51,030 --> 00:02:53,563 Tons of equipment from all the vendors speak OPC UA 69 00:02:53,563 --> 00:02:56,430 and Modbus is another extremely common protocol. 70 00:02:56,430 --> 00:02:58,830 So the breadth of impact of this malware set 71 00:02:58,830 --> 00:03:02,373 is more than we've seen in the past 10 years. 72 00:03:04,140 --> 00:03:05,550 Oop. There we go. 73 00:03:05,550 --> 00:03:08,190 So our lab, what we did was we set up 74 00:03:08,190 --> 00:03:10,960 an NX1P2 OMRON controller with a couple servo drives 75 00:03:10,960 --> 00:03:13,890 and a servo motor and then there's requisite stuff 76 00:03:13,890 --> 00:03:15,480 that we need to actually connect it to the network, 77 00:03:15,480 --> 00:03:18,480 like EtherCAT couplers and we have power supplies 78 00:03:18,480 --> 00:03:20,280 and all that kind of stuff. 79 00:03:20,280 --> 00:03:24,120 The Schneider Electric devices, we had a TM241 and TM251 80 00:03:24,120 --> 00:03:26,670 with two different versions of the firmware. 81 00:03:26,670 --> 00:03:28,770 We have an OPC UA server set up 82 00:03:28,770 --> 00:03:30,367 with our mock pipeline process. 83 00:03:30,367 --> 00:03:33,270 So it's a Kepware server with a bunch of Rockwell devices 84 00:03:33,270 --> 00:03:36,450 that are controlling a pipe and a pump. 85 00:03:36,450 --> 00:03:37,620 And then just some other hardware 86 00:03:37,620 --> 00:03:39,750 that we wanted to play with. 87 00:03:39,750 --> 00:03:41,880 So basically we just buy these devices 88 00:03:41,880 --> 00:03:44,130 off eBay or we asked a friend. 89 00:03:44,130 --> 00:03:47,370 Tom Van Norman over at ICS Village shipped me a 251 90 00:03:47,370 --> 00:03:48,930 in a pinch because I couldn't find one 91 00:03:48,930 --> 00:03:50,160 with the right firmware. 92 00:03:50,160 --> 00:03:51,780 Turned out, it still wasn't the right firmware. 93 00:03:51,780 --> 00:03:53,460 Ended up not being a problem. 94 00:03:53,460 --> 00:03:56,400 We tried to downgrade and it bricked the device. 95 00:03:56,400 --> 00:03:58,110 We could consider asking the vendor, 96 00:03:58,110 --> 00:04:00,720 but something to keep in mind with the industry is, 97 00:04:00,720 --> 00:04:02,730 in the regular IT world, there have been thousands 98 00:04:02,730 --> 00:04:05,130 and thousands and thousands of malware samples 99 00:04:05,130 --> 00:04:07,020 and these kinds of things are a little bit more common, 100 00:04:07,020 --> 00:04:09,450 but in the ICS space, aside from the deluge 101 00:04:09,450 --> 00:04:12,900 of ransomware, we have seven cases of ICS malware, 102 00:04:12,900 --> 00:04:14,940 and there's all kinds of other things 103 00:04:14,940 --> 00:04:16,804 that you have to worry about when talking to the vendor, 104 00:04:16,804 --> 00:04:19,650 they may not have the processes 105 00:04:19,650 --> 00:04:21,240 in place for even dealing with this. 106 00:04:21,240 --> 00:04:23,010 The NDAs can take some time. 107 00:04:23,010 --> 00:04:24,030 Personally, as an analyst, 108 00:04:24,030 --> 00:04:25,590 I have reporting deadlines and things. 109 00:04:25,590 --> 00:04:26,940 I need to get things to our customers. 110 00:04:26,940 --> 00:04:30,060 So generally refer to just purchase the stuff where we can, 111 00:04:30,060 --> 00:04:31,710 stay independent and then approach the vendors 112 00:04:31,710 --> 00:04:33,560 with the information once we have it. 113 00:04:35,880 --> 00:04:38,490 So when we go about analyzing this kind of thing, 114 00:04:38,490 --> 00:04:39,323 we had five pieces of malware, 115 00:04:39,323 --> 00:04:41,220 it was a big organizational task. 116 00:04:41,220 --> 00:04:42,053 We're basically focusing on a lot 117 00:04:42,053 --> 00:04:45,300 of the PLC malware first, statically analyzing it. 118 00:04:45,300 --> 00:04:47,130 So most of these toolkits were written in Python, 119 00:04:47,130 --> 00:04:49,683 a little bit of C++, pretty easy to read. 120 00:04:50,700 --> 00:04:52,050 So we do that static analysis 121 00:04:52,050 --> 00:04:53,760 while we're acquiring the hardware. 122 00:04:53,760 --> 00:04:56,400 We're gonna release initial details and mitigation advice 123 00:04:56,400 --> 00:04:59,760 and we do just a shit ton of runtime collection, 124 00:04:59,760 --> 00:05:02,220 collecting PCAPs of everything, 125 00:05:02,220 --> 00:05:03,810 check which techniques work, which don't, 126 00:05:03,810 --> 00:05:05,250 and then we're gonna release updated details, 127 00:05:05,250 --> 00:05:06,960 mitigation advice and then later on, 128 00:05:06,960 --> 00:05:08,850 all the follow on work that isn't as important 129 00:05:08,850 --> 00:05:10,470 for defending the networks. 130 00:05:10,470 --> 00:05:12,030 And that's gonna be things like actually 131 00:05:12,030 --> 00:05:14,130 stripping the firmware off the devices. 132 00:05:14,130 --> 00:05:15,990 Why we can't do that now while we're run time testing it? 133 00:05:15,990 --> 00:05:16,823 'Cause we could brick them. 134 00:05:16,823 --> 00:05:19,230 And like I said, sometimes they're not easy to find. 135 00:05:19,230 --> 00:05:23,130 So we're basically in step five at this point 136 00:05:23,130 --> 00:05:25,260 and that's been since, I think, April 137 00:05:25,260 --> 00:05:27,240 is when we started doing all of this. 138 00:05:27,240 --> 00:05:29,070 So the kinds of questions that we're asking ourselves 139 00:05:29,070 --> 00:05:30,840 before we get started with the lab testing, 140 00:05:30,840 --> 00:05:32,430 the main one, it's what's current, 141 00:05:32,430 --> 00:05:35,190 what do we have right now, does it work? 142 00:05:35,190 --> 00:05:36,750 Because we don't know yet. 143 00:05:36,750 --> 00:05:40,650 And then there's the CODESYS issue in EvilScholar. 144 00:05:40,650 --> 00:05:41,850 Does that actually work? 145 00:05:41,850 --> 00:05:44,160 Is that something specific to Schneider Electric 146 00:05:44,160 --> 00:05:47,520 or does it work on any CODESYS version 3 device? 147 00:05:47,520 --> 00:05:49,980 BadOmen has a module for manipulating 148 00:05:49,980 --> 00:05:52,110 servo drives and motors. 149 00:05:52,110 --> 00:05:54,360 We want to see if that worked on only the specific motor 150 00:05:54,360 --> 00:05:55,800 that was targeted in the malware 151 00:05:55,800 --> 00:05:58,860 or if it worked on other drives in that series. 152 00:05:58,860 --> 00:05:59,940 And then more generally, 153 00:05:59,940 --> 00:06:01,140 and this is the most important thing. 154 00:06:01,140 --> 00:06:03,660 Can these tools manipulate PLC logic? 155 00:06:03,660 --> 00:06:05,040 Meaning, can we actually manipulate 156 00:06:05,040 --> 00:06:06,570 this process specifically to create 157 00:06:06,570 --> 00:06:08,520 some kind of destructive effect? 158 00:06:08,520 --> 00:06:10,440 Then we think about future capabilities. 159 00:06:10,440 --> 00:06:13,590 Can we automate an attack with these tools? 160 00:06:13,590 --> 00:06:14,460 What do we think these tools 161 00:06:14,460 --> 00:06:16,260 are gonna look like in six months? 162 00:06:16,260 --> 00:06:18,560 We know we have them before they were actually used. 163 00:06:18,560 --> 00:06:20,100 So the implication there is that they're probably 164 00:06:20,100 --> 00:06:21,210 being actively developed. 165 00:06:21,210 --> 00:06:22,440 So these are gonna change by the time 166 00:06:22,440 --> 00:06:23,670 they actually get deployed 167 00:06:23,670 --> 00:06:25,980 and we wanna hypothesize what's not gonna look like 168 00:06:25,980 --> 00:06:27,960 so that we know what to look out for in the future. 169 00:06:27,960 --> 00:06:31,290 And then there's the last category, victimology, 170 00:06:31,290 --> 00:06:33,450 which is squarely in the vein of threat intel. 171 00:06:33,450 --> 00:06:35,343 Can we actually narrow down a target? 172 00:06:36,244 --> 00:06:38,070 (clears throat) 173 00:06:38,070 --> 00:06:40,080 So let's get to the fun stuff. 174 00:06:40,080 --> 00:06:42,530 So the first thing we started with was MouseHole. 175 00:06:44,820 --> 00:06:47,100 MouseHole is a remote console 176 00:06:47,100 --> 00:06:49,380 for interacting with OPC UA servers. 177 00:06:49,380 --> 00:06:52,740 So you can think about OPC as a translation mechanism. 178 00:06:52,740 --> 00:06:55,290 You speak to the OPC UA server and then it translates 179 00:06:55,290 --> 00:06:56,670 those messages to whatever language 180 00:06:56,670 --> 00:06:59,220 the other devices speak in order to manipulate them. 181 00:06:59,220 --> 00:07:00,270 It's the best way, you can just think 182 00:07:00,270 --> 00:07:01,890 about it as an intermediary. 183 00:07:01,890 --> 00:07:03,600 So what this tool can do is it can scan 184 00:07:03,600 --> 00:07:05,400 for OPC UA servers in the network. 185 00:07:05,400 --> 00:07:08,910 It can enumerate the OPC UA structure 186 00:07:08,910 --> 00:07:10,260 and change specific attributes 187 00:07:10,260 --> 00:07:12,270 that then change attributes on those devices. 188 00:07:12,270 --> 00:07:15,570 So this can be things like opening and closing a valve 189 00:07:15,570 --> 00:07:17,280 or maybe perhaps changing the speed 190 00:07:17,280 --> 00:07:18,720 of something or rotation direction. 191 00:07:18,720 --> 00:07:19,770 I mean, it can be all kinds of things. 192 00:07:19,770 --> 00:07:22,260 It just depends on what the device is. 193 00:07:22,260 --> 00:07:24,210 In general, this is a better implementation 194 00:07:24,210 --> 00:07:26,550 of the CrashOverride's attack. 195 00:07:26,550 --> 00:07:31,550 CrashOverride used the older OPC DA attack methodology. 196 00:07:31,860 --> 00:07:35,640 OPC DA is the older OPC protocol that was based on DCOM. 197 00:07:35,640 --> 00:07:37,800 The OPC foundation was like, "That's stupid," 198 00:07:37,800 --> 00:07:39,960 and they decided to make it cross-platform. 199 00:07:39,960 --> 00:07:41,523 and that turned into OPC UA. 200 00:07:42,810 --> 00:07:44,520 The structure of it is a little bit different, 201 00:07:44,520 --> 00:07:46,500 but it's basically what happened. 202 00:07:46,500 --> 00:07:48,480 So with MouseHole, it just worked. 203 00:07:48,480 --> 00:07:50,130 It's based on an open source library 204 00:07:50,130 --> 00:07:52,110 that was already well-vetted, I think. 205 00:07:52,110 --> 00:07:53,880 So it was very easy to get it to work 206 00:07:53,880 --> 00:07:55,110 and manipulate the process. 207 00:07:55,110 --> 00:07:57,303 We had no problems on our pipeline or lab. 208 00:07:58,800 --> 00:08:01,080 So after we verified it worked, 209 00:08:01,080 --> 00:08:03,180 I spoke with the analyst that was working on this 210 00:08:03,180 --> 00:08:06,030 and we decided let's automate an attack 211 00:08:06,030 --> 00:08:08,310 to produce a more real world attack scenario. 212 00:08:08,310 --> 00:08:10,830 The hypothesis being that you would deploy 213 00:08:10,830 --> 00:08:12,570 these remotely interactive utilities 214 00:08:12,570 --> 00:08:14,850 to recon the OPC server, figure out 215 00:08:14,850 --> 00:08:16,710 which parts you're gonna manipulate, 216 00:08:16,710 --> 00:08:19,650 pull the important reconnaissance tool back 217 00:08:19,650 --> 00:08:22,140 and then just wrap up a very small version of it 218 00:08:22,140 --> 00:08:23,730 and just throw that onto the box 219 00:08:23,730 --> 00:08:25,740 to actually disrupt the process. 220 00:08:25,740 --> 00:08:28,050 So I wanna show you a video here of our lab 221 00:08:28,050 --> 00:08:30,240 and this is us actually testing the malware 222 00:08:30,240 --> 00:08:32,253 so you can see how exciting my job is. 223 00:08:34,950 --> 00:08:36,450 Is it gonna play? There we go. 224 00:08:38,070 --> 00:08:39,990 So you'll see we have a water tank. 225 00:08:39,990 --> 00:08:42,120 There's a pump, there's a little valve 226 00:08:42,120 --> 00:08:44,160 and on the bottom right is our HMI screen. 227 00:08:44,160 --> 00:08:45,900 This is us actually running the malware 228 00:08:45,900 --> 00:08:48,150 and creating an unsafe state in our pipeline. 229 00:08:50,580 --> 00:08:52,740 See that? It was super exciting, right? 230 00:08:52,740 --> 00:08:55,140 So it turns out (chuckles) 231 00:08:55,140 --> 00:08:56,820 So it turns out if you ask your management 232 00:08:56,820 --> 00:08:59,160 if you can blow up a pipe and a water tank in your office, 233 00:08:59,160 --> 00:09:02,220 they tell you, they respond with a bunch of words 234 00:09:02,220 --> 00:09:05,490 like safety and liability and expenses. 235 00:09:05,490 --> 00:09:07,350 So what we're gonna do is, actually, 236 00:09:07,350 --> 00:09:09,120 the reason nothing actually went wrong there 237 00:09:09,120 --> 00:09:10,770 is that lab is specifically set up 238 00:09:10,770 --> 00:09:13,410 so that if we did overdo it, it wouldn't break. 239 00:09:13,410 --> 00:09:15,210 So the pump that's attached to that lab 240 00:09:15,210 --> 00:09:17,820 is a little baby pool pump and it can never 241 00:09:17,820 --> 00:09:19,470 over pressurize that pipe in a way 242 00:09:19,470 --> 00:09:21,060 that would actually cause a problem. 243 00:09:21,060 --> 00:09:23,820 And that's good because that pipeline process 244 00:09:23,820 --> 00:09:25,650 sits right behind our CFO. 245 00:09:25,650 --> 00:09:29,640 So it'd probably be a bad idea if we screwed it up. 246 00:09:29,640 --> 00:09:30,750 All right. 247 00:09:30,750 --> 00:09:32,700 Before we can actually talk about what an unsafe state is, 248 00:09:32,700 --> 00:09:34,500 I wanna show you what a safe shutdown looks like. 249 00:09:34,500 --> 00:09:36,240 So we're talking about a pipeline. 250 00:09:36,240 --> 00:09:38,280 So we have a few important things here. 251 00:09:38,280 --> 00:09:40,530 There's the pump and how fast it's pumping water. 252 00:09:40,530 --> 00:09:42,690 There's the valve position, whether or not it's on or off. 253 00:09:42,690 --> 00:09:44,700 And then there's the actual PSI of the pipe 254 00:09:44,700 --> 00:09:48,240 and in particular, the safe value for the PSI of that pipe. 255 00:09:48,240 --> 00:09:52,020 So there's a max PSI that this pipe is allowed 256 00:09:52,020 --> 00:09:54,720 to reach and any higher is considered unsafe 257 00:09:54,720 --> 00:09:58,350 and when that unsafe value is exceeded, 258 00:09:58,350 --> 00:10:00,037 the process is gonna take over and say, 259 00:10:00,037 --> 00:10:01,604 "Okay, we're gonna turn off the pump 260 00:10:01,604 --> 00:10:02,670 and we're gonna get everything back down 261 00:10:02,670 --> 00:10:05,130 so that you can restart that process and keep going." 262 00:10:05,130 --> 00:10:09,750 So this is what it looks like if we try to exceed that PSI. 263 00:10:09,750 --> 00:10:11,340 Oh man, why is it? Oh, there we go. 264 00:10:11,340 --> 00:10:13,350 We try to exceed the max PSI. 265 00:10:13,350 --> 00:10:14,250 We've slowed it down a bit, 266 00:10:14,250 --> 00:10:16,100 'cause otherwise it goes really fast. 267 00:10:16,980 --> 00:10:19,320 So you'll see there, we have the flow rate increasing, 268 00:10:19,320 --> 00:10:21,270 the max is about two gallons per minute 269 00:10:22,710 --> 00:10:23,820 and the next thing you're gonna see is 270 00:10:23,820 --> 00:10:26,700 that solenoid box, that's the green box at the bottom, 271 00:10:26,700 --> 00:10:29,460 is gonna turn red and switch to closed 272 00:10:29,460 --> 00:10:31,890 and then the pressure starts to increase. 273 00:10:31,890 --> 00:10:33,303 Where at 14 here. 274 00:10:34,230 --> 00:10:36,660 You'll see that it's probably already been exceeded at 15, 275 00:10:36,660 --> 00:10:38,460 the flow rate's already decreasing. 276 00:10:38,460 --> 00:10:40,650 So it's gonna go ahead and turn off the pump. 277 00:10:40,650 --> 00:10:42,870 So you're gonna see that go down dramatically. 278 00:10:42,870 --> 00:10:45,990 That's that 80 to the negative whatever stuff. 279 00:10:45,990 --> 00:10:48,480 And then the flow rate decreases and the pump switches off 280 00:10:48,480 --> 00:10:51,300 and then what the operator is gonna get is a reset alarm. 281 00:10:51,300 --> 00:10:55,380 So you're gonna see a new box pop up in blue there 282 00:10:55,380 --> 00:10:56,700 and then it's gonna turn red 283 00:10:56,700 --> 00:10:58,290 and it turns blue one more time 284 00:10:58,290 --> 00:10:59,610 and when it turns blue that last time, 285 00:10:59,610 --> 00:11:00,570 it means you can hit the button, 286 00:11:00,570 --> 00:11:02,970 clear the alarm and then start the pump again. 287 00:11:02,970 --> 00:11:04,740 So the operator's going to get an alert 288 00:11:04,740 --> 00:11:07,170 and the process is basically going to take care of itself 289 00:11:07,170 --> 00:11:09,520 to make sure that everything's okay to restart. 290 00:11:10,710 --> 00:11:12,900 So how do we actually attack this 291 00:11:12,900 --> 00:11:14,130 with the automated utility? 292 00:11:14,130 --> 00:11:17,220 Well, run MouseHole via the CLI 293 00:11:17,220 --> 00:11:19,410 to gather the important server and node information 294 00:11:19,410 --> 00:11:21,930 so we can understand what set points to manipulate. 295 00:11:21,930 --> 00:11:24,390 So we need to know what the max safe value is. 296 00:11:24,390 --> 00:11:26,430 That's called the high pressure set point 297 00:11:26,430 --> 00:11:27,510 that was set to 15. 298 00:11:27,510 --> 00:11:29,400 We want to increase that to a hundred. 299 00:11:29,400 --> 00:11:30,540 We need to change the pump speed. 300 00:11:30,540 --> 00:11:33,300 The pump speed was set to 80 by default. 301 00:11:33,300 --> 00:11:35,670 We're gonna increase that to a hundred. 302 00:11:35,670 --> 00:11:37,620 And then the last thing is closing the valve, 303 00:11:37,620 --> 00:11:40,920 which is the solenoid energize attribute. 304 00:11:40,920 --> 00:11:42,270 We're gonna change that from true to false 305 00:11:42,270 --> 00:11:44,430 and that's gonna end up closing the valve for us. 306 00:11:44,430 --> 00:11:48,720 So this is what it looks like on the HMI screen 307 00:11:48,720 --> 00:11:51,753 when you actually execute MouseHole on the pipeline. 308 00:11:52,800 --> 00:11:55,350 So you'll see the flow speed's already at max speed 309 00:12:03,810 --> 00:12:06,390 and then solenoid valve's gonna close. 310 00:12:06,390 --> 00:12:08,610 You're gonna see the pressure go up. 311 00:12:08,610 --> 00:12:10,500 So it's already past it at this point. 312 00:12:10,500 --> 00:12:12,750 You'll see that the flow rate starts to decrease 313 00:12:12,750 --> 00:12:14,340 and it's not decreasing 'cause the pumps off. 314 00:12:14,340 --> 00:12:16,020 It's decreasing because the pipe is full 315 00:12:16,020 --> 00:12:18,390 and there's nowhere for the fluid to go. 316 00:12:18,390 --> 00:12:20,820 So basically what's happening here is the pump 317 00:12:20,820 --> 00:12:23,460 is trying to get water into the pipe, 318 00:12:23,460 --> 00:12:25,470 but it's too high a pressure, it just can't fit it. 319 00:12:25,470 --> 00:12:28,080 It's trying to stuff your full closet. 320 00:12:28,080 --> 00:12:32,400 So what happens when that occurs, 321 00:12:32,400 --> 00:12:34,350 you're not able to throw pump fluid 322 00:12:34,350 --> 00:12:36,540 through that pipe and that's called deadheading 323 00:12:36,540 --> 00:12:39,510 and this can be a problem in certain situations. 324 00:12:39,510 --> 00:12:41,370 The reason being is when you deadhead a pipe, 325 00:12:41,370 --> 00:12:43,440 what happens is you add heat to the fluid 326 00:12:43,440 --> 00:12:45,000 that's being pumped. 327 00:12:45,000 --> 00:12:47,550 So if you're pumping water, this is okay 328 00:12:47,550 --> 00:12:48,780 because you're adding heat to water 329 00:12:48,780 --> 00:12:50,610 and all it'll do is like deteriorate seals 330 00:12:50,610 --> 00:12:53,250 and then it might get leaky and an engineer notices 331 00:12:53,250 --> 00:12:54,570 and they're like, "We should fix that," 332 00:12:54,570 --> 00:12:56,170 and not investigate any further. 333 00:12:58,140 --> 00:13:01,560 But if it's something combustible like mining slurry, 334 00:13:01,560 --> 00:13:06,120 which happened in Queensland in 2017, the pump blows up, 335 00:13:06,120 --> 00:13:07,593 which is obviously a problem. 336 00:13:09,360 --> 00:13:11,370 This is a very simple attack, I know, 337 00:13:11,370 --> 00:13:13,020 but this is just an example of what can happen 338 00:13:13,020 --> 00:13:15,210 in just like a regular pipeline process. 339 00:13:15,210 --> 00:13:17,280 The other reason I have this example included 340 00:13:17,280 --> 00:13:18,810 is it's important for the rest of the talk 341 00:13:18,810 --> 00:13:22,410 where we're gonna talk about PLC manipulation and control. 342 00:13:22,410 --> 00:13:27,150 The reason that this pump exploded in Queensland 343 00:13:27,150 --> 00:13:29,040 was that someone made a bad modification 344 00:13:29,040 --> 00:13:32,520 to the PLC logic that basically closed the valve 345 00:13:32,520 --> 00:13:35,270 and allowed the pump to pump against this closed valve. 346 00:13:36,360 --> 00:13:38,340 We're gonna move on to EvilScholar. 347 00:13:38,340 --> 00:13:42,930 which is the CODESYS version 3 utility command line. 348 00:13:42,930 --> 00:13:44,700 The main things we're gonna focus on here 349 00:13:44,700 --> 00:13:48,450 are the proxy module in EvilScholar, 350 00:13:48,450 --> 00:13:49,620 as well as the maintenance commands 351 00:13:49,620 --> 00:13:52,560 that allow me to transfer files and modify logic. 352 00:13:52,560 --> 00:13:55,430 So first thing to keep in mind here 353 00:13:55,430 --> 00:13:57,120 is that we're dealing with CODESYS. 354 00:13:57,120 --> 00:13:58,650 It's a device management protocol 355 00:13:58,650 --> 00:14:00,420 used by hundreds of vendors. 356 00:14:00,420 --> 00:14:02,640 Wide usage makes it a natural target. 357 00:14:02,640 --> 00:14:05,400 And essentially it's a custom TCP implementation 358 00:14:05,400 --> 00:14:07,833 on top of UDP with an Application layer. 359 00:14:09,810 --> 00:14:11,370 And basically the end sum of that 360 00:14:11,370 --> 00:14:14,250 is that CODESYS sucks and it's a pain in the ass. 361 00:14:14,250 --> 00:14:15,570 It's pseudo-routable. 362 00:14:15,570 --> 00:14:17,880 If you're using NAT, you don't get packets back 363 00:14:17,880 --> 00:14:20,580 and when you're trying to test a piece of malware 364 00:14:20,580 --> 00:14:22,350 and you're not sure how well it works, 365 00:14:22,350 --> 00:14:24,480 you're not sure if you're dealing with broken malware 366 00:14:24,480 --> 00:14:26,670 or if you're dealing with a broken network config, 367 00:14:26,670 --> 00:14:28,440 in particular, in our case, we had a lab 368 00:14:28,440 --> 00:14:30,420 where we could connect to the devices just fine. 369 00:14:30,420 --> 00:14:31,800 We had another lab where we couldn't connect 370 00:14:31,800 --> 00:14:32,943 to the devices at all. 371 00:14:33,900 --> 00:14:36,840 And this one, by far, took us the longest to sort out. 372 00:14:36,840 --> 00:14:38,550 We thought it was a firmware issue 373 00:14:38,550 --> 00:14:41,010 and it just turned out that out of the three utilities, 374 00:14:41,010 --> 00:14:43,366 EvilScholar's the dumb dumb in the group 375 00:14:43,366 --> 00:14:44,280 (audience laughing) 376 00:14:44,280 --> 00:14:48,150 and in part because the developer 377 00:14:48,150 --> 00:14:50,820 made poor assumptions about CODESYS. 378 00:14:50,820 --> 00:14:53,190 It made assumptions about the network size, 379 00:14:53,190 --> 00:14:54,570 the address sizing. 380 00:14:54,570 --> 00:14:56,970 Address sizing in CODESYS is variable length. 381 00:14:56,970 --> 00:14:59,400 So there's just a lot of issues there. 382 00:14:59,400 --> 00:15:03,060 So the default version of EvilScholar, 383 00:15:03,060 --> 00:15:06,450 we could connect to the 241 and the 251 with no fixes 384 00:15:06,450 --> 00:15:09,480 if the devices were on the network of the right size. 385 00:15:09,480 --> 00:15:13,020 We spent about two to three weeks fixing the Python code 386 00:15:13,020 --> 00:15:14,670 that generates the CODESYS traffic 387 00:15:14,670 --> 00:15:16,290 and we were able to get it to work 388 00:15:16,290 --> 00:15:18,210 with regular CODESYS version 3 389 00:15:18,210 --> 00:15:20,130 and then we were able to additionally connect 390 00:15:20,130 --> 00:15:22,920 to a Raspberry Pi running CODESYS, 391 00:15:22,920 --> 00:15:25,500 as well as a Hitachi EHV+, 392 00:15:25,500 --> 00:15:28,260 provided they were any little endian architecture stuff, 393 00:15:28,260 --> 00:15:31,140 because CODESYS also has fields that will flip endian 394 00:15:31,140 --> 00:15:33,060 that's depending on the target architecture. 395 00:15:33,060 --> 00:15:34,740 So we didn't go that far because it's not our job 396 00:15:34,740 --> 00:15:36,210 to develop malware or fix their shit. 397 00:15:36,210 --> 00:15:37,350 We were just trying to see exactly 398 00:15:37,350 --> 00:15:39,720 how far they had to go to make this work. 399 00:15:39,720 --> 00:15:42,930 So we were able to test all the plugins. 400 00:15:42,930 --> 00:15:44,850 And so this is just a really quick example here 401 00:15:44,850 --> 00:15:46,470 where, when we did things statically, 402 00:15:46,470 --> 00:15:50,040 we made a slight error and then after doing it dynamically, 403 00:15:50,040 --> 00:15:52,320 we were like, "Oh, this is actually what happens." 404 00:15:52,320 --> 00:15:53,610 And in hindsight you're like, "Well, fuck. 405 00:15:53,610 --> 00:15:55,800 Of course, that's what happens." 406 00:15:55,800 --> 00:15:58,620 So a PLCProxy, the idea here is that you have 407 00:15:58,620 --> 00:16:01,680 an internal PLC network behind an industrial firewall. 408 00:16:01,680 --> 00:16:03,630 The industrial firewall's job is to make sure 409 00:16:03,630 --> 00:16:05,190 that the configure, the EWS, 410 00:16:05,190 --> 00:16:07,980 the engineering workstations only talk to the PLCs 411 00:16:07,980 --> 00:16:10,140 for which they're responsible and no other ones. 412 00:16:10,140 --> 00:16:13,530 So the EvilScholar tool wants to talk 413 00:16:13,530 --> 00:16:16,020 to all these other PLCs from wherever it happens to be. 414 00:16:16,020 --> 00:16:18,960 So what it does is it queries for the CODESYS gateway 415 00:16:18,960 --> 00:16:22,410 to grab its IP address and it selects the network 416 00:16:22,410 --> 00:16:25,230 that the malware happens to not be on 417 00:16:25,230 --> 00:16:26,790 and creates a route to that network 418 00:16:26,790 --> 00:16:29,640 through the target, Schneider Electric PLC. 419 00:16:29,640 --> 00:16:31,020 So originally we thought it was actually 420 00:16:31,020 --> 00:16:32,580 sending traffic to the gateway. 421 00:16:32,580 --> 00:16:34,620 It turns out it's just sending traffic to the network 422 00:16:34,620 --> 00:16:38,790 and just letting the TM251 route to wherever it wants 423 00:16:38,790 --> 00:16:40,830 and that was the basic idea. 424 00:16:40,830 --> 00:16:42,330 It ended up being a minor error, 425 00:16:42,330 --> 00:16:44,130 because all we discovered was that the proxy thing 426 00:16:44,130 --> 00:16:47,130 was actually just nastier than we originally thought. 427 00:16:47,130 --> 00:16:49,050 Because as we are runtime testing, what we realized 428 00:16:49,050 --> 00:16:51,810 was that the 251 won't just proxy Modbus traffic, 429 00:16:51,810 --> 00:16:53,370 it'll proxy anything you give. 430 00:16:53,370 --> 00:16:55,830 So you can SSH through it, HTTP commands, 431 00:16:55,830 --> 00:16:58,260 arbitrary, other ICS protocols. 432 00:16:58,260 --> 00:17:01,863 It's just crazy, to say the least. 433 00:17:06,630 --> 00:17:08,580 I mean, that's basically it. 434 00:17:08,580 --> 00:17:10,050 It just got worse. 435 00:17:10,050 --> 00:17:12,960 Just not necessarily what you wanted the result to be. 436 00:17:12,960 --> 00:17:14,190 So we're gonna move on 437 00:17:14,190 --> 00:17:16,263 to the logic corruption pieces of this. 438 00:17:17,820 --> 00:17:19,120 Where are we at now? Okay. 439 00:17:21,570 --> 00:17:23,730 So EvilScholar lets you transfer files 440 00:17:23,730 --> 00:17:26,018 to and from the target device. 441 00:17:26,018 --> 00:17:28,530 So in general, we can pull the logic, 442 00:17:28,530 --> 00:17:31,200 modify it, put it back on the controller. 443 00:17:31,200 --> 00:17:33,240 Super easy thing for us to do with this tool. 444 00:17:33,240 --> 00:17:35,520 And it should be trivial to modify the logic, 445 00:17:35,520 --> 00:17:37,440 provided it's in a native assembly language 446 00:17:37,440 --> 00:17:39,060 and not an unknown bytecode. 447 00:17:39,060 --> 00:17:40,710 So some of these controllers have 448 00:17:41,640 --> 00:17:43,470 just their own bytecodes that you have to figure out 449 00:17:43,470 --> 00:17:44,840 before you can do this kind of attack. 450 00:17:44,840 --> 00:17:46,140 In this case, we were dealing with ARM. 451 00:17:46,140 --> 00:17:47,310 It was super easy. 452 00:17:47,310 --> 00:17:50,670 You see the red lights there mean the controller vectored. 453 00:17:50,670 --> 00:17:53,280 That error box there says that the controller 454 00:17:53,280 --> 00:17:55,260 is unable, sorry, the EWS is unable 455 00:17:55,260 --> 00:17:57,240 to connect to the controller. 456 00:17:57,240 --> 00:18:00,360 We were able to crash, manipulate logic on the controller, 457 00:18:00,360 --> 00:18:02,580 create various error and output states. 458 00:18:02,580 --> 00:18:04,440 In both cases, comms to the controller 459 00:18:04,440 --> 00:18:07,260 are not possible unless you power cycle the device. 460 00:18:07,260 --> 00:18:09,810 If you connect to the PLC before the crash, 461 00:18:09,810 --> 00:18:14,810 Schneider's SoMachine software will report 462 00:18:14,970 --> 00:18:17,220 that it knows there's something different on the PLC 463 00:18:17,220 --> 00:18:19,140 and only provide you with the option to override it 464 00:18:19,140 --> 00:18:21,450 and it won't let you pull the corrupted logic back 465 00:18:21,450 --> 00:18:23,370 so you can see what's going on. 466 00:18:23,370 --> 00:18:28,230 And this is something that's enforced by the EWS software. 467 00:18:28,230 --> 00:18:29,910 It's not enforced by the PLC. 468 00:18:29,910 --> 00:18:31,410 And we know that because EvilScholar 469 00:18:31,410 --> 00:18:33,540 can pull the bad logic for us. 470 00:18:33,540 --> 00:18:36,510 So in that sense, EvilScholar doubles as a forensics tool, 471 00:18:36,510 --> 00:18:39,750 which is super weird and not what I was expecting 472 00:18:39,750 --> 00:18:41,703 out of this malware analysis journey. 473 00:18:43,050 --> 00:18:45,560 The last little oddity here is that if you... 474 00:18:46,590 --> 00:18:48,810 Essentially, these controllers run an execution cycle. 475 00:18:48,810 --> 00:18:50,220 So there's a bunch of applications, 476 00:18:50,220 --> 00:18:52,470 which are basically programs and it calls each one 477 00:18:52,470 --> 00:18:54,630 and waits a minute for it to return. 478 00:18:54,630 --> 00:18:56,760 So on that first execution cycle, 479 00:18:56,760 --> 00:18:57,870 if you crash the controller, 480 00:18:57,870 --> 00:18:59,970 it just sets all the outputs to false. 481 00:18:59,970 --> 00:19:03,300 It doesn't set them to their default fail safe values, 482 00:19:03,300 --> 00:19:06,240 which is what you would expect a crash to get set to, 483 00:19:06,240 --> 00:19:08,940 because that default fail safe is where the process 484 00:19:08,940 --> 00:19:10,710 should be when it's not doing anything 485 00:19:10,710 --> 00:19:12,900 and then you flip the right values to start everything. 486 00:19:12,900 --> 00:19:14,220 So it just sets everything to false, 487 00:19:14,220 --> 00:19:16,710 which could have some unknown consequences 488 00:19:16,710 --> 00:19:18,420 for various processes. 489 00:19:18,420 --> 00:19:20,400 So that's a bit of a serious issue 490 00:19:20,400 --> 00:19:22,050 coming out of these controllers. 491 00:19:22,050 --> 00:19:23,550 I'm not able to say too much more here, 492 00:19:23,550 --> 00:19:25,440 'cause we're still working through the CVE process 493 00:19:25,440 --> 00:19:27,300 with Schneider and CODESYS. 494 00:19:27,300 --> 00:19:28,710 We reported on June 22nd 495 00:19:28,710 --> 00:19:30,590 but isn't quite long enough a window for me 496 00:19:30,590 --> 00:19:33,270 to be able to show screenshots of the code 497 00:19:33,270 --> 00:19:35,880 and what we manipulated and everything else. 498 00:19:35,880 --> 00:19:38,580 The good news is I can do that for BadOmen. 499 00:19:38,580 --> 00:19:41,190 So we're gonna focus on the logic corruption 500 00:19:41,190 --> 00:19:44,700 and manipulation with the BadOmen utility. 501 00:19:44,700 --> 00:19:46,380 BadOmen is again another remote shell 502 00:19:46,380 --> 00:19:48,750 for manipulating OMRON controllers 503 00:19:48,750 --> 00:19:51,720 and it primarily does this by an HTTP interface 504 00:19:51,720 --> 00:19:55,350 that's exported by the NX, NJ series controllers. 505 00:19:55,350 --> 00:19:56,580 The things we're gonna focus on here, 506 00:19:56,580 --> 00:19:59,460 in terms of its capabilities, are the transferring 507 00:19:59,460 --> 00:20:01,890 of logic, transferring of files, 508 00:20:01,890 --> 00:20:05,793 as well as the servo module for manipulating servo drives. 509 00:20:07,980 --> 00:20:09,930 So real quick background on the console. 510 00:20:09,930 --> 00:20:12,750 It takes advantage of a hard-coded credential vulnerability. 511 00:20:12,750 --> 00:20:17,070 So they can default log in to this HTTP server 512 00:20:17,070 --> 00:20:19,770 that's sitting on the NX1P2. 513 00:20:19,770 --> 00:20:21,900 The server has various CGI endpoints 514 00:20:21,900 --> 00:20:23,670 that's also used by Sysmax Studio, 515 00:20:23,670 --> 00:20:26,580 the EWS software for configuring the controller. 516 00:20:26,580 --> 00:20:30,210 So in this case, it primarily uses 517 00:20:30,210 --> 00:20:33,900 the cpu.fcgi and the ecat.fcgi. 518 00:20:33,900 --> 00:20:35,610 CPU for doing the standard downloading 519 00:20:35,610 --> 00:20:36,990 and transferring of files in logic 520 00:20:36,990 --> 00:20:39,330 and changing the controller state 521 00:20:39,330 --> 00:20:40,950 to run mode and program mode. 522 00:20:40,950 --> 00:20:43,050 And the ECAT is primarily for just wrapping up 523 00:20:43,050 --> 00:20:45,150 EtherCAT traffic to the NX1P2 524 00:20:45,150 --> 00:20:47,703 to send on the EtherCAT daisy chain. 525 00:20:49,320 --> 00:20:51,870 So same process that we did with EvilScholar, 526 00:20:51,870 --> 00:20:53,970 except we're just using different modules now. 527 00:20:53,970 --> 00:20:56,640 We took the backup module to grab the entire project file 528 00:20:56,640 --> 00:21:00,180 because the NX1P2 just has a backup command. 529 00:21:00,180 --> 00:21:01,500 So we just ran that. 530 00:21:01,500 --> 00:21:02,970 We unpacked it. We took a look. 531 00:21:02,970 --> 00:21:04,980 It turns out it's running QNX Neutrino. 532 00:21:04,980 --> 00:21:07,140 So the logic compiles the shared objects 533 00:21:07,140 --> 00:21:09,360 that gets mapped dynamically in the runtime. 534 00:21:09,360 --> 00:21:11,310 So the entry point was super easy to find. 535 00:21:11,310 --> 00:21:13,200 We just had to branch to a bad address 536 00:21:13,200 --> 00:21:15,270 and then we replaced the files on the controller 537 00:21:15,270 --> 00:21:18,060 and that resulted in another major fault. 538 00:21:18,060 --> 00:21:20,520 This major fault was similar to the EvilScholar warning 539 00:21:20,520 --> 00:21:25,520 in that we weren't able to upload the logic back, I guess. 540 00:21:26,460 --> 00:21:29,070 We weren't able to retrieve the bad logic. 541 00:21:29,070 --> 00:21:30,660 It prevents it, it says that there's no logic 542 00:21:30,660 --> 00:21:32,883 to retrieve once it crashes. 543 00:21:33,750 --> 00:21:36,630 I imagine this ever happened multiple times, 544 00:21:36,630 --> 00:21:38,520 even when we do some of the manipulation stuff 545 00:21:38,520 --> 00:21:39,780 and there's some kind of weird versioning 546 00:21:39,780 --> 00:21:44,070 where I think the EWS is saying you haven't done 547 00:21:44,070 --> 00:21:46,530 a transfer so there's nothing to pull back. 548 00:21:46,530 --> 00:21:48,833 Everything's still the same so why are you asking? 549 00:21:49,980 --> 00:21:51,150 The other thing is it prevents you 550 00:21:51,150 --> 00:21:53,460 from enabling program mode and without being able 551 00:21:53,460 --> 00:21:56,490 to enable program mode, you can't recover regularly. 552 00:21:56,490 --> 00:22:00,720 So we tried to restore from an SD card and that failed, 553 00:22:00,720 --> 00:22:02,130 but for whatever reason that allows you 554 00:22:02,130 --> 00:22:04,710 to set program mode again, then a factory reset 555 00:22:04,710 --> 00:22:06,600 and then you restore logic to the controller, 556 00:22:06,600 --> 00:22:08,940 which, it sounds like a medium pain in the ass, 557 00:22:08,940 --> 00:22:10,410 you listening to me, but if you imagine 558 00:22:10,410 --> 00:22:13,080 a plant engineer and this stuff happening, 559 00:22:13,080 --> 00:22:15,030 that's just the rest of his day 560 00:22:15,030 --> 00:22:16,650 and probably a call to the vendor 561 00:22:16,650 --> 00:22:18,090 and then wait time and everything else. 562 00:22:18,090 --> 00:22:20,220 So just this simple attack would cause 563 00:22:20,220 --> 00:22:21,750 some downtime for a plant and depending 564 00:22:21,750 --> 00:22:24,570 on what it's actually processing, lots of fucking money. 565 00:22:24,570 --> 00:22:28,080 So moving on. 566 00:22:28,080 --> 00:22:29,910 The servo module testing. 567 00:22:29,910 --> 00:22:32,400 So real quick before we actually talk about what this is. 568 00:22:32,400 --> 00:22:34,650 What's a servo drive and what's a servo motor? 569 00:22:34,650 --> 00:22:38,070 So a servo motor is just a motor and it spins a shaft 570 00:22:38,070 --> 00:22:40,410 and it can be connected to any number of things. 571 00:22:40,410 --> 00:22:43,137 So some of the examples that I've heard about 572 00:22:43,137 --> 00:22:45,150 are servos control wicket gates of dams 573 00:22:45,150 --> 00:22:48,300 for controlling water flow into a dam 574 00:22:48,300 --> 00:22:50,253 that can be in drone cameras. 575 00:22:51,840 --> 00:22:54,330 I think they actually help with creating 576 00:22:54,330 --> 00:22:56,430 basin subwoofers and things like that. 577 00:22:56,430 --> 00:22:58,470 Basically my general impression about servos 578 00:22:58,470 --> 00:22:59,760 from having analyzed this malware 579 00:22:59,760 --> 00:23:01,140 is that if you need something to move, 580 00:23:01,140 --> 00:23:03,000 there's probably a servo involved. 581 00:23:03,000 --> 00:23:07,350 So the servo drive provides power to a motor 582 00:23:07,350 --> 00:23:09,150 and it also handles all the comms 583 00:23:09,150 --> 00:23:13,353 and back and forth to the NX1P2 controller. 584 00:23:14,280 --> 00:23:15,780 The malware in particular targeted 585 00:23:15,780 --> 00:23:19,200 that 1SN10FECT servo drive. 586 00:23:19,200 --> 00:23:22,863 But this was a 380 to 480 VAC drive. 587 00:23:23,880 --> 00:23:26,970 We don't have a reliable source of 480 VAC power 588 00:23:26,970 --> 00:23:29,280 and that's also just super effing dangerous 589 00:23:29,280 --> 00:23:31,290 to just have sitting in an office. 590 00:23:31,290 --> 00:23:33,420 We didn't wanna electrocute the plant engineer 591 00:23:33,420 --> 00:23:36,660 that was so graciously helping me set all these things up. 592 00:23:36,660 --> 00:23:38,280 So we looked at the documentation, it turned out 593 00:23:38,280 --> 00:23:41,220 that there was 120 VAC model that we could hook up 594 00:23:41,220 --> 00:23:43,800 and 120 VAC is what regular residential 595 00:23:43,800 --> 00:23:45,480 and commercial real estate uses. 596 00:23:45,480 --> 00:23:46,590 So we hooked that up and we got 597 00:23:46,590 --> 00:23:48,330 the corresponding small motor 598 00:23:48,330 --> 00:23:49,830 and we were able to verify that the malware 599 00:23:49,830 --> 00:23:51,510 also worked on that smaller drive. 600 00:23:51,510 --> 00:23:53,700 So we did the rest of our testing on the small drive 601 00:23:53,700 --> 00:23:55,110 and also just for general safety, 602 00:23:55,110 --> 00:23:58,980 if we broke up bigger motor, bigger things could go wrong. 603 00:23:58,980 --> 00:24:02,553 So we stuck with the safer option here in our lab. 604 00:24:05,070 --> 00:24:08,190 So this is just a quick diagram of the setup. 605 00:24:08,190 --> 00:24:10,200 You have the computer talking to the CPU unit, 606 00:24:10,200 --> 00:24:12,390 which is the NX1P2, and it's gonna relay 607 00:24:12,390 --> 00:24:14,700 the EtherCAT commands down through the network 608 00:24:14,700 --> 00:24:18,030 to the actual servo drives and anything else 609 00:24:18,030 --> 00:24:20,190 that happens to be on that EtherCAT chain. 610 00:24:20,190 --> 00:24:22,440 So the actual servo module plugin, 611 00:24:22,440 --> 00:24:25,050 it turns out you can't monitor the fun stuff with it. 612 00:24:25,050 --> 00:24:27,120 You can't just change motor velocity 613 00:24:27,120 --> 00:24:28,653 and make things over spin. 614 00:24:29,520 --> 00:24:30,720 You can't change the direction. 615 00:24:30,720 --> 00:24:32,760 You can't change the direction of the motor live, 616 00:24:32,760 --> 00:24:34,020 even though that was a parameter. 617 00:24:34,020 --> 00:24:38,040 So what the EtherCAT module with the servo module does 618 00:24:38,040 --> 00:24:39,900 is it modifies particular parameters. 619 00:24:39,900 --> 00:24:41,940 Think about it as basic configuration. 620 00:24:41,940 --> 00:24:44,010 So there is a parameter for changing 621 00:24:44,010 --> 00:24:46,080 the direction of the motor, the way it spins, 622 00:24:46,080 --> 00:24:47,100 but you're not allowed to change that 623 00:24:47,100 --> 00:24:48,390 while the motor's actively running. 624 00:24:48,390 --> 00:24:49,770 Everything has to be turned off 625 00:24:49,770 --> 00:24:51,750 and in some weird configuration state. 626 00:24:51,750 --> 00:24:53,940 So there's not a lot you can do with this, 627 00:24:53,940 --> 00:24:56,700 but what you can do is you could be annoying 628 00:24:56,700 --> 00:24:58,020 and it would still cause downtime. 629 00:24:58,020 --> 00:25:00,600 For example, there's an excessive speed detection parameter. 630 00:25:00,600 --> 00:25:03,420 So if you set that to one rotation per minute 631 00:25:03,420 --> 00:25:05,790 and your process is running at 40 RPMs, 632 00:25:05,790 --> 00:25:08,580 the motor just stops and then they might be able 633 00:25:08,580 --> 00:25:10,110 to fix that and set it back and then you can just 634 00:25:10,110 --> 00:25:11,520 go back there and upset it again. 635 00:25:11,520 --> 00:25:13,290 So you can just keep needling them over and over again 636 00:25:13,290 --> 00:25:15,030 and just keep stopping all these servos 637 00:25:15,030 --> 00:25:17,340 or stop a bunch of them at once. 638 00:25:17,340 --> 00:25:19,080 So just with that, you could cause some issues, 639 00:25:19,080 --> 00:25:20,490 but generally, looking at this, 640 00:25:20,490 --> 00:25:22,530 I see it as a precursor to an attack 641 00:25:22,530 --> 00:25:24,480 where you actually screw with a motor. 642 00:25:24,480 --> 00:25:26,160 I'm gonna up the motor speed to max 643 00:25:26,160 --> 00:25:28,634 and I'm gonna disable all these warnings 644 00:25:28,634 --> 00:25:30,360 of the excessive speed detections 645 00:25:30,360 --> 00:25:32,883 so that you don't get a warning on the EWS 646 00:25:32,883 --> 00:25:35,190 that something is going wrong. 647 00:25:35,190 --> 00:25:37,710 And this worked out for us because we also wanna test 648 00:25:37,710 --> 00:25:41,820 logic manipulation with BadOmen on the NX1P2 649 00:25:41,820 --> 00:25:43,380 and it would be a lot more fun if we could 650 00:25:43,380 --> 00:25:48,380 over spin a servo motor than just setting outputs to false. 651 00:25:51,900 --> 00:25:52,943 Physical feedback when you're doing this 652 00:25:52,943 --> 00:25:54,960 is just a lot more interesting. 653 00:25:54,960 --> 00:25:58,650 So we know from crashing the logic 654 00:25:58,650 --> 00:26:00,990 that we can modify code on the device and send it back. 655 00:26:00,990 --> 00:26:03,330 And we also knew that the servo, the logic 656 00:26:03,330 --> 00:26:06,180 for the motor, just sits on the same NX1P2. 657 00:26:06,180 --> 00:26:08,220 So we should be able to pull that and modify it. 658 00:26:08,220 --> 00:26:10,320 So the first thing we had to do 659 00:26:10,320 --> 00:26:15,060 was actually configure the servo to run and do things. 660 00:26:15,060 --> 00:26:16,320 So that's ladder logic, that's what 661 00:26:16,320 --> 00:26:19,320 the visual programming language that engineers use. 662 00:26:19,320 --> 00:26:22,470 The main thing to keep in mind is that last box 663 00:26:22,470 --> 00:26:26,130 at the bottom is a function called MC_MoveRelative 664 00:26:26,130 --> 00:26:28,920 and it takes four arguments, 665 00:26:28,920 --> 00:26:31,890 distance, velocity, acceleration and deceleration. 666 00:26:31,890 --> 00:26:33,150 The important one being velocity, 667 00:26:33,150 --> 00:26:35,490 because that's how fast the motor is going to spin. 668 00:26:35,490 --> 00:26:37,170 So we created this program, 669 00:26:37,170 --> 00:26:38,760 we sent it to the controller 670 00:26:38,760 --> 00:26:41,130 and then we pulled the backup just like before 671 00:26:41,130 --> 00:26:42,570 and we decompiled the program. 672 00:26:42,570 --> 00:26:44,130 So the interesting thing here is, 673 00:26:44,130 --> 00:26:45,600 again, it was a shared object. 674 00:26:45,600 --> 00:26:47,820 You have all these symbols and function names 675 00:26:47,820 --> 00:26:50,640 and there was one called POU run function block. 676 00:26:50,640 --> 00:26:53,160 So if you just cross reference that, 677 00:26:53,160 --> 00:26:55,200 basically a function represents 678 00:26:55,200 --> 00:26:58,740 that particular ladder logic and it's in order. 679 00:26:58,740 --> 00:27:00,480 The first call is the call to the first function 680 00:27:00,480 --> 00:27:02,670 and the next one's the call to the timer that I set up. 681 00:27:02,670 --> 00:27:04,560 So the last thing I call is MoveRelative. 682 00:27:04,560 --> 00:27:06,840 You just scroll to the bottom like you're dealing with UPX 683 00:27:06,840 --> 00:27:08,640 and you're like, "Oh, there it is." 684 00:27:08,640 --> 00:27:10,740 And if you look at the bottom left there, 685 00:27:10,740 --> 00:27:14,610 you'll see a bunch of 64-bit numbers and one 32-bit number. 686 00:27:14,610 --> 00:27:19,320 Those are IEEE754, real values. 687 00:27:19,320 --> 00:27:21,060 And so you can convert those to figure out 688 00:27:21,060 --> 00:27:24,990 and it turns out that the V18 maps 689 00:27:24,990 --> 00:27:26,940 to the 40 degrees per second 690 00:27:26,940 --> 00:27:29,220 that I had set the motor velocity to. 691 00:27:29,220 --> 00:27:31,230 So I know degrees per second sounds weird. 692 00:27:31,230 --> 00:27:33,990 You don't set it in RPM so you have to do these conversions 693 00:27:33,990 --> 00:27:37,500 and there's some other ways to set the velocity. 694 00:27:37,500 --> 00:27:40,710 In the manual, they call it units of display 695 00:27:40,710 --> 00:27:43,560 and you have to go look up in your EWS 696 00:27:43,560 --> 00:27:44,943 exactly what the units are. 697 00:27:46,530 --> 00:27:50,370 We have the speed set to 40 degrees per second. 698 00:27:50,370 --> 00:27:52,500 I want it to spin at maximum. 699 00:27:52,500 --> 00:27:53,910 So you just do the conversion. 700 00:27:53,910 --> 00:27:56,190 The motor's max speed is 3,000 RPMs. 701 00:27:56,190 --> 00:27:58,680 You multiply that by six, you get 18,000. 702 00:27:58,680 --> 00:28:01,380 Convert it back to real, patch the instruction, 703 00:28:01,380 --> 00:28:04,353 it's ARM and then re-upload the program. 704 00:28:05,820 --> 00:28:07,683 And that's my next video. 705 00:28:09,390 --> 00:28:10,983 So this is what happens. 706 00:28:11,970 --> 00:28:15,570 So before we keep going, something to keep in mind. 707 00:28:15,570 --> 00:28:18,000 On the right, you have the ladder logic as it's running. 708 00:28:18,000 --> 00:28:19,290 Green means energized. 709 00:28:19,290 --> 00:28:21,690 I'm just gonna rewind it just a little bit here. 710 00:28:23,700 --> 00:28:26,193 Is it playing? No. 711 00:28:28,830 --> 00:28:32,563 Almost. There we go, okay. 712 00:28:35,940 --> 00:28:37,950 On the right, ladder logic, green means energized. 713 00:28:37,950 --> 00:28:39,780 That means the program is currently running. 714 00:28:39,780 --> 00:28:42,210 At the bottom here, I think you can make it out. 715 00:28:42,210 --> 00:28:44,100 I can't really see the screen from y'all's angle, 716 00:28:44,100 --> 00:28:45,390 but that's a motor velocity. 717 00:28:45,390 --> 00:28:47,130 So 40 degrees per second is anywhere 718 00:28:47,130 --> 00:28:50,700 from we've seen ranges from four to seven RPMs. 719 00:28:50,700 --> 00:28:54,330 This number is in RPMs because the EWS software hates you. 720 00:28:54,330 --> 00:28:55,380 I don't know. 721 00:28:55,380 --> 00:28:57,390 Everything's different in there. 722 00:28:57,390 --> 00:28:58,860 And then here is the actual servo motor 723 00:28:58,860 --> 00:29:00,690 that we had set up spinning and up here, 724 00:29:00,690 --> 00:29:02,400 we have the actual mode and the IP address. 725 00:29:02,400 --> 00:29:04,893 So I'll walk you through some of this now. 726 00:29:07,230 --> 00:29:08,480 So it's spinning at seven 727 00:29:10,410 --> 00:29:12,060 and now it's going into program mode, 728 00:29:12,060 --> 00:29:13,560 because BadOmen is transferring 729 00:29:13,560 --> 00:29:15,393 the modified logic to the servo. 730 00:29:22,350 --> 00:29:24,483 And now it's switching back into run mode. 731 00:29:25,770 --> 00:29:26,603 See that switch. 732 00:29:26,603 --> 00:29:28,530 Now everything's energized but there's a timer block there. 733 00:29:28,530 --> 00:29:30,080 So it takes about five seconds. 734 00:29:31,200 --> 00:29:32,430 And then once it's energized, 735 00:29:32,430 --> 00:29:34,680 it just starts over spinning or not over spinning. 736 00:29:34,680 --> 00:29:35,700 It's spinning in max speed. 737 00:29:35,700 --> 00:29:37,530 So that's at 3,000 RPMs. 738 00:29:37,530 --> 00:29:38,790 And all that happens automatically. 739 00:29:38,790 --> 00:29:40,260 It was literally, I think, two keystrokes 740 00:29:40,260 --> 00:29:42,210 or something to do that. 741 00:29:42,210 --> 00:29:45,690 We also tested spinning the servo motor, 742 00:29:45,690 --> 00:29:47,700 over spinning it at 5,000 and it turns out 743 00:29:47,700 --> 00:29:49,650 the motor will only spin at max. 744 00:29:49,650 --> 00:29:52,320 I don't know if that's a runtime decision, 745 00:29:52,320 --> 00:29:54,510 like a runtime check or if that's 746 00:29:54,510 --> 00:29:59,510 a mechanical limitation of the motor. 747 00:29:59,640 --> 00:30:01,350 We also tried sending negative values 748 00:30:01,350 --> 00:30:03,630 and then we actually got a proper runtime system error, 749 00:30:03,630 --> 00:30:05,370 which we had never seen before. 750 00:30:05,370 --> 00:30:06,420 The nice thing is, with BadOmen, 751 00:30:06,420 --> 00:30:09,150 we can actually access that runtime to look at it 752 00:30:09,150 --> 00:30:12,420 and see exactly what's emitting that error 753 00:30:12,420 --> 00:30:14,370 and maybe we can patch the runtime system to ignore it. 754 00:30:14,370 --> 00:30:16,410 So there's a lot of other things we could probably do here 755 00:30:16,410 --> 00:30:17,760 that would be pretty nasty, 756 00:30:18,690 --> 00:30:20,790 but also we had slide deadlines for DEF CON. 757 00:30:20,790 --> 00:30:23,403 So I can only do so much. 758 00:30:24,240 --> 00:30:26,070 Let's keep going here. 759 00:30:26,070 --> 00:30:29,760 Last thing here, I wanna talk 760 00:30:29,760 --> 00:30:31,470 about safety system corruption with BadOmen, 761 00:30:31,470 --> 00:30:33,240 which is just something that's pretty personal to me, 762 00:30:33,240 --> 00:30:35,070 'cause one of the first things I worked on 763 00:30:35,070 --> 00:30:38,820 at Dragos was a Trisis, which affected a safety system. 764 00:30:38,820 --> 00:30:40,680 And when we were looking at BadOmen, 765 00:30:40,680 --> 00:30:43,530 what we noticed was that you can hook up an SL3300 766 00:30:43,530 --> 00:30:46,790 on that same EtherCAT network to monitor the servo 767 00:30:46,790 --> 00:30:48,600 or anything else on that network. 768 00:30:48,600 --> 00:30:50,520 And so what we were wondering was, well, 769 00:30:50,520 --> 00:30:53,610 if the servo's logic sits on the NX1P2, 770 00:30:53,610 --> 00:30:55,440 does the safety systems logic do it? 771 00:30:55,440 --> 00:30:57,690 Or does the NX1P2 transfer it in such a way 772 00:30:57,690 --> 00:31:01,263 that we can manipulate it, free transfer, whatever? 773 00:31:02,610 --> 00:31:04,260 While we were setting up the safety system, 774 00:31:04,260 --> 00:31:06,120 what we noticed that it was very clear 775 00:31:06,120 --> 00:31:08,400 that setting up that safety system 776 00:31:08,400 --> 00:31:12,180 was different than manipulating the regular logic. 777 00:31:12,180 --> 00:31:16,080 So with the regular NX1P2 and the servo logic, 778 00:31:16,080 --> 00:31:18,300 you would put the thing in program mode, 779 00:31:18,300 --> 00:31:20,430 I think, take it offline and put it in program mode, 780 00:31:20,430 --> 00:31:22,950 make your manipulations, put it online, 781 00:31:22,950 --> 00:31:25,620 transfer everything over and then you hit run 782 00:31:25,620 --> 00:31:27,520 and then everything starts to execute. 783 00:31:28,470 --> 00:31:30,810 For the safety system transfer, 784 00:31:30,810 --> 00:31:32,670 there was this other weird convoluted process 785 00:31:32,670 --> 00:31:34,320 where you select the safety system, 786 00:31:34,320 --> 00:31:35,910 you put it in program mode, 787 00:31:35,910 --> 00:31:37,650 then you go to put it in debug stop mode 788 00:31:37,650 --> 00:31:39,810 and then it pops four different yellow pop ups, 789 00:31:39,810 --> 00:31:41,850 one of which is a password that was blank. 790 00:31:41,850 --> 00:31:44,940 And then you hit OK three or four times 791 00:31:44,940 --> 00:31:46,380 and then it goes in a debug stop 792 00:31:46,380 --> 00:31:49,980 and actually transfers your program over. 793 00:31:49,980 --> 00:31:51,870 But that still isn't gonna run it. 794 00:31:51,870 --> 00:31:53,130 You can't put it in run mode from there. 795 00:31:53,130 --> 00:31:54,690 From there, you had to put it in debug run 796 00:31:54,690 --> 00:31:56,550 to make sure that everything runs correctly 797 00:31:56,550 --> 00:31:58,260 then execute a safety validation, 798 00:31:58,260 --> 00:32:00,390 which does I still don't know what, 799 00:32:00,390 --> 00:32:01,620 because I don't know. 800 00:32:01,620 --> 00:32:02,970 I wasn't able to trigger an error in that. 801 00:32:02,970 --> 00:32:04,200 I wasn't sure what it was checking for, 802 00:32:04,200 --> 00:32:07,893 except for random variables that weren't being used. 803 00:32:09,450 --> 00:32:11,070 And after all of that, 804 00:32:11,070 --> 00:32:13,200 then you can put the safety controller in run mode. 805 00:32:13,200 --> 00:32:14,730 So these are all things we had never seen 806 00:32:14,730 --> 00:32:16,200 BadOmen even attempt to do. 807 00:32:16,200 --> 00:32:18,150 There were no reference to this in the code. 808 00:32:18,150 --> 00:32:19,350 We weren't really sure about it. 809 00:32:19,350 --> 00:32:20,730 And so we just went to digging 810 00:32:20,730 --> 00:32:23,010 and the way we did this was we created a program. 811 00:32:23,010 --> 00:32:24,510 So you'll notice that the program 812 00:32:24,510 --> 00:32:25,890 on the left looks a lot different 813 00:32:25,890 --> 00:32:27,900 than the ladder logic program we saw before. 814 00:32:27,900 --> 00:32:29,910 That's because it's another engineering language 815 00:32:29,910 --> 00:32:32,373 called the function block diagram networks. 816 00:32:33,360 --> 00:32:36,390 I'm not sure exactly why we have both of these languages, 817 00:32:36,390 --> 00:32:37,920 'cause they seem very similar to me 818 00:32:37,920 --> 00:32:39,630 and you can do similar things with them, 819 00:32:39,630 --> 00:32:41,370 but it reminds me of just circuit programming 820 00:32:41,370 --> 00:32:42,820 with a bunch of ands and ors. 821 00:32:43,890 --> 00:32:44,990 It's kind of the idea. 822 00:32:48,644 --> 00:32:50,820 Two of our services folks helped me write this program. 823 00:32:50,820 --> 00:32:51,903 It's a simple timer. 824 00:32:53,400 --> 00:32:55,320 And then we went through the process and steps 825 00:32:55,320 --> 00:32:57,060 of loading everything and we downloaded it 826 00:32:57,060 --> 00:32:58,470 and we started looking through the PCAP 827 00:32:58,470 --> 00:32:59,910 to see what was going on. 828 00:32:59,910 --> 00:33:01,530 And you'll notice highlighted red there 829 00:33:01,530 --> 00:33:04,020 in the payload area, you'll see references 830 00:33:04,020 --> 00:33:06,220 to the names for the function block diagram. 831 00:33:07,530 --> 00:33:10,500 And what we ended up realizing 832 00:33:10,500 --> 00:33:14,250 was that BadOmen, sorry Sysmax Studio, 833 00:33:14,250 --> 00:33:16,980 when it's transferring safety controller programs, 834 00:33:16,980 --> 00:33:21,980 uses a different CGI endpoint than we had seen before. 835 00:33:22,350 --> 00:33:26,040 So in this case it's using the nxbus.fcgi endpoint 836 00:33:26,040 --> 00:33:28,500 and there's a NX send and receive data 837 00:33:28,500 --> 00:33:32,580 and it's using a different endpoint 838 00:33:32,580 --> 00:33:34,230 and probably a different protocol. 839 00:33:34,230 --> 00:33:35,970 We dug in further and it actually looks 840 00:33:35,970 --> 00:33:37,890 like the safety controller is running, 841 00:33:37,890 --> 00:33:39,600 but this is kind of funny. 842 00:33:39,600 --> 00:33:44,190 CODESYS Safety run time under the covers, 843 00:33:44,190 --> 00:33:45,900 but the protocol that's actually involved there 844 00:33:45,900 --> 00:33:48,750 isn't CODESYS protocol so they're wrapping it 845 00:33:48,750 --> 00:33:49,740 a little bit differently there, 846 00:33:49,740 --> 00:33:51,330 but we're fairly certain that we have 847 00:33:51,330 --> 00:33:53,130 more CODESYS research on our hands, 848 00:33:53,130 --> 00:33:55,530 than just the safety side of it. 849 00:33:55,530 --> 00:33:57,720 So we're able to confirm that it only supports 850 00:33:57,720 --> 00:33:59,430 the CPU and ECAT endpoints. 851 00:33:59,430 --> 00:34:01,830 It can't manipulate the safety controllers. 852 00:34:01,830 --> 00:34:04,110 I'd like to say that that's good news, 853 00:34:04,110 --> 00:34:07,040 but again, we don't know when, how long... 854 00:34:08,100 --> 00:34:10,170 These tools are, obviously, have investment 855 00:34:10,170 --> 00:34:13,080 behind them and they're actively developing them 856 00:34:13,080 --> 00:34:14,700 and practically speaking, if you want 857 00:34:14,700 --> 00:34:16,260 to conduct a real world attack 858 00:34:16,260 --> 00:34:18,210 on say, a real servo, more than likely 859 00:34:18,210 --> 00:34:20,670 there's a safety system standing behind that servo, 860 00:34:20,670 --> 00:34:22,920 watching it to make sure it doesn't fuck anything up 861 00:34:22,920 --> 00:34:24,840 and then setting it back to the right state. 862 00:34:24,840 --> 00:34:26,220 So if you're gonna conduct a real attack, 863 00:34:26,220 --> 00:34:27,660 you have to manipulate the safety system 864 00:34:27,660 --> 00:34:29,280 and then manipulate the servo. 865 00:34:29,280 --> 00:34:31,080 Somehow you have to do both. 866 00:34:31,080 --> 00:34:33,780 And so there's no reason to believe 867 00:34:33,780 --> 00:34:35,130 that they aren't actively working 868 00:34:35,130 --> 00:34:37,410 on this type of safety system compromise. 869 00:34:37,410 --> 00:34:39,870 With the SL3300, it's one more endpoint. 870 00:34:39,870 --> 00:34:41,910 They've managed to get all the other endpoints right 871 00:34:41,910 --> 00:34:43,680 so I'm sure they'll be able to figure that out. 872 00:34:43,680 --> 00:34:46,980 So caveat there, unless you have a shitty network 873 00:34:46,980 --> 00:34:49,373 and then maybe they can mess with your servo. 874 00:34:50,430 --> 00:34:54,543 Cool. Alright. So we're almost done here. 875 00:34:55,920 --> 00:34:58,170 Talk about some of the real world usage here. 876 00:34:59,940 --> 00:35:01,890 So the endpoint here from running it 877 00:35:01,890 --> 00:35:04,800 or the main takeaway here from just running it 878 00:35:04,800 --> 00:35:06,600 is despite the CODESYS problems, 879 00:35:06,600 --> 00:35:08,670 we think those things are gonna get fixed. 880 00:35:08,670 --> 00:35:10,370 The safety system things, we think 881 00:35:11,370 --> 00:35:15,000 that's not gonna be a gap in their tool set anymore, 882 00:35:15,000 --> 00:35:15,930 but generally speaking, 883 00:35:15,930 --> 00:35:18,120 these utilities can cause problems now. 884 00:35:18,120 --> 00:35:20,040 And at the very least, at minimum, 885 00:35:20,040 --> 00:35:22,320 they're gonna cause financial problems, 886 00:35:22,320 --> 00:35:26,250 if not disruptive and destructive problems for a site. 887 00:35:26,250 --> 00:35:30,060 So what we think a theoretical attack scenario would look 888 00:35:30,060 --> 00:35:31,950 with any of these utilities is there would be 889 00:35:31,950 --> 00:35:35,040 an IT intrusion to a steal important process documentation. 890 00:35:35,040 --> 00:35:37,710 Figure out what the process is, what's important, 891 00:35:37,710 --> 00:35:39,540 where are the important bits that they have to manipulate 892 00:35:39,540 --> 00:35:41,460 in order to actually cause a problem. 893 00:35:41,460 --> 00:35:43,917 Maybe they've never attacked a mining facility before 894 00:35:43,917 --> 00:35:45,990 and they need to figure out what all this terminology means. 895 00:35:45,990 --> 00:35:47,160 There's a lot of research that's gonna have 896 00:35:47,160 --> 00:35:48,990 to go into it beforehand. 897 00:35:48,990 --> 00:35:50,400 Then they would deploy standard implants, 898 00:35:50,400 --> 00:35:52,560 blah, blah, blah, blah, until they deploy Pipedream. 899 00:35:52,560 --> 00:35:53,700 And then Pipedream would be used 900 00:35:53,700 --> 00:35:56,280 to recon the process, technically. 901 00:35:56,280 --> 00:35:58,110 What set points do I need to manipulate? 902 00:35:58,110 --> 00:36:00,870 What kind of equipment do I actually have on the network? 903 00:36:00,870 --> 00:36:02,430 Are there CODESYS devices here? 904 00:36:02,430 --> 00:36:04,590 Are there Modbus devices here? 905 00:36:04,590 --> 00:36:06,030 And they could bring all that data back. 906 00:36:06,030 --> 00:36:07,830 Now they have the process documentation 907 00:36:07,830 --> 00:36:10,380 and the actual technical data, put that together 908 00:36:10,380 --> 00:36:14,700 to develop a utility to put together a utility 909 00:36:14,700 --> 00:36:16,560 for causing that disruptive effect 910 00:36:16,560 --> 00:36:18,000 or if they're feeling lazy, 911 00:36:18,000 --> 00:36:20,583 just type in the commands and do it on the console. 912 00:36:25,080 --> 00:36:28,113 What I really wanna harp on here is that this really is, 913 00:36:29,279 --> 00:36:30,990 and I don't like to sound super hype, 914 00:36:30,990 --> 00:36:33,390 'cause if anybody or any of you will actually know me, 915 00:36:33,390 --> 00:36:35,520 I'm pretty fucking cynical about things 916 00:36:35,520 --> 00:36:39,180 and I don't like reading articles about malware analysis. 917 00:36:39,180 --> 00:36:40,593 This shit is serious. 918 00:36:42,060 --> 00:36:43,650 I've done this for 10 years and it's the first time 919 00:36:43,650 --> 00:36:46,650 I've opened up a piece of malware and just went, "Oh, fuck." 920 00:36:48,000 --> 00:36:49,980 And so this does have some serious implications 921 00:36:49,980 --> 00:36:53,190 for the ICS industry, for us in general, 922 00:36:53,190 --> 00:36:55,050 because it's attacking critical infrastructure, 923 00:36:55,050 --> 00:36:57,810 just in terms of the breadth of impact of this utility. 924 00:36:57,810 --> 00:36:59,100 It is the first time that we've seen 925 00:36:59,100 --> 00:37:01,380 malware devs attempt a plugin framework 926 00:37:01,380 --> 00:37:03,750 for specific PLCs and protocols. 927 00:37:03,750 --> 00:37:06,120 There is some precedent for this 928 00:37:06,120 --> 00:37:08,670 in ICSsploit and Metasploit SCADA. 929 00:37:08,670 --> 00:37:10,380 I would argue that these are a little bit different 930 00:37:10,380 --> 00:37:11,880 because they're a little bit more focused 931 00:37:11,880 --> 00:37:15,660 on the particular protocols versus a hodgepodge. 932 00:37:15,660 --> 00:37:20,490 You can't add a random Modbus exploit to BadOmen. 933 00:37:20,490 --> 00:37:22,140 That's not gonna make any sense, 934 00:37:22,140 --> 00:37:24,270 because the underlying libraries expect you 935 00:37:24,270 --> 00:37:27,060 to use the HTTP console. 936 00:37:27,060 --> 00:37:28,650 So it wouldn't really function that way, 937 00:37:28,650 --> 00:37:33,000 but you can add arbitrary plug-ins for OMRON controllers. 938 00:37:33,000 --> 00:37:34,050 Same thing with EvilScholar, 939 00:37:34,050 --> 00:37:35,490 almost all the plug-ins are expecting 940 00:37:35,490 --> 00:37:37,140 to use that CODESYS library. 941 00:37:37,140 --> 00:37:41,067 So you can't necessarily just add a plugin for... 942 00:37:41,067 --> 00:37:44,820 You can, maybe, but it's not as straightforward. 943 00:37:44,820 --> 00:37:45,690 You see what I'm saying. 944 00:37:45,690 --> 00:37:46,980 But you can very much straightforward add 945 00:37:46,980 --> 00:37:49,590 a plugin to do something else in CODESYS. 946 00:37:49,590 --> 00:37:51,060 The other thing to keep in mind 947 00:37:51,060 --> 00:37:53,520 is that this stuff is ICS process agnostic, 948 00:37:53,520 --> 00:37:56,910 which is an interesting thing if you're an adversary, 949 00:37:56,910 --> 00:38:00,240 because CODESYS is everywhere, 950 00:38:00,240 --> 00:38:01,950 so I imagine that more than likely now, 951 00:38:01,950 --> 00:38:04,380 you're not limited by your target selection anymore. 952 00:38:04,380 --> 00:38:06,060 You can say, "Well, maybe I wanna attack 953 00:38:06,060 --> 00:38:07,620 that mining facility over there 954 00:38:07,620 --> 00:38:09,240 and previously I was interested in electric, 955 00:38:09,240 --> 00:38:11,010 but they're both running Schneider Electric controllers 956 00:38:11,010 --> 00:38:12,900 and they're both speaking CODESYS version 3. 957 00:38:12,900 --> 00:38:15,663 So it made everything easier for them now. 958 00:38:16,650 --> 00:38:20,040 The last two points here I want to harp on 959 00:38:20,040 --> 00:38:21,660 is that again, these tools can be used 960 00:38:21,660 --> 00:38:22,860 for reconnaissance of the network 961 00:38:22,860 --> 00:38:24,360 and they can be used for research. 962 00:38:24,360 --> 00:38:26,100 So while they can do disruptive things, 963 00:38:26,100 --> 00:38:27,510 they are also tools that will accelerate 964 00:38:27,510 --> 00:38:29,700 the learning curve of the adversary, 965 00:38:29,700 --> 00:38:32,970 whether on your network or just in their lab at home. 966 00:38:32,970 --> 00:38:34,440 The next thing is these command line tools 967 00:38:34,440 --> 00:38:36,630 are super easy to use, which means that the people 968 00:38:36,630 --> 00:38:38,310 that are actually using the tools can just get 969 00:38:38,310 --> 00:38:39,900 a checklist of instructions and they can sit 970 00:38:39,900 --> 00:38:41,460 on a keyboard and just type them all in 971 00:38:41,460 --> 00:38:42,573 and everything's done. 972 00:38:43,860 --> 00:38:45,900 The people using these tools don't have to be as smart 973 00:38:45,900 --> 00:38:47,550 as the people that are developing them. 974 00:38:47,550 --> 00:38:49,830 And we haven't necessarily seen that so much either 975 00:38:49,830 --> 00:38:51,300 with a remotely interactive tool 976 00:38:51,300 --> 00:38:52,950 where they're trying to lower the bar for entry 977 00:38:52,950 --> 00:38:54,663 for achieving an attack. 978 00:38:57,120 --> 00:39:00,573 In terms of the industry here, 979 00:39:02,280 --> 00:39:03,870 really quick soapbox moment, 980 00:39:03,870 --> 00:39:06,210 is I really think that some of these proprietary protocols 981 00:39:06,210 --> 00:39:07,650 that the ICS vendors are using 982 00:39:07,650 --> 00:39:09,600 and proprietary OSS need to be opened up 983 00:39:09,600 --> 00:39:11,490 to security researchers a little bit better. 984 00:39:11,490 --> 00:39:13,170 I think it's kind of ridiculous. 985 00:39:13,170 --> 00:39:17,040 For example, with Trisis, I think it's absolutely batshit 986 00:39:17,040 --> 00:39:20,070 that I had to analyze a program, what was it, 987 00:39:20,070 --> 00:39:22,680 a Python program and then shellcode that ran an OS 988 00:39:22,680 --> 00:39:24,390 I'd never heard of and then extract the firmware 989 00:39:24,390 --> 00:39:25,950 and then analyze the operating system. 990 00:39:25,950 --> 00:39:28,620 Also, I could figure out one system call. 991 00:39:28,620 --> 00:39:30,630 That sounds fun from a research perspective, 992 00:39:30,630 --> 00:39:31,650 but when you're under the gun trying 993 00:39:31,650 --> 00:39:32,940 to get advice out to operators 994 00:39:32,940 --> 00:39:34,980 who are seriously worried about their safety, 995 00:39:34,980 --> 00:39:36,900 it's just seriously annoying. 996 00:39:36,900 --> 00:39:38,670 That project could have been so much quicker 997 00:39:38,670 --> 00:39:41,070 if someone had just handed me a CISCO reference. 998 00:39:43,239 --> 00:39:45,560 And that's what happens every time, 999 00:39:46,530 --> 00:39:47,610 the last two times, anyway, 1000 00:39:47,610 --> 00:39:48,717 we've had to analyze these kinds of things 1001 00:39:48,717 --> 00:39:50,340 and I think it would be a lot better, 1002 00:39:50,340 --> 00:39:51,960 because clearly the adversary is putting 1003 00:39:51,960 --> 00:39:54,090 in the effort to learn this anyway. 1004 00:39:54,090 --> 00:39:55,800 And we need to get ahead of the game, 1005 00:39:55,800 --> 00:39:57,570 because compared to the IT world, 1006 00:39:57,570 --> 00:40:00,690 our understanding of the heterogeneous operating systems 1007 00:40:00,690 --> 00:40:04,050 that ICS uses is so much smaller. 1008 00:40:04,050 --> 00:40:05,880 In our world, in the IT world. 1009 00:40:05,880 --> 00:40:07,710 It's with Windows, Mac, Linux, 1010 00:40:07,710 --> 00:40:09,360 I guess, iOS and Android. 1011 00:40:09,360 --> 00:40:11,730 Those are the five major ones and that's it. 1012 00:40:11,730 --> 00:40:14,220 And we've spent so many years studying this 1013 00:40:14,220 --> 00:40:17,910 that we already know how they all work for the most part. 1014 00:40:17,910 --> 00:40:20,637 With ICS, there's limited research on each of these OSS 1015 00:40:20,637 --> 00:40:21,990 and you end up having to pull out, 1016 00:40:21,990 --> 00:40:23,580 pull apart the firmware to really figure it out. 1017 00:40:23,580 --> 00:40:25,650 So I just think that the amount of time it takes 1018 00:40:25,650 --> 00:40:28,050 could be made a lot easier if some of these vendors 1019 00:40:28,050 --> 00:40:30,030 cooperated a little better with us. 1020 00:40:30,030 --> 00:40:31,770 The other thing is I think that the EWSs 1021 00:40:31,770 --> 00:40:34,170 and the vendor utilities should just create forensic tools 1022 00:40:34,170 --> 00:40:36,300 or allow the EWS to pull bad logic 1023 00:40:36,300 --> 00:40:38,940 so that IR people and forensics people 1024 00:40:38,940 --> 00:40:41,490 can send the logic over to people like me 1025 00:40:41,490 --> 00:40:42,577 or anyone and look at it and see, 1026 00:40:42,577 --> 00:40:44,637 "Oh, this program's actually doing X." 1027 00:40:45,750 --> 00:40:48,060 And in that sense, I think that the vendors 1028 00:40:48,060 --> 00:40:49,290 are so used to being the expert 1029 00:40:49,290 --> 00:40:50,820 and the people that get called in on the scenes 1030 00:40:50,820 --> 00:40:52,170 and that's starting to change, 1031 00:40:52,170 --> 00:40:54,300 where IR teams are starting to go on the scene. 1032 00:40:54,300 --> 00:40:56,610 Dragos goes on the scene, Mani goes on the scene, 1033 00:40:56,610 --> 00:40:58,380 but how often can they actually pull bad logic? 1034 00:40:58,380 --> 00:41:00,450 Sometimes, maybe they happen to find it, 1035 00:41:00,450 --> 00:41:01,920 but it's not so common or maybe they know enough 1036 00:41:01,920 --> 00:41:03,300 about the EWS that they can find 1037 00:41:03,300 --> 00:41:05,100 a project file sitting on a device. 1038 00:41:05,100 --> 00:41:06,630 Maybe they can see what went wrong. 1039 00:41:06,630 --> 00:41:08,550 But that isn't very common either. 1040 00:41:08,550 --> 00:41:09,930 And in some sense, I think that vendors need 1041 00:41:09,930 --> 00:41:11,730 to start recognizing they aren't the only experts 1042 00:41:11,730 --> 00:41:12,980 about this stuff anymore. 1043 00:41:14,340 --> 00:41:15,300 And then the last thing, 1044 00:41:15,300 --> 00:41:18,570 the fundamental problem with all of this is that PLCs 1045 00:41:18,570 --> 00:41:20,850 just don't do integrity checking in their code. 1046 00:41:20,850 --> 00:41:24,210 That's just the most basic problem that needs to get fixed. 1047 00:41:24,210 --> 00:41:26,700 As far as I can tell from the billions 1048 00:41:26,700 --> 00:41:28,050 of vulnerabilities that exist, 1049 00:41:28,050 --> 00:41:30,300 if there was just a modicum of integrity checking, 1050 00:41:30,300 --> 00:41:31,200 I think a lot of these things 1051 00:41:31,200 --> 00:41:32,430 would've been mitigated to begin with. 1052 00:41:32,430 --> 00:41:35,310 So that's all I have for you. 1053 00:41:35,310 --> 00:41:37,710 Just really quick so I don't get yelled at by people. 1054 00:41:37,710 --> 00:41:39,210 We do have mitigations for this 1055 00:41:39,210 --> 00:41:41,040 so if anybody's watching it online, there's a link. 1056 00:41:41,040 --> 00:41:43,650 You can go check those out at that website. 1057 00:41:43,650 --> 00:41:45,360 I didn't talk about it here because it's DEF CON 1058 00:41:45,360 --> 00:41:46,740 and I'm just like, "That's boring." 1059 00:41:46,740 --> 00:41:49,920 So if you're interested in that, go check that out there. 1060 00:41:49,920 --> 00:41:52,670 Otherwise, I'm finished, thank y'all for having me. 1061 00:41:52,670 --> 00:41:55,837 (audience applauding)