So I'm Jimmy Wiley. I'm a principal malware analyst at Dragos, and primarily my job is to search for and analyze malware that targets critical infrastructure companies and equipment.
So this could be lame shit like ransomware, or it could be the dangerous stuff like Trisis or Crashoveride, and now Pipedream, which is what I'm going to talk to you about today.
And mostly the sort of problems and the results from testing this toolkit in our lab.
So first we're going to start with a little bit of background, talk about what Pipedream's capabilities are, talk about the hardware that we acquired, the lab setup, and then the fun stuff, which is the actual results from testing in our lab, and then talk a little bit about real-world usage of this toolkit.
Real quick, before I go forward, I just want to acknowledge all the people that helped me with this. It would be disingenuous for me to represent all of this as my own work.
There was a whole team behind me helping with the analysis and the writing and just fielding my stupid questions about certain critical infrastructure.
And one more caveat, you can read that yourselves, but really the short summary here is I'm under an NDA, and I can only say certain things about the malware.
So in particular, I can't tell you where it came from, I can't show you screenshots of the malware, but I can do is show you what happens when we run it.
The important thing to take away from this, what I'm allowed to say is that we found the malware before it was used.
So in the ICS community, that does provide us with sort of a unique opportunity to start building defenses.
For this kind of stuff.
So what is PipeDream? It's the seventh ICS impacting malware after the S-word and Havix and Black Energy,
Crash Override, Trisis, Indestroyer 2.
What distinguishes PipeDream from all these other sample sets, it's the first cross-industry scalable, flexible framework, like a TAC framework for ICS equipment.
So,
there's five components in total, we're only going to focus on the three on the left, which are Evil Scholar, Bad Omen, and Mousehole.
All three of these are remotely interactive command line utilities for manipulating equipment.
Evil Scholar is a code assist version 3 attack tool with an initial targeting of Schneider electric controllers.
Bad Omen is a similar utility command line, but it targets the Omron controllers, and you have Mousehole, which is a tool for interacting with OPC UA servers.
So, when we talk about our lab setup and some of the equipment, these are all the impacted devices of sort of the initial set, what we know right now.
But the thing to take away from this slide is that column on the left, code assist, Modbus, and OPC UA are everywhere.
If you go to code assist's website, there's like hundreds of vendors that use code assist, and they have multiple types of equipment that they sell.
Tons of equipment from all the vendors speak OPC UA, and Modbus is another extremely common protocol.
So, the breadth of impact of this malware set is more than, like, we've seen in the past ten years.
So, our lab, what we did was we set up an NX1P2 Omron controller with a couple servo drives and a servo motor.
And then there's the requisite stuff that we need to actually connect it to the network, like EtherCAT couplers, and, you know, we have power supplies and all that kind of stuff.
The Schneider electric devices, we had a TM241 and a TM251 with two different versions of the firmware.
We have an OPC UA server set up with our mock pipeline process.
So, it's a Kepware server with a bunch of Rockwell devices that are controlling a pipe and a pump.
And then just some other hardware that we wanted to play with.
So, basically we just buy these devices off eBay, or we asked a friend.
Tom Van Norman over at ICS Village, like, shipped me a 251 in a pinch because I couldn't find one with the right firmware.
Turned out it still wasn't the right firmware. Ended up not being a problem.
We tried to downgrade and it bricked the device.
We could consider asking the vendor, but something to keep in mind with the industry is, you know, in the regular IT world there have been thousands and thousands and thousands of malware samples.
And these kinds of things are a little bit more common, but in the ICS space, aside from the deluge of ransomware, we have seven cases of ICS malware.
And there's all kinds of other things that you have to worry about when talking to the vendor.
They may not have the processes in place for even dealing with this.
The NDAs can take some time.
Personally, as an analyst, I have reporting deadlines and things.
I need to get things to our customers.
So generally, I prefer to just purchase the stuff where we can, stay independent, and then approach the vendors with the information once we have it.
So when we go about analyzing this kind of thing, we had five pieces of malware.
It was a big organizational task.
We're basically focusing on a lot of the PLC malware first, statically analyzing it.
So most of these toolkits are written in Python, a little bit of C++.
Pretty easy to read.
So we do that static analysis while we're acquiring the hardware.
We're going to release initial details.
The mitigation advice.
We do just a shit ton of runtime collection, collecting PCAPs of everything.
Check which techniques work, which don't.
And then we're going to release updated details, mitigation advice, and then later on all the follow-on work that isn't as important for defending the networks.
And that's going to be things like actually stripping the firmware off the devices.
We can't do that now while we're runtime testing it because we could break them.
And like I said, sometimes they're not easy to find.
So we're basically in step five at this point.
And that's been since I think April is when we started doing all of this.
So the kinds of questions that we're asking ourselves before we get started with the lab testing.
The main one, it's like what's current?
What do we have right now?
Does it work?
Because we don't know yet.
And then there's the Codasys issue in Evil Scholar.
Does that actually work?
Is that something specific to Schneider Electric?
Or does it work on any Codasys version 3 device?
Bad Omen has a module for manipulating servo drives and motors.
We wanted to see if that worked on only the specific motor that was targeted in the malware or if it worked on other drives in that series.
And then more generally, and this is the most important thing, can these tools manipulate PLC logic?
Meaning can we actually manipulate this process specifically to create some kind of destructive effect?
Then we think about future capabilities.
Can we automate an attack with these tools?
What do we think these tools are going to look like in six months?
We know we have them before they were actually used.
So sort of the implication there is that they're probably being actively developed.
So these are going to change by the time they actually get deployed.
And we want to hypothesize what's that going to look like so that we know what to look out for in the future.
And then there's the last category, victimology, which is squarely in the vein of threat intel.
Can we actually narrow down a target?
So let's get to the fun stuff.
So the first thing we started with was Mousehole.
Mousehole is a remote console for interacting with OPC UA servers.
So you can sort of think about like OPC as like a translation mechanism.
You speak to the OPC UA server and then it translates those messages to whatever language the other devices speak in order to manipulate them.
It's kind of the best way. You can just think about it as sort of like an intermediary.
So what this tool can do is it can scan for OPC servers in the network.
It can enumerate the OPC UA structure and change specific attributes that then change attributes on those devices.
So this can be things like opening and closing a valve, right?
Or maybe perhaps changing the speed of something.
Or rotation direction.
I mean, it can be all kinds of things. It just depends on what the device is.
In general, this is sort of a better implementation of the crash overrides attack.
Crash override used the older OPC DA attack methodology.
So OPC DA is the older OPC protocol that was based on DCOM.
The OPC foundation was like, that's stupid.
And they decided to make it cross platform and that turned into OPC UA.
The structure of it is a little bit different, but it's basically what happened.
So with Mousehole, it just worked.
It's based on an open source library that was already well vetted, I think.
So it was very easy to get it to work and manipulate the process.
We had no problems on our pipeline, our lab.
So after we verified it worked, I spoke with the analyst that was working on this.
And we decided, let's automate an attack to produce a more real world attack scenario.
The hypothesis being that you would deploy these remotely interactive utilities to sort of recon the OPC server.
Figure out which parts you're going to manipulate.
Pull the important, like, sort of reconnaissance tool back.
And then just like wrap up a very small version of it and just throw that onto the box to actually disrupt the process.
So I want to show you a video here of our lab.
And this is us actually testing the malware.
So you can see how exciting my job is.
So you'll see we have a water tank.
There's a pump.
There's a little valve.
And on the bottom right is our HMI screen.
This is us actually running the malware and creating an unsafe state in our pipeline.
See that?
It's super exciting, right?
So it turns out if you ask your management if you can blow up a pipe in a water tank in your office,
they tell you, they respond with a bunch of words like safety and liability and expenses.
So what we're going to do is actually, the reason nothing actually went wrong there is that lab is specifically set up so that if we did, like, overdo it, it wouldn't break.
So the pump that's attached to that lab is like a little baby pool pump.
And it can never, like, overpressurize that pipe in a way that would actually cause a problem.
And that's good because that pipeline process sits right behind our CFO.
So it would probably be a bad idea if we screwed it up.
All right.
So before we can actually talk about what an unsafe state is, I want to show you what a safe shutdown looks like.
So we're talking about a pipeline.
So we have a few important things here.
There's the pump and how fast it's pumping water.
There's the valve position, whether or not it's on or off.
And then there's the actual PSI of the pipe.
And in particular, the safe value for the PSI of that pipe.
So there's a max PSI that this pipe is allowed to reach.
And any higher is considered unsafe.
And when that unsafe value is exceeded, right, the process is going to take over and say, okay, we're going to turn off the pump.
And we're going to get everything back down so that you can restart that process and keep going.
So this is what it looks like if we try to exceed that PSI.
Oh, man, why is it?
Oh, there we go.
We try to exceed the max PSI.
We've slowed it down a bit because otherwise it, like, goes really fast.
So you'll see there we have the flow rate increasing.
The max is about two gallons per minute, right?
And the next thing you're going to see is that solenoid box, that's the green box at the bottom, is going to turn red and switch to closed.
And then the pressure starts to increase, right?
We're at 14 here.
And you'll see that it's probably already been exceeded to 15.
The flow rate is already decreasing.
So it's going to go ahead and turn off the pump.
So you're going to see that go down dramatically.
That's that 80 to the negative whatever stuff.
And then the flow rate decreases and the pump switches off.
And then what the operator is going to get is a reset alarm.
So you're going to see a new box pop up in blue there.
And then it's going to turn red.
And then it turns blue one more time.
And when it turns blue that last time, it means you can hit the button, clear the alarm, and then start the pump again.
Right?
So the operator is going to get an alert.
And the process is basically going to take care of itself to make sure that everything is okay to restart.
So how do we actually attack this with the automated utility?
Well, run mouse hole via the CLI to gather the important server and node information so we can understand what set points to manipulate.
Right?
So we need to know what the max safe value is.
That's called the high pressure set point.
That was set to 15.
We want to increase that to 100.
We need to change the pump speed.
The pump speed was set to 80 by default.
We're going to increase that to 100.
And then the last thing is closing the valve, which is the solenoid energize attribute.
We're going to change that from true to false.
And that's going to end up closing the valve for us.
So this is what it looks like on the HMI screen when you actually execute mouse hole on the pipeline.
So you'll see the flow speeds are ready at like max speed.
And then solenoid valve is going to close.
You're going to see the pressure go up.
So it's already past it at this point.
You'll see that the flow rate starts to decrease.
And it's not decreasing because the pump's off.
It's decreasing because the pipe is full and there's nowhere for the fluid to go.
Right?
So basically what's happening here is the pump is trying to get water into the pipe, but it's too high a pressure.
It just can't fit it.
It's like trying to stuff your full closet.
Right?
So what happens when that occurs, you're not able to throw pump fluid through that pipe.
And that's called deadheading.
And this can be a problem in certain situations.
The reason being is when you deadhead a pipe,
what happens is you add heat to the fluid that's being pumped.
So if you're pumping water, this is okay.
Because you're adding heat to water and all it'll do is like deteriorate seals.
And then it might get leaky and an engineer notices and they're like we should fix that.
And not investigate any further.
But if it's something combustible, like mining slurry, which happened in Queensland in 2017,
the pump blows up.
Right?
Which is obviously a problem.
Okay?
So this is a very simple attack.
But this is just an example of what can happen in just like a regular pipeline process.
The other reason I have this example included is it's important for the rest of the talk
where we're going to talk about PLC manipulation and control.
The reason that this pump exploded in Queensland was that someone made a bad modification
to the PLC logic that basically closed the valve and allowed the pump to pump against this closed valve.
So we're going to move on to Evil Scholar, which is the Code Assist version 3 utility.
The command line, the main things we're going to focus on here are the proxy module in Evil Scholar
as well as the maintenance commands that allow me to transfer files and modify logic.
So first thing to keep in mind here is that we're dealing with Code Assist.
It's a device management protocol used by hundreds of vendors.
Wide usage makes it a natural target.
And essentially it's a custom TCP implementation on top of UDP with an application layer.
And basically the end sum of that is that Code Assist sucks and it's a pain in the ass.
It's like pseudo-routable. If you're using NAT, you don't get packets back.
And when you're trying to test a piece of malware and you're not sure how well it works,
you're not sure if you're dealing with broken malware or if you're dealing with a broken network config.
In particular, in our case, we had a lab where we could connect to the devices just fine.
We had another lab where we couldn't connect to the devices at all.
And this one by far took us the longest to sort of sort out.
We thought it was a firmware issue.
And it just turned out that out of the three utilities, Evil Scholar is the dum-dum in the group.
And in part because the developer made poor assumptions about Code Assist.
It made assumptions about the network size, the address sizing.
Address sizing in Code Assist is variable length.
So there's just a lot of issues there.
So the default version of Evil Scholar, we could connect to the 241 and the 251 with no fixes.
If the devices were on the network of the right size.
We spent about two to three weeks fixing the Python code that generates the Code Assist traffic.
And we were able to get it to work with regular Code Assist version 3.
And then we were able to additionally connect to a Raspberry Pi running Code Assist as well as a Hitachi EHV+.
Provided there were any little endian architecture stuff.
Because Code Assist also has fields that will like flip endianness depending on the target architecture.
So we didn't go that far because it's not our job to develop malware or fix their shit.
We were just trying to see exactly how far they had to go to make this work.
So we were able to test all the plug-ins.
And so this is just a really quick example here where when we did things statically we made like a slight error.
And then after doing it dynamically we were like, oh, like this is actually what happens.
And in hindsight you're like, well, fuck, of course that's what happens.
So PLC proxy.
The idea here is that you have an internal PLC network behind an industrial firewall.
The industrial firewall's job is to make sure that the configure, the EWS, the engineering workstations only talk to the PLCs for which they're responsible and no other ones.
So if I want to talk to those other PLCs, the evil scholar tool wants to talk to all these other PLCs from wherever it happens to be.
So what it does is it queries for the Code Assist gateway to grab its IP address and it selects the network that the malware happens to not be on and creates a route to that network through the target Schneider Electric PLC.
So originally we thought it was actually like sending traffic to the gateway.
It turns out it's just sending traffic to the network and just letting the TM251 route to wherever it wants.
And that was kind of the basic idea.
It ended up being kind of a minor error because all we discovered was that the proxy thing was actually just nastier than we originally thought.
Because as we were runtime testing what we realized was that the 251 won't just proxy Modbus traffic, it'll proxy anything you give it.
So you can SSH through it, HTTP commands, arbitrary other ICS protocols.
It's just crazy, to say the least.
That's basically it.
It just got worse.
Not necessarily what you wanted the result to be.
We're going to move on to the logic corruption pieces of this.
So evil scholar lets you transfer files to and from the target device.
So you can pull the logic.
In general, you can pull the logic, modify it, put it back on the controller.
Super easy thing for us to do with this tool.
And it should be trivial to modify the logic, provided it's in a native assembly language and not an unknown bytecode.
So some of these controllers have just their own bytecodes that you have to figure out before you can do this kind of attack.
In this case, we were dealing with ARM.
It was super easy.
You see the red lights there mean the controller vectored.
That error box there says that the controller is unable.
Sorry.
The EWS is unable to connect to the controller, right?
We were able to crash, manipulate logic on the controller, create various error and output states.
In both cases, comms to the controller are not possible unless you power cycle the device.
If you connect to the PLC before the crash, Schneider's SoMachine software will report that it knows there's something different on the PLC and only provide you with the option to override it.
And it won't let you pull the corrupted logic back so you can see what's going on.
And this is something that's enforced by the EWS software.
It's not enforced by the PLC.
And we know that because Evil Scholar can pull the bad logic for us.
So in that sense, Evil Scholar doubles as a forensics tool, which is super weird.
And not what I was expecting out of this sort of malware analysis journey.
So the last little oddity here is that essentially these controllers run an execution cycle.
So there's a bunch of applications which are basically programs.
And it calls each one and waits a minute for it to return.
So on that first execution cycle, if you crash the controller, it just sets all the outputs to false.
It doesn't set them to their default fail-safe values, which is what you would expect a crash to get set to.
Because that default fail-safe is where the process should be when it's not doing anything.
And then you flip the right values to start everything.
So it just sets everything to false, which could have some unknown consequences for various processes.
So that's a bit of a serious issue coming out of these controllers.
So I'm not able to say too much more here because we're still working through the CVE process with Schneider and CodeAssist.
We report on June 22nd, but it isn't quite long enough a window for me to be able to show screenshots of the code and what we manipulated and everything else.
The good news is I can do that for BadOmen.
So we're going to focus on the sort of logic corruption and manipulation with the BadOmen utility.
So BadOmen is again another remote shell for manipulating Omron controllers.
And it primarily does this via an HTTP interface.
That's exported by the NXNJ series controllers.
The things we're going to focus on here in terms of its capabilities are the transferring of logic, transferring of files,
as well as the servo module for manipulating servo drives.
So real quick background on the console.
It takes advantage of a hard-coded credential vulnerability so they can default login to this HTTP server that's sitting on the NX1P2.
The server has various CGI endpoints that's also used by Sysmac Studio, the EWS software for configuring the controller.
So in this case it primarily uses the CPU.FCGI and the ECAT.FCGI.
CPU for doing the standard downloading and transferring of files and logic and changing the controller state to run mode and program mode.
And the ECAT is primarily for just wrapping up EtherCAT traffic to the NX1P2 to send on the EtherCAT daisy chain.
So same process that we did with Evil Scholar except we're just using different modules now.
We took the backup module to grab the entire project file because the NX1P2 just has a backup command so we just ran that.
We unpacked it, we took a look. It turns out it's running QNX Neutrino so the logic compiles the shared objects that gets mapped dynamically in the runtime.
So the entry point was super easy to find, we just had it branched to a bad address.
And then we replaced the files on the controller and that resulted in another major fault.
This major fault was similar to the Evil Scholar one in that we weren't able to download, upload the logic back I guess.
We weren't able to retrieve the bad logic.
It prevents it, it says that there's no logic to retrieve once it crashes.
I imagine this error happened multiple times even when we do some of the manipulation stuff.
And there's some kind of weird versioning where I think the EWS is saying you haven't done a transfer so there's nothing to pull back.
Everything's still the same so why are you asking?
The other thing is it prevents you from enabling program mode and without being able to enable program mode you can't recover regularly.
So we tried to restore from an SD card and that failed but for whatever reason that allows you to set program mode again.
Then a factory reset and then you restore logic to the controller.
It sounds like a medium pain in the ass, you listening to me, but if you imagine a plant engineer and this stuff happening that's just the rest of his day.
And probably a call to the vendor and then wait time and everything else.
So just this simple attack would cause some downtime for a plant and depending on what it's actually processing like lots of fucking money.
So moving on, the servo module testing.
So real quick before we actually talk about what this is, what's a servo drive and what's a servo motor?
So a servo motor is just a motor and it spins a shaft and it can be connected to any number of things.
So some of the examples that I've heard about are servos control widgets.
You can look at gates and dams for controlling like water flow into a dam.
They can be in drone cameras.
I think they actually help with like creating base and subwoofers and things like that.
Basically my general impression about servos from having analyzed this malware is that if you need something to move there's probably a servo involved.
So the servo drive provides power to a motor and it also handles all the comms and back and forth to the NX1P2 controller.
The malware in particular targeted that 1S10F ECT servo drive.
But this was a 380 to 480 VAC drive.
We don't have a reliable source of 480 VAC power and that's also like just super effing dangerous to just have sitting in an office.
We didn't want to like electrocute the plant engineer that was so graciously helping me set all these things up.
So we looked at the documentation and it turned out that there was a 120 VAC model that we could hook up.
And 120 VAC is what regular residential and commercial real estate uses.
So we hooked that up and we got the corresponding small motor and we were able to verify that the malware also worked on that smaller drive.
So we did the rest of our testing on the small drive and also like just for general safety if we broke a bigger motor like bigger things could go wrong.
So we stuck with like the safer option here in our lab.
So this is just a quick diagram of the setup.
You have the computer talking to the CPU unit which is the NX1P2.
And it's going to relay the EtherCAT commands down through the network to the actual servo drives and anything else that happens to be on that EtherCAT chain.
So the actual servo module plug-in, it turns out you can't monitor the fun stuff with it.
You can't just change motor velocity and like make things overspin.
You can't change the direction of the motor live even though that was a parameter.
So what the EtherCAT module, what the servo module does is it modifies particular parameters.
Think about it as basic configuration.
So there is a parameter for changing the direction of the motor, the way it spins.
But you're not allowed to change that while the motor is actively running.
Like everything has to be like turned off and in some weird configuration state.
So there's not a lot you can do with this.
But what you can do is kind of, you could be annoying, right, and it would still cause downtime.
For example, there's an excessive speed detection parameter.
So if you set that to one rotation per minute, right, and your process is running at 40 RPMs, the motor just stops.
And then, you know, they might be able to like fix that and set it back and then you can just go back there and set it again.
So you can just keep needling them over and over again and just keep stopping all these servos or stop a bunch of them at once.
So just with that, you could cause some issues.
But generally, like looking at this, I sort of see it as a precursor to an attack where you actually screw with the motor, right?
I'm going to up the motor speed to max and I'm going to disable all these warnings and the excessive speed detections so that you don't get like sort of a warning on the EWS that something is going wrong.
And this worked out for us.
Because we also want to test logic manipulation with Bad Omen on the NX1P2.
And it would be a lot more fun if we could like overspin a servo motor than just like setting outputs to false, right?
Like that's not, I mean, it's fun to I guess kind of see.
But like physical feedback when you're doing this is just a lot more interesting.
So we know from crashing the logic that we can modify code on the device and send it back.
And we also knew that the servo, the logic for the motor just sits on the same NX1P2.
So we should be able to pull that and modify it.
So the first thing we had to do was actually configure the servo to run and do things, right?
So that's lateral logic. That's the visual programming language that engineers use.
The main thing to keep in mind is that last box at the bottom is a function called MCMoveRelative.
And it takes four arguments. Distance, velocity, acceleration, and deceleration.
The important one being velocity because that's how fast the motor is going to spin.
So we created this program. We sent it to the controller.
And then we pulled the backup just like before and we decompiled the program.
So the interesting thing here is, again, it was a shared object.
You have all these symbols and like function names.
And there was one called POURunFunctionBlock.
So if you just like cross-reference that, like basically a function represents that particular lateral logic.
And it's like in order. Like the first call is a call to the first function.
Then the next one is a call to the timer that I set up.
So the last thing I call is moveRelative.
You just kind of scroll to the bottom like you're dealing with UPX.
And you're like, oh, there it is. Right?
And if you look at the bottom left there, you'll see a bunch of 64-bit numbers and like one 32-bit number.
Those are IEEE 754 real values. Right?
And so you can convert those to figure out.
And it turns out that the V18 maps to the 40 degrees per second that I had set the motor velocity to.
So I know degrees per second sounds weird.
You don't set it in RPM, so you have to do these conversions.
And there's some other ways to set the velocity.
In the manual they call it like units of display.
And you have to go look up in your EWS like exactly what the units are.
So we have the speed set to 40 degrees per second.
I want it to spin at maximum.
So you just do the conversion.
The motor's max speed is 3,000 RPMs.
You multiply that by 6, you get 18,000.
Convert it back to real.
Patch the instruction. It's ARM.
And then reupload the program.
And that's my next video.
So this is what happens.
So before we keep going.
Something to keep in mind.
On the right you have the ladder logic as it's running.
Green means energized.
I'm just going to rewind it just a little bit here.
Is it playing?
No.
Almost.
There we go.
Okay.
All right.
So yeah.
On the right, ladder logic, green means energized.
That means the program is currently running.
At the bottom here, I think you can make it out.
I can't really see the screen from y'all's angle.
But that's the motor velocity.
So 40 degrees per second is anywhere from like we've seen ranges from 4 to 7 RPMs.
This number is in RPMs because the EWS software hates you.
I don't know.
Like everything's different in there.
And then here is the actual servo motor that we had set up spinning.
And up here we have the actual mode and the IP address.
So I'll walk you through some of this now.
So it's spinning at 7.
And now it's going into program mode.
Because Bad Omen is transferring the modified logic to the servo.
And now it's switching back into run mode.
See that switch.
Now everything's energized but there's a timer block there.
So it takes about 5 seconds.
And then once it's energized, it just starts overspinning.
Or not overspinning, it's spinning at max speed.
So that's at 3,000 RPMs.
And all that happens automatically.
It was literally like I think two keystrokes or something to do that.
We also tested spinning the servo motor like overspinning it at 5,000.
And it turns out the motor will only spin at max.
I don't know if that's a runtime decision, like a runtime check.
Or if that's like a mechanical limitation.
We also tried sending negative values.
And then we actually got a proper runtime system error which we had never seen before.
The nice thing is with Bad Omen we can actually access that runtime to like look at it and see like exactly what's emitting that error.
Maybe we can patch the runtime system to ignore it.
So there's a lot of other things we could probably do here that would be pretty nasty.
But also, you know, we had slide deadlines for DEF CON.
So we can only do so much.
Let's keep going here.
So last thing here.
I want to talk about safety system corruption at Bad Omen which is just something that's pretty personal to me.
Because one of the first things I worked on at Dragos was a trisis which affected a safety system.
And when we were looking at Bad Omen what we noticed was that you can hook up an SL3300 on that same EtherCAT network to monitor the servo or anything else on that network.
And so what we were wondering was, well, if the servo's logic sits on the NX1P2, does the safety system's logic do it?
Or does the NX1P2 transfer it in such a way that we can manipulate it, you know, free transfer or whatever?
So while we were setting up the safety system, what we noticed that it was very clear that setting up that safety system was different than manipulating the regular logic.
So with the regular NX1P2 and the servo logic, you would put the thing in program mode, I think take it offline, put it in program mode, make your manipulations, put it online, transfer everything over.
And then you hit run and then everything starts to execute.
For the safety system transfer, there was this other weird convoluted process where you select the safety system, you put it in program mode.
Then you go to put it in debug stop mode and then it pops like four different yellow pop-ups, one of which is a password that was blank.
And then like you hit okay like three or four times and then it goes into debug stop and actually like transfers your program over.
But that still isn't going to run it. You can't put it in run mode from there.
So you put it in debug run to make sure that everything runs correctly, then execute a safety validation, which does I still don't know what.
Because I don't know, I wasn't able to trigger an error in that. I wasn't sure what it was checking for except for like random variables that weren't being used.
And after all of that, then you can put the safety controller in run mode.
So these are all things we had never seen Bad Owen even attempt to do. There were no reference to this in the code.
We weren't really sure about it and so we just went to digging. And the way we did this was we created a program.
So you'll notice that the program on the left looks a lot different than the ladder logic program we saw before.
That's because it's another engineering language called function block diagram networks.
And I'm not sure exactly why we have both of these languages because they seem very similar to me and you can do similar things with them.
But it reminds me of just like circuit programming with a bunch of ands and ors.
It's kind of the idea. So two of our services folks helped me write this program. It's a simple timer.
And then we went through the process and steps of loading everything and we downloaded it.
And we started looking through the PCAP to see what was going on.
And you'll notice highlighted red there in the sort of the payload area, you'll see references to the names for the function block diagram.
And what we ended up realizing was that Bad Omen, sorry, Sysmac Studio when it's transferring safety controller programs uses a different CGI endpoint than we had seen before.
So in this case it's using the nxbus.fcgi endpoint and there's a nx send and receive data.
And it's using a different protocol, a different endpoint and probably a different protocol.
We dug in further and it actually looks like the safety controller is running, this is kind of funny, code assist safety runtime under the covers.
But the protocol that's actually involved there isn't code assist protocol so they're wrapping it a little bit differently there.
But we're fairly certain that we have more code assist research on our hands.
But just like sort of the safety side of it.
So we're able to confirm that it only supports the CPU and ECAD endpoints.
It can't manipulate the safety controllers.
I'd like to say that that's good news.
But again, we don't know when, how long, these tools obviously have investment behind them and they're actively developing them.
And practically speaking, if you want to conduct a real world attack on say a real servo,
more than likely there's a safety system standing behind that servo watching it to make sure it doesn't fuck anything up
and then setting it back to the right state.
So if you're going to conduct a real attack, you have to manipulate the safety system and then manipulate the servo.
Somehow you have to do both.
And so I don't see, there's no reason to believe that they aren't actively working on this type of safety system compromise with the SL3300.
It's one more endpoint, they've managed to get all the other endpoints right.
So I'm sure they'll be able to figure that out.
So, caveat there, unless you have a shitty network
and then maybe they can mess with your servo.
Cool.
Alright, so we're almost done here.
I'm going to talk about some of the real world usage here.
So the endpoint here from running it, or sort of the main takeaway here from just running it,
is despite the CODIS's problems, we think those things are going to get fixed.
The safety system things, we think those are going to, that's not going to be a gap in their tool set anymore.
But generally speaking,
these utilities can cause problems now.
And at the very least, at minimum, they're going to cause financial problems,
if not disruptive and destructive problems for a site.
So what we sort of think like a theoretical attack scenario would look with any of these utilities,
is there would be an IT intrusion to steal important process documentation.
Figure out what the process is, what's important, right?
Where are the important bits that they have to manipulate in order to actually cause a problem, right?
Maybe they've never attacked like a mining facility before,
and they need to figure out what all this terminology means.
There's a lot of research that's going to have to go into it beforehand.
Then they would deploy standard implants, blah, blah, blah, blah, until they deploy Pipedream.
And then Pipedream would be used to recon the process technically.
What set points do I need to manipulate?
What kind of equipment do I actually have on the network, right?
Are there CODIS devices here? Are there Modbus devices here?
And they could bring all that data back.
Now they have the process documentation and the actual technical data,
put that together to develop a utility to actually,
like, to put together a utility for causing that disruptive effect.
Or, if they're feeling lazy, just type in the commands and do it on the console.
So, what I really want to harp on here is that this really is like,
and I don't like to sound super hype, because if any of you all actually know me,
I'm pretty fucking cynical about things.
And I don't like reading articles about malware analysis.
This shit is serious.
I've done this for ten years, and this is the first time I've opened up a piece of malware
and just went, oh, fuck, right?
And so this does have some serious implications for the ICS industry,
for us in general, because it's attacking critical infrastructure,
just in terms of the breadth of impact of this utility.
It is the first time that we've seen malware devs attempt a plug-in framework
for specific PLCs and protocols.
There is some precedent for this in ICSploit and Metasploit SCADA.
I would argue that these are a little bit different,
because they're a little bit more focused on the particular protocols
versus sort of like a hodgepodge, right?
You can't add like a random Modbus exploit to Bad Omen.
That's not going to make any sense, because the underlying libraries expect you to use
like the HTTP console, right?
So it wouldn't really function that way.
But you can add arbitrary plug-ins for Omron controllers, right?
Same thing with Evil Scholar.
Almost all the plug-ins are expecting to use that Codasys library,
so you can't necessarily just like add a plug-in for,
like you sort of can, maybe, but it's not as straightforward.
See what I'm saying?
But you can very much straightforward add a plug-in to do something else in Codasys.
The other thing to keep in mind is that this stuff is ICS process agnostic,
which is kind of an interesting thing if you're an adversary,
because, you know, Codasys is everywhere,
so I imagine that more than likely now you're not limited by your target selection anymore.
You can say like, well, maybe I want to attack that mining facility,
over there, and previously I was interested in electric,
but they're both running Schneider electric controllers,
and they're both speaking Codasys version 3.
So it sort of made everything easier for them now.
The last kind of two points here I want to harp on is that, again,
these tools can be used for reconnaissance of the network,
and they can be used for research.
So while they can do disruptive things,
they are also tools that will accelerate the learning curve of the adversary, right?
Whether on your network or just in their lab at home, right?
The next thing is these command line tools are super easy to use,
which means that the people that are actually using the tools
can just get like a checklist of instructions,
and they can sit on a keyboard and just type them all in,
and everything's done, right?
So the people using these tools don't have to be as smart
as the people that are developing them, right?
And we haven't necessarily seen that so much either
with like a remotely interactive tool
where they're trying to lower the bar for entry for achieving an attack.
In terms of, just in terms of the industry here,
like really quick soapbox moment,
is I really think that some of these like proprietary protocols
that the ICS vendors are using and proprietary OSs
need to be like opened up to security researchers a little bit better.
I think it's kind of ridiculous.
Like for example with Trisis,
I think it's absolutely batshit that I had to analyze like a program,
what was it, a Python program,
and then shellcode that ran an OS I had never heard of,
and then extract firmware, and then analyze the operating system,
all so I could figure out one system call, right?
That sounds fun from a research perspective,
but when you're under the gun trying to get advice out to operators
who are seriously worried about their safety,
it's like just seriously annoying.
That project could have been so much quicker
if someone had just handed me like a syscall reference, you know?
So, and that's what needs, that's what happens any time we have,
every time, the last two times anyway,
we've had to analyze these kinds of things,
and I think it would be a lot better
because clearly the adversary is putting in the effort to learn this anyway, right?
And we sort of need to get ahead of the game
because compared to the IT world,
our understanding of the heterogeneous operating systems that ICS uses
is so much better.
It's much smaller, right?
In our world, in the IT world, right?
It's like Windows, Macs, Linux, I guess iOS and Android, right?
Those are kind of the five major ones, and that's it, right?
And we've spent so many years studying this
that we already know how they all work, right?
For the most part.
With ICS, there's like limited research on each of these OSs,
and you end up having to pull apart the firmware to really figure it out.
So I just think that the amount of time it takes
could be made a lot easier
if some of these vendors cooperated a little better with us.
The other thing is I think that the EWSs
and the vendor utilities should just create forensics tools
or allow the EWSs to pull bad logic
so that IR people and forensics people can like
send the logic over to people like me or anyone
and look at it and see, oh, this program's actually doing X.
Right?
And in that sense, I think that the
the vendors are so used to being the expert
and the people that get called in on the scenes
and that's starting to change
where IR teams are starting to go on the scene.
Right? Dragos goes on the scene, Manny goes on the scene, right?
But how often can they actually pull bad logic?
Sometimes, maybe.
They happen to find it, but it's not so common,
or maybe they know enough about the EWS
that they can find a project file sitting on a device
and maybe they can see what went wrong.
Right? But that isn't very common either.
And in some sense, I think the vendors need to start recognizing
they aren't the only experts about this stuff anymore.
So, and then the last thing,
the sort of fundamental problem with all of this
is that like PLCs just don't do integrity checking in their code.
That's just the most basic problem that needs to get fixed.
As far as I can tell,
from the billions of vulnerabilities that exist,
if there was just a modicum of integrity checking,
I think a lot of these things would have been mitigated to begin with.
So, that's all I have for you.
Just really quick so I don't get yelled at by people.
We do have mitigations for this,
so if anybody's watching it online, there's a link.
You can go check those out at that website.
I didn't talk about it here because it's DEF CON,
and I'm just like, that's boring.
So, if you're interested in that, go check that out there.
Otherwise, I'm finished.
Thank you all for having me.