Good afternoon, everybody. Next up is Chris Sistrunk with what's the difference for DFIR
ins for ICS. So give it over to Chris. Thanks for coming. Can everybody hear me? Okay,
great. So hopefully you've been able to go see the ICS village and that's what we're here
representing. If you get a chance to, oh, thanks, nice. If you get a chance to go over there and
play around, actually try to plug in and see things because a lot of people don't get a chance
to do that and a lot of the equipment is cost prohibitive for most researchers. So go get a
chance to see that. So my name is Chris Sistrunk. I'm a professional engineer. I can talk
more about myself later. This is a lot of information to cram into a talk. But I'm going to
cover digital forensics and incident response like an overview. How many of you have ever done
digital forensics and incident response? I don't know if you've ever done digital forensics.
Okay. How many of you have done it for control systems? Yeah, very few people. When I gave this
talk at Black Hat last year, it was like three people raised their hand that had done for
embedded systems. So we're going to talk about DFIR for embedded systems and ICS, embedded
devices, what to collect, what to analyze and do some. Since I'm an electrical engineer, I work
for a power company. I'm going to talk a little bit more about that. I'm going to talk about the
power of the power. I'm going to talk about the power of the power of the power. And we're going to
talk about what we want to do. What we want to do is cover some examples for substation RTUs and
how to do digital forensics on those. And it covers a couple of operating systems there. So
what we want to do for incident response. We have find evil, assess the situation, define
objectives, collect evidence. This is from the book, the incident response book. And this is the
same framework that we want to do for any incident response. So if it's for IT or for the
or anything. It's almost akin to doing root cause analysis in engineering. So we do an
overview where we're collecting files, operating system, volatile and nonvolatile things,
network traffic, application data. We do an examination in digital forensics. We do an
analysis and reporting. So this is from the NIST standards in the incident response and
computer forensics book. Those are some really great resources. If you get a chance to go get
those books or get that standard, it's a free standard, NIST SP 800-86. So there's a lot of
traditional DFIR tools that are mature, red line and volatility. Anybody ever use those
before? I mean, I think it's great that the SANS ICS team has done some challenges where
they actually wanted you to use red line and volatility for an ICS type challenge. That's
great. And there's websites like virus total malware and a couple of others. We have cheat
sheets and books. But what do we have for control systems? We don't have any of those kind of
tools. So what is the difference for ICS? Hopefully that's the pun of the day. So for
assess the situation, you know, when, where, how is ICS affected? Define objectives, return
the control system to normal. So that's a great tool. And then there's a lot of other
things that you can do quickly and safely. Collect evidence. You know, ICS devices have real
time operating systems. I won't read the rest. But here's the difference. You have a similar
situation between IT, DFIR and control system DFIR for assess the situation. But for define the
objectives, it's physical processes. It's not just network. It's not just a data center. You have
to deal with, you know, a lot of different things. You have to deal with a lot of different things.
You have to deal with oil moving in a pipeline or, you know, power moving through power lines or
water through pipes. Manufacturing facilities where you have a chain of, you know, things
happening. Those are the things that are going to be important when you do DFIR. In collecting
evidence, there's a difference there. It must be collected manually most of the time because we
don't have any tools. And a lot of times you'll have to use the tools. You'll have to use the tools.
You'll have to use the tools that are in the engineering workstation like a software from a
vendor or something like that. When we do analysis, same thing. There's no ICS specific DFIR
tools. You can't use a red line or a volatility. You can if it's a Windows based HMI. You can do
that. But that's not the focus of this talk. We want to focus on the embedded systems and the
custom operating systems. Communicate the same. Develop a remediation plan. You know, we're
talking about how and when to regain control of the control system. ICS devices have
constraints. So you can't just go reimage a PLC like you can a laptop. Sometimes you have to load
firmware and the configuration and make sure you have, you know, a lot more things. So it's not
just willy‑nilly go just like we do in IT. Reimage the laptop, no big deal. So with control
systems, you have to really think about how you're going to do that. You have to really think about
how we do the remediation plan. Kick the bad guys out or if they're still attacking. Find out what
to do and do it safely. So let's talk about ICS anomalies. I know some of you in here know about
anomalies on control systems. But we want to say an anomaly of some kind has occurred. Increased
network activity, strange behavior, some kind of a failure. Now we need to investigate it. Is it
known bad? Is it unknown bad? Do we escalate this to a failure? Is it known bad? Is it unknown bad?
As a security incident, do we have to call in our security teams or notify EISAC or any, you know,
ICS cert or whoever? And who do we call? Do we call engineers, admins, PR, safety vendors, legal?
So here's an example of an incident. Might be an anomaly. So someone made a mistake in an example
configuration. So instead of 2400 ball, they put 2600 ball. So instead of 2400 ball, they put 2600
ball. And so how many times if you're an engineer, you've been called at 5 o'clock on Friday because
someone missed a configuration. And that's happened to me a lot. So then you have, you know,
maybe squirrels. That caused an incident. Or you may have some kind of unknown thing doing
something weird that you've never seen before. So don't reboot it. Don't reboot it.
A lot of times, unless it's a safety issue. Or there's some other issue that you can do to fix it.
But normally we want to preserve the forensics. And if you reboot the system, you lose the memory
and a lot of other things. So have you tried turning off it again? Don't do that. ICS collection
tools. So we don't have any ICS specific DFR tools, especially for embedded devices. But we do
have, you know, we have, you know, we have, you know, we have, you know, we have a lot of
other ways to collect the data manually. We can use TerraTerm or some terminal program to connect,
you know, Telnet or serial or whatever. Serial cable, right? Connect to the console. You can use
the software like from GE or Schweitzer or whoever, Siemens, to collect information about the
PLCs, the configurations, all that stuff. So you have to use the tools that are designed to go
with that system.
So what can we collect? Two types. Physical data and digital data. So whenever you have a response
that you need to do something, you need to figure out exactly what device it is and where it is and
what rack it's in or what panel or what platform, whatever it is. Any identifying information,
manufacturer, serial number, part number, name. Because most control systems, you try to have a
good asset inventory. But we all know that there's a lot of data in there. So you have to figure out
what's in there. There may not be an up-to-date asset inventory. And you've got to make sure that you
collect that right there and then. What kind of connections does it have? What are the LEDs on the
front and the back? They have diagnostic LEDs that are maybe green or red or, you know. With RTs
down, well, what are the lights? Are they green? Well, yeah. Well, it's good. There's something else
that's wrong. Power consumption, is it using more power than normal if you're monitoring that?
Evidence of tampering. So temperature running high, that's an interesting one. Mark Fabro
found a HMI that was running hot by using a FLIR camera. And it was hotter than the other HMI, and
they found out it had a trojan on it, and it was using more than normal CPU. So it made it run
hotter. So that's how they detected it, with the FLIR camera. Digital data. You want to have data,
to be able to get the running configuration, whatever user account sits in there, last
known good configuration. So if it's in a repository or if you have a CD or a thumb
drive, however they store it, if it's on the engineering workstation, try to find the
last known good configuration. What is the running firmware, approved firmware? Because
we've seen in the real world someone put in non-approved firmware. CPU usage percentage,
memory usage, RAM, storage, all that stuff. What running processes, just like you do with
Windows, you know, net stat and all these things, you can find out what running processes,
what running ports are active. Logs, if you have any, you might be able to do, you might not
have syslog but you can log in and look at the console logs. You may have to use terminal
server with a serial port. And then if you, if you have, if you have, if you have a
system that may be running VXWorks, you can do a memory dump. And I'll show you how we did
that. So what to analyze? Find evil or ways for evil to do evil. So we have a time base. So
things, you know, things you want to do fast and first and things that take more analysis. So,
you know, first responders, what do the user and event logs say? Sometimes an RTU may only have
255 spots in its buffer before it rolls over. So you've got to go grab that quick. What is the
configuration? Does it match the firmware? Is it firmware approved? Same thing. A lot of these
things that you got to go send a technician out there to collect or an engineer or whoever.
Is the configuration and logic correct for the process? So an engineer can look at the
configuration and say, oh, someone made a mistake. So that is not as fast as grabbing the configuration.
configuration. But you've got to do a little analysis there. Are the communications normal?
So you're going to have to look at a serial packet capture or Ethernet packet capture. And
that takes a little bit more time. And what takes even more slower, I mean, is things where
we're having to do getting a vendor, a digital forensic specialist, an embedded systems
analyst. So analyze embedded operating system files, capture data at rest, capture data in
transit. And, you know, in the case of, like, in destroyer, crash override or any, like,
Stuxnet, you know, this takes time. It takes a lot of time to analyze it and reproduce it.
Volatile memory. And to look for code injection or potential root kits. So keep these things in
mind when you have a DFR process for control systems. So let's do DFR and two substation
RTUs. The one on the left is a D20 from GE. The one on the right is a Schweitzer.
And if you are working in the electrical sector, you've seen these. So time to read the manual.
So that's the Apollo manual. And you have to have good manuals. So let's look at the GE D20
MX manual. Anybody here ever seen a D20 before? Okay. I didn't bring it. But this is just an
example. If you have different PLCs, you don't have to know this stuff. But just remember, go
look in the manuals that you have when you get back home. Because you may have, when you get back
to work, when you look at the manuals, you may have all the same information. So this is going to
show you. So we found out the specs. It has an embedded power quick pro. It tells you how
much memory it has. It's running VX works. And then, you know, we've got the tools there. We've
got the software, SG config. I can use terminal. I can use WinSCP. And I can look at the, you know,
the manual that came with it. And then in chapter 11 in there, it actually shows how to
troubleshoot serial communications, firmware version, mismatches, shell commands and logs. So
in this manual, it actually shows you how to do some digital forensics. So here's some examples
that it has some shells that belongs to GE. And then there's the VX works shells. So let's look
at some of the error logs.
Whenever an RTU technician first logs in, they want to look at the error logs. So you see, you
know, some diagnostic information. If there's any errors in the configuration, they're going to show
up right here. I don't know if you can see my laser pointer. But you can see the top part, it
tells some information about the Ethernet ports. And radius is not configured. So there's some
errors. But if there's other errors, they would show up in that screen. Now the user
log is really nice. You can export it to a CSV file. And it also shows every command. So if there's,
you know, what they type and all that stuff. So it's very, very important for digital forensics.
So the power of the three shells, there's a D20 shell and two VX work shells. And a lot of these
shells, sometimes GE, the factory, may want to use these commands with a technician on site. So just
be careful when you're in these things. Other, like Ovation and other GE devices, they use VX
works. So some of these same things apply. So anybody remember this movie? He doesn't know how to use
the three C shells. So you got to know how to use the shells. So the main shell shows you all these
different commands. So you can just type in help and it tells you in the D20. And then what you can do
is look in the user manual. And if you type IFINF, it will tell you what the Ethernet port is. And if
you look in the manual, it tells you what that command does. So read your manuals. And this is great
for an attacker, too, because if you see, you can also do things like restart, inject memory. You
can dump the memory. You can do a lot of things from this. So it's very juicy for you. And you can
use this for an attacker's perspective. So let's do data collection. We can get the running
configuration. It's a very common task. Last known good configuration. Like I said before, you can
type in IMG to get the running firmware. You can query the processes. You can query the
performance. And get the CPU usage. All these things that we would do in Windows forensics. You
just have to type these in manually. So let's go ahead and get started. So let's get started.
You can do a serial analyzer. Remember what we had before, Wireshark? You have to look in
serial analyzer. So we can look in the RTU and say it can tell us this is an example DMP3 string.
So to and from. And we get this and send it off for troubleshooting if we were in the field, if
there was a communication problem. Now, this is really great. VxWorks allows you to dump the
memory. So you can do a serial analyzer. You can do a serial analyzer. You can do a serial analyzer.
You can do a system information dump and it tells you the start of where the memory is in the
addresses. So you need to know that information when you get ready to dump the information. And
you type in D and you can see I put a D, a bunch of zeros, and then FF at the end. So I just
grabbed the first 255 bytes of the memory. But in the D20, you can only do a serial analyzer.
You can't do it over serial. They turned off doing it over SSH. So there's a gig of RAM and it
takes a while to dump all the memory. But we're going to show you some more commands in VxWorks.
Because, like, the Mars Rover uses VxWorks. A lot of these, it's the most popular embedded
operating system outside of like Android. So VxWorks is everywhere and you can use these same
commands. You can, in the, in the, in the, in the, in the, in the, in the, in the, in the
D20 that runs VxWorks, but anything that runs VxWorks, if they have it enabled in the firmware,
you can run a PCAP straight from the command line, which is really great for forensics.
Okay. Now we're going to put on our evil hat for a minute. And we're going to do a live memory
code injection and memory dump on the D20. So we injected M and then we're going to run the
memory to, to do a memory edit. And then we put in dead beef. And you can kind of see it there.
And then we can do D and dump the memory and we see dead beef when we dumped it. So there's no
live demo here. We just took screenshots. So we show that if you are an attacker and you have
console access to this RTU, you can inject memory just like you can do with Windows or Linux or
whatever. Okay. Now we need tools to enable us to perform DFR for ICS and embedded devices. So
with FRI, we got, I got with my good friend at, on the Flare team, they are reverse engineers, and
I said, can you write us a tool that will dump memory out of VxWorks and then do a differential
analysis on that. So we've written it and I'm going to show you the links to it and show you
some things about it.
Okay.
So we wrote a VxWorks DFR tool. It has a bunch of, it's all in Python and it's up on our
GitHub. So there's the address ICSmemcollect on the Fri GitHub. What it can do, it will read and
write to memory of the device. It can cache the live memory locally. And it can compare the
system image. So from GE, you get a firmware set when you download it. And then when you download
it, you get a firmware that is there to compare the memory. And what we do is we compare the
memory that we don't against with the firmware image. And then it does that comparison. It can
work over serial, TCP IP and it supports caching. So if you're say on a D20, you have to do it
over serial. And it takes a long time. So we had some problems dumping it. And it takes like a
six days. Because the fastest you can go is 115, 200 baud. If you could do it over SSH, it would be
a lot faster to dump the memory. So we allowed to resume if connectivity is lost. And it works on
anything that looks like a Python file object. So cache files, memory dumps, sparse memory maps
and other special objects. So here's an example. Josh Triplett explains this a little bit better.
Than I can. If you go back and watch the black hat presentation from last year, you can hear his
explanation. But here's a, it says Python validate image, disk image, VX works. And it looks at
the cache. And you can see a section of the memory says IP firewall start. So in VX works, that
is where it starts the firewall. So what we did is we started the firewall. So what we did is we
injected something to make the firewall not start. So we turned the firewall off in memory. And
so here's some more about the, I'll show you, we're going to show you the outputs from the tool.
But it uses, it loads everything. It works with the architecture of that device, which is really
cool. I don't know how all that works. But it works with the architecture of that device, which is
really cool. It basically pulls from these two things. It loads everything in capstone.
Basically it assembles and disassembles all of the things that go into the firmware for that
instruction set for that processor. Now the plans for the future, we want to do better
documentation. Testing with other embedded devices using VX works like Ovation controller. Or, you
know, for the GE digital relay. Expand the tool to work on other devices and other embedded
operating systems. And refine the scripts in the easy to use modules. Maybe you can have a
volatility plug-in of some kind. And allow for feedback. So if you are interested in using this,
go to that FireEye GitHub and you can download it and you'll have to compile it and do a lot of
things. But if you have any questions, feel free to reach out to me. I'll be happy to answer any of
your questions about it. Just send us an e-mail and we'll help. Okay. Let's go to the Switzer
R-TAC. So this is we wanted to cover a Linux embedded device. And this one doesn't give us any
of the shells like the VX works does. They protected the command line. But it runs on a power PC.
Has a gig of RAM and embedded Linux. And, you know, it has the instruction manual that you can get.
And the software for it. And you can use the web browser and terminal for SSH. So let's do some
collection here. We have to look at the digital data so we can get the running configuration. User
counts, running firmware. All that stuff is on the screen when you log in to it on the
computer. And then you can reset the device so you can log in. And I've highlighted over, there's a
button at the top. If you highlight over it, it says if the password jumper is in or not. So it's
just something to keep track of. If an attacker was physically there, they could put that
password jumper in. And then get into your system. But then it's a physical security problem.
So here's an example of the screens that we did from whenever we did an attack with DMP3. So back in,
2003, 2013, we, Adam Crane and I were doing DMP3 fuzzing. And we had a Switzer 3530. And it was
vulnerable to DMP3. It basically didn't validate the incoming DMP3 messages. So this is a
screenshot, just a little snippet of the web page. And when we sent a DMP3 packet, unsolicited, it was
made to CPU to 100%, which is not normal. Then you can see the memory usage was very large. And
then overtime the RAM crept up to a 100%. And then what happened was, the hardware watchdog
rebooted, you can see the middle screen there, this application has failed, continue anyway.
When it rebooted, the configuration was gone. So you can see that the configuration on the right side, this screen, the
scratched through and it's got a failed project error message. And you can see that the memory
went back to nothing. And then the storage usage was zero kilobytes. So it was really
strange that this message caused this thing to happen. But we could monitor that. If we
were monitoring these things, we could have some really good digital forensics and incident
response capabilities. So data collection, you can get several different reports and logs and
especially syslog. So a good friend of mine, Nathan, he, what he did was he took syslog, but in
the Swyter device you can make anything go into syslog. So he did all kinds of cool things to
custom make syslog. So if you have a device and you want to make it go into syslog, you can
do that. Someone plugged in the Ethernet cable, he could write that to syslog. If you unplug
it, it writes it to syslog. If you, if a port is active and then he tied in a door switch to the
cabinet. So he tied that into syslog. So all these really cool things that you can do with this
device. It talks about all these different reports on the devices that's connected to it. And
you can access it, Microsoft access. So you can run some database type things. It does a lot of
things. There's no Linux shell. So that's, that's sad. But there's pros and cons to that. It
means that an attacker can't write memory to it like they could with a D20 or any VxWorks device.
We can also capture packets just like what the D20 could. And you can, it ties right into
Wireshark. So it's really great. And this is an example message on the right. And it can do, they
wrote serial Wireshark detectors for cell fast messaging, IntelliGear 8979. Those are two
serial protocols that they wrote Wireshark parsers for. Which is really great. Because we can
tunnel those over TCPIP now to make things work. As we upgrade our systems to newer things. So
we're getting close to the end of the talk. For further reading about digital forensics and VxWorks
and other embedded systems, you can go to the link in the show notes. You can download the
files. They can get to the end of the talk. So you can see more. Had a blog post about VxWorks
seven years ago. They wrote a Medusploit module for VxWorks remote memory dump. It does the
same thing that our tool does. But it's you can look it up. WBRPC memory dump is the module
name. David O'Dell wrote a blog post about QNX. And that's a really good blog post to check out.
out. And then ICS has recommended practices for ICS forensics. If you need to take a picture of
that. Go to the next one. Travis Goodspeed, hopefully you guys know Travis. He wrote some
very good information about MSP 430. So he did some digital forensics on that software family
and the hardware family there with that specific chip. And Ralph Langner did a really great
paper, especially in forensics data about Stuxnet. So if you read his paper to kill the
centrifuge, it's really great. And then if you read Sands' Duck 5 about the Ukrainian power grid
attack, included parts of writing firmware of the embedded serial devices. So just more things
that's embedded. That's what we've got to get to. Does anybody have any questions?
Yeah. So I've presented this at Black Hat. So you can download the slides from Black Hat. If you
want these with the ICS Village logo, that's fine. But, yeah, the video from Black Hat is up as
well. You can see Josh's explanation is a lot better about the IP firewalls and the IP
firewall start and how he injected that and doing the Python memory differential analysis,
which is really great. Hopefully this will inspire people to write more tools for embedded
systems. And we need a community around that. So please try to install some of these things. And
if you have that ability, and then maybe we can work on making more modules for this type of
tool. Any other questions?
Okay.
Thank you so much for coming. And come see the ICS Village. We appreciate it.