Thank you very much for the invite.
I've never been to DEF CON or Black Hat before,
so this is kind of an interesting phenomena for me.
Thanks.
I'm an engineer that got into security
rather than the other way around.
So what you're going to hear
is the engineering view of the world
and what's going on and what's needed and what's missing.
I've deliberately put this slide up.
Everything we're talking about is this guy right here.
If you saw that in the village, that's a sensor.
In this case, it's a 4 to 20 milliamp sensor.
It's probably measuring flow.
They put it in a line.
It could be measuring pressure, flow, temperature, picket.
What we're talking about are fingers.
This is what measures things.
And it's going to the brain, which would be the controllers.
If you cut your fingers off, the brain can't do anything.
And that's what all of this is about.
I'm going to take it now back to the beginning.
Okay.
So, like I say, I wanted to thank Larry for inviting me
because this is a really important area
and one of the biggest things you're going to get out of this.
Is we've got to, got to, got to have the engineers
and the IT security people working together.
And that is not happening in most cases.
Because what we're talking about here is pure engineering.
It is not IT.
You've probably heard all of this.
The big thing is that last bullet and what's in bold.
Control systems monitor and control things.
Physical processes.
It doesn't matter what it is.
It doesn't matter if it's a power plant, a pipe, a human body.
If you've got physical processes, you know,
and you've got to monitor, for example,
temperature or pressure or flow or radiation,
and then you want to control it, that's a control system.
Industrial was a poor term.
We didn't know what else to call it.
Because all of the building controls here are essentially ICS.
Because you're monitoring temperature.
You're monitoring, you know, steam flow, water flow, humidity.
All of the things you think about.
There's nothing different.
So everything we're talking about here crosses every border.
But the big thing is, most of you, if you're coming from the IT world,
you have the CIA paradigm.
You know, confidentiality, integrity, availability.
It isn't just that it's backwards for us.
It's actually missing the most important letter.
And that's the letter S.
And the letter S is for safety.
Because in our world, we can kill people.
That's the most important.
Everything we do is reliability and safety.
Confidentiality, really one of the only places it's important,
is if you have a meter, like a smart meter in a house or something like that.
For almost everything else, confidentiality is very, very,
very trivial.
When you put it in a database and store it somewhere,
then it becomes IT and confidentiality is a big thing.
We're talking about data in motion.
So one of the big things, and this is a problem in Europe.
Europe is totally confidentiality based or privacy based.
In the control system world, that's not us.
This is just a Venn diagram I've had for years.
Part of our problem is, you've got two worlds coming together.
The IT world and the control system world.
And you'll notice the number of arrows.
Most of the people that are working this are coming from IT.
Very few are coming from the domain.
And part of it is, it's unfortunate,
but control system people generally do not like security.
It's an impediment to what they do.
So you have...
You don't have near enough people who actually understand the domain
even wanting to be in this.
And we need to figure out what to do.
There is a really, really, really small dot
where you have actual ICS security experts.
And by the way, one of the things I always tell everybody,
if somebody tells you they're a SCADA security expert,
run the other way.
It generally means they learned how to spell SCADA within a year or so.
Because there's one other thing.
How many people would go to Macworld and only want to talk about PCs?
Sounds kind of silly, doesn't it?
When you talk about SCADA, SCADA is not every control system.
SCADA is only one type of control system.
And it isn't used in a lot of places.
So when you use the term SCADA,
and you don't use it right,
you will lose all credibility really fast.
We are here in, you could either say the cyber-physical world,
but really we're in the physical cyber world.
Everything we're talking about is physical.
Cyber happens to play into it.
And you need to address it from that perspective.
People hear about advanced persistent threats.
When you hear about advanced persistent threats,
when you look at things like Stuxnet,
those were hard-coded default passwords
that were part of the design of that controller.
That wasn't an APT.
In a sense, that was a persistent design vulnerability.
You can't change it.
And by the way, this isn't just Siemens.
We have a lot of PLC manufacturers
that have, if you will,
used the password,
as part of their actual software computing platform.
So if you change the hard-coded default password,
that PLC falls on the floor dead.
This is not like what you would normally think.
Well, geez, how stupid can you be?
This was never, remember,
we didn't start out life worrying about security.
We worry about reliability and safety.
And a lot of this came because we were trying to capture,
meantime, between failure data,
the only way to do that
was to have a backdoor in
to be able to continue to capture
and get that data.
Totally different reason.
Okay?
In IT, you normally want to stop data.
You want to do a denial of service.
When you're in the control system world,
you don't, and you want to do something bad.
It's not a denial of service.
You want to call the police.
You want to cause boom in the night.
So what you want to do
is you want to take control of the process.
Or you want to take control of the operator displays
and have the operator do the wrong thing.
It is not a denial of service.
What you're looking at, level 012,
this is all from the Purdue reference model.
This has nothing to do with the seven-layer OSI stack.
Okay?
So what you're talking about,
this is the process.
So this is like a boiler.
Or it's like the electrical wires.
Okay?
Or this is like, you know, a,
you know, the refinery process.
Here, level 1,
is basically,
the real-time measurement.
This is normally,
now again, years ago,
you could have subdivided this easily.
What happened with Stuxnet,
that was level 2.
Now, many years ago,
there was a distinct difference.
Now, with networking and with ethernet, et cetera,
it's blurring.
Level 2, which used to be,
totally isolated from the internet,
is no longer isolated from the internet.
So it starts blurring when you start talking about
levels 2, 3, 4 on up.
And it's even starting to blur
even when you get down to here.
Because you start talking about sensors on chips
and things like that.
Okay?
But what we're really worried about
is when you can compromise a system
by physics,
you can't stop that.
One of the things Aurora does is use physics
to cause essentially
very, very, very large torques and current spikes
in milliseconds.
There's nothing you're gonna do about that.
That's how, by the way,
you bring the grid down for about 9 to 18 months.
That's what we're talking about.
This is not,
this is not a day or two or anything else.
This is how you break very large equipment
like generators, motors, transformers.
These are really long, you know,
term equipment that,
and you don't make very many of these,
and these are almost always very specific
to the specific process.
And there aren't very many vendors
who even make this anymore,
and a lot of them are offshore.
So when you start breaking things,
there's a lot of questions that come in.
Now, again, our communications are not native IP.
You're talking about, say, Modbus over IP,
or DNP over IP.
Pick something, but it's not native IP.
What we do is we use IP as a router.
We use IP as a wrapper, okay?
And there's work going on,
but these protocols are not, like I say, native IP.
And the real protocols we're worried about here
aren't Modbus or DNP.
They're things like HART or wireless HART
or PROFIBUS, Fieldbus.
These are your level zero, level one sensor protocols.
There's not only no security in them,
they were designed almost to be insecure.
You'll find a lot of these have things like XML.
So one of the other questions is, so what?
Well, you can get to them.
The I.O., the input, output cards, you'll see them.
I mean, when you go to the board,
you'll see things like Phoenix Contact or whatever.
It allows the instruments to communicate,
to communicate bi-directionally.
You'd think about things like data diodes.
Well, that doesn't do anything
when you're talking about sensors, especially smart sensors.
Smart sensors have to communicate bi-directionally.
Smart meaning it's digital,
and it's done all of the conversions at the sensor layer.
And you'll notice, again,
engineers can no longer simply measure
the output analog signal.
They need to communicate with the transmitter
and read the digital signal.
Well, there's no air gap.
It's impossible to have an air gap.
You know, everybody thinks, hey, I did this,
therefore I'm fine.
You can't do that.
And the whole point about the presentation,
there is almost no cyber security in them.
There's no authentication.
And from a cyber perspective, there's no forensics.
One of the things that's really important
to get across, as an engineer,
I don't care if the process fails
because it was a system problem,
or because it was hacked.
The process still failed.
We need to understand when the process is starting to fail.
And what you'll find in our world
is often the only difference
between malicious versus unintentional
is the motivation
of somebody.
Why did they do what they did?
It's different.
And we need to think different.
We need to think about how do you keep the boiler
within a prescribed temperature and pressure range,
regardless of why.
And like I say, the protocols are vulnerable.
I put this up.
This was an ICS cert notification.
Last revised.
July 25th, 2017.
We're talking Thursday.
That's PROFINET.
You know, when people think,
hey, you know, this isn't vulnerable.
Yes, it is.
We've had any number of demonstrations
of hacking, heart, wireless heart.
A friend of mine just got his doctorate
from the Air Force Institute of Technology
at Wright-Patterson Air Force Base.
He's now at Oak Ridge.
His doctoral thesis was on, you know,
hacking heart-enabled process sensors.
What they're gonna be doing next year,
even though he's already proven it,
is the next one going through the PhD program
is gonna go directly after wireless heart,
even though we already know that's vulnerable.
His thesis happened to have been on wired heart.
So just so people understand,
not only are we talking about a problem,
it's a very exploitable problem.
So, just want you to understand that.
Control system cyber incidents are real.
It was my database.
Been well over 950 to date, okay.
These are not vulnerabilities.
These are real incidents.
And you could see the impacts ranging all over the place.
And you can see, you know,
this is complete destruction of a turbine.
That happens to be Aurora.
That's DC Metro.
metro train crash, San Bruno, Bellingham, that was Marucci.
That was an offshore oil platform
that they hacked the buoyancy
and tilted an offshore oil rig.
This is really different than things you're talking about.
Gee, somebody stole my data.
This is the dam in Russia.
I don't even want to try to pronounce it.
That lifted a 1,000-ton turbine off its pedestal.
There were 75 people killed there.
In fact, the Russians originally thought we did that.
That happens to be a brand-new Navy ship
that, because of control system cyber problems,
on its maiden voyage,
had to be towed in the last 40 miles into Norfolk.
So when you start looking,
you can only...
You can also see there's a lot of different industries here.
This is just kind of my summary.
There have been over 1,000 deaths to date
and well over $50 billion in direct damage.
In fact, I'll give you an example of what we're talking about,
and it also flows into what this presentation is about.
I think most people know what Stuxnet is.
It was changing the logic in a controller
to change the process
and then changing the logic back so nobody would know.
You know what I just described?
Volkswagen.
And by the way, Fiat, Harley-Davidson, all of them,
they all did exactly that
because they couldn't meet the environmental controls.
And when I had first...
was looking at all of this about, quote-unquote,
ICS sensors not being secure or authenticated,
I called one of the professors at UC San Diego
who was doing the analysis on Volkswagen and Fiat.
And I asked him,
are the sensors in cars,
do they have authentication or security?
You're right, no.
No.
There's nothing specific
about this.
But think about it.
If the sensor data can't be trusted,
what does it say about every single thing
you're doing with network monitoring?
If you're a doctor and you cannot trust
your temperature or blood pressure readings,
how can you make a diagnosis?
That's where we are here.
And you go think about Microsoft with the Azure Cloud
or any of the others.
What are they...
What's their...
What's their assumption on everything?
The sensor data is authenticated and trusted.
That's the entire...
That is the assumption for the entire Azure Cloud.
So think about what I just said
with respect to the ICS world and cloud computing.
It's really that simple.
Here's the techie that explains most everything.
What you have over here...
What you have over here is what most people realize.
Here's your...
You know, your normal network...
Your operator displays.
This is all Windows.
Everything here is generally IT.
Prior to Stuxnet,
don't even look there.
Since Stuxnet, people are really worried about the PLC.
Okay?
Only...
You know, this is a valve.
Here's a motor.
That's a sensor.
This thing here,
that's what makes that 1970s or 80s
valve a 2015 valve.
This is the smart electronics.
That's the smart electronics for the motor.
Same thing with the sensor.
And that's all remotely accessible.
Okay?
So what you've got here,
there's no security.
Now,
here's our process.
That's the most important thing in the world to us.
This stuff starts...
as serial.
It ends up going through the serial to Ethernet converter.
And by the way, what did the Russians hack
both in the... in the U.S. and in the Ukraine?
It was the serial to Ethernet converters.
Only they used it to go up into the network,
not down...
to the sensors.
Here is where all of your network monitoring starts.
They assume whatever happens here is correct.
That's the flaw in every single network anomaly detection program today.
They assume when you start here that that sensor value is correct.
And they have no way of knowing that it isn't.
Now, I showed you that picture there.
Now, this is the Holy Grail.
The Holy Grail is to know...
In fact, I'll give it to you this way.
About four or five weeks ago,
Honda shut down an assembly plant in Japan.
It cost them a thousand cars.
Why?
They found WannaCry on the networks.
There wasn't one single alarm on the factory floor.
They did a self-denial of service
without having a clue whether it affected them or not.
This is where we are today.
You cannot, from here,
make any real...
view of what's going on in the real process.
Because all you've got is
the data that's going from that serial to Ethernet converter.
The Holy Grail is to be able to look at the process
and cross-correlate it to the network anomaly.
To be able to say,
hey, I found malware.
Is my process changing?
Assuming it shouldn't,
and you're watching the sensors
and you know precisely what's happening.
Why do anything?
This is probably the most important thing in the world
when you talk about the industrial world
or the commercial world.
When do I care?
Today, they cannot make that statement.
This is just the sensor.
And again, here's the point.
What you're doing
is you're coming back here.
You're coming back in.
And if you change things like zero and span,
you can prevent a sensor from ever reaching a set point
or to reach a set point long before it should.
And you will never know that
from network monitoring.
This happens to be a smart sensor.
The same thing.
You get into here.
You make your changes.
Nobody will ever know.
Now,
what happens,
this is what I was doing 20 some odd years ago
when I was still at the Electric Power Research Institute
before I ever heard of security.
We were worried about doing root cause analysis on failures.
But what we were doing was a snapshot in time
doing postmortem.
Well, there's technology now
that allows you to do this in real time.
This is where the Holy Grail is coming.
But what's happening is
this is what's filtered out
as you go through the serial Ethernet converter.
That's why nobody here on the network side
is ever going to be able to really understand what's happening.
What you're getting essentially
is the absolute value of the signal.
Not what's actually happening in the process.
This is what is completely and totally different.
Here are real things.
And two weeks ago I was at a power plant.
It had tripped.
They had tripped it.
In other words, shut it down.
Very large plant.
A pressure sensor reached a set point.
Why?
A sensing line clogged.
It is impossible to see that from network monitoring.
But if you're looking at the process,
it becomes very obvious
something is going on with that sensor.
This is something I worked on a long time ago.
This is a nuclear plant
where a safety relief valve didn't lift.
This is safety.
This is as much safety as you can get.
Why didn't it lift?
Because the sensor could never reach its set point.
What happened?
That's Tom Sock, the dam in Missouri.
Basically failed about five or six years ago.
What happened?
It had a couple of level sensors
attached to the wall of the dam.
The tie-downs broke,
so the sensors came out from the wall.
When they came out from the wall,
the level readings changed.
Because you measure level by elevation head.
You measure level by height.
Well, the height changed.
So the SCADA system took the data,
took it as real,
had no reason not to.
And when it said the level was low,
it turned the pumps on until the reservoir overfilled
and the earthen dam collapsed.
There's no hypothetical here.
This is the stuff that every bit of this
has actually happened.
You know, this was one of the biggest
fires in the UK,
which was a refinery fire
because the sensors didn't work
to tell them that the tank was being overfilled.
These are huge catastrophic problems.
The controller can't do anything,
or it's different.
The controller will do exactly
what the sensor is telling them
that the process seems to be.
This is what is reality,
and this is what network monitoring
can't do today.
Because it doesn't have a view into the process.
Part of what's going on is this.
Sensors, actuators, and drives to this day
are still being made with no security
and no authentication.
Why?
They're engineering systems.
What our vendors do is they have the security done
over here where you're worried about the network.
Over here where they do the engineering
for the engineering systems,
there are no security requirements.
That's to this day.
This is a huge problem.
In IT, it may be all about packets.
For ICS, it is not.
IoT.
You know, IoT is a funny animal.
You know, what is IoT?
If it's Fitbits and refrigerators, who cares?
But in a sense, we're part of IoT.
We were IoT 20 years ago.
20 years before there was IoT.
And they're certainly not looking at anything like this.
And then here's, even ICS-CERT,
they put out their 2016, you know, statistics.
There is no mention whatsoever
of anything at level zero and one.
The only thing they talk about are network issues.
And it's the same thing from 2015 and 14,
and you can keep on going.
There's nothing that's ever changed.
And what needs to be done,
you've got to get the engineers involved.
And reliability and safety have to be number one.
Confidentiality is fine, maybe.
But if you don't take reliability and safety there,
why bother?
And the other thing to realize,
everything we're talking about in the sensor world
is all reliability and safety.
It's really not cyber.
But it's the most important input to cyber.
Because how can you monitor anything
if you don't trust the input?
And that's what this is all about.
We have, you know, what do you do about supply chain
when you talk about this layer?
What do you do about risk assessment methodology?
Risk assessment, certainly vulnerability assessment,
assumes you're secure and it's a gap analysis
away from what you think is secure.
When these have zero security,
what do you do about a vulnerability assessment
with zero?
It's effectively infinite.
Like I say, the ultimate holy grail
is to correlate process anomaly detection
with network anomaly detection.
Short of that, we're going to continue to have
the Honda issue.
And this occurred, by the way, 10, 15 years ago
with Code Red and Slammer.
We had utilities shut down.
Not utilities.
Paper and pulp plants, you name it.
They shut them down
because somebody said there's malware out there.
Not because there was anything
within the process that changed.
Nobody knew.
You want a funny form of ransomware,
go do this.
But malware, nobody has any idea what it means.
So with that,
any questions?
Martina, I was going to put this back on.
Here are some of those,
because I know you were late.
These are some of the real
sensor-related
incidents that have already occurred.
I just went through my database quickly.
This is not all of them by any stretch.
But like I say, you're talking about
hydro, water, waste water,
fossil,
nuclear.
This was essentially mining,
refineries.
This is all over the world.
This is not
a utility problem.
It's not a North American problem.
It's not any one vendor's problem.
Since we're talking about
issues with sensors,
what's the current state of practices,
best practices or normal practices
as it relates to,
what's the mindset so far
of redundancy in sensors?
Multiple sensors, same type,
multiple different types,
out of hand.
Well, one of the things,
and then I'm going to have to get going
because I've got to go to that thing too,
where Chris is.
Part of the reason I brought up risk assessments,
we've not done very good risk assessments.
One of the things,
how could a single pressure sensor
shut down
a large power plant?
That's,
from an engineer's perspective,
something is wrong here.
Now,
part of what it's saying is
you better understand
what is really truly critical.
The other is,
what happens if you put
redundant sensors
but they're on the same,
if you will,
Ethernet LAN?
That's exactly right.
I'm working with some technology
to try to really understand
what's going on at the sensor layer.
But,
first and foremost,
you've got to have a good risk assessment.
If you don't know what's really critical,
you're dead on arrival.
It's a really good question
and this is such a great answer
to your question.
One pressure sensor
isn't supposed to shut down
a multi-unit power plant.
And it did.
Yeah?
So if we're talking about security
from the sensor perspective,
what do you think is the solution to that?
Because it's like
you're trying to be aware of process issues
and you have to use other sensors.
So how do you use sensors
to protect your sensors?
This is the only way I know.
I'm going back to what I did years ago.
So, I mean,
you need a sensor to detect
that kind of stuff, right?
Well, you need monitoring
to detect what's going on.
I'm working with one or two small companies
that are actually looking at this
and they've actually done
some really neat, interesting work.
And that's what's why I'm saying
we have a shot at the Holy Grail
of being able to correlate
the actual process
to the malware.
And without that, it's not possible.
We're mitigating some of that risk
with things like Moxa firewalls
through those devices
and other devices like the Moxas.
And the evolution,
there's no IDS for this.
They have Pi.
The Pi is to collect the numbers from...
Yeah, don't forget,
where is Pi getting its data?
From there.
That's what I was trying to explain.
This is the official Catch-22.
Everything you're doing starts at the sensor.
If you compromise the sensor,
everything you look at downstream...
Okay?
I just had this discussion with somebody
about, well, hey, I have redundancy
because I have the historian.
Yeah, but where did the historian data come from?
That's what I heard
because I got in yesterday
and I didn't have a chance...
Yes.
Yes.
Absolutely.
And that's why I wanted to put this back up
because you came in late.
But this is where we are.
And by the way,
the serial to Ethernet converters
were how the Russians put malware
into our U.S. grids
in October 2014
and it's still there.
It's how the Russians put the malware
into the Ukraine.
I will say...
Let's go to the Russians.
Let's go.
Adversaries.
I will say...
I will say the attackers.
There you go.
Mark, you're right.
How they get to the U.S.
It's how they get to the U.S., right?
It's how the attackers implement it.
And I apologize...
You're right.
Okay.
Last question
because we have to wrap up.
And I have to go, too.
Just a quick comment.
I grew up in Alaska,
I worked on crude oil fields, and we were doing a retrofit of one of Arco's units,
and one of my men who was out in the facility was climbing up, and he tripped, hit his foot, hit the sensor,
and it sent the entire station into flare.
All the processing went straight up, and the whole ground started shaking like there was an earthquake.
And because we knew what had happened, we were able to tell the control center,
and they were able to shut it down, that they would have had to troubleshoot that thing for hours
to try to figure out what was going on.
And that's, you know, for what it's worth, the city of San Francisco was brought to its knees April 21st
by one relay and one substation.
That should have never, ever, ever, ever happened.
Joe, are you going to be back in the village later?
Yeah, I just want to go to that console.
Do you have any other questions?
Are you going to be back in the ice in this place?