All right, so what is the problem? I think one of the things that I really like so much
about where DEF CON has grown up with the villages is that when you're looking at industrial
control systems, one of the big challenges is not only the technical intimidation of
where do I even start to understand how to access this code, look at these protocols
and understand this equipment, but there's also a barrier to entry of this equipment
is hard to find. It's very expensive. And so being able to get your hands on it and
to touch it and to be able to start to play with it is a barrier to being able to start
that research and have an open dialogue between the security research community and of course
the vendors and the corporations that use this equipment. And the villages of course
have all of these things there set up for you with experts happy to walk you through,
let you touch it, ask those questions. So I encourage any of you who haven't been to
the ICS village or any of the other villages, please do so because this is that kind of
opportunity where the kinds of things that we're going to talk about, you get to go actually
physically touch that and it's not just abstract. The second year, it was a bit more, there's
a lot of us on this wire. We continue to see further digging in. The opportunity is being
taken, but it needs to be encouraged. We're still way beyond the eight ball. And Atlas
aside for the most part, this is not sexy. So it's not the challenges. Strike the pose
again, please. That says industrial control systems. Thanks for dropping the beat, man.
It's not sexy. And a lot of the challenges and a lot of the solutions and a lot of things
that need to be done are not sexy. I mean, when you start looking at a lot of the conversation
that's happening in the research community at large, sexy, zero day being dropped, SWIFT
being compromised, sexy, man speed infrastructure. We have another mic? Don't drop it. Far enough.
You don't know how far I'm going to go. All right. So sexy, the SWIFT, the various things,
banking and maybe, well, destruction of dams is pretty cool, but it doesn't dive into the
problem deeply enough. But a number of years ago we did have some sexy parts of the grid
and it was called AMI, advanced metering infrastructure. The concepts were relatively new, they were
exciting, they had ARA grant money, they had backing, they had momentum, and yet industrial
control system sat on the back burner and every IOU I spoke with, IOU, industrial utility,
everyone that I spoke with just kind of shut the conversation. Oh, no, that's something very
different. We don't have money for that at the moment, but we're working on it. The other part
of the not sexy, of course, is we're people. We do what we're incentivized to do. Part of that,
of course, is we're pursuing our individual passions, but starting to take that challenge of
cost and limited ability to start to do this, and in the last two years we've started to see
the rise of the bug bounty concept and crowd-sourced vulnerability disclosure programs.
Where are those for this kind of community? Where's that kind of incentive? You want to touch on
longevity? Yeah, I thought you were hitting longevity. I was going to, but I forgot it, so I'm
passing it back to you. There, I touched on it. Longevity. We have industrial control systems.
Hey, big surprise. They're built to withstand many decades. Okay, it's not surprising, because
these systems control that which makes our life what it is. An attack, not to be too
accelerant, but a certain type of attack could potentially change the way we live. So, do we get
into this? Crap, I'm forgetting what's in the next slide. So, forgive me if I say things twice.
So, what we have is a problem of systems that were created, designed 20 years ago, put in
place 15 years ago, but they've got a 20 to 30 year lifespan that has been amortized by
utilities and people deploying, anybody who's deploying ICS. The manufacturers have been
strong armed into the lowest possible price or it will go somewhere else. They didn't build in
money to maintain these systems for long enough. They were never intended to be connected to
the internet, but as we all know, every IP network wants to and will eventually be connected
to the internet. So, the where's Waldo of industrial control systems is if you're looking at a
box and you want to know whether it's an industrial control system or just a traditional
computer, if it's at least 20 years old, it's an industrial control system. I mean, we're
talking top of the line 30 megahertz. That's an impressive piece of equipment. And actually,
funny story here, just since we've been talking about insecurity, we'd actually had props as a
part of this presentation where keeping in line with OSHA, which prescribes the standards for
folks that are working with dangerous things like industrial control systems and SCADA, etc.,
that we would have hard hats on and safety vests. They were stolen. So, that just kind of
shows you that you got to have physical security at all times as well. So, God grant me serenity.
What can we change? What can you change? This is an actual statistic that we made up that 93
percent of all code is shit. And that is the technical jargon for it. And we made that up, but
we're not wrong. And that code is code built on top of code built on top of code. I think one
of the more interesting conversations I've ever had with a manufacturer was through some
level of acquisition, they had code through some little subsystem of one of their boards.
And they're like, nobody knows what that code is. Nobody has any idea. Whoever wrote that is
long gone through some acquisition of some time ago. We have no idea. We just kind of let it
go and we sort of know how to poke it and occasionally it pokes back, but that's it. So, we
have to accept that crappy code is going to make it into the systems that protect our lives
and provide for the way that we like to live. Do we give up on code? Do we say, oh, it's crap?
No. Just recognize there is a natural tendency towards crap, chaos. Entropy. Entropy.
Well, couldn't we just keep them off the internet? Sure. I'd like to see you do that. So,
people. People. Oh, people. Management and people on the ground don't know how to talk to
each other. You can't do that. It's stupid. If you had any brains in your head, you would
not have even suggested that. You can't do so many less things. Anyway. That was loud. You
almost... Bet you don't want to hear me anymore. No, you almost threw off my pacemaker. That
would be the next talk. That's another talk. We do medical devices. The challenge here is the
classic nerd versus operations and management. Nerds speak nerd. They're kind of abusive
actually. Unchecked. We kind of beat each other up. Why? Because we were beaten up as kids.
We're nerds. Now that I have some power, man. Meanwhile, management is there ensuring the
function of the operation, right? They're caring about revenue. They're caring about the
business and they're trying to do that. You make that sound really sexy. As an engineer, I'm
kind of offended. I don't know what the interpretive dance is for dollar sign. I do,
actually. We're in Vegas. Management is focused on operations and the nerd guy runs down
and is like, hey, boss, we have this huge problem that we just found out. There's this issue
with code and immediately starts running down into the technical bits. Management glazes
it over, waits for him to leave the room. You know what happened? I don't know what happened.
Stupid or not, the management has a job and they have things that are very important to them.
Maybe their paycheck. Maybe they actually care about security or at least safety and the
mission of their organization or if you have a great environment, they may actually care
about you. However, what's the secret to communication? Understanding the other side. I like to
say that there are no crazy people on this planet. You just don't understand the system of
reference or their decision making. There are no crazy people. What's important to you? What's
in it for me? Oh, you had to go there. A little bit later we're going to go through a little
bit more granularity about the different kinds of stakeholders in this and the pieces of the
supply chain. But fundamentally you have this challenge of this, the two sides speaking Greek.
Nerd speaking nerd, management speaking operations and in the middle of this gap. And this is
often times where security research companies like us come in between because we're there to
bridge that gap between. We have the technical understanding of working with the equipment at
a very granular level and understanding the implications there. And then importantly we can
tie it back to the business context and impact because that's the key. That's how you got to
bridge that. When you're thinking about, it's not code, it's the impact of the code. Compromise
is not compromise is not compromise. Compromise or a hack or an exploit only has value within
how does that vulnerability affect the business, affect the operation. That impact helps you
understand prioritization. That impact helps you understand the correct remedial solution
because it's not just a question of, there are lots of different ways to fix problems. It's
fixing it the right way for the context of how it's being used. And you don't want to take
responsibility as an engineer for something that is going to get your boss fired. It's his
responsibility. Your responsibility is to provide information, to understand and to bring
along. A very dear friend of mine last night at our party said, Atlas, we've had a couple of
drinks so we're all kind of reclining by the hot tub. He says, Atlas, what's important to you?
How do we fix this? What are you thinking? And I didn't have this canned but the three things
that came out of my mouth were education, motivation and partnership. I didn't realize that
that applied so well to what we're going to talk about today. Go ahead.
But wait, there's more. So as we discussed, we talked about the agent incentivization
problem where is the security community really incentivized to become engaged in this. And
then of course looking at where's the clear mandate coming from? Who is in fact pushing
that? Because this is a top-down problem. You're not going to have the average consumer, which
everyone in this room is. Last time I checked, we all use electricity and water. Everyone in
here is a consumer and a stakeholder. And we as a large group are not really able to
effectively advocate for something that is a very complex problem and set of problems. This is
something that needs to be top-down driven. Accountability?
Yeah, no, I don't. I'm not a consumer. I live in a yurt. Solar power, dude.
Ryan, how's that yurt coming?
So there's no clear mandate. Who is responsible for fixing these problems? The way that we
live affects all of us, as he just said. But who has to fix this? Is this the vendors, the
people who made PLCs, RTUs? Is it the utilities? Is it their problem? I've seen some pretty
amazing engineering going on in utilities. Is it their problem? How about regulatory
agencies? Show of hands. Who wants the regulatory agencies to fix this problem for you?
Oh, I love this. This is great. Go, Joe. He's saying if it's regulatory, it will be check the
box. It's the only way that they know how to do it. And in fact, they're incentivized with
both carrot and stick to make it check the box. Very good answer. I mean, there are a couple
different agencies. We'll talk about that in a minute. Go ahead. So the first thing, let's
address the problems that we can right now. There are things we can do. We actually want to
have a 5, 10, 15 year plan. Hopefully not 15. Please, Lord, let us fix this faster. But there
are things we can do right now. Show of hands. Anyone want to give an offer? Any ideas?
I can build my house on stilts so that when the dam breaks, I'm good to go. I like the idea. I
already have my house built. I'm not sure that I want to jack it all up. Anyone else? Tim, I
know you. No, I'm just kidding. So where do we start? Well, we begin by assessing what the
problem is in our own backyard. Every one of the people on the last page, even the
regulators, we want to become aware of the problems that we can fix right now. For example,
if we are still using Modbus, probably a good idea to get that off the wire. I don't care if
you're doing serial. Okay, there are worse things to use. But if you're using Modbus right
now over Ethernet, please stop. There are other options. And I'm sure that there are business
decisions that made Modbus the right thing and it takes money and time and effort. But it's
something that we can fix. If the protocols that you use offer security, lab it up, make sure we
don't roll out some big change and take down the grid or the water supply or back up the poop.
I mean, we're talking about more than just the electric grid, right? But lab it up, test it,
migrate to something that at least offers security, even if it's not perfect. DMP3 has tons
of bugs. But at least it, as he stepped on the chair, I was thinking he was going to call out
the fact that you may be still kind of surrounded by shit, but if you can get a little higher,
you can be in it less. Pick something better. If you have systems that are, have replacement
upgrades and you can put into place a better system, maybe a system that actually has a
vendor that's still around or a vendor that you trust to partner with you, this is a good
first step. There are many things that you or we, whoever is doing this assessment, can sit down
and identify, within the next year, we can do this. Let's talk to the right people, work it
out, get something as part of a bigger process. To emphasize, this is the all is not lost slide.
You can start somewhere and have that vision of where things can get better, circle the wagons
where you can establish control and then iterate from there. Best practices. I mean,
so, a few other things that we can throw into the mix that you can do within the next year to
two. Authentication is a problem we all deal with. There are many technologies available
because everybody is struggling with it. Two-factor authentication for the right systems
is a good first start. If you have to use passwords and not tokens or keys or fobs, please.
The next step, guys, I know that we didn't mean to plug the grid into the internet, but we've
done it. We didn't design our systems and our networks to withstand the type of onslaught that
a nation state is willing to offer. In fact, the state of many organizations, many people here
without state funding could do it. So, I'd like to lay out the idea of a multi-tiered castle. At
the very center, with the most restrictive protections in place, you have crown jewels, you
have head ends, you have things that actually change significant parts of life. Outside of
that, you have slightly less trusted, less protected systems, but still very highly
protected. And so on and so forth. I recommend three to five layers at least. And these layers
play into a potential for protection. So if you have a system that there's absolutely no way
that you can have assurance that it is protected, then it falls in the outer ring. For example,
pole top devices. Anyone here that's been at DEF CON, you're not, you cannot possibly argue with
me that a system being on top of a pole is not secure. Unless it's in a compound with guns.
Patching policy. This is just regular SEC 101. The systems that are being used to pivot into
attack your critical infrastructure, generally people's workstations, check an email, surf
on the web, these systems have to be maintained and protected. Now this is not going to keep
off everybody. I'm going to try not to throw out nation's names because I know everybody's
represented here. But patching, keep at least the end days done. And then there are
configurations in our PLCs and RTUs that again can be used to turn on security features.
Encryption and strong authentication and other things to lock down access that you
shouldn't have available on your PLC. Now there are many who will say encryption takes time
and I need point blah blah blah milliseconds response time. If your critical systems can't
handle that type of waylay and processing power, those might fit on your list to replace.
Monitor all the things, all the comms. So this is while end points might not be the place
that you can do that monitoring, certainly there can be transmission paths and one of the
challenges with this of course is not all of the communications are going to be wired. You have
to keep in fact that a lot of things are going to have RF communications and you need to
start thinking of the physical security and the physical presence of those things. Please
don't allow your cell phones to connect to your network and talk to your head end. There's
one good rule of thumb. But if you do have cell phones connected to a network, it better be
on something you can monitor really well. Hunting, who here has heard of hunting? Okay,
about one percent of you. Hunting is the act of scouring your network. No penetration
testing has to go into this if you're worried about breaking things. Simply monitoring
traffic patterns. Traffic can be cataloged, identified and put into a database and
analyzed. For example, there's no reason why workstation in the office needs to talk
directly to head end. If you have this type of communication or other strange communication,
A, that's something you want to understand better. B, it may be the thing that tells who's
on your network. Fuck Modbus. We've beaten Modbus. Modbus is unauthenticated. And perfectly
sane to kick the shit out of here. There are others like it, but this is Modbus. Please
choose something that has some sort of protectiveness. If you have critical systems
that must use an insecure protocol, maybe because you're worried about latency, you should
have armed guards around everywhere that that network touches. This is the self-promotion
slide, so I'm just throwing that out there because this is some of the stuff that we do. But
it's more, even if you don't do it with us, please understand, we have yet to get to the
point where I need to throw binary exploitation against most control systems the way we do
your laptop, the way we do your servers, the way we do against firewalls and routers. We
don't have to do that. It's not hard enough to warrant that kind of work. But there will come
a time, hopefully very rapidly, where that's not the case. As we fix the fixable, digging
into hardware, software and firmware, that was sexy, identifying weaknesses in code, weaknesses
in design of hardware, weaknesses in what your vendors are shipping out, like hard-coded
passwords, these still exist. Assessments of all three things will provide you, when we finally
get our poop in a group, something that is defendable. And if we don't start now, it will
just be that much longer before we get to something that's defendable. But I've got limited
budget. Well then prioritize. If you have systems that are most widely deployed, those should
make pretty high on the list. If you have significant systems that control large,
potentially irreplaceable parts of your systems, that may make a very good top place. You can
get to the older stuff later. You guys know how to think about boundaries, right? We do the
most important first, we then back up. We don't stop until we're done. But we get the best
that we can for now. So while we list three device categories here, of loose categories,
there are also three kinds of assessments. White box, gray box, and black box. And I list those
because those also require careful consideration for what you're trying to accomplish. A black
box assessment is saying, hey, you're just like somebody else out there anywhere and you are
required to do all of the open systems intelligence and gathering to try to figure out what kind
of information can you gain. And then from there start to work that to then be able to
determine how you're going to conduct your research for exploitation. Gray box is going to
be a mix of you have access to all of the documentation, so there's kind of that in
between. And some of the factors there are going to be trying to understand how much is
obfuscation your friend, and obscurity is your friend, versus how much of it do you really
want to just get straight to the technical research and understanding, which is what white
box is going to do. And there's also one other thing to point out. This is not check box
things. Check boxes get you the first layer of an assessment like this, because you can
actually put into place a list of things that you want to check to gain knowledge about the
next steps. Once you're past that, you need people who are creatively evil, and you need them
to have the time to sit in their creativity and think, oh, hey, Tim, do you think that we could
do blah, blah, blah, blah, blah? Yeah? Let's go do it. Oh, this is my slide. Sorry. So as we
said before, we need devices that we can deploy that have security baked in from vendors
who think about security. We also need, we need to reassess what we're doing for comms,
because there are radios that have encryption enabled. Almost by default. Oh, but it's
really easy to enable them. You simply have to choose one of 256 standard encryption keys.
That doesn't sound like a lot. Hey, it's AES fucking 128. With one of 256 different keys. And
please, understand, one of the things that, actually one of my favorite parts of my career
has been breaking into banks, breaking into government buildings, and doing things with
cyber that aren't typical pen testing things. You may think of them as red teaming. If so,
that's great. But some of our assessments have really gone, oh, well, I can hop the fence.
Okay, here's the device, you're going to get there. You don't have to hop the fence, I know
that you can do it. And we've gotten our hands on a lot of attacks after. In fact, the reason
that I call out MDS is because they're deplorable in many of their at least older stuff
that's still widely deployed. For being able to use one radio that you purchase off eBay to
attack all the others, and even brute force within 18 hours, guaranteed to be on any network.
So, the people. We've talked about the people, we've talked about the people through multiple
things. Where are these folks, right? So, first of all, I think that this is, again, what I
love about DEF CON is this is the meeting of those passionate folks who really do care about
this problem for various reasons. Either you're motivated by how we can all help humanity
together, or you have a particular interest of the matrix metaphor of going down the rabbit
hole and seeing how far it can go technically. Your vendors should be having security-minded
people in their hiring process. They should be actively pursuing not just bullshit bingo
security, but true security. And utilities. They need to be able to call bullshit. They need to
be able to have the knowledge and skill to say, that's crap. Fix that. Or better yet, get out
of here, these other vendors, they have people who know what they're doing. So, go ahead, I'll
let you take the top. How to fix for the who's who. We have two regulatory agencies, FERC and
NERC, and there is a challenge there where there is a mix between the overlap of
responsibilities and a difference between suggestion and compliance. And while we talked
about compliance being a challenge from a checklist perspective, we also talked about the
fact that this is a top-down mandate that needs to push and encourage this action. And that is
the purpose of government and legislation. And that's where these organizations need to come
into more of a common agreement with the vendors in the industry to help push that. Now, they
need to be light-handed right now. We've already got some very heavy-handed things in place
that do not, that actually can detract from true security. They have to be careful how they
engage, but FERC and NERC do play a very important role. Our public services commission, our
public utility commissions, these are the state-run organizations that kind of keep our
utilities in check. They are the protectors of the consumers. Now, I have seen some
significant investment from several PSCs into researching cyber security. They want to
understand. They want to know how they can fix the problem. And they have a big role. They
control how much a utility can make for a year. They also allow a utility to do a rate hike,
which allows them to charge more per kilowatt hour or per gallon of water or whatever. In
the interest of maintaining a grid, a water system, a water treatment system that services
our needs. But our vendors, they're saying, where the hell is the money coming from? The
utilities don't want security. They're not willing to pay for it. I can't make it unless
they're willing to pay for it. So utilities, phony up. Utilities are saying we don't have
any vendors that provide things that we can put into place that are defensible. They have
crap. And this is what I have to deal with. Encourage. Educate. Make the decision. And in
fact, you have a lot of sway. The unfortunate side is we have decades of utilities beating up
vendors for the lowest price. We have lost the partnership that is provided for our
country and our world. We got to get it back. And so that concludes the amount of time we
have for this. This was a lot to condense into a short talk. So I'm Bryson. I'm Atlas. And
we're Grimm. Thank you.