start	end	text
0	5960	All right, so what is the problem? I think one of the things that I really like so much
5960	10360	about where DEF CON has grown up with the villages is that when you're looking at industrial
10360	16220	control systems, one of the big challenges is not only the technical intimidation of
16220	21000	where do I even start to understand how to access this code, look at these protocols
21000	26460	and understand this equipment, but there's also a barrier to entry of this equipment
26460	32240	is hard to find. It's very expensive. And so being able to get your hands on it and
32240	37400	to touch it and to be able to start to play with it is a barrier to being able to start
37400	42180	that research and have an open dialogue between the security research community and of course
42180	46980	the vendors and the corporations that use this equipment. And the villages of course
46980	51560	have all of these things there set up for you with experts happy to walk you through,
51560	56060	let you touch it, ask those questions. So I encourage any of you who haven't been to
56060	60080	the ICS village or any of the other villages, please do so because this is that kind of
60080	63860	opportunity where the kinds of things that we're going to talk about, you get to go actually
63860	81740	physically touch that and it's not just abstract. The second year, it was a bit more, there's
81760	87360	a lot of us on this wire. We continue to see further digging in. The opportunity is being
87360	93020	taken, but it needs to be encouraged. We're still way beyond the eight ball. And Atlas
93020	100560	aside for the most part, this is not sexy. So it's not the challenges. Strike the pose
100560	108960	again, please. That says industrial control systems. Thanks for dropping the beat, man.
112520	116560	It's not sexy. And a lot of the challenges and a lot of the solutions and a lot of things
116560	121580	that need to be done are not sexy. I mean, when you start looking at a lot of the conversation
121580	127220	that's happening in the research community at large, sexy, zero day being dropped, SWIFT
127220	139320	being compromised, sexy, man speed infrastructure. We have another mic? Don't drop it. Far enough.
139660	149540	You don't know how far I'm going to go. All right. So sexy, the SWIFT, the various things,
149540	155380	banking and maybe, well, destruction of dams is pretty cool, but it doesn't dive into the
155380	160820	problem deeply enough. But a number of years ago we did have some sexy parts of the grid
160820	166580	and it was called AMI, advanced metering infrastructure. The concepts were relatively new, they were
166580	172940	exciting, they had ARA grant money, they had backing, they had momentum, and yet industrial
172940	180080	control system sat on the back burner and every IOU I spoke with, IOU, industrial utility,
180080	184520	everyone that I spoke with just kind of shut the conversation. Oh, no, that's something very
184520	190160	different. We don't have money for that at the moment, but we're working on it. The other part
190160	194960	of the not sexy, of course, is we're people. We do what we're incentivized to do. Part of that,
194960	198660	of course, is we're pursuing our individual passions, but starting to take that challenge of
198660	204500	cost and limited ability to start to do this, and in the last two years we've started to see
204500	210340	the rise of the bug bounty concept and crowd-sourced vulnerability disclosure programs.
210820	217420	Where are those for this kind of community? Where's that kind of incentive? You want to touch on
217420	225140	longevity? Yeah, I thought you were hitting longevity. I was going to, but I forgot it, so I'm
225140	236460	passing it back to you. There, I touched on it. Longevity. We have industrial control systems.
236460	244360	Hey, big surprise. They're built to withstand many decades. Okay, it's not surprising, because
244360	252440	these systems control that which makes our life what it is. An attack, not to be too
255340	264560	accelerant, but a certain type of attack could potentially change the way we live. So, do we get
264560	270520	into this? Crap, I'm forgetting what's in the next slide. So, forgive me if I say things twice.
271370	279220	So, what we have is a problem of systems that were created, designed 20 years ago, put in
279220	286140	place 15 years ago, but they've got a 20 to 30 year lifespan that has been amortized by
286140	293860	utilities and people deploying, anybody who's deploying ICS. The manufacturers have been
293860	301320	strong armed into the lowest possible price or it will go somewhere else. They didn't build in
301900	307360	money to maintain these systems for long enough. They were never intended to be connected to
307360	314920	the internet, but as we all know, every IP network wants to and will eventually be connected
314920	320400	to the internet. So, the where's Waldo of industrial control systems is if you're looking at a
320400	323820	box and you want to know whether it's an industrial control system or just a traditional
323820	328420	computer, if it's at least 20 years old, it's an industrial control system. I mean, we're
328420	336280	talking top of the line 30 megahertz. That's an impressive piece of equipment. And actually,
336280	341480	funny story here, just since we've been talking about insecurity, we'd actually had props as a
341480	348180	part of this presentation where keeping in line with OSHA, which prescribes the standards for
348180	352220	folks that are working with dangerous things like industrial control systems and SCADA, etc.,
352220	357900	that we would have hard hats on and safety vests. They were stolen. So, that just kind of
357900	366400	shows you that you got to have physical security at all times as well. So, God grant me serenity.
367180	374620	What can we change? What can you change? This is an actual statistic that we made up that 93
380760	386580	percent of all code is shit. And that is the technical jargon for it. And we made that up, but
386580	393560	we're not wrong. And that code is code built on top of code built on top of code. I think one
393560	397560	of the more interesting conversations I've ever had with a manufacturer was through some
397560	402640	level of acquisition, they had code through some little subsystem of one of their boards.
402640	407960	And they're like, nobody knows what that code is. Nobody has any idea. Whoever wrote that is
407960	412960	long gone through some acquisition of some time ago. We have no idea. We just kind of let it
412960	418720	go and we sort of know how to poke it and occasionally it pokes back, but that's it. So, we
418720	423200	have to accept that crappy code is going to make it into the systems that protect our lives
423200	430000	and provide for the way that we like to live. Do we give up on code? Do we say, oh, it's crap?
430000	439140	No. Just recognize there is a natural tendency towards crap, chaos. Entropy. Entropy.
439140	446820	Well, couldn't we just keep them off the internet? Sure. I'd like to see you do that. So,
446820	455580	people. People. Oh, people. Management and people on the ground don't know how to talk to
455580	461600	each other. You can't do that. It's stupid. If you had any brains in your head, you would
461600	471300	not have even suggested that. You can't do so many less things. Anyway. That was loud. You
471300	476740	almost... Bet you don't want to hear me anymore. No, you almost threw off my pacemaker. That
476740	482920	would be the next talk. That's another talk. We do medical devices. The challenge here is the
482920	490680	classic nerd versus operations and management. Nerds speak nerd. They're kind of abusive
490680	495440	actually. Unchecked. We kind of beat each other up. Why? Because we were beaten up as kids.
495440	503740	We're nerds. Now that I have some power, man. Meanwhile, management is there ensuring the
503740	507740	function of the operation, right? They're caring about revenue. They're caring about the
507740	512080	business and they're trying to do that. You make that sound really sexy. As an engineer, I'm
512080	516820	kind of offended. I don't know what the interpretive dance is for dollar sign. I do,
516820	529660	actually. We're in Vegas. Management is focused on operations and the nerd guy runs down
529660	533940	and is like, hey, boss, we have this huge problem that we just found out. There's this issue
533940	539560	with code and immediately starts running down into the technical bits. Management glazes
539560	545120	it over, waits for him to leave the room. You know what happened? I don't know what happened.
545120	551840	Stupid or not, the management has a job and they have things that are very important to them.
552040	557320	Maybe their paycheck. Maybe they actually care about security or at least safety and the
557320	563120	mission of their organization or if you have a great environment, they may actually care
563120	572500	about you. However, what's the secret to communication? Understanding the other side. I like to
572500	579300	say that there are no crazy people on this planet. You just don't understand the system of
579300	586100	reference or their decision making. There are no crazy people. What's important to you? What's
586100	592640	in it for me? Oh, you had to go there. A little bit later we're going to go through a little
592640	597600	bit more granularity about the different kinds of stakeholders in this and the pieces of the
597600	604620	supply chain. But fundamentally you have this challenge of this, the two sides speaking Greek.
604620	609500	Nerd speaking nerd, management speaking operations and in the middle of this gap. And this is
609500	613740	often times where security research companies like us come in between because we're there to
613740	618220	bridge that gap between. We have the technical understanding of working with the equipment at
618220	622620	a very granular level and understanding the implications there. And then importantly we can
622620	628640	tie it back to the business context and impact because that's the key. That's how you got to
628640	633960	bridge that. When you're thinking about, it's not code, it's the impact of the code. Compromise
633960	640220	is not compromise is not compromise. Compromise or a hack or an exploit only has value within
640220	645740	how does that vulnerability affect the business, affect the operation. That impact helps you
645740	652680	understand prioritization. That impact helps you understand the correct remedial solution
653340	658320	because it's not just a question of, there are lots of different ways to fix problems. It's
658320	662960	fixing it the right way for the context of how it's being used. And you don't want to take
662960	668800	responsibility as an engineer for something that is going to get your boss fired. It's his
668800	674880	responsibility. Your responsibility is to provide information, to understand and to bring
674880	681300	along. A very dear friend of mine last night at our party said, Atlas, we've had a couple of
681300	687420	drinks so we're all kind of reclining by the hot tub. He says, Atlas, what's important to you?
687420	696020	How do we fix this? What are you thinking? And I didn't have this canned but the three things
696020	705780	that came out of my mouth were education, motivation and partnership. I didn't realize that
705780	710040	that applied so well to what we're going to talk about today. Go ahead.
710940	717360	But wait, there's more. So as we discussed, we talked about the agent incentivization
717360	724540	problem where is the security community really incentivized to become engaged in this. And
724540	730000	then of course looking at where's the clear mandate coming from? Who is in fact pushing
730000	735020	that? Because this is a top-down problem. You're not going to have the average consumer, which
735680	742540	everyone in this room is. Last time I checked, we all use electricity and water. Everyone in
742540	747480	here is a consumer and a stakeholder. And we as a large group are not really able to
747480	752260	effectively advocate for something that is a very complex problem and set of problems. This is
752260	756940	something that needs to be top-down driven. Accountability?
757800	764040	Yeah, no, I don't. I'm not a consumer. I live in a yurt. Solar power, dude.
764220	765820	Ryan, how's that yurt coming?
768540	776740	So there's no clear mandate. Who is responsible for fixing these problems? The way that we
776740	783840	live affects all of us, as he just said. But who has to fix this? Is this the vendors, the
783840	790260	people who made PLCs, RTUs? Is it the utilities? Is it their problem? I've seen some pretty
790260	795520	amazing engineering going on in utilities. Is it their problem? How about regulatory
795520	801240	agencies? Show of hands. Who wants the regulatory agencies to fix this problem for you?
802360	824860	Oh, I love this. This is great. Go, Joe. He's saying if it's regulatory, it will be check the
824860	830080	box. It's the only way that they know how to do it. And in fact, they're incentivized with
830080	837980	both carrot and stick to make it check the box. Very good answer. I mean, there are a couple
837980	845860	different agencies. We'll talk about that in a minute. Go ahead. So the first thing, let's
845860	853020	address the problems that we can right now. There are things we can do. We actually want to
853020	859900	have a 5, 10, 15 year plan. Hopefully not 15. Please, Lord, let us fix this faster. But there
859900	865060	are things we can do right now. Show of hands. Anyone want to give an offer? Any ideas?
867080	873420	I can build my house on stilts so that when the dam breaks, I'm good to go. I like the idea. I
873420	880360	already have my house built. I'm not sure that I want to jack it all up. Anyone else? Tim, I
880360	886280	know you. No, I'm just kidding. So where do we start? Well, we begin by assessing what the
886280	894960	problem is in our own backyard. Every one of the people on the last page, even the
894960	901160	regulators, we want to become aware of the problems that we can fix right now. For example,
901160	907580	if we are still using Modbus, probably a good idea to get that off the wire. I don't care if
907580	914440	you're doing serial. Okay, there are worse things to use. But if you're using Modbus right
914440	921200	now over Ethernet, please stop. There are other options. And I'm sure that there are business
921200	926440	decisions that made Modbus the right thing and it takes money and time and effort. But it's
926440	935640	something that we can fix. If the protocols that you use offer security, lab it up, make sure we
935640	943480	don't roll out some big change and take down the grid or the water supply or back up the poop.
944440	953040	I mean, we're talking about more than just the electric grid, right? But lab it up, test it,
953040	958840	migrate to something that at least offers security, even if it's not perfect. DMP3 has tons
958840	965000	of bugs. But at least it, as he stepped on the chair, I was thinking he was going to call out
965000	971520	the fact that you may be still kind of surrounded by shit, but if you can get a little higher,
971520	979540	you can be in it less. Pick something better. If you have systems that are, have replacement
979540	985520	upgrades and you can put into place a better system, maybe a system that actually has a
985520	992880	vendor that's still around or a vendor that you trust to partner with you, this is a good
992880	1000480	first step. There are many things that you or we, whoever is doing this assessment, can sit down
1000480	1007300	and identify, within the next year, we can do this. Let's talk to the right people, work it
1007300	1016560	out, get something as part of a bigger process. To emphasize, this is the all is not lost slide.
1017000	1022540	You can start somewhere and have that vision of where things can get better, circle the wagons
1022540	1032980	where you can establish control and then iterate from there. Best practices. I mean,
1033530	1040620	so, a few other things that we can throw into the mix that you can do within the next year to
1040620	1048120	two. Authentication is a problem we all deal with. There are many technologies available
1048120	1053020	because everybody is struggling with it. Two-factor authentication for the right systems
1053020	1060220	is a good first start. If you have to use passwords and not tokens or keys or fobs, please.
1062140	1069600	The next step, guys, I know that we didn't mean to plug the grid into the internet, but we've
1069600	1077380	done it. We didn't design our systems and our networks to withstand the type of onslaught that
1077520	1085300	a nation state is willing to offer. In fact, the state of many organizations, many people here
1085300	1094940	without state funding could do it. So, I'd like to lay out the idea of a multi-tiered castle. At
1094940	1101180	the very center, with the most restrictive protections in place, you have crown jewels, you
1101180	1107740	have head ends, you have things that actually change significant parts of life. Outside of
1107740	1114080	that, you have slightly less trusted, less protected systems, but still very highly
1114080	1123060	protected. And so on and so forth. I recommend three to five layers at least. And these layers
1123060	1130160	play into a potential for protection. So if you have a system that there's absolutely no way
1130160	1137980	that you can have assurance that it is protected, then it falls in the outer ring. For example,
1137980	1146100	pole top devices. Anyone here that's been at DEF CON, you're not, you cannot possibly argue with
1146100	1155440	me that a system being on top of a pole is not secure. Unless it's in a compound with guns.
1157260	1164800	Patching policy. This is just regular SEC 101. The systems that are being used to pivot into
1164800	1171220	attack your critical infrastructure, generally people's workstations, check an email, surf
1171220	1176780	on the web, these systems have to be maintained and protected. Now this is not going to keep
1176780	1182320	off everybody. I'm going to try not to throw out nation's names because I know everybody's
1182320	1191180	represented here. But patching, keep at least the end days done. And then there are
1191180	1199060	configurations in our PLCs and RTUs that again can be used to turn on security features.
1199060	1205120	Encryption and strong authentication and other things to lock down access that you
1205120	1210240	shouldn't have available on your PLC. Now there are many who will say encryption takes time
1210240	1217860	and I need point blah blah blah milliseconds response time. If your critical systems can't
1217860	1225460	handle that type of waylay and processing power, those might fit on your list to replace.
1228390	1234950	Monitor all the things, all the comms. So this is while end points might not be the place
1234950	1240530	that you can do that monitoring, certainly there can be transmission paths and one of the
1240530	1245830	challenges with this of course is not all of the communications are going to be wired. You have
1245830	1249970	to keep in fact that a lot of things are going to have RF communications and you need to
1249970	1253410	start thinking of the physical security and the physical presence of those things. Please
1253410	1257570	don't allow your cell phones to connect to your network and talk to your head end. There's
1257570	1262730	one good rule of thumb. But if you do have cell phones connected to a network, it better be
1262730	1269890	on something you can monitor really well. Hunting, who here has heard of hunting? Okay,
1269890	1276510	about one percent of you. Hunting is the act of scouring your network. No penetration
1276510	1281750	testing has to go into this if you're worried about breaking things. Simply monitoring
1281750	1289650	traffic patterns. Traffic can be cataloged, identified and put into a database and
1289650	1297470	analyzed. For example, there's no reason why workstation in the office needs to talk
1297470	1305330	directly to head end. If you have this type of communication or other strange communication,
1305330	1313150	A, that's something you want to understand better. B, it may be the thing that tells who's
1313150	1326510	on your network. Fuck Modbus. We've beaten Modbus. Modbus is unauthenticated. And perfectly
1326510	1334150	sane to kick the shit out of here. There are others like it, but this is Modbus. Please
1334150	1341290	choose something that has some sort of protectiveness. If you have critical systems
1341290	1348030	that must use an insecure protocol, maybe because you're worried about latency, you should
1348030	1360500	have armed guards around everywhere that that network touches. This is the self-promotion
1360500	1364460	slide, so I'm just throwing that out there because this is some of the stuff that we do. But
1365020	1372340	it's more, even if you don't do it with us, please understand, we have yet to get to the
1372340	1380700	point where I need to throw binary exploitation against most control systems the way we do
1380700	1384940	your laptop, the way we do your servers, the way we do against firewalls and routers. We
1384940	1394640	don't have to do that. It's not hard enough to warrant that kind of work. But there will come
1394780	1401320	a time, hopefully very rapidly, where that's not the case. As we fix the fixable, digging
1401320	1410120	into hardware, software and firmware, that was sexy, identifying weaknesses in code, weaknesses
1410120	1417220	in design of hardware, weaknesses in what your vendors are shipping out, like hard-coded
1417220	1427160	passwords, these still exist. Assessments of all three things will provide you, when we finally
1427160	1433300	get our poop in a group, something that is defendable. And if we don't start now, it will
1433300	1440820	just be that much longer before we get to something that's defendable. But I've got limited
1440820	1447540	budget. Well then prioritize. If you have systems that are most widely deployed, those should
1447540	1454880	make pretty high on the list. If you have significant systems that control large,
1454880	1463940	potentially irreplaceable parts of your systems, that may make a very good top place. You can
1463940	1469660	get to the older stuff later. You guys know how to think about boundaries, right? We do the
1469660	1476180	most important first, we then back up. We don't stop until we're done. But we get the best
1476180	1483060	that we can for now. So while we list three device categories here, of loose categories,
1483060	1488240	there are also three kinds of assessments. White box, gray box, and black box. And I list those
1488240	1493680	because those also require careful consideration for what you're trying to accomplish. A black
1493680	1499080	box assessment is saying, hey, you're just like somebody else out there anywhere and you are
1499080	1504180	required to do all of the open systems intelligence and gathering to try to figure out what kind
1504180	1509020	of information can you gain. And then from there start to work that to then be able to
1509020	1513120	determine how you're going to conduct your research for exploitation. Gray box is going to
1513120	1516700	be a mix of you have access to all of the documentation, so there's kind of that in
1516700	1521240	between. And some of the factors there are going to be trying to understand how much is
1521240	1526780	obfuscation your friend, and obscurity is your friend, versus how much of it do you really
1526780	1530800	want to just get straight to the technical research and understanding, which is what white
1530800	1536960	box is going to do. And there's also one other thing to point out. This is not check box
1536960	1543240	things. Check boxes get you the first layer of an assessment like this, because you can
1543240	1548980	actually put into place a list of things that you want to check to gain knowledge about the
1548980	1556740	next steps. Once you're past that, you need people who are creatively evil, and you need them
1556740	1566460	to have the time to sit in their creativity and think, oh, hey, Tim, do you think that we could
1566460	1580140	do blah, blah, blah, blah, blah? Yeah? Let's go do it. Oh, this is my slide. Sorry. So as we
1580140	1586480	said before, we need devices that we can deploy that have security baked in from vendors
1586480	1594800	who think about security. We also need, we need to reassess what we're doing for comms,
1594800	1600280	because there are radios that have encryption enabled. Almost by default. Oh, but it's
1600280	1608020	really easy to enable them. You simply have to choose one of 256 standard encryption keys.
1609180	1618900	That doesn't sound like a lot. Hey, it's AES fucking 128. With one of 256 different keys. And
1618900	1623400	please, understand, one of the things that, actually one of my favorite parts of my career
1623400	1627100	has been breaking into banks, breaking into government buildings, and doing things with
1627100	1632780	cyber that aren't typical pen testing things. You may think of them as red teaming. If so,
1632780	1638920	that's great. But some of our assessments have really gone, oh, well, I can hop the fence.
1638920	1642960	Okay, here's the device, you're going to get there. You don't have to hop the fence, I know
1642960	1648380	that you can do it. And we've gotten our hands on a lot of attacks after. In fact, the reason
1648380	1654760	that I call out MDS is because they're deplorable in many of their at least older stuff
1654760	1663260	that's still widely deployed. For being able to use one radio that you purchase off eBay to
1663260	1671460	attack all the others, and even brute force within 18 hours, guaranteed to be on any network.
1674330	1678170	So, the people. We've talked about the people, we've talked about the people through multiple
1678170	1683250	things. Where are these folks, right? So, first of all, I think that this is, again, what I
1683250	1687650	love about DEF CON is this is the meeting of those passionate folks who really do care about
1687650	1692290	this problem for various reasons. Either you're motivated by how we can all help humanity
1692290	1697950	together, or you have a particular interest of the matrix metaphor of going down the rabbit
1697950	1706630	hole and seeing how far it can go technically. Your vendors should be having security-minded
1706630	1713270	people in their hiring process. They should be actively pursuing not just bullshit bingo
1713270	1720110	security, but true security. And utilities. They need to be able to call bullshit. They need to
1720110	1728630	be able to have the knowledge and skill to say, that's crap. Fix that. Or better yet, get out
1728630	1735750	of here, these other vendors, they have people who know what they're doing. So, go ahead, I'll
1735750	1741850	let you take the top. How to fix for the who's who. We have two regulatory agencies, FERC and
1741850	1746050	NERC, and there is a challenge there where there is a mix between the overlap of
1746050	1750870	responsibilities and a difference between suggestion and compliance. And while we talked
1750870	1754870	about compliance being a challenge from a checklist perspective, we also talked about the
1754870	1760470	fact that this is a top-down mandate that needs to push and encourage this action. And that is
1760470	1764430	the purpose of government and legislation. And that's where these organizations need to come
1764430	1768870	into more of a common agreement with the vendors in the industry to help push that. Now, they
1768870	1774150	need to be light-handed right now. We've already got some very heavy-handed things in place
1774150	1779690	that do not, that actually can detract from true security. They have to be careful how they
1779690	1784650	engage, but FERC and NERC do play a very important role. Our public services commission, our
1784650	1789330	public utility commissions, these are the state-run organizations that kind of keep our
1789330	1794810	utilities in check. They are the protectors of the consumers. Now, I have seen some
1794810	1803710	significant investment from several PSCs into researching cyber security. They want to
1803710	1808330	understand. They want to know how they can fix the problem. And they have a big role. They
1808330	1814830	control how much a utility can make for a year. They also allow a utility to do a rate hike,
1814830	1820850	which allows them to charge more per kilowatt hour or per gallon of water or whatever. In
1820850	1826910	the interest of maintaining a grid, a water system, a water treatment system that services
1826910	1833670	our needs. But our vendors, they're saying, where the hell is the money coming from? The
1833670	1837590	utilities don't want security. They're not willing to pay for it. I can't make it unless
1837590	1843390	they're willing to pay for it. So utilities, phony up. Utilities are saying we don't have
1843390	1848190	any vendors that provide things that we can put into place that are defensible. They have
1848190	1855930	crap. And this is what I have to deal with. Encourage. Educate. Make the decision. And in
1855930	1865510	fact, you have a lot of sway. The unfortunate side is we have decades of utilities beating up
1865510	1872610	vendors for the lowest price. We have lost the partnership that is provided for our
1872610	1879350	country and our world. We got to get it back. And so that concludes the amount of time we
1879350	1884730	have for this. This was a lot to condense into a short talk. So I'm Bryson. I'm Atlas. And
1884730	1887630	we're Grimm. Thank you.