start	end	text
0	4660	Welcome to the talk on detecting lateral movement in zero-touch environments. We're going to dive
4660	12840	into how that lateral movement works, how to detect it, and why you really want a zero-touch
12840	20760	environment. So before we jump in too much, some disclaimers and acknowledgments. First off,
20760	25960	I work for the MITRE Corporation, but I am not representing or speaking for them today.
27080	32440	All of the research done in this talk was performed in the pursuit of a master's degree
32440	37940	through SANS, and so thank you to them for supporting the research, and thank you to my
37940	44700	advisor, but I'm also not speaking for them today. And lastly, thank you to my wife Madeline,
44700	53520	whom I'm also not speaking for today. So apart from all of that, who am I?
54680	63240	My happy place professionally is in the middle of security and DevOps. And that has quite naturally
63240	70940	led me to doing a lot of work with the cloud, automating not just applications and operations,
70940	76860	but the security portions of that. A little bit of background on me. I wrote my first
76860	82460	vulnerable code in elementary school. I was writing a blogging platform for my family,
82460	88740	and they were quite surprised when it was hacked through a SQL injection that I inadvertently
88740	95440	added. I began learning to write exploit code in middle school, when I received as a present
95440	101540	the book Hacking the Art of Exploitation. And I've been involved in security one way or another
101540	107940	ever since. This is my first time speaking at DEF CON, and I am super excited that the
107940	115060	Cloud Village has me here. And one last little kind of personal philosophy thing is I really
115060	120780	enjoy learning through hacking. I imagine a lot of you do, too. And you'll definitely see that
120780	126160	flavor as we go through the presentation today of looking at things from both an attacker's
126160	133920	point of view and a defender's point of view. So why should I care about DevOps if you're
133920	139600	not already sold? Well, if you're running any applications or any services,
140600	146420	you've got to deal with it. That's the way it is now. Welcome to the world. However,
147640	152080	that's true regardless of whether you're hosting applications for the outside world,
152080	158540	just for yourself, whoever it is. If you're a red team and you need to spin up infrastructure to
158540	166140	support a penetration test or something like that, DevOps to the rescue. Blue team, you're
166140	172040	trying to keep continuous monitoring and alerting of what's going on in your environment, DevOps.
172860	180300	Application owner, trying to make sure your application is stable and available and secure,
180300	187480	DevOps. Another great perk for us is that DevOps is essentially cloud native now. All of the major
187480	196340	cloud providers have offered ways to automate their environments. And so being able to develop
196340	203640	infrastructure as code, applications as code, security compliance as code, monitoring as code,
203640	208240	all of these are really enabled by the cloud. They can be done in other environments,
208240	215280	but the cloud makes it easy. And lastly, DevOps is better for security. We're integrating security
215280	219760	earlier in the process, rather than trying to layer it on later.
222380	227720	So I promised an attacker's point of view. Today, we're going to look at kind of a simplified
227720	232220	example environment. I'm sure, you know, you've seen a bunch of other presentations that have got
232220	238500	the hard to read, squint and see, that's AWS, right? Not Azure. We're simplifying things for
238500	244780	this talk. You're somewhere out on the internet, and you really want to get to an application server.
245620	252260	Might want to get in and then exfil some of the sensitive data on it, you might want to get in
252260	257940	and use their computing resources to mine cryptocurrency. Whatever the reason is,
257940	264620	you're going for that application server. So there's three main options we're going to look at.
264720	270160	First is straight through the front door, directly in from the internet hitting the application
270160	277480	server. Pretty simple. Yeah, there's maybe a firewall or an IDS or something that you have
277480	285640	to evade. But overall, pretty direct, pretty simple. Except it's not 2003 anymore, and we've
285640	292080	gotten better at protecting our applications and the servers they run on. So while it's still
292080	299280	possible, and there are plenty of big, well-known events where it's happened, it's not as popular or
300500	307400	as reliable as it used to be. So that brings us to our second option, and that's, let's compromise
307400	313440	the workstation of an employee, and then move laterally to the application server we really
313440	320020	want to be on. For the purposes of this presentation, I'm going to say we phished that
320020	328240	employee, but it could be any number of things. From there, most organizations have a bastion host
328240	335140	that employees have to log into before they can get into their production environment. That's a
335140	341080	pretty common practice, and so we see that on the screen. Our second option is compromise an employee
341080	349120	workstation, use valid credentials that that employee has access to, move through the bastion
349120	357040	host to the application server. And then our third option is to take the route that DevOps uses to
357040	363480	publish applications itself. From that employee workstation, push something into the source code
363480	370300	repo, get it through the test servers undetected, have the configuration server deploy it for us.
371300	378020	Or maybe we can compromise each of those servers in turn in order to get there in the end.
378600	384200	In any case, that's a lot more hops to have to go through and a lot more chances to be detected
384200	392040	or fail in some other way. So we're going to focus on detecting number two, both because it's pretty
392040	399920	common and because that's the easiest route that attackers have, and so that's most likely what
399920	405680	they're going to go for. And this being common is borne out by repeated studies. We've seen
405680	412180	reports from FireEye and the Verizon data breach report say that social engineering and then using
412180	421670	valid credentials is incredibly popular for malicious actors right now. So what does it look
421670	429070	to do a traditional application deployment from that employee's point of view? Developers give
429070	435690	ops a deployment package and hopefully some install instructions, and then ops logs into
435690	442070	the app server through the bastion host and manually installs the software. Pretty simple.
442790	446750	And then you get a patch. Well, you've got to log in and install the patch.
446930	450650	There's an update to one of the dependencies. We've got to log in and install that.
451050	455810	Anytime anything changes, someone's logging into that application server.
459850	465510	So that's, you know, pretty normal, but not ideal, and we'll get into that.
467410	473530	Traditional lateral movement takes advantage of that fact. Because you've got people logging in
473530	483250	to the server manually, they have highly privileged credentials. And very often,
483250	488570	those credentials are stored in plain text on their employee workstations. Exactly what we
488570	494750	were talking about compromising earlier. That might take the form of SSH keys,
494750	501810	it might take the form of API tokens or keys. And, you know, especially relevant for the cloud
501810	509350	village, that's the way AWS stores local credentials. Plain text file in that path on the
509350	517510	screen. Attackers then use those plain text credentials from the station they've already
518030	524710	compromised to move further into your environment and do whatever evil they want to get up to.
524750	533150	They've hit their target. So what is ZeroTouch and why does it apply to any of this?
533570	540630	So Google are the ones who have defined ZeroTouch networking and ZeroTouch production concepts.
540630	546770	But they're far from the only people who use them. This is a technique used by many mature
546770	552430	DevOps organizations to automate as much as possible and get humans out of their
552430	561130	production environments. What does that look like? Here's the three rules that Google use
561130	568530	to define their ZeroTouch production. And the first one is really big. Every change
568530	577560	is made by automation. No humans are involved. By getting that human traffic out of the environment,
578620	586340	we can do a lot of really interesting but simple things to detect when attackers are trying to
586340	594980	pivot into our protected environment. The second rule, pre-validated by software, is important
594980	600460	because if we try to go to that attack path number three, the one that uses the DevOps pipeline
600460	605480	itself, that means we're checking to see if there's anything malicious before we deploy.
605480	612720	Now, that doesn't always work. But it gives us as defenders a better opportunity than someone
612720	618190	who's just pivoting through the bastion host and hiding in the noise. And then lastly,
619440	627260	emergency changes can be made via a break glass mechanism. But they have to be auditable.
627340	635960	We'll come back to that. And what this looks like in our picture is essentially disallowing
635960	643000	communication from that bastion host to the application server. And all normal traffic is
643000	648100	going through that path number three where it goes through the source repo and the test servers
648100	653860	and the configuration servers. So we've got good control over what's being deployed to the
653860	664320	application server and good confidence that it's not malicious. All right. So now let's look at
664320	670300	lateral movement in each of these two types of environments. We've compromised our employee
670300	677120	workstation, found some valid credentials that we can use to pivot. And in this case, we're going
677120	683460	to look at the traditional path where operators are regularly logging into the environment to
683460	689900	deploy patches, update software, any of those kinds of things. I'm about to play an animation.
690280	694300	And there are going to be several connections shown on the screen by arrows.
694320	699300	Try and find the one that looks a little bit different than all the rest.
699640	708640	This is going to be our malicious lateral movement. Did you see it?
710020	717880	Have you seen it yet? It's a little bit tricky to find. And this is assuming that it stands out at
717880	724200	all. If you're using an encrypted communication channel like SSH or HTTPS, which are super
724200	734620	common, this use of valid credentials might not stand out in any way. Now let's compare that to
735320	741560	someone trying to use lateral movement on a zero touch network. Same thing. I'm going to display
741560	746500	several arrows in an animation and see if you can pick out the one that looks a little bit different
746500	755410	and indicates our malicious lateral movement. I'll bet that was a lot easier. And you maybe even
755410	762050	spotted it before the animation finished playing. In this case, it stands out like a sore thumb
762050	768750	because those valid credentials, even if the channel is encrypted, are going an entirely
768750	778690	different route through the network than normal, regular, valid traffic would. So how do we go
778690	785790	about detecting that from a programmatic and automatic standpoint? First, we need to define
785790	792130	the protected servers. In the case of our example, that's just the application server. But it could
792130	796770	be a lot of things. Could be application servers, web servers, the DevOps servers themselves.
797090	803990	Anything that humans shouldn't be logging into on a regular basis. Then we need to define all
803990	809810	of the human access points into the environment. Again, in this example, that's just the bastion
809810	814710	host. But depending on your network architecture, there might be any number of things that you need
814710	821930	to put into that list. And then watch for any connections made from those manual access points
821930	829450	to your protected servers. Real simple, but very effective if it's not supposed to be happening.
829450	836190	At that point, you can alert, investigate, whatever your normal process is, and enjoy having detected
836190	843790	malicious lateral movement. Now, I mentioned earlier that auditing was important. And the
843790	849690	reason that this is still a valid attack path into the environment is most organizations can't
849690	855470	get rid of their bastion hosts entirely. They're still needed for incident response or forensics,
855470	862410	some kind of unusual emergency purpose. And so, even though that's valid traffic, this method
862410	869770	would still alert on it. However, that's not a problem, because anytime that emergency traffic
869770	876670	does occur, we wanted it to be auditable anyway. And this is helpfully generating those audit alerts
876670	884690	for us. So, now we're going to cross our fingers and see if the live demo goes well.
890750	898630	What I've got up on the screen are three lab machines. One, a bastion host,
898630	903590	one, our application server, and one, a puppet configuration server.
905210	908870	What I'm going to do, just to prove to you all that this is actually working,
908870	914970	and that I've got nothing up my sleeves today, is we're going to put some valid traffic into
914970	922130	the environment. What this puppet command will do is generate traffic between the puppet configuration
922130	928990	server and the application server, as well as traffic out to the wider internet over both SSH
928990	933630	and HTTPS. And we'll keep that going throughout the rest of the demonstration.
936150	952010	Then, we'll go over to the bastion host and try to execute our... let me fix that real quick.
952010	961070	Then we'll go over to the bastion host and... nope. My routes are wrong today,
961070	964550	so I'm going to play the prerecorded version of the demo instead.
972580	982550	All right, but you'll get all the same information here. So, for the demo, we're going to look at a
982550	990950	couple of scenarios. First up, let me tell you what you're seeing. In this environment, I've got
990950	997390	a bastion host that's used for manually accessing it when needed, an application server, and a
997390	1002910	configuration server running puppet. So, just to make sure that there's nothing up my sleeves,
1003490	1009190	what we're going to do for this demo is run some puppet configuration in the background.
1010250	1015990	This will be traffic both within the environment to the puppet server, and external to the
1015990	1022310	environment SSH connections and HTTPS connections out to the wider internet. So, let's get that
1022310	1032050	started. And then, while this runs, jump over to the bastion host, and as you can see, we've already
1032050	1038990	compromised the bastion host. And we see some SSH credentials that we found, and in fact, some SSH
1038990	1048050	credentials for the application server that has the goodies we want to get to. So, we'll run our evil
1048770	1057810	script file. And, you know, in real life, this might be installing a crypto miner or any number of things.
1057810	1065270	In this case, just modifying the configuration files that Puppet is managing to show that we've
1065270	1073230	actually gotten in and had the ability to do evil if we wanted to. So, now we're going to jump over to
1074270	1085810	the network monitor. This is a plain instance of Security Onion with all of the usual defaults and
1085810	1096130	alerts installed. And what we're looking at right now are the ZIP notices. And let me turn on the
1096130	1106870	refresh. But you'll see a whole lot of nothing. Zero. No results found. No results found. If we
1106870	1123950	switch over to Squirt and refresh it, same thing. No result. No result. And instance of a network
1123950	1134470	monitor. It's not going to alert on connections like this. Let me enable the rules we've been
1134470	1147480	talking about. And then we'll come back and we'll try this again. And now that I've enabled the
1147480	1153540	zero-touch rules, let's try it again and see what's changed. As you can see, the lab server
1153540	1159000	is still emulating real traffic by reaching out to the Puppet configuration server and to the
1159000	1166880	external internet. Let's go over to the Bastion and run our evil script again. Same thing, utilizing
1166880	1175960	those compromised SSH credentials to reach in and modify the app server. Now we'll jump over to
1176540	1183300	the network monitor. And as you can see, we're starting to get alerts. Zeke has noticed
1184700	1192700	a couple of connections from the Bastion host to the app server. And if we switch over,
1192700	1202800	you'll see the same thing in Squirt as well. Notice how they work slightly differently.
1202800	1210440	The Squirt gives us many more alerts because it's looking individually at the packets traversing
1210440	1216240	the network versus Zeke gives us total number of connections. But either way, we're seeing
1216240	1222440	the alerts that we're violating this zero-touch policy and that it's a good indicator of lateral
1222440	1240650	movement. All right, so you can see from the demo, you know, this isn't something that adds to your
1240650	1246340	ability to detect lateral movement as long as you conform to the expectations of the environment.
1247080	1251260	Demonstrated here with Zeke and Snort, but there's no reason you can't use
1251940	1258640	any of your tools of choice. But this is the cloud village. So what about cloud?
1259780	1265730	Totally still doable in the cloud, despite the fact I use VMware for my lab environment.
1267120	1273520	If you're not familiar, all the major cloud providers provide some form of traffic mirroring,
1273520	1278720	whatever they call that, traffic mirroring in AWS, virtual network tap in Azure,
1279280	1285640	and packet mirroring for Google Cloud Platform. And that'll give you full network visibility.
1285640	1289540	You can install something just like Security Onion or your favorite network monitor
1291040	1298540	feeding directly off of that. You can also install your own within the environment. And
1298540	1305240	because this is looking for such simple things as connections, we don't even need the full
1305240	1311940	traffic in order to tell when this activity is occurring. We can just use flow logs. And so
1311940	1318800	that'll cut down on the amount of traffic that we have to capture if we go that route. Now,
1318800	1324120	given that the cloud is a much more dynamic environment than traditional data centers,
1324120	1328420	this means that tagging an asset inventory is really important to get right.
1329240	1335020	You have to know what those protected servers are and what the human access points are in order to
1335020	1341080	detect connections between them. And if you've got automatic scaling or something like that,
1341080	1345700	keeping those lists up to date can be a little bit tricky, but it's definitely doable,
1346520	1354100	especially since those lists in most tools can be entire subnets rather than specific machines.
1355440	1361820	But there are also some challenges with the cloud. And the big one is visibility into
1361820	1369960	management traffic. With something like Puppet or Ansible, your traditional configuration servers,
1369960	1374300	that's all going over a network that you can view. However, in the cloud,
1374300	1379180	some of your management traffic goes over the management plane in the background.
1379180	1386800	And these features like traffic mirroring are not able to capture that data for you.
1386800	1392680	It all happens in AWS or in Azure without you being able to see it.
1392720	1398020	And that poses a visibility challenge. Doesn't mean that it's not worth doing,
1398020	1404360	but you have to understand how your environment usually works in order to be able to protect it.
1407150	1412870	So, some next steps if you're sold on this idea. First off, if you're not Zerotouch yet,
1412870	1420870	do it. It's got a lot of benefits aside from this particular detection. It's hard to do,
1420870	1425610	and it's hard to get right, but it is worth it. And then you can implement this detection on
1425610	1433290	your platform of choice. I showed off Snort and Zeek here, but it's very simple. There's no need
1433290	1437310	and buy specialized software, buy specialized hardware,
1437310	1441290	install anything you're probably not already using anyway.
1442610	1449330	And then when you do implement this detection, tailor it to your specific environment. Here I
1449330	1455250	was looking for any connections between the human access points and our protected servers. But
1456670	1461610	your environment may or may not be able to support that depending on its specific architecture.
1462150	1468570	The good news is that this approach is valid for a lot of different options, and you just need to
1468570	1474170	figure out which ones work specifically for you. You might only turn it on for SSH connections.
1474170	1479350	You might only look at particular areas of your network that are high risk.
1479410	1483270	However you do it, make sure you tailor it, because that'll give you the most value.
1484390	1491590	And then there are ways to extend this work far beyond what I demonstrated in today's video.
1492140	1498480	For example, correlate it with other events that are generated off of suspicious traffic.
1498830	1505530	Imagine tying this alert about potential lateral movement with potential data exfiltration that's
1505530	1513070	generated by Zeke, and then presenting your security team with a map of how the attacker
1513720	1518970	moved through your entire network. It's pretty powerful once you start combining it with other
1518970	1528430	things. Lastly, I want to leave you with some lessons learned from doing this research project.
1528430	1533270	First off, you have to know your network. That's the basis of this entire project,
1533270	1539250	is you have to know your network well enough to know what's normal and what's abnormal.
1539370	1546170	And then putting in things to look for abnormalities specific to your environment
1546170	1550190	as opposed to things that apply to everyone's environment.
1550830	1555890	I learned this principle first from Mick Douglas and Justin Henderson, who teach the
1555890	1563930	sans-tactical-seam class. And it really resonated, it was really powerful in being able to do things
1563930	1569710	that don't come pre-built with your software, because you're looking for things specific to you.
1570230	1575350	And then second, don't be afraid to look for stupid simple things on your network.
1576170	1580670	All we're looking for in this is the presence of a connection.
1581970	1588490	But knowing our environment the way we do, that lets us say there's something weird going on,
1588490	1592650	I need to investigate. And that is super valuable.
1594830	1601370	So, thank you. If you have any questions after the talk and the Q&A is done,
1601370	1605910	best place to reach me is on Twitter. My handle's on the screen right now.
1606170	1610530	And then we'll see if we've got any questions live.
1610530	1616250	Do you happen to know any tools that might help implement this kind of a
1616250	1622930	zero-touch environment or getting to know your network better or something along those lines?
1622930	1627750	Sure. So, getting to know your network better, you're just going to have to get familiar with
1627750	1633490	how these operate. And my biggest recommendation is if you're starting to do DevOps as a security
1633490	1641290	person, is make time to spend with the operations and the developers. Because they're going to know
1641290	1646550	how things normally go. And you'll be able to learn from them what they expect. And then use
1646550	1653930	your knowledge of security to figure out what shouldn't be going on. In terms of tools to help
1653930	1661290	implement this, you know, typical DevOps tools, there's a periodic table of DevOps that's out
1661290	1668430	there, I think by Xenia Labs, that shows a lot of the different options. For the demo today, I was
1668430	1675190	using Puppet as a configuration server to manage the configuration of the application server and
1675190	1682210	making sure that it had the files and configuration settings that I expected. There are a lot of them,
1682730	1689070	apart from Puppet, Ansible, SaltStack, things of that nature. And then for building out a
1689070	1694270	whole DevOps pipeline, look at that periodic table. You'll need several different things
1694270	1699730	along there to handle your source code and your testing and things like that.