start	end	text
13790	17650	Hey AppSecVillage, welcome back and get ready for another talk. Coming up next
17650	20570	we've got Can't Touch This! Detecting Lateral Movement in Zero-Touch
20570	25510	Environments. Philip Marlowe is going to be presenting this. Philip is a
25510	29430	cybersecurity and DevOps engineer. He helps organizations understand how to
29430	34070	adopt DevOps practices to increase their security rather than sacrifice it in
34070	38990	the name of speed. Philip holds several security, cloud, and agile certifications
38990	41570	and is currently pursuing a master's degree in information security
41570	46890	engineering at SANS Technology Institute. Please give it up for Philip Marlowe.
47230	52290	Hello everyone and welcome to Can't Touch This! Detecting Lateral Movement in
52290	56250	Zero-Touch Environments. In this talk we're going to talk some about DevOps
56250	61710	and how to detect attackers trying to move into your production environments.
63570	69490	Before we get started, a few disclaimers and acknowledgments. First off, I work for
69490	75810	the MITRE Corporation, but nothing I'm saying today reflects their opinion.
76590	80290	Also, I'm a student at the SANS Technology Institute working on my
80290	86210	master's degree. Huge shout out to my faculty research advisor, but I'm not
86210	92550	speaking for her or SANS today either. And lastly, big thank you to my wife
92550	97150	Madeline. Once again, not speaking for her in this presentation today.
98910	106130	So who am I? I'm Philip and professionally I really enjoy the intersection of
106130	112150	security and DevOps. Yes, you can call it DevSecOps if you want. I don't really
112150	118090	care either way. I wrote my first vulnerable code in elementary school.
118090	123970	I was writing a blogging platform for my family and right in there, because I
123970	128270	didn't know any better, put a SQL injection vulnerability. That was a big
128270	134170	surprise for them when they found out we've been hacked because of that. I
134170	139030	first began learning to exploit applications in middle school when I
139030	143810	received as a present the book, Hacking the Art of Exploitation. And I've been
143810	150130	involved in application security one way or another ever since then. Today is
150130	156570	my first time as a DEF CON speaker, and I'm super excited to be here. And one of
156570	161690	the things I really enjoy is being able to learn through hacking. And you'll see
161870	165770	a flavor of that throughout this presentation as we take both the
165770	174920	attacker and the defender point of view. So if you're not already on board, why
174920	182680	should I care about DevOps? Well, for one, you probably don't have any choice. If
182680	188040	you're running any kind of application, that's the way it is now. You probably
188040	194180	just need to learn about it. You know, if your red team standing up command and
194180	199000	control infrastructure, helps to have DevOps. Your blue team trying to
199000	203920	implement monitoring and detection helps to have DevOps. Application security
203920	213680	DevOps is the way of life now. And, you know, provides all these security
213680	220520	benefits on top of reliability benefits and maintainability benefits for the
220520	228080	other parts of the business. So let's take a look at a very simplified
228080	233920	environment, and let's take a look at it from the attackers point of view. We want
233920	239300	to get to something on the application server. Maybe we want to steal data off
239300	245800	of it, or maybe we just want to use the hefty server power for crypto mining.
246440	253060	Either way, we've got three basic options. Option one is to just go straight in the
253060	256900	front door and attack the application itself in order to get to the access to
256900	263180	the server. This is pretty simple. Maybe there's a firewall or IDS you have
263180	268220	to evade, but, you know, you're not making a lot of hops around to finally get to
268220	276300	where you want to be. But it's also not 2002 anymore. The situation's gotten a
276300	281200	lot better in how we protect our application servers. So while this once
281200	288140	might have been the best way in, it's not really anymore. Report after report from
288140	294260	places like FireEye and Verizon have shown a much more popular way into
294260	300940	networks these days is by social engineering users in one way or another.
300940	309980	In this case, we're going to take a look at phishing a operator or developer on
309980	315300	their employee workstation to give us our initial access. Then in method two,
315940	321220	we're going to move laterally through the bastion host and finally to the app
321220	330360	server. We'll talk in a couple of slides about why this works so well. But before
330360	335080	we get there, let's talk a little bit about option three. With DevOps
335080	340780	implemented these days, another way into that protected app server is through the
340780	346240	DevOps pipeline itself. But there are a lot more hops here and a lot more ways
346240	354880	that can go wrong and a lot more opportunities to detect this attempt. It
354880	360700	goes through source control repos, test servers, and configuration servers before
360700	370540	we ever get to our intended target. So how does traditional application
370540	377160	deployment work? Well, developers give Ops a deployment package, and if we're
377160	382100	lucky, some install instructions. And then Ops is going to go and log into the
382100	388040	application server, manually install that software. Pretty simple. And then an
388040	394340	update comes along, and you have to go and manually log in and update. And then
394340	400320	the security patch comes along, and you have to go log in and update. And this
400320	406520	manual logging in and updating is going to be a real problem for detecting
406520	412540	lateral movement. So what does traditional lateral movement look like?
414840	419780	Because they're logging in to do configuration and updates all the time,
419780	427420	Ops has highly privileged credentials. This might take the form of SSH keys, or
427420	434800	API tokens. And oftentimes, they're stored in plain text on their employee
434800	441420	workstations. If you've already phished that user, those are really easy and
441420	446980	really juicy targets to steal those credentials and move further into the
446980	456590	environment. All right, so that sounds like it's a problem. What is this zero
456590	464310	touch concept? Google defined zero touch networking and zero touch production. And
464310	469270	they've given several presentations on what that means to them and how they've
469270	476910	implemented it in their environment. Overall, it's about how we can abstract
476910	485050	away managing the production environment, so that we're not doing those manual
485050	492950	logins, change things, configure and update anymore. And while Google is one
492950	498030	of the only companies to use the zero touch nomenclature, a lot of companies do
498030	504630	this. Often, it's just a result of being very mature in their DevOps practices,
504630	513710	that everything is handled by automation, rather than human users. So zero touch
513710	520010	deployment looks very different. Every change is made by automation and
520010	527090	pre-validated. This means no humans logging in. And that's key for our talk
527090	533570	today. You'll also notice on Google's presentation that there is an exception.
533570	538610	And that's when something is made by an audited break glass mechanism. We'll talk
538730	545970	a little bit more about that later as well. So what is the zero touch deployment
545970	553530	look like in practice? Well, rather than having a user log into the app server via
553710	558430	a bastion host, since that's no longer allowed, every change has to go through
558430	569000	that DevOps deployment pipeline. Looking at how attackers are going to move
569000	575380	laterally, however, let's compare and contrast between the traditional and the
575380	583180	zero touch deployments. So I'm about to play an animation, and keep an eye out
583180	589440	for the malicious connection between the bastion host and the app server. It's
589440	598480	going to be a slightly different colored arrow. Did you catch it? Have you seen it
598480	605340	yet? It's kind of tricky to find. And this is when it stands out in some way. If
605340	612420	we're looking at encrypted connections like SSH or HTTPS, it may not stand out
612420	618060	in any way, unless you're doing some kind of break and inspect in the middle. Now
618060	626740	let's compare it to zero touch. In the zero touch deployment, you're going to see
626860	633140	a bunch of arrows again, showing normal benign traffic. And you'll also see one
633140	636900	arrow that'll look a little different, being our attacker trying to move
636900	644200	laterally. See if it's easier to spot this time. I'll bet that was a lot easier,
644200	649280	and as soon as it came up, you noticed it. This is the whole concept that we want
649280	655740	to use writing this detection. So how do we actually go about detecting this
655740	660920	lateral movement? There's a few things we need to do. First, we need to define
660920	666120	all of the protected servers. In our simplified environment, that's just the
666120	670980	app server. But it might also include DevOps automation servers, it might
670980	676520	include web servers, database servers, all of which might be behind load balancers.
676600	682820	Whatever humans should not be logging into, that goes in this list. We also
682820	687640	need to define all of the human access points into the environment. Again, in our
687640	691800	simplified environment, that's just the bastion host. But depending on your
691800	696520	network architecture, you might need to include other sets of load balance
696520	702060	machines, virtual desktops, employee workstations, anything of that nature.
702060	707940	That goes in the second list. And then all we do to detect is look for any
707940	716580	connections between those two sets of servers. Then alert, investigate, enjoy
716580	725080	detecting this lateral movement. Let's take a quick look at what this looks
725080	736860	like in practice. So for the demo, we're going to look at a couple of scenarios.
737100	743560	First up, let me tell you what you're seeing. In this environment, I've got a
743560	748440	bastion host that's used for manually accessing it when needed, an application
748440	754340	server, and a configuration server running Puppet. So just to make sure that
754340	759880	there's nothing up my sleeves, what we're going to do for this demo is run some
759880	766160	Puppet configuration in the background. This will be traffic both within the
766160	770080	environment to the Puppet server and external to the environment, SSH
770080	775300	connections and HTTPS connections out to the wider internet. So let's get that
775300	784100	started. And then while this runs, jump over to the bastion host. And as you can
784100	788660	see, we've already compromised the bastion host. And we see some SSH
788660	793860	credentials that we found, and in fact, some SSH credentials for the application
793860	803080	server that has the goodies we want to get to. So we'll run our evil script file.
803760	810160	And, you know, in real life, this might be installing a crypto miner or any number
810160	816140	of things. In this case, just modifying the configuration files that Puppet is
816140	821260	managing to show that we've actually gotten in and had the ability to do evil
821260	830320	if we wanted to. So now we're going to jump over to the network monitor. This is
831680	840200	a plain instance of Security Onion with all of the usual defaults and alerts
840200	848200	installed. And what we're looking at right now are the Zeek notices. And let me
848200	855740	turn on the refresh. But you'll see a whole lot of nothing. Zero. No results
855740	869240	found. No results found. If we switch over to squirt and refresh it, same thing. No
869240	876400	result. No result. And this is exactly what we expect from a default instance
876400	885030	of a network monitor. It's not going to alert on connections like this. Let me
885030	890370	enable the rules we've been talking about, and then we'll come back and we'll
890370	902560	try this again. And now that I've enabled the zero touch rules, let's try it again
902560	908060	and see what's changed. As you can see, the lab server is still emulating real
908060	912380	traffic by reaching out to the Puppet configuration server and to the external
912380	919480	internet. Let's go over to the Bastion and run our evil script again. Same thing,
919480	924300	utilizing those compromised SSH credentials to reach in and modify the
924300	932940	app server. Now we'll jump over to the network monitor, and as you can see, we're
932940	939880	starting to get alerts. Zeek has noticed a couple of connections from the Bastion
939880	949560	host to the app server. And if we switch over, you'll see the same thing in Squirt
949560	958180	as well. Notice how they work slightly differently. The Squirt gives us many
958180	963500	more alerts because it's looking individually at the packets traversing
963500	968660	the network, versus Zeek gives us total number of connections. But either way,
968660	973320	we're seeing the alerts that we're violating this zero-touch policy, and
973320	982980	that it's a good indicator of lateral movement. So you want to start doing this
982980	989200	in your environment as well. Well, first off, if you're not zero-touch yet, you
989200	994320	should move that direction. Implement zero-touch. It's got a lot of benefits
994320	1001160	besides this one. When you're ready to implement this detection, use your
1001160	1005720	platform of choice. Whatever you're already using is going to be best.
1005820	1011400	There's no need to go get specialized software or hardware in order to do this.
1011400	1019640	Use your existing IDS or network monitor. It's a very simple detection to include.
1020420	1026330	But then, also pay attention to how you tailor it for your specific environment.
1026870	1032250	In the example, it was looking for all TCP connections, because I wanted to catch
1032250	1037290	the broadest range of lateral movement. But that might not make sense for your
1037290	1042530	environment. So maybe you need to tailor it down to only look for SSH connections
1043390	1049190	or, you know, whatever makes sense for your environment specifically.
1050110	1055050	And then, the detections I showed were all very simple. They just looked for that
1055050	1061310	connection and alerted. But that can be just the beginning of a bigger process.
1061310	1067570	Imagine with Zeek, instead of raising a notice, if we correlated that with other
1067570	1072730	suspicious traffic that we'd seen, maybe some data exfiltration traffic,
1072730	1078690	and then we could automatically provide analysts with, here's the map of the
1078690	1084770	attacker's path through the network. Starting to add some of those higher-end
1084770	1091310	capabilities really makes this powerful. And finally, I wanted to leave you with
1091310	1095670	some lessons learned from this project. And there are two that really jumped out
1095670	1103830	at me. The first is know your network. And this is not just know your networking,
1103830	1110890	it is know your specific network. Justin Henderson and Mick Douglas, who teach the
1110890	1116150	Tactical SIEM class for SANS, really nailed this, and I learned this from them.
1116150	1121510	So, huge shout-out to them. Knowing the specifics of your network lets you do
1121510	1127090	really cool things in monitoring it. Things that aren't possible in generic
1127090	1134390	networks, such as the detection we talked about today. And then, secondly,
1134390	1139830	don't be afraid to look for stupid, simple things. All we were looking at today was
1139830	1146090	the presence of a connection. Tying together these two lessons learned is
1146090	1151670	really this whole project, is looking for something simple like the presence of a
1151670	1156870	connection and knowing your network well enough to be able to say,
1156870	1165250	that's unusual, that's a problem. Thank you so much for attending the
1165250	1169870	presentation today. Shout-out again to the SANS team for their help and support
1169870	1176530	with the research project. If you've got any questions about this project, AppSec,
1176530	1182150	DevOps, the SANS Masters program, or anything related, the best way to get in
1182150	1187450	contact with me is via Twitter. My handle's on the screen now. If you're
1187450	1190890	viewing the live presentation, I look forward to chatting with you during the
1190890	1197670	Q&A immediately afterwards in Discord. Again, I'm Philip Marlowe. Thanks for
1197670	1198330	coming.