Welcome to the talk on detecting lateral movement in zero-touch environments. We're going to dive
into how that lateral movement works, how to detect it, and why you really want a zero-touch
environment. So before we jump in too much, some disclaimers and acknowledgments. First off,
I work for the MITRE Corporation, but I am not representing or speaking for them today.
All of the research done in this talk was performed in the pursuit of a master's degree
through SANS, and so thank you to them for supporting the research, and thank you to my
advisor, but I'm also not speaking for them today. And lastly, thank you to my wife Madeline,
whom I'm also not speaking for today. So apart from all of that, who am I?
My happy place professionally is in the middle of security and DevOps. And that has quite naturally
led me to doing a lot of work with the cloud, automating not just applications and operations,
but the security portions of that. A little bit of background on me. I wrote my first
vulnerable code in elementary school. I was writing a blogging platform for my family,
and they were quite surprised when it was hacked through a SQL injection that I inadvertently
added. I began learning to write exploit code in middle school, when I received as a present
the book Hacking the Art of Exploitation. And I've been involved in security one way or another
ever since. This is my first time speaking at DEF CON, and I am super excited that the
Cloud Village has me here. And one last little kind of personal philosophy thing is I really
enjoy learning through hacking. I imagine a lot of you do, too. And you'll definitely see that
flavor as we go through the presentation today of looking at things from both an attacker's
point of view and a defender's point of view. So why should I care about DevOps if you're
not already sold? Well, if you're running any applications or any services,
you've got to deal with it. That's the way it is now. Welcome to the world. However,
that's true regardless of whether you're hosting applications for the outside world,
just for yourself, whoever it is. If you're a red team and you need to spin up infrastructure to
support a penetration test or something like that, DevOps to the rescue. Blue team, you're
trying to keep continuous monitoring and alerting of what's going on in your environment, DevOps.
Application owner, trying to make sure your application is stable and available and secure,
DevOps. Another great perk for us is that DevOps is essentially cloud native now. All of the major
cloud providers have offered ways to automate their environments. And so being able to develop
infrastructure as code, applications as code, security compliance as code, monitoring as code,
all of these are really enabled by the cloud. They can be done in other environments,
but the cloud makes it easy. And lastly, DevOps is better for security. We're integrating security
earlier in the process, rather than trying to layer it on later.
So I promised an attacker's point of view. Today, we're going to look at kind of a simplified
example environment. I'm sure, you know, you've seen a bunch of other presentations that have got
the hard to read, squint and see, that's AWS, right? Not Azure. We're simplifying things for
this talk. You're somewhere out on the internet, and you really want to get to an application server.
Might want to get in and then exfil some of the sensitive data on it, you might want to get in
and use their computing resources to mine cryptocurrency. Whatever the reason is,
you're going for that application server. So there's three main options we're going to look at.
First is straight through the front door, directly in from the internet hitting the application
server. Pretty simple. Yeah, there's maybe a firewall or an IDS or something that you have
to evade. But overall, pretty direct, pretty simple. Except it's not 2003 anymore, and we've
gotten better at protecting our applications and the servers they run on. So while it's still
possible, and there are plenty of big, well-known events where it's happened, it's not as popular or
as reliable as it used to be. So that brings us to our second option, and that's, let's compromise
the workstation of an employee, and then move laterally to the application server we really
want to be on. For the purposes of this presentation, I'm going to say we phished that
employee, but it could be any number of things. From there, most organizations have a bastion host
that employees have to log into before they can get into their production environment. That's a
pretty common practice, and so we see that on the screen. Our second option is compromise an employee
workstation, use valid credentials that that employee has access to, move through the bastion
host to the application server. And then our third option is to take the route that DevOps uses to
publish applications itself. From that employee workstation, push something into the source code
repo, get it through the test servers undetected, have the configuration server deploy it for us.
Or maybe we can compromise each of those servers in turn in order to get there in the end.
In any case, that's a lot more hops to have to go through and a lot more chances to be detected
or fail in some other way. So we're going to focus on detecting number two, both because it's pretty
common and because that's the easiest route that attackers have, and so that's most likely what
they're going to go for. And this being common is borne out by repeated studies. We've seen
reports from FireEye and the Verizon data breach report say that social engineering and then using
valid credentials is incredibly popular for malicious actors right now. So what does it look
to do a traditional application deployment from that employee's point of view? Developers give
ops a deployment package and hopefully some install instructions, and then ops logs into
the app server through the bastion host and manually installs the software. Pretty simple.
And then you get a patch. Well, you've got to log in and install the patch.
There's an update to one of the dependencies. We've got to log in and install that.
Anytime anything changes, someone's logging into that application server.
So that's, you know, pretty normal, but not ideal, and we'll get into that.
Traditional lateral movement takes advantage of that fact. Because you've got people logging in
to the server manually, they have highly privileged credentials. And very often,
those credentials are stored in plain text on their employee workstations. Exactly what we
were talking about compromising earlier. That might take the form of SSH keys,
it might take the form of API tokens or keys. And, you know, especially relevant for the cloud
village, that's the way AWS stores local credentials. Plain text file in that path on the
screen. Attackers then use those plain text credentials from the station they've already
compromised to move further into your environment and do whatever evil they want to get up to.
They've hit their target. So what is ZeroTouch and why does it apply to any of this?
So Google are the ones who have defined ZeroTouch networking and ZeroTouch production concepts.
But they're far from the only people who use them. This is a technique used by many mature
DevOps organizations to automate as much as possible and get humans out of their
production environments. What does that look like? Here's the three rules that Google use
to define their ZeroTouch production. And the first one is really big. Every change
is made by automation. No humans are involved. By getting that human traffic out of the environment,
we can do a lot of really interesting but simple things to detect when attackers are trying to
pivot into our protected environment. The second rule, pre-validated by software, is important
because if we try to go to that attack path number three, the one that uses the DevOps pipeline
itself, that means we're checking to see if there's anything malicious before we deploy.
Now, that doesn't always work. But it gives us as defenders a better opportunity than someone
who's just pivoting through the bastion host and hiding in the noise. And then lastly,
emergency changes can be made via a break glass mechanism. But they have to be auditable.
We'll come back to that. And what this looks like in our picture is essentially disallowing
communication from that bastion host to the application server. And all normal traffic is
going through that path number three where it goes through the source repo and the test servers
and the configuration servers. So we've got good control over what's being deployed to the
application server and good confidence that it's not malicious. All right. So now let's look at
lateral movement in each of these two types of environments. We've compromised our employee
workstation, found some valid credentials that we can use to pivot. And in this case, we're going
to look at the traditional path where operators are regularly logging into the environment to
deploy patches, update software, any of those kinds of things. I'm about to play an animation.
And there are going to be several connections shown on the screen by arrows.
Try and find the one that looks a little bit different than all the rest.
This is going to be our malicious lateral movement. Did you see it?
Have you seen it yet? It's a little bit tricky to find. And this is assuming that it stands out at
all. If you're using an encrypted communication channel like SSH or HTTPS, which are super
common, this use of valid credentials might not stand out in any way. Now let's compare that to
someone trying to use lateral movement on a zero touch network. Same thing. I'm going to display
several arrows in an animation and see if you can pick out the one that looks a little bit different
and indicates our malicious lateral movement. I'll bet that was a lot easier. And you maybe even
spotted it before the animation finished playing. In this case, it stands out like a sore thumb
because those valid credentials, even if the channel is encrypted, are going an entirely
different route through the network than normal, regular, valid traffic would. So how do we go
about detecting that from a programmatic and automatic standpoint? First, we need to define
the protected servers. In the case of our example, that's just the application server. But it could
be a lot of things. Could be application servers, web servers, the DevOps servers themselves.
Anything that humans shouldn't be logging into on a regular basis. Then we need to define all
of the human access points into the environment. Again, in this example, that's just the bastion
host. But depending on your network architecture, there might be any number of things that you need
to put into that list. And then watch for any connections made from those manual access points
to your protected servers. Real simple, but very effective if it's not supposed to be happening.
At that point, you can alert, investigate, whatever your normal process is, and enjoy having detected
malicious lateral movement. Now, I mentioned earlier that auditing was important. And the
reason that this is still a valid attack path into the environment is most organizations can't
get rid of their bastion hosts entirely. They're still needed for incident response or forensics,
some kind of unusual emergency purpose. And so, even though that's valid traffic, this method
would still alert on it. However, that's not a problem, because anytime that emergency traffic
does occur, we wanted it to be auditable anyway. And this is helpfully generating those audit alerts
for us. So, now we're going to cross our fingers and see if the live demo goes well.
What I've got up on the screen are three lab machines. One, a bastion host,
one, our application server, and one, a puppet configuration server.
What I'm going to do, just to prove to you all that this is actually working,
and that I've got nothing up my sleeves today, is we're going to put some valid traffic into
the environment. What this puppet command will do is generate traffic between the puppet configuration
server and the application server, as well as traffic out to the wider internet over both SSH
and HTTPS. And we'll keep that going throughout the rest of the demonstration.
Then, we'll go over to the bastion host and try to execute our... let me fix that real quick.
Then we'll go over to the bastion host and... nope. My routes are wrong today,
so I'm going to play the prerecorded version of the demo instead.
All right, but you'll get all the same information here. So, for the demo, we're going to look at a
couple of scenarios. First up, let me tell you what you're seeing. In this environment, I've got
a bastion host that's used for manually accessing it when needed, an application server, and a
configuration server running puppet. So, just to make sure that there's nothing up my sleeves,
what we're going to do for this demo is run some puppet configuration in the background.
This will be traffic both within the environment to the puppet server, and external to the
environment SSH connections and HTTPS connections out to the wider internet. So, let's get that
started. And then, while this runs, jump over to the bastion host, and as you can see, we've already
compromised the bastion host. And we see some SSH credentials that we found, and in fact, some SSH
credentials for the application server that has the goodies we want to get to. So, we'll run our evil
script file. And, you know, in real life, this might be installing a crypto miner or any number of things.
In this case, just modifying the configuration files that Puppet is managing to show that we've
actually gotten in and had the ability to do evil if we wanted to. So, now we're going to jump over to
the network monitor. This is a plain instance of Security Onion with all of the usual defaults and
alerts installed. And what we're looking at right now are the ZIP notices. And let me turn on the
refresh. But you'll see a whole lot of nothing. Zero. No results found. No results found. If we
switch over to Squirt and refresh it, same thing. No result. No result. And instance of a network
monitor. It's not going to alert on connections like this. Let me enable the rules we've been
talking about. And then we'll come back and we'll try this again. And now that I've enabled the
zero-touch rules, let's try it again and see what's changed. As you can see, the lab server
is still emulating real traffic by reaching out to the Puppet configuration server and to the
external internet. Let's go over to the Bastion and run our evil script again. Same thing, utilizing
those compromised SSH credentials to reach in and modify the app server. Now we'll jump over to
the network monitor. And as you can see, we're starting to get alerts. Zeke has noticed
a couple of connections from the Bastion host to the app server. And if we switch over,
you'll see the same thing in Squirt as well. Notice how they work slightly differently.
The Squirt gives us many more alerts because it's looking individually at the packets traversing
the network versus Zeke gives us total number of connections. But either way, we're seeing
the alerts that we're violating this zero-touch policy and that it's a good indicator of lateral
movement. All right, so you can see from the demo, you know, this isn't something that adds to your
ability to detect lateral movement as long as you conform to the expectations of the environment.
Demonstrated here with Zeke and Snort, but there's no reason you can't use
any of your tools of choice. But this is the cloud village. So what about cloud?
Totally still doable in the cloud, despite the fact I use VMware for my lab environment.
If you're not familiar, all the major cloud providers provide some form of traffic mirroring,
whatever they call that, traffic mirroring in AWS, virtual network tap in Azure,
and packet mirroring for Google Cloud Platform. And that'll give you full network visibility.
You can install something just like Security Onion or your favorite network monitor
feeding directly off of that. You can also install your own within the environment. And
because this is looking for such simple things as connections, we don't even need the full
traffic in order to tell when this activity is occurring. We can just use flow logs. And so
that'll cut down on the amount of traffic that we have to capture if we go that route. Now,
given that the cloud is a much more dynamic environment than traditional data centers,
this means that tagging an asset inventory is really important to get right.
You have to know what those protected servers are and what the human access points are in order to
detect connections between them. And if you've got automatic scaling or something like that,
keeping those lists up to date can be a little bit tricky, but it's definitely doable,
especially since those lists in most tools can be entire subnets rather than specific machines.
But there are also some challenges with the cloud. And the big one is visibility into
management traffic. With something like Puppet or Ansible, your traditional configuration servers,
that's all going over a network that you can view. However, in the cloud,
some of your management traffic goes over the management plane in the background.
And these features like traffic mirroring are not able to capture that data for you.
It all happens in AWS or in Azure without you being able to see it.
And that poses a visibility challenge. Doesn't mean that it's not worth doing,
but you have to understand how your environment usually works in order to be able to protect it.
So, some next steps if you're sold on this idea. First off, if you're not Zerotouch yet,
do it. It's got a lot of benefits aside from this particular detection. It's hard to do,
and it's hard to get right, but it is worth it. And then you can implement this detection on
your platform of choice. I showed off Snort and Zeek here, but it's very simple. There's no need
and buy specialized software, buy specialized hardware,
install anything you're probably not already using anyway.
And then when you do implement this detection, tailor it to your specific environment. Here I
was looking for any connections between the human access points and our protected servers. But
your environment may or may not be able to support that depending on its specific architecture.
The good news is that this approach is valid for a lot of different options, and you just need to
figure out which ones work specifically for you. You might only turn it on for SSH connections.
You might only look at particular areas of your network that are high risk.
However you do it, make sure you tailor it, because that'll give you the most value.
And then there are ways to extend this work far beyond what I demonstrated in today's video.
For example, correlate it with other events that are generated off of suspicious traffic.
Imagine tying this alert about potential lateral movement with potential data exfiltration that's
generated by Zeke, and then presenting your security team with a map of how the attacker
moved through your entire network. It's pretty powerful once you start combining it with other
things. Lastly, I want to leave you with some lessons learned from doing this research project.
First off, you have to know your network. That's the basis of this entire project,
is you have to know your network well enough to know what's normal and what's abnormal.
And then putting in things to look for abnormalities specific to your environment
as opposed to things that apply to everyone's environment.
I learned this principle first from Mick Douglas and Justin Henderson, who teach the
sans-tactical-seam class. And it really resonated, it was really powerful in being able to do things
that don't come pre-built with your software, because you're looking for things specific to you.
And then second, don't be afraid to look for stupid simple things on your network.
All we're looking for in this is the presence of a connection.
But knowing our environment the way we do, that lets us say there's something weird going on,
I need to investigate. And that is super valuable.
So, thank you. If you have any questions after the talk and the Q&A is done,
best place to reach me is on Twitter. My handle's on the screen right now.
And then we'll see if we've got any questions live.
Do you happen to know any tools that might help implement this kind of a
zero-touch environment or getting to know your network better or something along those lines?
Sure. So, getting to know your network better, you're just going to have to get familiar with
how these operate. And my biggest recommendation is if you're starting to do DevOps as a security
person, is make time to spend with the operations and the developers. Because they're going to know
how things normally go. And you'll be able to learn from them what they expect. And then use
your knowledge of security to figure out what shouldn't be going on. In terms of tools to help
implement this, you know, typical DevOps tools, there's a periodic table of DevOps that's out
there, I think by Xenia Labs, that shows a lot of the different options. For the demo today, I was
using Puppet as a configuration server to manage the configuration of the application server and
making sure that it had the files and configuration settings that I expected. There are a lot of them,
apart from Puppet, Ansible, SaltStack, things of that nature. And then for building out a
whole DevOps pipeline, look at that periodic table. You'll need several different things
along there to handle your source code and your testing and things like that.