Hey AppSecVillage, welcome back and get ready for another talk. Coming up next
we've got Can't Touch This! Detecting Lateral Movement in Zero-Touch
Environments. Philip Marlowe is going to be presenting this. Philip is a
cybersecurity and DevOps engineer. He helps organizations understand how to
adopt DevOps practices to increase their security rather than sacrifice it in
the name of speed. Philip holds several security, cloud, and agile certifications
and is currently pursuing a master's degree in information security
engineering at SANS Technology Institute. Please give it up for Philip Marlowe.
Hello everyone and welcome to Can't Touch This! Detecting Lateral Movement in
Zero-Touch Environments. In this talk we're going to talk some about DevOps
and how to detect attackers trying to move into your production environments.
Before we get started, a few disclaimers and acknowledgments. First off, I work for
the MITRE Corporation, but nothing I'm saying today reflects their opinion.
Also, I'm a student at the SANS Technology Institute working on my
master's degree. Huge shout out to my faculty research advisor, but I'm not
speaking for her or SANS today either. And lastly, big thank you to my wife
Madeline. Once again, not speaking for her in this presentation today.
So who am I? I'm Philip and professionally I really enjoy the intersection of
security and DevOps. Yes, you can call it DevSecOps if you want. I don't really
care either way. I wrote my first vulnerable code in elementary school.
I was writing a blogging platform for my family and right in there, because I
didn't know any better, put a SQL injection vulnerability. That was a big
surprise for them when they found out we've been hacked because of that. I
first began learning to exploit applications in middle school when I
received as a present the book, Hacking the Art of Exploitation. And I've been
involved in application security one way or another ever since then. Today is
my first time as a DEF CON speaker, and I'm super excited to be here. And one of
the things I really enjoy is being able to learn through hacking. And you'll see
a flavor of that throughout this presentation as we take both the
attacker and the defender point of view. So if you're not already on board, why
should I care about DevOps? Well, for one, you probably don't have any choice. If
you're running any kind of application, that's the way it is now. You probably
just need to learn about it. You know, if your red team standing up command and
control infrastructure, helps to have DevOps. Your blue team trying to
implement monitoring and detection helps to have DevOps. Application security
DevOps is the way of life now. And, you know, provides all these security
benefits on top of reliability benefits and maintainability benefits for the
other parts of the business. So let's take a look at a very simplified
environment, and let's take a look at it from the attackers point of view. We want
to get to something on the application server. Maybe we want to steal data off
of it, or maybe we just want to use the hefty server power for crypto mining.
Either way, we've got three basic options. Option one is to just go straight in the
front door and attack the application itself in order to get to the access to
the server. This is pretty simple. Maybe there's a firewall or IDS you have
to evade, but, you know, you're not making a lot of hops around to finally get to
where you want to be. But it's also not 2002 anymore. The situation's gotten a
lot better in how we protect our application servers. So while this once
might have been the best way in, it's not really anymore. Report after report from
places like FireEye and Verizon have shown a much more popular way into
networks these days is by social engineering users in one way or another.
In this case, we're going to take a look at phishing a operator or developer on
their employee workstation to give us our initial access. Then in method two,
we're going to move laterally through the bastion host and finally to the app
server. We'll talk in a couple of slides about why this works so well. But before
we get there, let's talk a little bit about option three. With DevOps
implemented these days, another way into that protected app server is through the
DevOps pipeline itself. But there are a lot more hops here and a lot more ways
that can go wrong and a lot more opportunities to detect this attempt. It
goes through source control repos, test servers, and configuration servers before
we ever get to our intended target. So how does traditional application
deployment work? Well, developers give Ops a deployment package, and if we're
lucky, some install instructions. And then Ops is going to go and log into the
application server, manually install that software. Pretty simple. And then an
update comes along, and you have to go and manually log in and update. And then
the security patch comes along, and you have to go log in and update. And this
manual logging in and updating is going to be a real problem for detecting
lateral movement. So what does traditional lateral movement look like?
Because they're logging in to do configuration and updates all the time,
Ops has highly privileged credentials. This might take the form of SSH keys, or
API tokens. And oftentimes, they're stored in plain text on their employee
workstations. If you've already phished that user, those are really easy and
really juicy targets to steal those credentials and move further into the
environment. All right, so that sounds like it's a problem. What is this zero
touch concept? Google defined zero touch networking and zero touch production. And
they've given several presentations on what that means to them and how they've
implemented it in their environment. Overall, it's about how we can abstract
away managing the production environment, so that we're not doing those manual
logins, change things, configure and update anymore. And while Google is one
of the only companies to use the zero touch nomenclature, a lot of companies do
this. Often, it's just a result of being very mature in their DevOps practices,
that everything is handled by automation, rather than human users. So zero touch
deployment looks very different. Every change is made by automation and
pre-validated. This means no humans logging in. And that's key for our talk
today. You'll also notice on Google's presentation that there is an exception.
And that's when something is made by an audited break glass mechanism. We'll talk
a little bit more about that later as well. So what is the zero touch deployment
look like in practice? Well, rather than having a user log into the app server via
a bastion host, since that's no longer allowed, every change has to go through
that DevOps deployment pipeline. Looking at how attackers are going to move
laterally, however, let's compare and contrast between the traditional and the
zero touch deployments. So I'm about to play an animation, and keep an eye out
for the malicious connection between the bastion host and the app server. It's
going to be a slightly different colored arrow. Did you catch it? Have you seen it
yet? It's kind of tricky to find. And this is when it stands out in some way. If
we're looking at encrypted connections like SSH or HTTPS, it may not stand out
in any way, unless you're doing some kind of break and inspect in the middle. Now
let's compare it to zero touch. In the zero touch deployment, you're going to see
a bunch of arrows again, showing normal benign traffic. And you'll also see one
arrow that'll look a little different, being our attacker trying to move
laterally. See if it's easier to spot this time. I'll bet that was a lot easier,
and as soon as it came up, you noticed it. This is the whole concept that we want
to use writing this detection. So how do we actually go about detecting this
lateral movement? There's a few things we need to do. First, we need to define
all of the protected servers. In our simplified environment, that's just the
app server. But it might also include DevOps automation servers, it might
include web servers, database servers, all of which might be behind load balancers.
Whatever humans should not be logging into, that goes in this list. We also
need to define all of the human access points into the environment. Again, in our
simplified environment, that's just the bastion host. But depending on your
network architecture, you might need to include other sets of load balance
machines, virtual desktops, employee workstations, anything of that nature.
That goes in the second list. And then all we do to detect is look for any
connections between those two sets of servers. Then alert, investigate, enjoy
detecting this lateral movement. Let's take a quick look at what this looks
like in practice. So for the demo, we're going to look at a couple of scenarios.
First up, let me tell you what you're seeing. In this environment, I've got a
bastion host that's used for manually accessing it when needed, an application
server, and a configuration server running Puppet. So just to make sure that
there's nothing up my sleeves, what we're going to do for this demo is run some
Puppet configuration in the background. This will be traffic both within the
environment to the Puppet server and external to the environment, SSH
connections and HTTPS connections out to the wider internet. So let's get that
started. And then while this runs, jump over to the bastion host. And as you can
see, we've already compromised the bastion host. And we see some SSH
credentials that we found, and in fact, some SSH credentials for the application
server that has the goodies we want to get to. So we'll run our evil script file.
And, you know, in real life, this might be installing a crypto miner or any number
of things. In this case, just modifying the configuration files that Puppet is
managing to show that we've actually gotten in and had the ability to do evil
if we wanted to. So now we're going to jump over to the network monitor. This is
a plain instance of Security Onion with all of the usual defaults and alerts
installed. And what we're looking at right now are the Zeek notices. And let me
turn on the refresh. But you'll see a whole lot of nothing. Zero. No results
found. No results found. If we switch over to squirt and refresh it, same thing. No
result. No result. And this is exactly what we expect from a default instance
of a network monitor. It's not going to alert on connections like this. Let me
enable the rules we've been talking about, and then we'll come back and we'll
try this again. And now that I've enabled the zero touch rules, let's try it again
and see what's changed. As you can see, the lab server is still emulating real
traffic by reaching out to the Puppet configuration server and to the external
internet. Let's go over to the Bastion and run our evil script again. Same thing,
utilizing those compromised SSH credentials to reach in and modify the
app server. Now we'll jump over to the network monitor, and as you can see, we're
starting to get alerts. Zeek has noticed a couple of connections from the Bastion
host to the app server. And if we switch over, you'll see the same thing in Squirt
as well. Notice how they work slightly differently. The Squirt gives us many
more alerts because it's looking individually at the packets traversing
the network, versus Zeek gives us total number of connections. But either way,
we're seeing the alerts that we're violating this zero-touch policy, and
that it's a good indicator of lateral movement. So you want to start doing this
in your environment as well. Well, first off, if you're not zero-touch yet, you
should move that direction. Implement zero-touch. It's got a lot of benefits
besides this one. When you're ready to implement this detection, use your
platform of choice. Whatever you're already using is going to be best.
There's no need to go get specialized software or hardware in order to do this.
Use your existing IDS or network monitor. It's a very simple detection to include.
But then, also pay attention to how you tailor it for your specific environment.
In the example, it was looking for all TCP connections, because I wanted to catch
the broadest range of lateral movement. But that might not make sense for your
environment. So maybe you need to tailor it down to only look for SSH connections
or, you know, whatever makes sense for your environment specifically.
And then, the detections I showed were all very simple. They just looked for that
connection and alerted. But that can be just the beginning of a bigger process.
Imagine with Zeek, instead of raising a notice, if we correlated that with other
suspicious traffic that we'd seen, maybe some data exfiltration traffic,
and then we could automatically provide analysts with, here's the map of the
attacker's path through the network. Starting to add some of those higher-end
capabilities really makes this powerful. And finally, I wanted to leave you with
some lessons learned from this project. And there are two that really jumped out
at me. The first is know your network. And this is not just know your networking,
it is know your specific network. Justin Henderson and Mick Douglas, who teach the
Tactical SIEM class for SANS, really nailed this, and I learned this from them.
So, huge shout-out to them. Knowing the specifics of your network lets you do
really cool things in monitoring it. Things that aren't possible in generic
networks, such as the detection we talked about today. And then, secondly,
don't be afraid to look for stupid, simple things. All we were looking at today was
the presence of a connection. Tying together these two lessons learned is
really this whole project, is looking for something simple like the presence of a
connection and knowing your network well enough to be able to say,
that's unusual, that's a problem. Thank you so much for attending the
presentation today. Shout-out again to the SANS team for their help and support
with the research project. If you've got any questions about this project, AppSec,
DevOps, the SANS Masters program, or anything related, the best way to get in
contact with me is via Twitter. My handle's on the screen now. If you're
viewing the live presentation, I look forward to chatting with you during the
Q&A immediately afterwards in Discord. Again, I'm Philip Marlowe. Thanks for
coming.