[00:13.790 --> 00:17.650]  Hey AppSecVillage, welcome back and get ready for another talk. Coming up next
[00:17.650 --> 00:20.570]  we've got Can't Touch This! Detecting Lateral Movement in Zero-Touch
[00:20.570 --> 00:25.510]  Environments. Philip Marlowe is going to be presenting this. Philip is a
[00:25.510 --> 00:29.430]  cybersecurity and DevOps engineer. He helps organizations understand how to
[00:29.430 --> 00:34.070]  adopt DevOps practices to increase their security rather than sacrifice it in
[00:34.070 --> 00:38.990]  the name of speed. Philip holds several security, cloud, and agile certifications
[00:38.990 --> 00:41.570]  and is currently pursuing a master's degree in information security
[00:41.570 --> 00:46.890]  engineering at SANS Technology Institute. Please give it up for Philip Marlowe.
[00:47.230 --> 00:52.290]  Hello everyone and welcome to Can't Touch This! Detecting Lateral Movement in
[00:52.290 --> 00:56.250]  Zero-Touch Environments. In this talk we're going to talk some about DevOps
[00:56.250 --> 01:01.710]  and how to detect attackers trying to move into your production environments.
[01:03.570 --> 01:09.490]  Before we get started, a few disclaimers and acknowledgments. First off, I work for
[01:09.490 --> 01:15.810]  the MITRE Corporation, but nothing I'm saying today reflects their opinion.
[01:16.590 --> 01:20.290]  Also, I'm a student at the SANS Technology Institute working on my
[01:20.290 --> 01:26.210]  master's degree. Huge shout out to my faculty research advisor, but I'm not
[01:26.210 --> 01:32.550]  speaking for her or SANS today either. And lastly, big thank you to my wife
[01:32.550 --> 01:37.150]  Madeline. Once again, not speaking for her in this presentation today.
[01:38.910 --> 01:46.130]  So who am I? I'm Philip and professionally I really enjoy the intersection of
[01:46.130 --> 01:52.150]  security and DevOps. Yes, you can call it DevSecOps if you want. I don't really
[01:52.150 --> 01:58.090]  care either way. I wrote my first vulnerable code in elementary school.
[01:58.090 --> 02:03.970]  I was writing a blogging platform for my family and right in there, because I
[02:03.970 --> 02:08.270]  didn't know any better, put a SQL injection vulnerability. That was a big
[02:08.270 --> 02:14.170]  surprise for them when they found out we've been hacked because of that. I
[02:14.170 --> 02:19.030]  first began learning to exploit applications in middle school when I
[02:19.030 --> 02:23.810]  received as a present the book, Hacking the Art of Exploitation. And I've been
[02:23.810 --> 02:30.130]  involved in application security one way or another ever since then. Today is
[02:30.130 --> 02:36.570]  my first time as a DEF CON speaker, and I'm super excited to be here. And one of
[02:36.570 --> 02:41.690]  the things I really enjoy is being able to learn through hacking. And you'll see
[02:41.870 --> 02:45.770]  a flavor of that throughout this presentation as we take both the
[02:45.770 --> 02:54.920]  attacker and the defender point of view. So if you're not already on board, why
[02:54.920 --> 03:02.680]  should I care about DevOps? Well, for one, you probably don't have any choice. If
[03:02.680 --> 03:08.040]  you're running any kind of application, that's the way it is now. You probably
[03:08.040 --> 03:14.180]  just need to learn about it. You know, if your red team standing up command and
[03:14.180 --> 03:19.000]  control infrastructure, helps to have DevOps. Your blue team trying to
[03:19.000 --> 03:23.920]  implement monitoring and detection helps to have DevOps. Application security
[03:23.920 --> 03:33.680]  DevOps is the way of life now. And, you know, provides all these security
[03:33.680 --> 03:40.520]  benefits on top of reliability benefits and maintainability benefits for the
[03:40.520 --> 03:48.080]  other parts of the business. So let's take a look at a very simplified
[03:48.080 --> 03:53.920]  environment, and let's take a look at it from the attackers point of view. We want
[03:53.920 --> 03:59.300]  to get to something on the application server. Maybe we want to steal data off
[03:59.300 --> 04:05.800]  of it, or maybe we just want to use the hefty server power for crypto mining.
[04:06.440 --> 04:13.060]  Either way, we've got three basic options. Option one is to just go straight in the
[04:13.060 --> 04:16.900]  front door and attack the application itself in order to get to the access to
[04:16.900 --> 04:23.180]  the server. This is pretty simple. Maybe there's a firewall or IDS you have
[04:23.180 --> 04:28.220]  to evade, but, you know, you're not making a lot of hops around to finally get to
[04:28.220 --> 04:36.300]  where you want to be. But it's also not 2002 anymore. The situation's gotten a
[04:36.300 --> 04:41.200]  lot better in how we protect our application servers. So while this once
[04:41.200 --> 04:48.140]  might have been the best way in, it's not really anymore. Report after report from
[04:48.140 --> 04:54.260]  places like FireEye and Verizon have shown a much more popular way into
[04:54.260 --> 05:00.940]  networks these days is by social engineering users in one way or another.
[05:00.940 --> 05:09.980]  In this case, we're going to take a look at phishing a operator or developer on
[05:09.980 --> 05:15.300]  their employee workstation to give us our initial access. Then in method two,
[05:15.940 --> 05:21.220]  we're going to move laterally through the bastion host and finally to the app
[05:21.220 --> 05:30.360]  server. We'll talk in a couple of slides about why this works so well. But before
[05:30.360 --> 05:35.080]  we get there, let's talk a little bit about option three. With DevOps
[05:35.080 --> 05:40.780]  implemented these days, another way into that protected app server is through the
[05:40.780 --> 05:46.240]  DevOps pipeline itself. But there are a lot more hops here and a lot more ways
[05:46.240 --> 05:54.880]  that can go wrong and a lot more opportunities to detect this attempt. It
[05:54.880 --> 06:00.700]  goes through source control repos, test servers, and configuration servers before
[06:00.700 --> 06:10.540]  we ever get to our intended target. So how does traditional application
[06:10.540 --> 06:17.160]  deployment work? Well, developers give Ops a deployment package, and if we're
[06:17.160 --> 06:22.100]  lucky, some install instructions. And then Ops is going to go and log into the
[06:22.100 --> 06:28.040]  application server, manually install that software. Pretty simple. And then an
[06:28.040 --> 06:34.340]  update comes along, and you have to go and manually log in and update. And then
[06:34.340 --> 06:40.320]  the security patch comes along, and you have to go log in and update. And this
[06:40.320 --> 06:46.520]  manual logging in and updating is going to be a real problem for detecting
[06:46.520 --> 06:52.540]  lateral movement. So what does traditional lateral movement look like?
[06:54.840 --> 06:59.780]  Because they're logging in to do configuration and updates all the time,
[06:59.780 --> 07:07.420]  Ops has highly privileged credentials. This might take the form of SSH keys, or
[07:07.420 --> 07:14.800]  API tokens. And oftentimes, they're stored in plain text on their employee
[07:14.800 --> 07:21.420]  workstations. If you've already phished that user, those are really easy and
[07:21.420 --> 07:26.980]  really juicy targets to steal those credentials and move further into the
[07:26.980 --> 07:36.590]  environment. All right, so that sounds like it's a problem. What is this zero
[07:36.590 --> 07:44.310]  touch concept? Google defined zero touch networking and zero touch production. And
[07:44.310 --> 07:49.270]  they've given several presentations on what that means to them and how they've
[07:49.270 --> 07:56.910]  implemented it in their environment. Overall, it's about how we can abstract
[07:56.910 --> 08:05.050]  away managing the production environment, so that we're not doing those manual
[08:05.050 --> 08:12.950]  logins, change things, configure and update anymore. And while Google is one
[08:12.950 --> 08:18.030]  of the only companies to use the zero touch nomenclature, a lot of companies do
[08:18.030 --> 08:24.630]  this. Often, it's just a result of being very mature in their DevOps practices,
[08:24.630 --> 08:33.710]  that everything is handled by automation, rather than human users. So zero touch
[08:33.710 --> 08:40.010]  deployment looks very different. Every change is made by automation and
[08:40.010 --> 08:47.090]  pre-validated. This means no humans logging in. And that's key for our talk
[08:47.090 --> 08:53.570]  today. You'll also notice on Google's presentation that there is an exception.
[08:53.570 --> 08:58.610]  And that's when something is made by an audited break glass mechanism. We'll talk
[08:58.730 --> 09:05.970]  a little bit more about that later as well. So what is the zero touch deployment
[09:05.970 --> 09:13.530]  look like in practice? Well, rather than having a user log into the app server via
[09:13.710 --> 09:18.430]  a bastion host, since that's no longer allowed, every change has to go through
[09:18.430 --> 09:29.000]  that DevOps deployment pipeline. Looking at how attackers are going to move
[09:29.000 --> 09:35.380]  laterally, however, let's compare and contrast between the traditional and the
[09:35.380 --> 09:43.180]  zero touch deployments. So I'm about to play an animation, and keep an eye out
[09:43.180 --> 09:49.440]  for the malicious connection between the bastion host and the app server. It's
[09:49.440 --> 09:58.480]  going to be a slightly different colored arrow. Did you catch it? Have you seen it
[09:58.480 --> 10:05.340]  yet? It's kind of tricky to find. And this is when it stands out in some way. If
[10:05.340 --> 10:12.420]  we're looking at encrypted connections like SSH or HTTPS, it may not stand out
[10:12.420 --> 10:18.060]  in any way, unless you're doing some kind of break and inspect in the middle. Now
[10:18.060 --> 10:26.740]  let's compare it to zero touch. In the zero touch deployment, you're going to see
[10:26.860 --> 10:33.140]  a bunch of arrows again, showing normal benign traffic. And you'll also see one
[10:33.140 --> 10:36.900]  arrow that'll look a little different, being our attacker trying to move
[10:36.900 --> 10:44.200]  laterally. See if it's easier to spot this time. I'll bet that was a lot easier,
[10:44.200 --> 10:49.280]  and as soon as it came up, you noticed it. This is the whole concept that we want
[10:49.280 --> 10:55.740]  to use writing this detection. So how do we actually go about detecting this
[10:55.740 --> 11:00.920]  lateral movement? There's a few things we need to do. First, we need to define
[11:00.920 --> 11:06.120]  all of the protected servers. In our simplified environment, that's just the
[11:06.120 --> 11:10.980]  app server. But it might also include DevOps automation servers, it might
[11:10.980 --> 11:16.520]  include web servers, database servers, all of which might be behind load balancers.
[11:16.600 --> 11:22.820]  Whatever humans should not be logging into, that goes in this list. We also
[11:22.820 --> 11:27.640]  need to define all of the human access points into the environment. Again, in our
[11:27.640 --> 11:31.800]  simplified environment, that's just the bastion host. But depending on your
[11:31.800 --> 11:36.520]  network architecture, you might need to include other sets of load balance
[11:36.520 --> 11:42.060]  machines, virtual desktops, employee workstations, anything of that nature.
[11:42.060 --> 11:47.940]  That goes in the second list. And then all we do to detect is look for any
[11:47.940 --> 11:56.580]  connections between those two sets of servers. Then alert, investigate, enjoy
[11:56.580 --> 12:05.080]  detecting this lateral movement. Let's take a quick look at what this looks
[12:05.080 --> 12:16.860]  like in practice. So for the demo, we're going to look at a couple of scenarios.
[12:17.100 --> 12:23.560]  First up, let me tell you what you're seeing. In this environment, I've got a
[12:23.560 --> 12:28.440]  bastion host that's used for manually accessing it when needed, an application
[12:28.440 --> 12:34.340]  server, and a configuration server running Puppet. So just to make sure that
[12:34.340 --> 12:39.880]  there's nothing up my sleeves, what we're going to do for this demo is run some
[12:39.880 --> 12:46.160]  Puppet configuration in the background. This will be traffic both within the
[12:46.160 --> 12:50.080]  environment to the Puppet server and external to the environment, SSH
[12:50.080 --> 12:55.300]  connections and HTTPS connections out to the wider internet. So let's get that
[12:55.300 --> 13:04.100]  started. And then while this runs, jump over to the bastion host. And as you can
[13:04.100 --> 13:08.660]  see, we've already compromised the bastion host. And we see some SSH
[13:08.660 --> 13:13.860]  credentials that we found, and in fact, some SSH credentials for the application
[13:13.860 --> 13:23.080]  server that has the goodies we want to get to. So we'll run our evil script file.
[13:23.760 --> 13:30.160]  And, you know, in real life, this might be installing a crypto miner or any number
[13:30.160 --> 13:36.140]  of things. In this case, just modifying the configuration files that Puppet is
[13:36.140 --> 13:41.260]  managing to show that we've actually gotten in and had the ability to do evil
[13:41.260 --> 13:50.320]  if we wanted to. So now we're going to jump over to the network monitor. This is
[13:51.680 --> 14:00.200]  a plain instance of Security Onion with all of the usual defaults and alerts
[14:00.200 --> 14:08.200]  installed. And what we're looking at right now are the Zeek notices. And let me
[14:08.200 --> 14:15.740]  turn on the refresh. But you'll see a whole lot of nothing. Zero. No results
[14:15.740 --> 14:29.240]  found. No results found. If we switch over to squirt and refresh it, same thing. No
[14:29.240 --> 14:36.400]  result. No result. And this is exactly what we expect from a default instance
[14:36.400 --> 14:45.030]  of a network monitor. It's not going to alert on connections like this. Let me
[14:45.030 --> 14:50.370]  enable the rules we've been talking about, and then we'll come back and we'll
[14:50.370 --> 15:02.560]  try this again. And now that I've enabled the zero touch rules, let's try it again
[15:02.560 --> 15:08.060]  and see what's changed. As you can see, the lab server is still emulating real
[15:08.060 --> 15:12.380]  traffic by reaching out to the Puppet configuration server and to the external
[15:12.380 --> 15:19.480]  internet. Let's go over to the Bastion and run our evil script again. Same thing,
[15:19.480 --> 15:24.300]  utilizing those compromised SSH credentials to reach in and modify the
[15:24.300 --> 15:32.940]  app server. Now we'll jump over to the network monitor, and as you can see, we're
[15:32.940 --> 15:39.880]  starting to get alerts. Zeek has noticed a couple of connections from the Bastion
[15:39.880 --> 15:49.560]  host to the app server. And if we switch over, you'll see the same thing in Squirt
[15:49.560 --> 15:58.180]  as well. Notice how they work slightly differently. The Squirt gives us many
[15:58.180 --> 16:03.500]  more alerts because it's looking individually at the packets traversing
[16:03.500 --> 16:08.660]  the network, versus Zeek gives us total number of connections. But either way,
[16:08.660 --> 16:13.320]  we're seeing the alerts that we're violating this zero-touch policy, and
[16:13.320 --> 16:22.980]  that it's a good indicator of lateral movement. So you want to start doing this
[16:22.980 --> 16:29.200]  in your environment as well. Well, first off, if you're not zero-touch yet, you
[16:29.200 --> 16:34.320]  should move that direction. Implement zero-touch. It's got a lot of benefits
[16:34.320 --> 16:41.160]  besides this one. When you're ready to implement this detection, use your
[16:41.160 --> 16:45.720]  platform of choice. Whatever you're already using is going to be best.
[16:45.820 --> 16:51.400]  There's no need to go get specialized software or hardware in order to do this.
[16:51.400 --> 16:59.640]  Use your existing IDS or network monitor. It's a very simple detection to include.
[17:00.420 --> 17:06.330]  But then, also pay attention to how you tailor it for your specific environment.
[17:06.870 --> 17:12.250]  In the example, it was looking for all TCP connections, because I wanted to catch
[17:12.250 --> 17:17.290]  the broadest range of lateral movement. But that might not make sense for your
[17:17.290 --> 17:22.530]  environment. So maybe you need to tailor it down to only look for SSH connections
[17:23.390 --> 17:29.190]  or, you know, whatever makes sense for your environment specifically.
[17:30.110 --> 17:35.050]  And then, the detections I showed were all very simple. They just looked for that
[17:35.050 --> 17:41.310]  connection and alerted. But that can be just the beginning of a bigger process.
[17:41.310 --> 17:47.570]  Imagine with Zeek, instead of raising a notice, if we correlated that with other
[17:47.570 --> 17:52.730]  suspicious traffic that we'd seen, maybe some data exfiltration traffic,
[17:52.730 --> 17:58.690]  and then we could automatically provide analysts with, here's the map of the
[17:58.690 --> 18:04.770]  attacker's path through the network. Starting to add some of those higher-end
[18:04.770 --> 18:11.310]  capabilities really makes this powerful. And finally, I wanted to leave you with
[18:11.310 --> 18:15.670]  some lessons learned from this project. And there are two that really jumped out
[18:15.670 --> 18:23.830]  at me. The first is know your network. And this is not just know your networking,
[18:23.830 --> 18:30.890]  it is know your specific network. Justin Henderson and Mick Douglas, who teach the
[18:30.890 --> 18:36.150]  Tactical SIEM class for SANS, really nailed this, and I learned this from them.
[18:36.150 --> 18:41.510]  So, huge shout-out to them. Knowing the specifics of your network lets you do
[18:41.510 --> 18:47.090]  really cool things in monitoring it. Things that aren't possible in generic
[18:47.090 --> 18:54.390]  networks, such as the detection we talked about today. And then, secondly,
[18:54.390 --> 18:59.830]  don't be afraid to look for stupid, simple things. All we were looking at today was
[18:59.830 --> 19:06.090]  the presence of a connection. Tying together these two lessons learned is
[19:06.090 --> 19:11.670]  really this whole project, is looking for something simple like the presence of a
[19:11.670 --> 19:16.870]  connection and knowing your network well enough to be able to say,
[19:16.870 --> 19:25.250]  that's unusual, that's a problem. Thank you so much for attending the
[19:25.250 --> 19:29.870]  presentation today. Shout-out again to the SANS team for their help and support
[19:29.870 --> 19:36.530]  with the research project. If you've got any questions about this project, AppSec,
[19:36.530 --> 19:42.150]  DevOps, the SANS Masters program, or anything related, the best way to get in
[19:42.150 --> 19:47.450]  contact with me is via Twitter. My handle's on the screen now. If you're
[19:47.450 --> 19:50.890]  viewing the live presentation, I look forward to chatting with you during the
[19:50.890 --> 19:57.670]  Q&A immediately afterwards in Discord. Again, I'm Philip Marlowe. Thanks for
[19:57.670 --> 19:58.330]  coming.