WEBVTT 00:00.000 --> 00:03.960 Welcome to the talk on detecting lateral movement in zero-touch environments. 00:03.960 --> 00:13.940 We're going to dive into how that lateral movement works, how to detect it, and why you really want a zero-touch environment. 00:15.320 --> 00:19.600 So before we jump in too much, some disclaimers and acknowledgments. 00:19.600 --> 00:25.960 First off, I work for the MITRE Corporation, but I am not representing or speaking for them today. 00:27.080 --> 00:41.300 All of the research done in this talk was performed in the pursuit of a master's degree through SANS, and so thank you to them for supporting the research, and thank you to my advisor, but I'm also not speaking for them today. 00:41.700 --> 00:47.820 And lastly, thank you to my wife Madeline, whom I'm also not speaking for today. 00:49.940 --> 00:53.520 So apart from all of that, who am I? 00:54.680 --> 01:00.900 My happy place professionally is in the middle of security and DevOps. 01:01.140 --> 01:13.500 And that has quite naturally led me to doing a lot of work with the cloud, automating not just applications and operations, but the security portions of that. 01:14.200 --> 01:16.020 A little bit of background on me. 01:16.020 --> 01:19.360 I wrote my first vulnerable code in elementary school. 01:19.360 --> 01:29.760 I was writing a blogging platform for my family, and they were quite surprised when it was hacked through a SQL injection that I inadvertently added. 01:29.780 --> 01:38.140 I began learning to write exploit code in middle school, when I received as a present the book Hacking the Art of Exploitation. 01:38.380 --> 01:42.920 And I've been involved in security one way or another ever since. 01:43.380 --> 01:49.860 This is my first time speaking at DEF CON, and I am super excited that the Cloud Village has me here. 01:50.620 --> 01:56.920 And one last little kind of personal philosophy thing is I really enjoy learning through hacking. 01:57.120 --> 01:59.240 I imagine a lot of you do, too. 01:59.240 --> 02:08.700 And you'll definitely see that flavor as we go through the presentation today of looking at things from both an attacker's point of view and a defender's point of view. 02:10.980 --> 02:15.260 So why should I care about DevOps if you're not already sold? 02:15.940 --> 02:21.920 Well, if you're running any applications or any services, you've got to deal with it. 02:21.920 --> 02:23.360 That's the way it is now. 02:23.440 --> 02:25.080 Welcome to the world. 02:25.640 --> 02:34.680 However, that's true regardless of whether you're hosting applications for the outside world, just for yourself, whoever it is. 02:35.640 --> 02:45.180 If you're a red team and you need to spin up infrastructure to support a penetration test or something like that, DevOps to the rescue. 02:45.340 --> 02:52.040 Blue team, you're trying to keep continuous monitoring and alerting of what's going on in your environment, DevOps. 02:52.860 --> 03:01.200 Application owner, trying to make sure your application is stable and available and secure, DevOps. 03:01.920 --> 03:06.600 Another great perk for us is that DevOps is essentially cloud native now. 03:06.600 --> 03:14.080 All of the major cloud providers have offered ways to automate their environments. 03:14.080 --> 03:26.640 And so being able to develop infrastructure as code, applications as code, security compliance as code, monitoring as code, all of these are really enabled by the cloud. 03:26.640 --> 03:30.040 They can be done in other environments, but the cloud makes it easy. 03:30.620 --> 03:34.000 And lastly, DevOps is better for security. 03:34.000 --> 03:39.760 We're integrating security earlier in the process, rather than trying to layer it on later. 03:42.380 --> 03:44.860 So I promised an attacker's point of view. 03:45.740 --> 03:49.180 Today, we're going to look at kind of a simplified example environment. 03:49.180 --> 03:55.780 I'm sure, you know, you've seen a bunch of other presentations that have got the hard to read, squint and see, that's AWS, right? 03:55.780 --> 03:56.600 Not Azure. 03:57.160 --> 03:59.380 We're simplifying things for this talk. 03:59.840 --> 04:04.780 You're somewhere out on the internet, and you really want to get to an application server. 04:05.620 --> 04:16.200 Might want to get in and then exfil some of the sensitive data on it, you might want to get in and use their computing resources to mine cryptocurrency. 04:16.380 --> 04:20.200 Whatever the reason is, you're going for that application server. 04:20.900 --> 04:24.620 So there's three main options we're going to look at. 04:24.720 --> 04:31.200 First is straight through the front door, directly in from the internet hitting the application server. 04:31.760 --> 04:33.140 Pretty simple. 04:33.140 --> 04:38.480 Yeah, there's maybe a firewall or an IDS or something that you have to evade. 04:38.480 --> 04:41.720 But overall, pretty direct, pretty simple. 04:42.180 --> 04:50.560 Except it's not 2003 anymore, and we've gotten better at protecting our applications and the servers they run on. 04:50.700 --> 05:03.340 So while it's still possible, and there are plenty of big, well-known events where it's happened, it's not as popular or as reliable as it used to be. 05:03.540 --> 05:14.740 So that brings us to our second option, and that's, let's compromise the workstation of an employee, and then move laterally to the application server we really want to be on. 05:16.120 --> 05:23.260 For the purposes of this presentation, I'm going to say we phished that employee, but it could be any number of things. 05:24.340 --> 05:34.560 From there, most organizations have a bastion host that employees have to log into before they can get into their production environment. 05:34.560 --> 05:37.780 That's a pretty common practice, and so we see that on the screen. 05:37.780 --> 05:50.680 Our second option is compromise an employee workstation , use valid credentials that that employee has access to, move through the bastion host to the application server. 05:51.260 --> 05:59.040 And then our third option is to take the route that DevOps uses to publish applications itself. 05:59.140 --> 06:10.300 From that employee workstation, push something into the source code repo, get it through the test servers undetected, have the configuration server deploy it for us. 06:11.300 --> 06:18.020 Or maybe we can compromise each of those servers in turn in order to get there in the end. 06:18.600 --> 06:26.380 In any case, that's a lot more hops to have to go through and a lot more chances to be detected or fail in some other way. 06:26.940 --> 06:41.300 So we're going to focus on detecting number two, both because it's pretty common and because that's the easiest route that attackers have, and so that's most likely what they're going to go for. 06:41.440 --> 06:44.960 And this being common is borne out by repeated studies. 06:44.960 --> 06:58.020 We've seen reports from FireEye and the Verizon data breach report say that social engineering and then using valid credentials is incredibly popular for malicious actors right now. 07:00.350 --> 07:06.090 So what does it look to do a traditional application deployment from that employee's point of view? 07:07.870 --> 07:20.590 Developers give ops a deployment package and hopefully some install instructions, and then ops logs into the app server through the bastion host and manually installs the software. 07:20.670 --> 07:22.070 Pretty simple. 07:22.790 --> 07:24.410 And then you get a patch. 07:24.410 --> 07:26.750 Well, you've got to log in and install the patch. 07:26.930 --> 07:28.790 There's an update to one of the dependencies. 07:28.790 --> 07:30.650 We've got to log in and install that. 07:31.050 --> 07:35.810 Anytime anything changes, someone's logging into that application server. 07:39.850 --> 07:45.510 So that's, you know, pretty normal, but not ideal, and we'll get into that. 07:47.410 --> 07:51.390 Traditional lateral movement takes advantage of that fact. 07:51.390 --> 08:00.570 Because you've got people logging in to the server manually, they have highly privileged credentials. 08:01.070 --> 08:07.730 And very often, those credentials are stored in plain text on their employee workstations. 08:07.730 --> 08:11.370 Exactly what we were talking about compromising earlier. 08:11.390 --> 08:18.450 That might take the form of SSH keys, it might take the form of API tokens or keys. 08:18.930 --> 08:27.050 And, you know, especially relevant for the cloud village, that's the way AWS stores local credentials. 08:27.050 --> 08:30.210 Plain text file in that path on the screen. 08:32.290 --> 08:44.710 Attackers then use those plain text credentials from the station they've already compromised to move further into your environment and do whatever evil they want to get up to. 08:44.750 --> 08:46.710 They've hit their target. 08:48.510 --> 08:53.150 So what is ZeroTouch and why does it apply to any of this? 08:53.570 --> 09:00.630 So Google are the ones who have defined ZeroTouch networking and ZeroTouch production concepts. 09:00.630 --> 09:03.670 But they're far from the only people who use them. 09:04.050 --> 09:14.370 This is a technique used by many mature DevOps organizations to automate as much as possible and get humans out of their production environments. 09:14.810 --> 09:17.210 What does that look like? 09:17.390 --> 09:23.890 Here's the three rules that Google use to define their ZeroTouch production. 09:23.890 --> 09:27.030 And the first one is really big. 09:27.650 --> 09:31.170 Every change is made by automation. 09:31.170 --> 09:33.790 No humans are involved. 09:33.910 --> 09:50.740 By getting that human traffic out of the environment, we can do a lot of really interesting but simple things to detect when attackers are trying to pivot into our protected environment. 09:51.740 --> 10:05.480 The second rule, pre-validated by software, is important because if we try to go to that attack path number three, the one that uses the DevOps pipeline itself, that means we're checking to see if there's anything malicious before we deploy. 10:05.480 --> 10:07.840 Now, that doesn't always work. 10:07.840 --> 10:16.360 But it gives us as defenders a better opportunity than someone who's just pivoting through the bastion host and hiding in the noise. 10:16.920 --> 10:23.700 And then lastly, emergency changes can be made via a break glass mechanism. 10:24.000 --> 10:27.260 But they have to be auditable. 10:27.340 --> 10:29.140 We'll come back to that. 10:32.320 --> 10:40.700 And what this looks like in our picture is essentially disallowing communication from that bastion host to the application server. 10:41.160 --> 10:50.000 And all normal traffic is going through that path number three where it goes through the source repo and the test servers and the configuration servers. 10:50.000 --> 10:58.920 So we've got good control over what's being deployed to the application server and good confidence that it's not malicious. 11:01.920 --> 11:02.380 All right. 11:02.380 --> 11:08.020 So now let's look at lateral movement in each of these two types of environments. 11:08.460 --> 11:15.860 We've compromised our employee workstation, found some valid credentials that we can use to pivot. 11:16.000 --> 11:27.520 And in this case, we're going to look at the traditional path where operators are regularly logging into the environment to deploy patches, update software, any of those kinds of things. 11:27.760 --> 11:29.900 I'm about to play an animation. 11:30.280 --> 11:34.300 And there are going to be several connections shown on the screen by arrows. 11:34.320 --> 11:39.300 Try and find the one that looks a little bit different than all the rest. 11:39.640 --> 11:43.200 This is going to be our malicious lateral movement. 11:47.120 --> 11:48.640 Did you see it? 11:50.020 --> 11:51.680 Have you seen it yet? 11:52.340 --> 11:54.840 It's a little bit tricky to find. 11:55.100 --> 11:58.500 And this is assuming that it stands out at all. 11:58.500 --> 12:10.460 If you're using an encrypted communication channel like SSH or HTTPS, which are super common, this use of valid credentials might not stand out in any way. 12:12.820 --> 12:19.460 Now let's compare that to someone trying to use lateral movement on a zero touch network. 12:19.480 --> 12:20.840 Same thing. 12:20.840 --> 12:29.360 I'm going to display several arrows in an animation and see if you can pick out the one that looks a little bit different and indicates our malicious lateral movement. 12:32.350 --> 12:34.490 I'll bet that was a lot easier. 12:34.490 --> 12:38.130 And you maybe even spotted it before the animation finished playing. 12:39.530 --> 12:54.670 In this case, it stands out like a sore thumb because those valid credentials, even if the channel is encrypted, are going an entirely different route through the network than normal, regular, valid traffic would. 12:57.150 --> 13:03.350 So how do we go about detecting that from a programmatic and automatic standpoint? 13:03.530 --> 13:07.790 First, we need to define the protected servers. 13:07.810 --> 13:11.630 In the case of our example, that's just the application server. 13:11.710 --> 13:12.910 But it could be a lot of things. 13:12.910 --> 13:16.770 Could be application servers, web servers, the DevOps servers themselves. 13:17.090 --> 13:21.550 Anything that humans shouldn't be logging into on a regular basis. 13:22.350 --> 13:26.530 Then we need to define all of the human access points into the environment. 13:26.830 --> 13:30.430 Again, in this example, that's just the bastion host. 13:30.430 --> 13:36.350 But depending on your network architecture, there might be any number of things that you need to put into that list. 13:37.210 --> 13:44.210 And then watch for any connections made from those manual access points to your protected servers. 13:45.130 --> 13:49.450 Real simple, but very effective if it's not supposed to be happening. 13:49.450 --> 13:58.350 At that point, you can alert, investigate, whatever your normal process is, and enjoy having detected malicious lateral movement. 13:58.610 --> 14:03.190 Now, I mentioned earlier that auditing was important. 14:03.190 --> 14:11.930 And the reason that this is still a valid attack path into the environment is most organizations can't get rid of their bastion hosts entirely. 14:11.930 --> 14:18.810 They're still needed for incident response or forensics, some kind of unusual emergency purpose. 14:19.050 --> 14:25.430 And so, even though that's valid traffic, this method would still alert on it. 14:25.430 --> 14:34.350 However, that's not a problem, because anytime that emergency traffic does occur, we wanted it to be auditable anyway. 14:34.350 --> 14:37.850 And this is helpfully generating those audit alerts for us. 14:39.790 --> 14:44.690 So, now we're going to cross our fingers and see if the live demo goes well. 14:50.750 --> 14:56.170 What I've got up on the screen are three lab machines. 14:56.330 --> 15:03.590 One, a bastion host, one, our application server, and one, a puppet configuration server. 15:05.210 --> 15:16.170 What I'm going to do, just to prove to you all that this is actually working, and that I've got nothing up my sleeves today, is we're going to put some valid traffic into the environment. 15:16.190 --> 15:30.690 What this puppet command will do is generate traffic between the puppet configuration server and the application server, as well as traffic out to the wider internet over both SSH and HTTPS. 15:30.690 --> 15:33.630 And we'll keep that going throughout the rest of the demonstration. 15:36.150 --> 15:42.430 Then, we'll go over to the bastion host and try to execute our... 15:46.390 --> 15:52.010 let me fix that real quick. 15:52.010 --> 15:58.890 Then we'll go over to the bastion host and... nope. 15:59.050 --> 16:04.550 My routes are wrong today, so I'm going to play the prerecorded version of the demo instead. 16:12.580 --> 16:16.900 All right, but you'll get all the same information here. 16:20.370 --> 16:23.810 So, for the demo, we're going to look at a couple of scenarios. 16:24.170 --> 16:26.950 First up, let me tell you what you're seeing. 16:27.410 --> 16:39.670 In this environment, I've got a bastion host that's used for manually accessing it when needed, an application server, and a configuration server running puppet. 16:39.870 --> 16:49.190 So, just to make sure that there's nothing up my sleeves, what we're going to do for this demo is run some puppet configuration in the background. 16:50.250 --> 17:00.870 This will be traffic both within the environment to the puppet server, and external to the environment SSH connections and HTTPS connections out to the wider internet. 17:01.390 --> 17:03.170 So, let's get that started. 17:05.070 --> 17:14.170 And then, while this runs, jump over to the bastion host, and as you can see, we've already compromised the bastion host. 17:14.170 --> 17:23.590 And we see some SSH credentials that we found, and in fact, some SSH credentials for the application server that has the goodies we want to get to. 17:24.430 --> 17:30.050 So, we'll run our evil script file. 17:30.830 --> 17:37.810 And, you know, in real life, this might be installing a crypto miner or any number of things. 17:37.810 --> 17:49.650 In this case, just modifying the configuration files that Puppet is managing to show that we've actually gotten in and had the ability to do evil if we wanted to. 17:50.710 --> 17:56.630 So, now we're going to jump over to the network monitor. 17:56.630 --> 18:08.790 This is a plain instance of Security Onion with all of the usual defaults and alerts installed. 18:08.790 --> 18:13.210 And what we're looking at right now are the ZIP notices. 18:13.370 --> 18:17.090 And let me turn on the refresh. 18:17.090 --> 18:21.430 But you'll see a whole lot of nothing. 18:21.430 --> 18:22.230 Zero. 18:22.230 --> 18:23.550 No results found. 18:23.550 --> 18:25.210 No results found. 18:25.730 --> 18:35.990 If we switch over to Squirt and refresh it, same thing. 18:36.010 --> 18:37.170 No result. 18:37.170 --> 18:38.590 No result. 18:39.290 --> 18:44.890 And instance of a network monitor. 18:44.890 --> 18:48.010 It's not going to alert on connections like this. 18:51.430 --> 18:55.470 Let me enable the rules we've been talking about. 18:55.470 --> 18:59.090 And then we'll come back and we'll try this again. 19:06.220 --> 19:11.120 And now that I've enabled the zero-touch rules, let's try it again and see what's changed. 19:11.120 --> 19:20.360 As you can see, the lab server is still emulating real traffic by reaching out to the Puppet configuration server and to the external internet. 19:20.540 --> 19:25.020 Let's go over to the Bastion and run our evil script again. 19:25.280 --> 19:32.440 Same thing, utilizing those compromised SSH credentials to reach in and modify the app server. 19:33.340 --> 19:38.020 Now we'll jump over to the network monitor. 19:38.020 --> 19:41.860 And as you can see, we're starting to get alerts. 19:42.540 --> 19:50.200 Zeke has noticed a couple of connections from the Bastion host to the app server. 19:50.200 --> 19:57.900 And if we switch over, you'll see the same thing in Squirt as well. 19:58.660 --> 20:02.800 Notice how they work slightly differently. 20:02.800 --> 20:14.840 The Squirt gives us many more alerts because it's looking individually at the packets traversing the network versus Zeke gives us total number of connections. 20:14.940 --> 20:23.540 But either way, we're seeing the alerts that we're violating this zero-touch policy and that it's a good indicator of lateral movement. 20:32.810 --> 20:46.340 All right, so you can see from the demo, you know, this isn't something that adds to your ability to detect lateral movement as long as you conform to the expectations of the environment. 20:47.080 --> 20:54.540 Demonstrated here with Zeke and Snort, but there's no reason you can't use any of your tools of choice. 20:54.580 --> 20:57.000 But this is the cloud village. 20:57.000 --> 20:58.640 So what about cloud? 20:59.780 --> 21:05.730 Totally still doable in the cloud, despite the fact I use VMware for my lab environment. 21:07.120 --> 21:22.720 If you're not familiar, all the major cloud providers provide some form of traffic mirroring, whatever they call that, traffic mirroring in AWS, virtual network tap in Azure, and packet mirroring for Google Cloud Platform. 21:22.780 --> 21:25.640 And that'll give you full network visibility. 21:25.640 --> 21:33.380 You can install something just like Security Onion or your favorite network monitor feeding directly off of that. 21:34.800 --> 21:38.440 You can also install your own within the environment. 21:38.440 --> 21:49.460 And because this is looking for such simple things as connections, we don't even need the full traffic in order to tell when this activity is occurring. 21:49.460 --> 21:51.680 We can just use flow logs. 21:51.760 --> 21:56.700 And so that'll cut down on the amount of traffic that we have to capture if we go that route. 21:57.460 --> 22:08.420 Now, given that the cloud is a much more dynamic environment than traditional data centers, this means that tagging an asset inventory is really important to get right. 22:09.240 --> 22:16.920 You have to know what those protected servers are and what the human access points are in order to detect connections between them. 22:16.920 --> 22:34.100 And if you've got automatic scaling or something like that, keeping those lists up to date can be a little bit tricky, but it's definitely doable, especially since those lists in most tools can be entire subnets rather than specific machines. 22:35.440 --> 22:38.840 But there are also some challenges with the cloud. 22:38.840 --> 22:43.300 And the big one is visibility into management traffic. 22:44.040 --> 22:52.740 With something like Puppet or Ansible, your traditional configuration servers, that's all going over a network that you can view. 22:52.800 --> 22:59.180 However, in the cloud, some of your management traffic goes over the management plane in the background. 22:59.180 --> 23:06.800 And these features like traffic mirroring are not able to capture that data for you. 23:06.800 --> 23:12.680 It all happens in AWS or in Azure without you being able to see it. 23:12.720 --> 23:15.460 And that poses a visibility challenge. 23:15.740 --> 23:24.360 Doesn't mean that it's not worth doing, but you have to understand how your environment usually works in order to be able to protect it. 23:27.150 --> 23:30.370 So, some next steps if you're sold on this idea. 23:30.570 --> 23:34.170 First off, if you're not Zerotouch yet, do it. 23:34.170 --> 23:38.350 It's got a lot of benefits aside from this particular detection. 23:38.750 --> 23:43.430 It's hard to do, and it's hard to get right, but it is worth it. 23:43.650 --> 23:47.170 And then you can implement this detection on your platform of choice. 23:47.710 --> 23:52.570 I showed off Snort and Zeek here, but it's very simple. 23:52.570 --> 24:01.290 There's no need and buy specialized software, buy specialized hardware, install anything you're probably not already using anyway. 24:02.610 --> 24:08.710 And then when you do implement this detection, tailor it to your specific environment. 24:08.710 --> 24:14.970 Here I was looking for any connections between the human access points and our protected servers. 24:15.030 --> 24:21.610 But your environment may or may not be able to support that depending on its specific architecture. 24:22.150 --> 24:31.070 The good news is that this approach is valid for a lot of different options, and you just need to figure out which ones work specifically for you. 24:31.070 --> 24:34.170 You might only turn it on for SSH connections. 24:34.170 --> 24:39.350 You might only look at particular areas of your network that are high risk. 24:39.410 --> 24:43.270 However you do it, make sure you tailor it, because that'll give you the most value. 24:44.390 --> 24:51.590 And then there are ways to extend this work far beyond what I demonstrated in today's video. 24:52.140 --> 24:58.480 For example, correlate it with other events that are generated off of suspicious traffic. 24:58.830 --> 25:16.030 Imagine tying this alert about potential lateral movement with potential data exfiltration that's generated by Zeke, and then presenting your security team with a map of how the attacker moved through your entire network. 25:16.030 --> 25:20.070 It's pretty powerful once you start combining it with other things. 25:22.370 --> 25:28.430 Lastly, I want to leave you with some lessons learned from doing this research project. 25:28.430 --> 25:31.210 First off, you have to know your network. 25:31.210 --> 25:39.250 That's the basis of this entire project, is you have to know your network well enough to know what's normal and what's abnormal. 25:39.370 --> 25:50.190 And then putting in things to look for abnormalities specific to your environment as opposed to things that apply to everyone's environment. 25:50.830 --> 25:58.210 I learned this principle first from Mick Douglas and Justin Henderson, who teach the sans-tactical-seam class. 25:58.910 --> 26:09.710 And it really resonated, it was really powerful in being able to do things that don't come pre-built with your software, because you're looking for things specific to you. 26:10.230 --> 26:15.350 And then second, don't be afraid to look for stupid simple things on your network. 26:16.170 --> 26:20.670 All we're looking for in this is the presence of a connection. 26:21.970 --> 26:30.210 But knowing our environment the way we do, that lets us say there's something weird going on, I need to investigate. 26:30.270 --> 26:32.650 And that is super valuable. 26:34.830 --> 26:36.730 So, thank you. 26:37.010 --> 26:43.430 If you have any questions after the talk and the Q&A is done, best place to reach me is on Twitter. 26:43.450 --> 26:45.910 My handle's on the screen right now. 26:46.170 --> 26:50.530 And then we'll see if we've got any questions live. 26:50.530 --> 27:00.970 Do you happen to know any tools that might help implement this kind of a zero -touch environment or getting to know your network better or something along those lines? 27:02.430 --> 27:03.190 Sure . 27:03.190 --> 27:08.790 So, getting to know your network better, you're just going to have to get familiar with how these operate. 27:08.790 --> 27:19.790 And my biggest recommendation is if you're starting to do DevOps as a security person, is make time to spend with the operations and the developers. 27:20.010 --> 27:22.710 Because they're going to know how things normally go. 27:22.710 --> 27:25.690 And you'll be able to learn from them what they expect. 27:25.690 --> 27:31.070 And then use your knowledge of security to figure out what shouldn't be going on. 27:31.890 --> 27:45.910 In terms of tools to help implement this, you know, typical DevOps tools, there's a periodic table of DevOps that's out there, I think by Xenia Labs, that shows a lot of the different options. 27:46.590 --> 28:00.470 For the demo today, I was using Puppet as a configuration server to manage the configuration of the application server and making sure that it had the files and configuration settings that I expected. 28:00.470 --> 28:07.970 There are a lot of them, apart from Puppet, Ansible, SaltStack, things of that nature. 28:07.970 --> 28:12.610 And then for building out a whole DevOps pipeline, look at that periodic table. 28:12.610 --> 28:19.730 You'll need several different things along there to handle your source code and your testing and things like that.