1 00:00:13,790 --> 00:00:17,010 Hey AppSecVillage, welcome back and get ready for another talk. 2 00:00:17,010 --> 00:00:18,850 Coming up next we've got Can't Touch This! 3 00:00:18,850 --> 00:00:21,490 Detecting Lateral Movement in Zero-Touch Environments. 4 00:00:21,490 --> 00:00:23,590 Philip Marlowe is going to be presenting this. 5 00:00:23,590 --> 00:00:27,090 Philip is a cybersecurity and DevOps engineer. 6 00:00:27,090 --> 00:00:35,270 He helps organizations understand how to adopt DevOps practices to increase their security rather than sacrifice it in the name of speed. 7 00:00:35,510 --> 00:00:44,370 Philip holds several security, cloud, and agile certifications and is currently pursuing a master's degree in information security engineering at SANS Technology Institute. 8 00:00:44,370 --> 00:00:46,890 Please give it up for Philip Marlowe. 9 00:00:47,230 --> 00:00:50,490 Hello everyone and welcome to Can't Touch This! 10 00:00:50,490 --> 00:00:54,050 Detecting Lateral Movement in Zero-Touch Environments. 11 00:00:54,050 --> 00:01:01,710 In this talk we're going to talk some about DevOps and how to detect attackers trying to move into your production environments. 12 00:01:03,570 --> 00:01:07,810 Before we get started, a few disclaimers and acknowledgments. 13 00:01:07,810 --> 00:01:15,810 First off, I work for the MITRE Corporation, but nothing I'm saying today reflects their opinion. 14 00:01:16,590 --> 00:01:21,650 Also, I'm a student at the SANS Technology Institute working on my master's degree. 15 00:01:22,290 --> 00:01:29,890 Huge shout out to my faculty research advisor, but I'm not speaking for her or SANS today either. 16 00:01:29,970 --> 00:01:33,610 And lastly, big thank you to my wife Madeline. 17 00:01:33,610 --> 00:01:37,150 Once again, not speaking for her in this presentation today. 18 00:01:38,910 --> 00:01:40,070 So who am I? 19 00:01:40,070 --> 00:01:48,530 I'm Philip and professionally I really enjoy the intersection of security and DevOps. 20 00:01:48,730 --> 00:01:51,570 Yes, you can call it DevSecOps if you want. 21 00:01:51,590 --> 00:01:53,530 I don't really care either way. 22 00:01:53,990 --> 00:01:58,090 I wrote my first vulnerable code in elementary school. 23 00:01:58,090 --> 00:02:07,630 I was writing a blogging platform for my family and right in there, because I didn't know any better, put a SQL injection vulnerability. 24 00:02:07,750 --> 00:02:12,930 That was a big surprise for them when they found out we've been hacked because of that. 25 00:02:13,650 --> 00:02:22,850 I first began learning to exploit applications in middle school when I received as a present the book, Hacking the Art of Exploitation. 26 00:02:22,950 --> 00:02:28,310 And I've been involved in application security one way or another ever since then. 27 00:02:29,330 --> 00:02:34,630 Today is my first time as a DEF CON speaker, and I'm super excited to be here. 28 00:02:35,070 --> 00:02:40,690 And one of the things I really enjoy is being able to learn through hacking. 29 00:02:40,690 --> 00:02:48,230 And you'll see a flavor of that throughout this presentation as we take both the attacker and the defender point of view. 30 00:02:52,200 --> 00:02:56,600 So if you're not already on board, why should I care about DevOps? 31 00:02:57,540 --> 00:03:02,100 Well, for one, you probably don't have any choice. 32 00:03:02,120 --> 00:03:07,280 If you're running any kind of application, that's the way it is now. 33 00:03:07,280 --> 00:03:09,840 You probably just need to learn about it. 34 00:03:10,980 --> 00:03:17,500 You know, if your red team standing up command and control infrastructure, helps to have DevOps. 35 00:03:17,500 --> 00:03:22,540 Your blue team trying to implement monitoring and detection helps to have DevOps. 36 00:03:22,780 --> 00:03:28,080 Application security DevOps is the way of life now. 37 00:03:29,500 --> 00:03:42,340 And, you know, provides all these security benefits on top of reliability benefits and maintainability benefits for the other parts of the business. 38 00:03:44,760 --> 00:03:52,780 So let's take a look at a very simplified environment, and let's take a look at it from the attackers point of view. 39 00:03:53,000 --> 00:03:57,020 We want to get to something on the application server. 40 00:03:57,400 --> 00:04:05,800 Maybe we want to steal data off of it, or maybe we just want to use the hefty server power for crypto mining. 41 00:04:06,440 --> 00:04:09,780 Either way, we've got three basic options. 42 00:04:10,580 --> 00:04:17,800 Option one is to just go straight in the front door and attack the application itself in order to get to the access to the server. 43 00:04:18,740 --> 00:04:20,460 This is pretty simple. 44 00:04:20,840 --> 00:04:29,540 Maybe there's a firewall or IDS you have to evade, but, you know, you're not making a lot of hops around to finally get to where you want to be. 45 00:04:30,260 --> 00:04:33,800 But it's also not 2002 anymore. 46 00:04:34,640 --> 00:04:39,240 The situation's gotten a lot better in how we protect our application servers. 47 00:04:39,300 --> 00:04:44,900 So while this once might have been the best way in, it's not really anymore. 48 00:04:46,320 --> 00:05:00,940 Report after report from places like FireEye and Verizon have shown a much more popular way into networks these days is by social engineering users in one way or another. 49 00:05:00,940 --> 00:05:13,480 In this case, we're going to take a look at phishing a operator or developer on their employee workstation to give us our initial access. 50 00:05:13,480 --> 00:05:22,180 Then in method two, we're going to move laterally through the bastion host and finally to the app server. 51 00:05:23,240 --> 00:05:29,280 We'll talk in a couple of slides about why this works so well. 52 00:05:29,700 --> 00:05:34,020 But before we get there, let's talk a little bit about option three. 53 00:05:34,060 --> 00:05:42,660 With DevOps implemented these days, another way into that protected app server is through the DevOps pipeline itself. 54 00:05:42,760 --> 00:05:52,520 But there are a lot more hops here and a lot more ways that can go wrong and a lot more opportunities to detect this attempt. 55 00:05:54,080 --> 00:06:03,880 It goes through source control repos, test servers, and configuration servers before we ever get to our intended target. 56 00:06:07,660 --> 00:06:12,340 So how does traditional application deployment work? 57 00:06:12,780 --> 00:06:19,180 Well, developers give Ops a deployment package, and if we're lucky, some install instructions. 58 00:06:19,620 --> 00:06:25,320 And then Ops is going to go and log into the application server, manually install that software. 59 00:06:25,480 --> 00:06:26,620 Pretty simple. 60 00:06:27,080 --> 00:06:33,120 And then an update comes along, and you have to go and manually log in and update. 61 00:06:33,560 --> 00:06:39,160 And then the security patch comes along, and you have to go log in and update. 62 00:06:39,500 --> 00:06:48,560 And this manual logging in and updating is going to be a real problem for detecting lateral movement. 63 00:06:49,320 --> 00:06:52,540 So what does traditional lateral movement look like? 64 00:06:54,840 --> 00:07:02,860 Because they're logging in to do configuration and updates all the time, Ops has highly privileged credentials. 65 00:07:03,340 --> 00:07:09,180 This might take the form of SSH keys, or API tokens. 66 00:07:10,480 --> 00:07:16,300 And oftentimes, they're stored in plain text on their employee workstations. 67 00:07:16,540 --> 00:07:28,180 If you've already phished that user, those are really easy and really juicy targets to steal those credentials and move further into the environment. 68 00:07:31,230 --> 00:07:34,910 All right, so that sounds like it's a problem. 69 00:07:35,190 --> 00:07:38,310 What is this zero touch concept? 70 00:07:39,010 --> 00:07:43,890 Google defined zero touch networking and zero touch production. 71 00:07:44,030 --> 00:07:51,610 And they've given several presentations on what that means to them and how they've implemented it in their environment. 72 00:07:52,350 --> 00:08:09,870 Overall, it's about how we can abstract away managing the production environment, so that we're not doing those manual logins, change things, configure and update anymore. 73 00:08:10,530 --> 00:08:19,110 And while Google is one of the only companies to use the zero touch nomenclature, a lot of companies do this. 74 00:08:19,190 --> 00:08:30,230 Often, it's just a result of being very mature in their DevOps practices, that everything is handled by automation, rather than human users. 75 00:08:32,190 --> 00:08:36,330 So zero touch deployment looks very different. 76 00:08:36,950 --> 00:08:41,730 Every change is made by automation and pre-validated. 77 00:08:42,250 --> 00:08:44,710 This means no humans logging in. 78 00:08:44,710 --> 00:08:48,210 And that's key for our talk today. 79 00:08:48,550 --> 00:08:53,570 You'll also notice on Google's presentation that there is an exception. 80 00:08:53,570 --> 00:08:58,050 And that's when something is made by an audited break glass mechanism. 81 00:08:58,050 --> 00:09:01,110 We'll talk a little bit more about that later as well. 82 00:09:03,490 --> 00:09:08,290 So what is the zero touch deployment look like in practice? 83 00:09:08,370 --> 00:09:21,810 Well, rather than having a user log into the app server via a bastion host, since that's no longer allowed, every change has to go through that DevOps deployment pipeline. 84 00:09:25,580 --> 00:09:37,380 Looking at how attackers are going to move laterally, however, let's compare and contrast between the traditional and the zero touch deployments. 85 00:09:37,540 --> 00:09:49,100 So I'm about to play an animation, and keep an eye out for the malicious connection between the bastion host and the app server. 86 00:09:49,100 --> 00:09:52,220 It's going to be a slightly different colored arrow. 87 00:09:54,680 --> 00:09:56,600 Did you catch it? 88 00:09:57,420 --> 00:09:59,320 Have you seen it yet? 89 00:09:59,800 --> 00:10:01,600 It's kind of tricky to find. 90 00:10:01,600 --> 00:10:04,800 And this is when it stands out in some way. 91 00:10:04,800 --> 00:10:16,180 If we're looking at encrypted connections like SSH or HTTPS, it may not stand out in any way, unless you're doing some kind of break and inspect in the middle. 92 00:10:17,500 --> 00:10:21,700 Now let's compare it to zero touch. 93 00:10:22,660 --> 00:10:31,680 In the zero touch deployment, you're going to see a bunch of arrows again, showing normal benign traffic. 94 00:10:32,020 --> 00:10:37,880 And you'll also see one arrow that'll look a little different, being our attacker trying to move laterally. 95 00:10:38,000 --> 00:10:40,620 See if it's easier to spot this time. 96 00:10:42,800 --> 00:10:46,500 I'll bet that was a lot easier, and as soon as it came up, you noticed it. 97 00:10:47,260 --> 00:10:52,580 This is the whole concept that we want to use writing this detection. 98 00:10:53,040 --> 00:10:57,020 So how do we actually go about detecting this lateral movement? 99 00:10:57,020 --> 00:10:58,920 There's a few things we need to do. 100 00:10:59,060 --> 00:11:02,820 First, we need to define all of the protected servers. 101 00:11:03,000 --> 00:11:07,160 In our simplified environment, that's just the app server. 102 00:11:07,180 --> 00:11:16,520 But it might also include DevOps automation servers, it might include web servers, database servers, all of which might be behind load balancers. 103 00:11:16,600 --> 00:11:21,240 Whatever humans should not be logging into, that goes in this list. 104 00:11:21,960 --> 00:11:26,460 We also need to define all of the human access points into the environment. 105 00:11:26,960 --> 00:11:30,300 Again, in our simplified environment, that's just the bastion host. 106 00:11:30,300 --> 00:11:42,060 But depending on your network architecture, you might need to include other sets of load balance machines, virtual desktops, employee workstations, anything of that nature. 107 00:11:42,060 --> 00:11:44,000 That goes in the second list. 108 00:11:44,840 --> 00:11:52,140 And then all we do to detect is look for any connections between those two sets of servers. 109 00:11:53,100 --> 00:11:59,180 Then alert, investigate, enjoy detecting this lateral movement. 110 00:12:02,960 --> 00:12:06,900 Let's take a quick look at what this looks like in practice. 111 00:12:13,300 --> 00:12:16,860 So for the demo, we're going to look at a couple of scenarios. 112 00:12:17,100 --> 00:12:20,060 First up, let me tell you what you're seeing. 113 00:12:20,160 --> 00:12:32,560 In this environment, I've got a bastion host that's used for manually accessing it when needed, an application server, and a configuration server running Puppet. 114 00:12:32,640 --> 00:12:42,580 So just to make sure that there's nothing up my sleeves, what we're going to do for this demo is run some Puppet configuration in the background. 115 00:12:43,100 --> 00:12:53,860 This will be traffic both within the environment to the Puppet server and external to the environment, SSH connections and HTTPS connections out to the wider internet. 116 00:12:54,340 --> 00:12:56,220 So let's get that started. 117 00:12:58,440 --> 00:13:03,520 And then while this runs, jump over to the bastion host. 118 00:13:03,620 --> 00:13:06,980 And as you can see, we've already compromised the bastion host. 119 00:13:06,980 --> 00:13:16,620 And we see some SSH credentials that we found, and in fact, some SSH credentials for the application server that has the goodies we want to get to. 120 00:13:17,420 --> 00:13:23,080 So we'll run our evil script file. 121 00:13:23,760 --> 00:13:30,780 And, you know, in real life, this might be installing a crypto miner or any number of things. 122 00:13:30,780 --> 00:13:42,880 In this case, just modifying the configuration files that Puppet is managing to show that we've actually gotten in and had the ability to do evil if we wanted to. 123 00:13:43,600 --> 00:13:49,760 So now we're going to jump over to the network monitor. 124 00:13:49,760 --> 00:14:01,740 This is a plain instance of Security Onion with all of the usual defaults and alerts installed. 125 00:14:01,800 --> 00:14:06,080 And what we're looking at right now are the Zeek notices. 126 00:14:06,520 --> 00:14:10,160 And let me turn on the refresh. 127 00:14:10,160 --> 00:14:14,240 But you'll see a whole lot of nothing. 128 00:14:14,240 --> 00:14:15,140 Zero. 129 00:14:15,140 --> 00:14:16,500 No results found. 130 00:14:16,500 --> 00:14:17,900 No results found. 131 00:14:18,820 --> 00:14:28,640 If we switch over to squirt and refresh it, same thing. 132 00:14:28,640 --> 00:14:30,080 No result. 133 00:14:30,080 --> 00:14:31,400 No result. 134 00:14:31,920 --> 00:14:37,980 And this is exactly what we expect from a default instance of a network monitor. 135 00:14:37,980 --> 00:14:41,100 It's not going to alert on connections like this. 136 00:14:44,170 --> 00:14:52,150 Let me enable the rules we've been talking about, and then we'll come back and we'll try this again. 137 00:14:59,020 --> 00:15:03,880 And now that I've enabled the zero touch rules, let's try it again and see what's changed. 138 00:15:03,880 --> 00:15:13,380 As you can see, the lab server is still emulating real traffic by reaching out to the Puppet configuration server and to the external internet. 139 00:15:13,600 --> 00:15:18,080 Let's go over to the Bastion and run our evil script again. 140 00:15:18,380 --> 00:15:25,480 Same thing, utilizing those compromised SSH credentials to reach in and modify the app server. 141 00:15:26,380 --> 00:15:34,900 Now we'll jump over to the network monitor, and as you can see, we're starting to get alerts. 142 00:15:35,520 --> 00:15:43,240 Zeek has noticed a couple of connections from the Bastion host to the app server. 143 00:15:43,400 --> 00:15:50,860 And if we switch over, you'll see the same thing in Squirt as well. 144 00:15:51,820 --> 00:15:55,820 Notice how they work slightly differently. 145 00:15:56,240 --> 00:16:07,800 The Squirt gives us many more alerts because it's looking individually at the packets traversing the network, versus Zeek gives us total number of connections. 146 00:16:07,800 --> 00:16:16,560 But either way, we're seeing the alerts that we're violating this zero-touch policy, and that it's a good indicator of lateral movement. 147 00:16:21,340 --> 00:16:24,640 So you want to start doing this in your environment as well. 148 00:16:24,860 --> 00:16:31,060 Well, first off, if you're not zero-touch yet, you should move that direction. 149 00:16:31,580 --> 00:16:32,820 Implement zero-touch. 150 00:16:32,820 --> 00:16:36,160 It's got a lot of benefits besides this one. 151 00:16:37,460 --> 00:16:42,560 When you're ready to implement this detection, use your platform of choice. 152 00:16:42,560 --> 00:16:45,720 Whatever you're already using is going to be best. 153 00:16:45,820 --> 00:16:51,400 There's no need to go get specialized software or hardware in order to do this. 154 00:16:51,400 --> 00:16:54,880 Use your existing IDS or network monitor. 155 00:16:55,160 --> 00:16:59,640 It's a very simple detection to include. 156 00:17:00,420 --> 00:17:06,330 But then, also pay attention to how you tailor it for your specific environment. 157 00:17:06,870 --> 00:17:15,170 In the example, it was looking for all TCP connections, because I wanted to catch the broadest range of lateral movement. 158 00:17:15,350 --> 00:17:18,290 But that might not make sense for your environment. 159 00:17:18,350 --> 00:17:29,190 So maybe you need to tailor it down to only look for SSH connections or, you know, whatever makes sense for your environment specifically. 160 00:17:30,110 --> 00:17:34,190 And then, the detections I showed were all very simple. 161 00:17:34,190 --> 00:17:37,290 They just looked for that connection and alerted. 162 00:17:37,290 --> 00:17:41,310 But that can be just the beginning of a bigger process. 163 00:17:41,310 --> 00:18:01,330 Imagine with Zeek, instead of raising a notice, if we correlated that with other suspicious traffic that we'd seen, maybe some data exfiltration traffic, and then we could automatically provide analysts with, here's the map of the attacker's path through the network. 164 00:18:02,010 --> 00:18:07,830 Starting to add some of those higher-end capabilities really makes this powerful. 165 00:18:09,330 --> 00:18:13,990 And finally, I wanted to leave you with some lessons learned from this project. 166 00:18:13,990 --> 00:18:16,610 And there are two that really jumped out at me. 167 00:18:16,890 --> 00:18:20,430 The first is know your network. 168 00:18:20,890 --> 00:18:26,550 And this is not just know your networking, it is know your specific network. 169 00:18:28,050 --> 00:18:36,150 Justin Henderson and Mick Douglas, who teach the Tactical SIEM class for SANS, really nailed this, and I learned this from them. 170 00:18:36,150 --> 00:18:38,010 So, huge shout-out to them. 171 00:18:38,890 --> 00:18:44,590 Knowing the specifics of your network lets you do really cool things in monitoring it. 172 00:18:44,730 --> 00:18:51,470 Things that aren't possible in generic networks, such as the detection we talked about today. 173 00:18:52,670 --> 00:18:57,770 And then, secondly, don't be afraid to look for stupid, simple things. 174 00:18:57,770 --> 00:19:02,030 All we were looking at today was the presence of a connection. 175 00:19:03,510 --> 00:19:20,090 Tying together these two lessons learned is really this whole project, is looking for something simple like the presence of a connection and knowing your network well enough to be able to say, that's unusual, that's a problem. 176 00:19:22,410 --> 00:19:25,890 Thank you so much for attending the presentation today . 177 00:19:26,270 --> 00:19:31,650 Shout-out again to the SANS team for their help and support with the research project. 178 00:19:31,750 --> 00:19:44,670 If you've got any questions about this project, AppSec, DevOps, the SANS Masters program, or anything related, the best way to get in contact with me is via Twitter. 179 00:19:44,670 --> 00:19:46,650 My handle's on the screen now. 180 00:19:46,770 --> 00:19:54,310 If you're viewing the live presentation, I look forward to chatting with you during the Q&A immediately afterwards in Discord. 181 00:19:54,710 --> 00:19:57,030 Again, I'm Philip Marlowe. 182 00:19:57,190 --> 00:19:58,330 Thanks for coming.