1 00:00:01,350 --> 00:00:04,950 So, just a quick rundown of what I'm going to go through here. 2 00:00:05,330 --> 00:00:07,990 I'm going to introduce myself a little bit more. 3 00:00:08,150 --> 00:00:13,290 And then, if you're not too familiar with bastion hosts or jump hosts, don't worry about it. 4 00:00:13,290 --> 00:00:18,470 I'm going to go through a quick overview of those and talk about why people use them. 5 00:00:18,470 --> 00:00:22,090 Then I'll go through some problems with bastion hosts. 6 00:00:22,190 --> 00:00:26,430 In particular, I want to go through the SSH multiplexing attack. 7 00:00:26,430 --> 00:00:37,090 And then, we're going to go through some alternatives to bastion hosts in three major cloud providers, AWS, GCP, and Azure. 8 00:00:38,350 --> 00:00:41,570 So, like Jaron mentioned, my name is Colin Estep. 9 00:00:41,910 --> 00:00:45,410 I worked on the security team at Apple. 10 00:00:45,590 --> 00:00:48,250 I also worked in security at Netflix. 11 00:00:48,250 --> 00:00:52,670 So, I had kind of a baptism by fire at Netflix learning AWS. 12 00:00:52,810 --> 00:00:55,310 And I really loved it. 13 00:00:55,310 --> 00:01:02,290 So, I sort of continued that at a startup called SIF Security, where we built some cloud security software. 14 00:01:02,530 --> 00:01:08,630 And now, I do threat research at Netscope, the Netscope Threat Labs team. 15 00:01:09,030 --> 00:01:14,010 And when I'm not working, I'm probably hanging out with my four kids. 16 00:01:14,010 --> 00:01:16,290 And I included a couple pictures here. 17 00:01:16,290 --> 00:01:17,710 I like to snowboard. 18 00:01:17,710 --> 00:01:19,490 That's at Lake Tahoe. 19 00:01:19,490 --> 00:01:22,530 And I also like taking my 4Runner off-road. 20 00:01:22,530 --> 00:01:26,550 So, whenever I manage to do those things, that's a lot of fun. 21 00:01:26,690 --> 00:01:32,290 So, if anyone has some cool mods that they've done to their off-road vehicle, I would love to see that. 22 00:01:33,250 --> 00:01:35,090 So, let's go ahead and jump right in. 23 00:01:35,090 --> 00:01:37,290 So, what are bastion hosts? 24 00:01:38,970 --> 00:01:44,830 I grabbed this diagram from an old AWS post because it's really nice and simple. 25 00:01:44,930 --> 00:01:52,450 So, it shows an AWS environment where you have one VPC, and you have a private subnet and a public subnet. 26 00:01:52,450 --> 00:02:06,290 And basically, what's happening here is the only way that the bastion host users, meaning your corporate users that need to access these systems, the only way that they can get to those instances is through these bastion hosts. 27 00:02:06,350 --> 00:02:20,170 So, the Linux instances that are running sensitive workloads for you are actually in a private subnet that's not accessible from the Internet, at least for these management protocols like SSH. 28 00:02:20,170 --> 00:02:30,330 And so, the only way that you can get to them is through creating your own bastion hosts in a public subnet that does expose SSH to your users. 29 00:02:30,810 --> 00:02:34,250 And so, what's the benefit here? 30 00:02:34,250 --> 00:02:43,910 So, the idea is that you're limiting the exposure of servers to the Internet and hopefully reducing your footprint, your attack surface. 31 00:02:43,910 --> 00:02:50,170 So, it can prevent brute force attacks on SSH for most of your infrastructure. 32 00:02:50,470 --> 00:02:57,130 Usually, those bastion hosts, the best practice is to make them simply be bastion hosts. 33 00:02:57,130 --> 00:03:02,450 So, you're running EC2 instances in AWS that are not doing anything else. 34 00:03:02,450 --> 00:03:06,550 They should not be running a website, they should not be running a database or anything like that. 35 00:03:06,550 --> 00:03:10,190 They should just be there for SSH access. 36 00:03:10,190 --> 00:03:17,030 And they should be locked down, they should be hardened, and they should have good monitoring around them. 37 00:03:17,090 --> 00:03:25,750 And that's one of the things I like about this is that you have some centralized access where you can set up really verbose logging and monitoring. 38 00:03:25,950 --> 00:03:30,850 And it's useful, it's not going to just flood the sock with random events. 39 00:03:30,870 --> 00:03:36,450 There are probably events that you want to respond to if they're happening on your bastion, if there's something weird. 40 00:03:37,790 --> 00:03:42,310 So, you may be wondering, okay, it sounds like there's alternatives. 41 00:03:42,310 --> 00:03:45,310 Maybe bastion hosts are sort of an older thing. 42 00:03:45,310 --> 00:03:46,970 Why is this still relevant? 43 00:03:46,970 --> 00:03:57,050 Well, you know, out of the customer base at Netscope, we looked at a lot of the customers who expose instances to routable IP addresses at all. 44 00:03:57,050 --> 00:04:03,530 And we found that almost half of the exposed instances actually allow SSH ingress. 45 00:04:03,530 --> 00:04:14,030 So, to me, this kind of signals that people are still running bastion hosts, people may even be running SSH on sensitive workloads directly. 46 00:04:14,150 --> 00:04:18,370 And so this is definitely something that's still around and still relevant. 47 00:04:18,370 --> 00:04:24,750 And we should look at this a little bit more closely to find what's the best approach for each case. 48 00:04:25,830 --> 00:04:29,330 So with that, let's talk about some problems with bastions. 49 00:04:29,330 --> 00:04:33,890 Again, you're maintaining EC2 instances yourself. 50 00:04:33,890 --> 00:04:36,770 So you're responsible for the OS patching. 51 00:04:36,830 --> 00:04:43,250 You're responsible for the configuration of both the network ports and the OS. 52 00:04:43,310 --> 00:04:49,990 It could become expensive if you need to distribute bastions in different regions and support a large user base. 53 00:04:49,990 --> 00:04:55,510 You're running these compute instances that could, you know, rack up on your bill. 54 00:04:56,690 --> 00:04:59,790 And you still need to manage the access all by yourself. 55 00:04:59,790 --> 00:05:03,510 So that includes managing SSH certificates. 56 00:05:03,510 --> 00:05:06,130 Maybe you have an identity provider you want to connect to. 57 00:05:06,130 --> 00:05:08,150 You've got to connect all that plumbing. 58 00:05:08,150 --> 00:05:10,690 You've got to worry about the MFA being in place. 59 00:05:10,690 --> 00:05:20,030 If you enforce MFA when someone's using your bastion hosts in order to gain access to your infrastructure, you're still dealing with all of that. 60 00:05:20,030 --> 00:05:26,270 Plus, it is vulnerable to this SSH multiplexing attack that we're going to talk about next. 61 00:05:26,270 --> 00:05:31,910 So these are just a few of the problems that you could run into if you're running your own bastion hosts. 62 00:05:33,070 --> 00:05:40,390 So before I talk about attacks, let's just talk about SSH multiplexing itself and what this actually is. 63 00:05:40,390 --> 00:05:49,350 It's really a feature of OpenSSH that allows you to reuse a TCP connection for more than one SSH session. 64 00:05:49,350 --> 00:05:57,470 So if it's configured in your SSH client, that will essentially save a file to your file system. 65 00:05:57,470 --> 00:06:03,450 And you can just invoke the SSH session again by calling up that file. 66 00:06:03,450 --> 00:06:05,110 So I have a little example here. 67 00:06:05,110 --> 00:06:08,390 There's just a small config snippet on the left side. 68 00:06:08,390 --> 00:06:12,830 And that shows how easy it is to set this up for multiplexing. 69 00:06:12,830 --> 00:06:16,930 So at the bottom, you'll see control persists 240 minutes. 70 00:06:16,930 --> 00:06:23,830 So that's how long I can keep that connection active and then still call the SSH session back up. 71 00:06:23,870 --> 00:06:27,490 And then on the right hand side, I have the command that I could use to do it. 72 00:06:27,490 --> 00:06:32,470 So I'm basically just calling the file and then telling it what server to use. 73 00:06:32,570 --> 00:06:39,190 And I've re-established my SSH connection, and I don't need to enter a password for the key. 74 00:06:39,190 --> 00:06:42,610 I don't need to use MFA, even if MFA is being enforced. 75 00:06:42,610 --> 00:06:44,310 It's already established. 76 00:06:44,310 --> 00:06:49,850 So it's kind of a shortcut to just get straight into a shell. 77 00:06:52,070 --> 00:06:56,590 So now that we've talked about multiplexing, a little bit about how that works. 78 00:06:56,590 --> 00:06:58,550 Now let's talk about the attack. 79 00:06:58,550 --> 00:07:03,970 So basically, the attack is an attacker could compromise one of your users. 80 00:07:05,050 --> 00:07:15,070 So a phishing attack or some other thing gaining shell access on an endpoint that is being used by one of your users. 81 00:07:15,970 --> 00:07:24,210 Once they get that shell access, it's pretty easy for them, like I showed, to set up this config for the SSH client. 82 00:07:24,330 --> 00:07:30,430 And the end user may be completely unaware, because it's not really going to affect anything on their end. 83 00:07:30,430 --> 00:07:37,370 So the next time they establish SSH to the bastion, it's getting saved to that socket file. 84 00:07:37,450 --> 00:07:45,070 And now the attacker can call that back up and re-establish the connection to the bastion host and circumvent MFA. 85 00:07:45,270 --> 00:07:50,270 And from there, it's pretty trivial from the bastion host to get to the actual servers. 86 00:07:50,270 --> 00:08:07,730 So even if your end users have really, really great hygiene and are not saving PII onto their laptops or whatever, this is a pretty easy way to sort of gain access to where those good things live. 87 00:08:08,410 --> 00:08:13,670 And I just wanted to give credit to the NCC group for a blog post about this. 88 00:08:13,670 --> 00:08:17,590 So on the bottom right hand corner of my slide, you'll see a link to their blog post. 89 00:08:17,590 --> 00:08:21,750 It was a really useful blog post when I was discovering this. 90 00:08:21,830 --> 00:08:24,690 So this is not something I cooked up myself. 91 00:08:26,410 --> 00:08:35,570 So now that we've talked about bastion hosts, hopefully you guys were able to follow what those are and why there might be some problems with them. 92 00:08:35,570 --> 00:08:38,150 I wanted to present you with some solutions. 93 00:08:39,310 --> 00:08:46,470 Before I dive into the details of each solution, I wanted to just sort of cover some common attributes that I found. 94 00:08:46,470 --> 00:08:51,750 And these are not necessarily all requirements that I set out to satisfy. 95 00:08:51,750 --> 00:08:56,850 These were just things that I discovered as I was exploring these different solutions. 96 00:08:57,150 --> 00:09:02,030 And I think it's good to keep these in mind as we examine each one. 97 00:09:02,030 --> 00:09:13,610 So the first one is that your end users are not actually establishing SSH or RDP connections straight up from their endpoints. 98 00:09:13,610 --> 00:09:17,930 They're going over HTTPS to start these connections. 99 00:09:20,190 --> 00:09:25,170 The next two items are really sort of related to having bastion hosts as well. 100 00:09:25,170 --> 00:09:33,530 In an ideal world, you're not going to need to have public IP addresses assigned to your sensitive workloads in order to gain access to them. 101 00:09:34,010 --> 00:09:38,710 But, you know, again, if you're running your own bastion hosts, you have to take care of all that plumbing. 102 00:09:38,710 --> 00:09:41,210 You have to make sure that it can communicate. 103 00:09:41,970 --> 00:09:47,630 The third point is, again, having to do with network exposure. 104 00:09:47,630 --> 00:09:52,610 You're not exposing these network ports to external routable IP addresses. 105 00:09:52,990 --> 00:09:57,710 With these alternative solutions, in some cases, you don't have to expose it at all. 106 00:09:58,450 --> 00:10:01,050 And it removes the risk. 107 00:10:01,050 --> 00:10:06,510 These new solutions, well, they're relatively new compared to bastion hosts. 108 00:10:06,510 --> 00:10:14,650 These alternative solutions sort of remove the risk of the SSH multiplexing attack as I presented it. 109 00:10:14,710 --> 00:10:27,190 I will point out, if you have an attacker that got remote shell on the client endpoint and they're advanced enough, I'm sure they're going to find another way to peer into HTTPS traffic or something else. 110 00:10:27,190 --> 00:10:34,170 But the low-hanging fruit of this SSH multiplexing attack will essentially go away. 111 00:10:34,170 --> 00:10:41,270 And then lastly, the logging provided by these cloud services is pretty good. 112 00:10:41,350 --> 00:10:44,390 Right out of the box, you kind of get a lot for free. 113 00:10:44,390 --> 00:10:46,950 So I'll cover that. 114 00:10:46,950 --> 00:10:51,230 So with that, let's dive into the actual services themselves. 115 00:10:51,230 --> 00:11:00,190 So I'm going to cover Session Manager, which is part of a bigger service called AWS Systems Manager in AWS. 116 00:11:00,190 --> 00:11:06,150 And then GCP has OSLogin and the Identity-Aware Proxy or IAP. 117 00:11:06,570 --> 00:11:09,410 And then Azure has the Bastion service. 118 00:11:09,410 --> 00:11:11,730 So these are the three that I'm going to cover. 119 00:11:13,690 --> 00:11:16,050 So first, let's start with AWS. 120 00:11:17,350 --> 00:11:23,310 Like I mentioned, Session Manager is part of AWS Systems Manager or SSM. 121 00:11:23,590 --> 00:11:29,010 And this is the only service that I'm going to show you that actually relies on an agent. 122 00:11:29,010 --> 00:11:39,370 So in order for you to use the model that I'm going to go through, you actually have to deploy SSM agent to each EC2 instance that you want to be able to access. 123 00:11:39,870 --> 00:11:48,990 But if you're an AWS customer who's already using Systems Manager, this might be a non-issue because you're probably already going to have it on there. 124 00:11:49,690 --> 00:11:54,290 It does not actually use SSH at all or RDP by default. 125 00:11:54,290 --> 00:12:00,690 And it can provide you full session logs, which sort of stands out from the other services. 126 00:12:01,910 --> 00:12:06,750 So with that, let's just sort of get a high-level view of how this works once it's configured. 127 00:12:06,770 --> 00:12:12,030 There's a lot of steps to get this configured, and I'm not going to go through that because that would take a long time. 128 00:12:13,110 --> 00:12:20,110 I do have a blog post that came out a couple days ago about this specific setup. 129 00:12:20,110 --> 00:12:23,590 And so that takes you into more gritty detail of how you get this working. 130 00:12:24,170 --> 00:12:29,950 But for the sake of this presentation, I'm just going to walk through the scenario so you get a high-level view of it. 131 00:12:30,310 --> 00:12:36,790 So in this scenario, I've got a bunch of EC2 instances in a private subnet in a VPC. 132 00:12:36,790 --> 00:12:43,470 So that means these EC2 instances are not exposed to the internet, at least for any sort of workload purposes. 133 00:12:44,330 --> 00:12:48,150 And I want to keep it that way, you know, especially for this access. 134 00:12:48,150 --> 00:12:55,590 So basically, within the VPC, I can create VPC endpoints. 135 00:12:55,610 --> 00:13:06,420 And this is something called AWS Private Link, where you set up a virtual sort of endpoint for APIs that you want to interact with, AWS APIs. 136 00:13:06,850 --> 00:13:14,130 And they live in the VPC, so the traffic is not going over the internet. 137 00:13:14,130 --> 00:13:21,670 So the traffic between the EC2 instances and what says SSM endpoints in my diagram is actually staying within AWS. 138 00:13:21,670 --> 00:13:23,590 And it's over HTTPS. 139 00:13:23,590 --> 00:13:33,250 So in this case, I didn't even have to create an opening for SSH within the VPC or within the subnet. 140 00:13:33,250 --> 00:13:36,420 SSH is completely locked down in this example. 141 00:13:36,890 --> 00:13:39,770 And I don't have traffic going over the internet. 142 00:13:39,770 --> 00:13:46,150 The only place that I have traffic going over the internet is the client accessing the Session Manager. 143 00:13:46,590 --> 00:13:52,210 So the client side on the far right hand side, you'll see I have three methods listed out. 144 00:13:52,210 --> 00:13:56,590 These are the main methods to launch a session with Session Manager. 145 00:13:56,590 --> 00:14:00,430 The first two are both through the AWS Web UI. 146 00:14:00,430 --> 00:14:02,930 One is within the EC2 console. 147 00:14:02,930 --> 00:14:05,890 One is within the Session Manager console. 148 00:14:05,890 --> 00:14:19,350 The reason why I listed those out here is because you can actually put the permissions for each client at such a granular level that you can specify if you don't want them using the Session Manager console. 149 00:14:19,350 --> 00:14:25,250 You can say, hey, they can only launch sessions through the EC2 console, for example. 150 00:14:25,870 --> 00:14:29,550 Or just through the AWS command line interface. 151 00:14:29,550 --> 00:14:34,030 And so in the case of the AWS command line interface, you can do that. 152 00:14:34,030 --> 00:14:39,570 But for some reason, you need to install a plugin in order for that to work with Session Manager. 153 00:14:41,450 --> 00:14:44,430 But those are three ways of doing it. 154 00:14:44,550 --> 00:14:50,170 And it's communicated over HTTPS as well on that side of things. 155 00:14:50,170 --> 00:14:52,910 So Session Manager kind of sits in the middle. 156 00:14:52,910 --> 00:14:58,330 The service itself is sitting in the middle, providing that shell access to the end client. 157 00:15:00,490 --> 00:15:08,730 So once you're doing this, once you're actually using that for sessions, how do you sort of keep track of what's happening? 158 00:15:08,730 --> 00:15:20,590 So the great thing here is for anyone who's not familiar, AWS has a logging facility called CloudTrail, which keeps tabs on what's happening in your account if you enable it. 159 00:15:21,310 --> 00:15:29,470 And you save it off, you can look at these CloudTrail logs, and it provides a nice audit trail of everything your users are doing. 160 00:15:29,530 --> 00:15:37,530 And so by default, without you configuring anything else, SSM will log events around the beginning and ending of sessions. 161 00:15:37,850 --> 00:15:49,330 And these sessions contain some nice metadata about, I mean, these events contain some nice metadata about the sessions, such as the AWS user that's making these requests. 162 00:15:50,130 --> 00:15:56,390 In the cases where multi-factor is in the workflow, that will be logged as well. 163 00:15:56,550 --> 00:16:11,550 I'll just point out if your user has the AWS CLI, and they are using their API key, and MFA is not being enforced there, obviously, that's a way around using MFA. 164 00:16:11,750 --> 00:16:13,770 So that's something to be aware of. 165 00:16:13,770 --> 00:16:19,750 It also includes the EC2 instance ID that's being targeted for access. 166 00:16:19,870 --> 00:16:30,850 And then the other basics that you're usually going to see for these sort of audit events, like the IP address it's coming from, the timestamps, and whether it's allowed or denied. 167 00:16:32,250 --> 00:16:38,290 Now, another great thing about Session Manager is it goes beyond just that CloudTrail logging. 168 00:16:38,290 --> 00:16:52,770 If you want full session logs, and what I mean by that is if you want to see, once the users get on the system, what they've typed in and what the results were, you can actually obtain that pretty easily. 169 00:16:52,770 --> 00:17:04,290 So once you configure that logging in Session Manager, you're just going to want to elect where to save the logs. 170 00:17:04,290 --> 00:17:18,250 So if whether you do data lake from S3 buckets, or you use CloudWatch to ship logs off somewhere else, or whatever your case may be, you can use either one of those options to save off the SSM logs. 171 00:17:18,250 --> 00:17:34,610 And in my example here, I'm just showing again, in the case that I have EC2 instances that I want to stay private, and minimize internet traffic, I can set up endpoints and get those logs shipped off to where they need to go without it going through the internet. 172 00:17:34,610 --> 00:17:36,930 So it's a really great feature. 173 00:17:37,870 --> 00:17:52,870 And one thing I do want to point out is that, excuse me, if one thing that they do provide in Session Manager is the ability to say tunnel SSH over those HTTPS connections. 174 00:17:52,870 --> 00:18:03,190 And if that happens, if you elect to do that, the only shortfall here is you won't have these full session logs because it's encrypted traffic with SSH. 175 00:18:05,530 --> 00:18:09,410 Okay, so next, we're going to go over the GCP solutions. 176 00:18:09,410 --> 00:18:12,510 That's using OS login and IAP. 177 00:18:15,830 --> 00:18:21,870 So in GCP, you don't need any agents to be deployed to your virtual machines. 178 00:18:22,230 --> 00:18:24,850 I will point out it's very easy to set up. 179 00:18:24,850 --> 00:18:33,470 So if your organization is a G Suite organization that uses GCP, it's very easy to set this up. 180 00:18:33,690 --> 00:18:35,730 It's like a matter of minutes. 181 00:18:35,730 --> 00:18:45,550 And it's great because you can pair your local OS level users that are SSHing in with the Google IAM user. 182 00:18:45,570 --> 00:18:49,830 It also provides LDAP and AD support if you need to do that. 183 00:18:50,190 --> 00:18:55,870 And again, it logs metadata of the sessions for free, very similar to AWS. 184 00:18:55,870 --> 00:18:57,950 And it's super easy. 185 00:18:57,950 --> 00:19:07,070 If you use two-factor authentication for G Suite, you can basically do like a checkbox and enable it for SSH as well. 186 00:19:09,310 --> 00:19:12,510 So we'll go through how this works a little bit. 187 00:19:12,510 --> 00:19:17,770 So again, you have servers in a VPC that are private. 188 00:19:17,770 --> 00:19:20,770 And I don't want these exposed to the internet. 189 00:19:20,770 --> 00:19:25,190 So there's no public IP addresses associated with them, or any of that. 190 00:19:25,310 --> 00:19:35,850 And what I have to do is create an... well, I have to enable Identity-Aware Proxy or IAP in the project itself. 191 00:19:35,850 --> 00:19:40,330 So you'll notice here in the diagram, I have the Google Cloud Platform box. 192 00:19:40,330 --> 00:19:42,870 And then I have a project box within that. 193 00:19:42,870 --> 00:19:50,090 And so I put IAP in there on purpose, because you have to create that in each project you want to use it. 194 00:19:51,050 --> 00:20:01,030 And so once you've enabled it, essentially, for anyone who's not aware, in GCP, you have to enable the APIs you want to use, with a few exceptions. 195 00:20:01,030 --> 00:20:03,950 But for the most part, you have to enable everything you want to use. 196 00:20:03,950 --> 00:20:07,510 And so IAP is one of those that you have to enable it first. 197 00:20:07,510 --> 00:20:10,870 So the admin or whoever will enable that. 198 00:20:10,970 --> 00:20:15,270 And now you can say that you want to use it with TCP forwarding. 199 00:20:15,290 --> 00:20:24,810 And that's where it gives you pretty good access to SSH into your servers directly. 200 00:20:24,910 --> 00:20:34,490 So you hit IAP with HTTPS coming from your user, and then it creates the tunnel for you to your server. 201 00:20:34,490 --> 00:20:40,050 And then OS login is a second service that you can be using for logging into the servers. 202 00:20:40,050 --> 00:20:43,450 So IAP is basically just sort of the networking piece. 203 00:20:43,450 --> 00:20:56,230 So the servers that you have in your VPC have to allow the IAP endpoint range, the IP address range, to access port 22 on the servers. 204 00:20:56,290 --> 00:21:00,910 But other than that, you can have SSH locked down and blocked for everything else. 205 00:21:00,910 --> 00:21:06,410 And then OS login is sort of the SSH off portion of it. 206 00:21:06,890 --> 00:21:10,110 So once you hit the servers, it goes and checks. 207 00:21:10,150 --> 00:21:16,830 If OS login is enabled for those machines, you can do it on the individual machine level or in the project. 208 00:21:16,950 --> 00:21:25,590 And if the user who's making the Google IAM user who's making the request is authorized by OS login, they'll be able to establish a connection. 209 00:21:25,590 --> 00:21:42,350 Now, one of the great things that I like about this is even though it's still using SSH, if the OS login is all set up, and the user hasn't even set up SSH keys on these machines, it'll set up an ephemeral pair of keys for you and establish the connection. 210 00:21:42,950 --> 00:21:48,270 And so you still don't have to manage SSH keys, even though it is using SSH. 211 00:21:49,030 --> 00:22:02,170 And on the bottom of the diagram, I'm just showing basically for your overview here that SSH traffic is not being allowed from the internet at all. 212 00:22:04,630 --> 00:22:08,070 So what do you get from this? 213 00:22:08,070 --> 00:22:11,370 You're getting, again, in the default audit logs. 214 00:22:11,710 --> 00:22:13,830 Within Google, there's two different types. 215 00:22:13,830 --> 00:22:17,730 There's administrative action, I think is what it's called, and then data access. 216 00:22:17,730 --> 00:22:24,770 And when IAP is authorizing a session for a user, we'll see entries in the data access logs. 217 00:22:24,770 --> 00:22:34,730 So the audit logs that you get for free in GCP will contain events where you're establishing and removing connections. 218 00:22:34,750 --> 00:22:47,430 And so that'll contain the same kind of info we had with AWS, where it's got the primary email of the Google identity, the destination IP import, instance ID, and all the rest. 219 00:22:49,270 --> 00:22:54,710 Now, if you want additional logs, I borrowed this diagram from the GCP documentation. 220 00:22:55,270 --> 00:23:04,810 It's kind of a complicated looking diagram just to show, basically, you can install a logging agent on your virtual machines. 221 00:23:04,870 --> 00:23:12,630 And you can set it up to take things like syslog and process that and send it off to Stackdriver Logging. 222 00:23:12,630 --> 00:23:17,070 Stackdriver Logging is where you have those audit logs that I was just talking about. 223 00:23:17,070 --> 00:23:21,670 So Stackdriver is sort of the centralized place that you go in GCP to look at logs. 224 00:23:21,830 --> 00:23:25,550 And so it's just showing here that it's fairly straightforward. 225 00:23:25,550 --> 00:23:36,530 If your virtual machines don't have internet access, you need to enable private access, Google private access, and that'll allow it to hit the logging API endpoints. 226 00:23:36,910 --> 00:23:40,510 And the logging agent will ship these logs off for you. 227 00:23:40,530 --> 00:23:46,650 And, you know, you can do, if you want debug level logs on things, you can get that off of your endpoints. 228 00:23:46,650 --> 00:23:53,870 But that's basically your option here if you wanted to get additional things from the virtual machines. 229 00:23:54,730 --> 00:23:56,390 So that's it for Google. 230 00:23:56,390 --> 00:23:58,630 It's really sort of quick and easy. 231 00:23:58,710 --> 00:24:03,470 The next one we'll go through is in Azure, the Azure bastion hosts. 232 00:24:03,910 --> 00:24:11,190 So basically, in Azure, you're again connecting over HTTPS to the bastion. 233 00:24:11,190 --> 00:24:14,910 And then that connects to the servers via SSH or RDP. 234 00:24:15,470 --> 00:24:20,030 You still need to maintain your SSH certificates with this particular one. 235 00:24:20,290 --> 00:24:26,150 And you expose the SSH port to internal traffic so that the bastion can get to it. 236 00:24:26,790 --> 00:24:30,490 A great feature here is that it works with just-in-time access. 237 00:24:30,590 --> 00:24:39,650 So in case you're not aware of that, even though you need to enable the SSH port from internal traffic, you can use it with just-in-time. 238 00:24:39,650 --> 00:24:44,470 So just-in-time is where you block SSH in your firewall rules. 239 00:24:44,470 --> 00:24:53,310 But anytime you want to establish an SSH connection, you request access, and then it gets granted through the just-in-time service. 240 00:24:53,490 --> 00:24:58,930 And that enables SSH for, say, an hour or three hours, whatever it is. 241 00:24:58,950 --> 00:25:03,130 And then it goes back to being blocked automatically after the time expires. 242 00:25:04,550 --> 00:25:07,350 And I borrowed this diagram from Azure. 243 00:25:07,350 --> 00:25:11,610 I thought it was pretty good, so I didn't bother reinventing the wheel here. 244 00:25:13,130 --> 00:25:20,710 This shows the user hitting the Azure portal, and it establishes a connection on the Azure bastion. 245 00:25:20,710 --> 00:25:28,750 So in your virtual network where you want to have this level of access, you have to create an Azure bastion instance. 246 00:25:28,750 --> 00:25:31,570 Now, that's not an instance of a virtual machine. 247 00:25:31,570 --> 00:25:34,650 It's an abstracted sort of instance. 248 00:25:34,650 --> 00:25:38,450 So you're not maintaining the OS-level patching or any of that. 249 00:25:38,450 --> 00:25:41,690 It's just sitting there for your use. 250 00:25:41,710 --> 00:25:46,310 And it has to sit in its own subnet called Azure bastion subnet. 251 00:25:46,690 --> 00:25:56,570 And so you access that through HTTPS, and then the bastion allows you to access SSH on your virtual machine. 252 00:25:58,850 --> 00:26:00,670 And so what does this look like? 253 00:26:00,670 --> 00:26:04,370 I mentioned you have to kind of still manage the SSH certs. 254 00:26:04,370 --> 00:26:11,030 So what this looks like here is this is a screenshot from me going through the virtual machine. 255 00:26:11,450 --> 00:26:16,410 I'm looking at the virtual machine in my Azure portal, and I say I want to connect to it. 256 00:26:16,410 --> 00:26:17,850 And it gives me three options. 257 00:26:17,850 --> 00:26:20,270 It says RDP, SSH, and bastion. 258 00:26:20,390 --> 00:26:30,830 So connecting via bastion, I put in the local username for my SSH user, and then I can put in my private key here. 259 00:26:30,830 --> 00:26:34,330 But the public key needs to be sitting on that virtual machine. 260 00:26:35,570 --> 00:26:39,270 And you may be asking, does this support AD? 261 00:26:39,270 --> 00:26:41,330 And it does not currently. 262 00:26:41,770 --> 00:26:45,730 I saw that Microsoft says that they're working on that. 263 00:26:45,730 --> 00:26:53,610 So you can set up AD shell access, but you can't do it through a bastion. 264 00:26:56,030 --> 00:26:59,290 So what do you get when you're using the bastion host? 265 00:26:59,290 --> 00:27:07,870 So one unique thing that you have here that stands out from the other two cloud providers is you can see active sessions as they are happening. 266 00:27:08,050 --> 00:27:12,770 And if you are an admin, you can actually delete sessions. 267 00:27:12,770 --> 00:27:22,270 So if a user contacts you and says, hey, I'm noticing something weird and you think there's a compromise, you can come in here and you can knock sessions off. 268 00:27:22,590 --> 00:27:24,990 So that was kind of interesting to see that. 269 00:27:25,070 --> 00:27:27,070 And this is what the screen looks like. 270 00:27:27,070 --> 00:27:34,290 So you can see here I have one active connection to call in server and I can see the target IP address. 271 00:27:34,290 --> 00:27:40,690 And that IP address is obviously not routable because I wanted it to stay in a private VNet. 272 00:27:41,110 --> 00:27:44,850 And it's using the SSH protocol. 273 00:27:44,850 --> 00:27:47,810 So you can see some of the good details there. 274 00:27:49,290 --> 00:27:53,550 And again, with this provider, you get similar support. 275 00:27:53,550 --> 00:27:55,530 So Azure makes it really easy. 276 00:27:55,530 --> 00:28:01,470 You don't get it quite for free, but it's so easy to set up and send it off that it's pretty much for free. 277 00:28:01,570 --> 00:28:06,910 So you just have to tell Azure where you want to send the audit logs from the bastion. 278 00:28:07,350 --> 00:28:12,490 And so these are the three choices, log analytics, a storage account or an event hub. 279 00:28:12,670 --> 00:28:17,430 And I did not see any facility for setting up full session logs at this time. 280 00:28:18,010 --> 00:28:21,510 Because again, it's using SSH, so it's going to be encrypted. 281 00:28:22,350 --> 00:28:26,470 So if you sent it off to log analytics, this is kind of what it would look like. 282 00:28:26,470 --> 00:28:35,830 Again, you get some useful metadata around the connections themselves and when they're ended and the protocols they're using. 283 00:28:37,770 --> 00:28:41,330 So that's it for the deep dive. 284 00:28:41,930 --> 00:28:48,610 So just to kind of wrap things up here, running your own bastion hosts may not be necessary. 285 00:28:48,610 --> 00:28:59,950 If you're looking at rolling something out, if your organization is migrating to the cloud and you're thinking about rolling out bastion hosts, you should really look at these alternatives. 286 00:28:59,950 --> 00:29:01,490 You may not need to. 287 00:29:03,050 --> 00:29:10,050 If you need to, for some reason, just at least be aware that these SSH multiplexing attacks exist. 288 00:29:10,050 --> 00:29:13,530 And so there's ways of dealing with it. 289 00:29:13,530 --> 00:29:21,270 But I think personally, the best way of dealing with it is just not to have to manage your own bastion hosts. 290 00:29:22,090 --> 00:29:28,530 Cloud providers and even third party vendors provide some great alternatives for managing this level of access. 291 00:29:28,550 --> 00:29:31,390 So it's definitely worth looking at. 292 00:29:32,370 --> 00:29:40,690 These solutions that are out there, they can help with basically all aspects of the bastion host issue. 293 00:29:41,270 --> 00:29:46,690 If you're running your own bastions, you're running more compute instances that you have to manage. 294 00:29:46,690 --> 00:29:51,390 You have to manage the patching schedule, you have to deal with the costs, all of that. 295 00:29:51,450 --> 00:29:56,930 And then there's identity management that comes into play and the logging and monitoring. 296 00:29:56,930 --> 00:30:02,230 I was surprised at the level of logging and monitoring you can get from these cloud providers right out of the box. 297 00:30:02,390 --> 00:30:05,410 So it's definitely something worth considering there. 298 00:30:06,670 --> 00:30:09,330 So with that, I'll take any questions. 299 00:30:09,570 --> 00:30:21,090 One thing I wanted to plug here is if this sort of content is useful to you, I highly encourage you to follow our blog, the Netscope Threat Labs blog. 300 00:30:21,090 --> 00:30:29,110 We post things about threats that we find, but we also post stuff that we just find interesting, things that we dream up. 301 00:30:29,310 --> 00:30:32,850 It could be some useful content. 302 00:30:32,850 --> 00:30:38,310 So I highly encourage anybody who's interested to check that out and follow us on Twitter. 303 00:30:38,310 --> 00:30:42,670 You can reach out to me on Twitter or LinkedIn if you have any more questions. 304 00:30:42,670 --> 00:30:51,750 Or even if you're like, hey, I've been looking at these things and what do you think of some other issue I didn't bring up here? 305 00:30:51,750 --> 00:30:56,650 I love to nerd out about cloud stuff, cloud security.