start	end	text
1350	4950	So, just a quick rundown of what I'm going to go through here.
5330	7990	I'm going to introduce myself a little bit more.
8150	12290	And then, if you're not too familiar with bastion hosts or jump hosts,
12290	15030	don't worry about it. I'm going to go through a quick overview of those
15630	18470	and talk about why people use them.
18470	22090	Then I'll go through some problems with bastion hosts.
22190	26430	In particular, I want to go through the SSH multiplexing attack.
26430	32430	And then, we're going to go through some alternatives to bastion hosts
32430	37090	in three major cloud providers, AWS, GCP, and Azure.
38350	41570	So, like Jaron mentioned, my name is Colin Estep.
41910	45410	I worked on the security team at Apple.
45590	48250	I also worked in security at Netflix.
48250	52670	So, I had kind of a baptism by fire at Netflix learning AWS.
52810	55310	And I really loved it.
55310	59690	So, I sort of continued that at a startup called SIF Security,
59690	62290	where we built some cloud security software.
62530	66410	And now, I do threat research at Netscope,
66410	68630	the Netscope Threat Labs team.
69030	74010	And when I'm not working, I'm probably hanging out with my four kids.
74010	76290	And I included a couple pictures here.
76290	79490	I like to snowboard. That's at Lake Tahoe.
79490	82530	And I also like taking my 4Runner off-road.
82530	86550	So, whenever I manage to do those things, that's a lot of fun.
86690	90870	So, if anyone has some cool mods that they've done to their off-road vehicle,
90870	92290	I would love to see that.
93250	97290	So, let's go ahead and jump right in. So, what are bastion hosts?
98970	102290	I grabbed this diagram from an old AWS post
102290	104830	because it's really nice and simple.
104930	108650	So, it shows an AWS environment where you have one VPC,
109090	112450	and you have a private subnet and a public subnet.
112450	117690	And basically, what's happening here is the only way that the bastion host users,
117690	121450	meaning your corporate users that need to access these systems,
121450	126290	the only way that they can get to those instances is through these bastion hosts.
126350	131550	So, the Linux instances that are running sensitive workloads for you
131550	136310	are actually in a private subnet that's not accessible from the Internet,
136310	140170	at least for these management protocols like SSH.
140170	145170	And so, the only way that you can get to them is through creating your own bastion hosts
145170	150330	in a public subnet that does expose SSH to your users.
150810	154250	And so, what's the benefit here?
154250	160570	So, the idea is that you're limiting the exposure of servers to the Internet
160570	163910	and hopefully reducing your footprint, your attack surface.
163910	170170	So, it can prevent brute force attacks on SSH for most of your infrastructure.
170470	177130	Usually, those bastion hosts, the best practice is to make them simply be bastion hosts.
177130	182450	So, you're running EC2 instances in AWS that are not doing anything else.
182450	186550	They should not be running a website, they should not be running a database or anything like that.
186550	190190	They should just be there for SSH access.
190190	197030	And they should be locked down, they should be hardened, and they should have good monitoring around them.
197090	201450	And that's one of the things I like about this is that you have some centralized access
201450	205750	where you can set up really verbose logging and monitoring.
205950	210850	And it's useful, it's not going to just flood the sock with random events.
210870	216450	There are probably events that you want to respond to if they're happening on your bastion, if there's something weird.
217790	222310	So, you may be wondering, okay, it sounds like there's alternatives.
222310	225310	Maybe bastion hosts are sort of an older thing.
225310	226970	Why is this still relevant?
226970	230730	Well, you know, out of the customer base at Netscope,
230730	237050	we looked at a lot of the customers who expose instances to routable IP addresses at all.
237050	243530	And we found that almost half of the exposed instances actually allow SSH ingress.
243530	248910	So, to me, this kind of signals that people are still running bastion hosts,
248910	254030	people may even be running SSH on sensitive workloads directly.
254150	258370	And so this is definitely something that's still around and still relevant.
258370	264750	And we should look at this a little bit more closely to find what's the best approach for each case.
265830	269330	So with that, let's talk about some problems with bastions.
269330	273890	Again, you're maintaining EC2 instances yourself.
273890	276770	So you're responsible for the OS patching.
276830	283250	You're responsible for the configuration of both the network ports and the OS.
283310	289990	It could become expensive if you need to distribute bastions in different regions and support a large user base.
289990	295510	You're running these compute instances that could, you know, rack up on your bill.
296690	299790	And you still need to manage the access all by yourself.
299790	303510	So that includes managing SSH certificates.
303510	306130	Maybe you have an identity provider you want to connect to.
306130	308150	You've got to connect all that plumbing.
308150	310690	You've got to worry about the MFA being in place.
310690	317470	If you enforce MFA when someone's using your bastion hosts in order to gain access to your infrastructure,
317470	320030	you're still dealing with all of that.
320030	326270	Plus, it is vulnerable to this SSH multiplexing attack that we're going to talk about next.
326270	331910	So these are just a few of the problems that you could run into if you're running your own bastion hosts.
333070	340390	So before I talk about attacks, let's just talk about SSH multiplexing itself and what this actually is.
340390	349350	It's really a feature of OpenSSH that allows you to reuse a TCP connection for more than one SSH session.
349350	357470	So if it's configured in your SSH client, that will essentially save a file to your file system.
357470	363450	And you can just invoke the SSH session again by calling up that file.
363450	368390	So I have a little example here. There's just a small config snippet on the left side.
368390	372830	And that shows how easy it is to set this up for multiplexing.
372830	376930	So at the bottom, you'll see control persists 240 minutes.
376930	383830	So that's how long I can keep that connection active and then still call the SSH session back up.
383870	387490	And then on the right hand side, I have the command that I could use to do it.
387490	392470	So I'm basically just calling the file and then telling it what server to use.
392570	399190	And I've re-established my SSH connection, and I don't need to enter a password for the key.
399190	404310	I don't need to use MFA, even if MFA is being enforced. It's already established.
404310	409850	So it's kind of a shortcut to just get straight into a shell.
412070	416590	So now that we've talked about multiplexing, a little bit about how that works.
416590	418550	Now let's talk about the attack.
418550	423970	So basically, the attack is an attacker could compromise one of your users.
425050	435070	So a phishing attack or some other thing gaining shell access on an endpoint that is being used by one of your users.
435970	444210	Once they get that shell access, it's pretty easy for them, like I showed, to set up this config for the SSH client.
444330	450430	And the end user may be completely unaware, because it's not really going to affect anything on their end.
450430	457370	So the next time they establish SSH to the bastion, it's getting saved to that socket file.
457450	465070	And now the attacker can call that back up and re-establish the connection to the bastion host and circumvent MFA.
465270	470270	And from there, it's pretty trivial from the bastion host to get to the actual servers.
470270	479770	So even if your end users have really, really great hygiene and are not saving PII onto their laptops or whatever,
479770	487730	this is a pretty easy way to sort of gain access to where those good things live.
488410	493670	And I just wanted to give credit to the NCC group for a blog post about this.
493670	497590	So on the bottom right hand corner of my slide, you'll see a link to their blog post.
497590	501750	It was a really useful blog post when I was discovering this.
501830	504690	So this is not something I cooked up myself.
506410	515570	So now that we've talked about bastion hosts, hopefully you guys were able to follow what those are and why there might be some problems with them.
515570	518150	I wanted to present you with some solutions.
519310	526470	Before I dive into the details of each solution, I wanted to just sort of cover some common attributes that I found.
526470	531750	And these are not necessarily all requirements that I set out to satisfy.
531750	536850	These were just things that I discovered as I was exploring these different solutions.
537150	542030	And I think it's good to keep these in mind as we examine each one.
542030	553610	So the first one is that your end users are not actually establishing SSH or RDP connections straight up from their endpoints.
553610	557930	They're going over HTTPS to start these connections.
560190	565170	The next two items are really sort of related to having bastion hosts as well.
565170	573530	In an ideal world, you're not going to need to have public IP addresses assigned to your sensitive workloads in order to gain access to them.
574010	578710	But, you know, again, if you're running your own bastion hosts, you have to take care of all that plumbing.
578710	581210	You have to make sure that it can communicate.
581970	587630	The third point is, again, having to do with network exposure.
587630	592610	You're not exposing these network ports to external routable IP addresses.
592990	597710	With these alternative solutions, in some cases, you don't have to expose it at all.
598450	601050	And it removes the risk.
601050	606510	These new solutions, well, they're relatively new compared to bastion hosts.
606510	614650	These alternative solutions sort of remove the risk of the SSH multiplexing attack as I presented it.
614710	627190	I will point out, if you have an attacker that got remote shell on the client endpoint and they're advanced enough, I'm sure they're going to find another way to peer into HTTPS traffic or something else.
627190	634170	But the low-hanging fruit of this SSH multiplexing attack will essentially go away.
634170	641270	And then lastly, the logging provided by these cloud services is pretty good.
641350	644390	Right out of the box, you kind of get a lot for free.
644390	646950	So I'll cover that.
646950	651230	So with that, let's dive into the actual services themselves.
651230	660190	So I'm going to cover Session Manager, which is part of a bigger service called AWS Systems Manager in AWS.
660190	666150	And then GCP has OSLogin and the Identity-Aware Proxy or IAP.
666570	669410	And then Azure has the Bastion service.
669410	671730	So these are the three that I'm going to cover.
673690	676050	So first, let's start with AWS.
677350	683310	Like I mentioned, Session Manager is part of AWS Systems Manager or SSM.
683590	689010	And this is the only service that I'm going to show you that actually relies on an agent.
689010	699370	So in order for you to use the model that I'm going to go through, you actually have to deploy SSM agent to each EC2 instance that you want to be able to access.
699870	708990	But if you're an AWS customer who's already using Systems Manager, this might be a non-issue because you're probably already going to have it on there.
709690	714290	It does not actually use SSH at all or RDP by default.
714290	720690	And it can provide you full session logs, which sort of stands out from the other services.
721910	726750	So with that, let's just sort of get a high-level view of how this works once it's configured.
726770	732030	There's a lot of steps to get this configured, and I'm not going to go through that because that would take a long time.
733110	740110	I do have a blog post that came out a couple days ago about this specific setup.
740110	743590	And so that takes you into more gritty detail of how you get this working.
744170	749950	But for the sake of this presentation, I'm just going to walk through the scenario so you get a high-level view of it.
750310	756790	So in this scenario, I've got a bunch of EC2 instances in a private subnet in a VPC.
756790	763470	So that means these EC2 instances are not exposed to the internet, at least for any sort of workload purposes.
764330	768150	And I want to keep it that way, you know, especially for this access.
768150	775590	So basically, within the VPC, I can create VPC endpoints.
775610	786420	And this is something called AWS Private Link, where you set up a virtual sort of endpoint for APIs that you want to interact with, AWS APIs.
786850	794130	And they live in the VPC, so the traffic is not going over the internet.
794130	801670	So the traffic between the EC2 instances and what says SSM endpoints in my diagram is actually staying within AWS.
801670	803590	And it's over HTTPS.
803590	813250	So in this case, I didn't even have to create an opening for SSH within the VPC or within the subnet.
813250	816420	SSH is completely locked down in this example.
816890	819770	And I don't have traffic going over the internet.
819770	826150	The only place that I have traffic going over the internet is the client accessing the Session Manager.
826590	832210	So the client side on the far right hand side, you'll see I have three methods listed out.
832210	836590	These are the main methods to launch a session with Session Manager.
836590	840430	The first two are both through the AWS Web UI.
840430	842930	One is within the EC2 console.
842930	845890	One is within the Session Manager console.
845890	859350	The reason why I listed those out here is because you can actually put the permissions for each client at such a granular level that you can specify if you don't want them using the Session Manager console.
859350	865250	You can say, hey, they can only launch sessions through the EC2 console, for example.
865870	869550	Or just through the AWS command line interface.
869550	874030	And so in the case of the AWS command line interface, you can do that.
874030	879570	But for some reason, you need to install a plugin in order for that to work with Session Manager.
881450	884430	But those are three ways of doing it.
884550	890170	And it's communicated over HTTPS as well on that side of things.
890170	892910	So Session Manager kind of sits in the middle.
892910	898330	The service itself is sitting in the middle, providing that shell access to the end client.
900490	908730	So once you're doing this, once you're actually using that for sessions, how do you sort of keep track of what's happening?
908730	920590	So the great thing here is for anyone who's not familiar, AWS has a logging facility called CloudTrail, which keeps tabs on what's happening in your account if you enable it.
921310	929470	And you save it off, you can look at these CloudTrail logs, and it provides a nice audit trail of everything your users are doing.
929530	937530	And so by default, without you configuring anything else, SSM will log events around the beginning and ending of sessions.
937850	949330	And these sessions contain some nice metadata about, I mean, these events contain some nice metadata about the sessions, such as the AWS user that's making these requests.
950130	956390	In the cases where multi-factor is in the workflow, that will be logged as well.
956550	971550	I'll just point out if your user has the AWS CLI, and they are using their API key, and MFA is not being enforced there, obviously, that's a way around using MFA.
971750	973770	So that's something to be aware of.
973770	979750	It also includes the EC2 instance ID that's being targeted for access.
979870	990850	And then the other basics that you're usually going to see for these sort of audit events, like the IP address it's coming from, the timestamps, and whether it's allowed or denied.
992250	998290	Now, another great thing about Session Manager is it goes beyond just that CloudTrail logging.
998290	1012770	If you want full session logs, and what I mean by that is if you want to see, once the users get on the system, what they've typed in and what the results were, you can actually obtain that pretty easily.
1012770	1024290	So once you configure that logging in Session Manager, you're just going to want to elect where to save the logs.
1024290	1038250	So if whether you do data lake from S3 buckets, or you use CloudWatch to ship logs off somewhere else, or whatever your case may be, you can use either one of those options to save off the SSM logs.
1038250	1047490	And in my example here, I'm just showing again, in the case that I have EC2 instances that I want to stay private, and minimize internet traffic,
1047490	1056930	I can set up endpoints and get those logs shipped off to where they need to go without it going through the internet. So it's a really great feature.
1057870	1072870	And one thing I do want to point out is that, excuse me, if one thing that they do provide in Session Manager is the ability to say tunnel SSH over those HTTPS connections.
1072870	1083190	And if that happens, if you elect to do that, the only shortfall here is you won't have these full session logs because it's encrypted traffic with SSH.
1085530	1092510	Okay, so next, we're going to go over the GCP solutions. That's using OS login and IAP.
1095830	1104850	So in GCP, you don't need any agents to be deployed to your virtual machines. I will point out it's very easy to set up.
1104850	1115730	So if your organization is a G Suite organization that uses GCP, it's very easy to set this up. It's like a matter of minutes.
1115730	1125550	And it's great because you can pair your local OS level users that are SSHing in with the Google IAM user.
1125570	1135870	It also provides LDAP and AD support if you need to do that. And again, it logs metadata of the sessions for free, very similar to AWS.
1135870	1147070	And it's super easy. If you use two-factor authentication for G Suite, you can basically do like a checkbox and enable it for SSH as well.
1149310	1157770	So we'll go through how this works a little bit. So again, you have servers in a VPC that are private.
1157770	1165190	And I don't want these exposed to the internet. So there's no public IP addresses associated with them, or any of that.
1165310	1175850	And what I have to do is create an... well, I have to enable Identity-Aware Proxy or IAP in the project itself.
1175850	1182870	So you'll notice here in the diagram, I have the Google Cloud Platform box. And then I have a project box within that.
1182870	1190090	And so I put IAP in there on purpose, because you have to create that in each project you want to use it.
1191050	1201030	And so once you've enabled it, essentially, for anyone who's not aware, in GCP, you have to enable the APIs you want to use, with a few exceptions.
1201030	1207510	But for the most part, you have to enable everything you want to use. And so IAP is one of those that you have to enable it first.
1207510	1215270	So the admin or whoever will enable that. And now you can say that you want to use it with TCP forwarding.
1215290	1224810	And that's where it gives you pretty good access to SSH into your servers directly.
1224910	1234490	So you hit IAP with HTTPS coming from your user, and then it creates the tunnel for you to your server.
1234490	1240050	And then OS login is a second service that you can be using for logging into the servers.
1240050	1243450	So IAP is basically just sort of the networking piece.
1243450	1256230	So the servers that you have in your VPC have to allow the IAP endpoint range, the IP address range, to access port 22 on the servers.
1256290	1260910	But other than that, you can have SSH locked down and blocked for everything else.
1260910	1266410	And then OS login is sort of the SSH off portion of it.
1266890	1276830	So once you hit the servers, it goes and checks. If OS login is enabled for those machines, you can do it on the individual machine level or in the project.
1276950	1285590	And if the user who's making the Google IAM user who's making the request is authorized by OS login, they'll be able to establish a connection.
1285590	1296890	Now, one of the great things that I like about this is even though it's still using SSH, if the OS login is all set up, and the user hasn't even set up SSH keys on these machines,
1296890	1302350	it'll set up an ephemeral pair of keys for you and establish the connection.
1302950	1308270	And so you still don't have to manage SSH keys, even though it is using SSH.
1309030	1322170	And on the bottom of the diagram, I'm just showing basically for your overview here that SSH traffic is not being allowed from the internet at all.
1324630	1331370	So what do you get from this? You're getting, again, in the default audit logs.
1331710	1337730	Within Google, there's two different types. There's administrative action, I think is what it's called, and then data access.
1337730	1344770	And when IAP is authorizing a session for a user, we'll see entries in the data access logs.
1344770	1354730	So the audit logs that you get for free in GCP will contain events where you're establishing and removing connections.
1354750	1367430	And so that'll contain the same kind of info we had with AWS, where it's got the primary email of the Google identity, the destination IP import, instance ID, and all the rest.
1369270	1374710	Now, if you want additional logs, I borrowed this diagram from the GCP documentation.
1375270	1384810	It's kind of a complicated looking diagram just to show, basically, you can install a logging agent on your virtual machines.
1384870	1392630	And you can set it up to take things like syslog and process that and send it off to Stackdriver Logging.
1392630	1397070	Stackdriver Logging is where you have those audit logs that I was just talking about.
1397070	1401670	So Stackdriver is sort of the centralized place that you go in GCP to look at logs.
1401830	1405550	And so it's just showing here that it's fairly straightforward.
1405550	1416530	If your virtual machines don't have internet access, you need to enable private access, Google private access, and that'll allow it to hit the logging API endpoints.
1416910	1420510	And the logging agent will ship these logs off for you.
1420530	1426650	And, you know, you can do, if you want debug level logs on things, you can get that off of your endpoints.
1426650	1433870	But that's basically your option here if you wanted to get additional things from the virtual machines.
1434730	1438630	So that's it for Google. It's really sort of quick and easy.
1438710	1443470	The next one we'll go through is in Azure, the Azure bastion hosts.
1443910	1451190	So basically, in Azure, you're again connecting over HTTPS to the bastion.
1451190	1454910	And then that connects to the servers via SSH or RDP.
1455470	1460030	You still need to maintain your SSH certificates with this particular one.
1460290	1466150	And you expose the SSH port to internal traffic so that the bastion can get to it.
1466790	1470490	A great feature here is that it works with just-in-time access.
1470590	1479650	So in case you're not aware of that, even though you need to enable the SSH port from internal traffic, you can use it with just-in-time.
1479650	1484470	So just-in-time is where you block SSH in your firewall rules.
1484470	1493310	But anytime you want to establish an SSH connection, you request access, and then it gets granted through the just-in-time service.
1493490	1498930	And that enables SSH for, say, an hour or three hours, whatever it is.
1498950	1503130	And then it goes back to being blocked automatically after the time expires.
1504550	1511610	And I borrowed this diagram from Azure. I thought it was pretty good, so I didn't bother reinventing the wheel here.
1513130	1520710	This shows the user hitting the Azure portal, and it establishes a connection on the Azure bastion.
1520710	1528750	So in your virtual network where you want to have this level of access, you have to create an Azure bastion instance.
1528750	1534650	Now, that's not an instance of a virtual machine. It's an abstracted sort of instance.
1534650	1538450	So you're not maintaining the OS-level patching or any of that.
1538450	1546310	It's just sitting there for your use. And it has to sit in its own subnet called Azure bastion subnet.
1546690	1556570	And so you access that through HTTPS, and then the bastion allows you to access SSH on your virtual machine.
1558850	1564370	And so what does this look like? I mentioned you have to kind of still manage the SSH certs.
1564370	1571030	So what this looks like here is this is a screenshot from me going through the virtual machine.
1571450	1576410	I'm looking at the virtual machine in my Azure portal, and I say I want to connect to it.
1576410	1580270	And it gives me three options. It says RDP, SSH, and bastion.
1580390	1590830	So connecting via bastion, I put in the local username for my SSH user, and then I can put in my private key here.
1590830	1594330	But the public key needs to be sitting on that virtual machine.
1595570	1601330	And you may be asking, does this support AD? And it does not currently.
1601770	1605730	I saw that Microsoft says that they're working on that.
1605730	1613610	So you can set up AD shell access, but you can't do it through a bastion.
1616030	1619290	So what do you get when you're using the bastion host?
1619290	1627870	So one unique thing that you have here that stands out from the other two cloud providers is you can see active sessions as they are happening.
1628050	1632770	And if you are an admin, you can actually delete sessions.
1632770	1642270	So if a user contacts you and says, hey, I'm noticing something weird and you think there's a compromise, you can come in here and you can knock sessions off.
1642590	1647070	So that was kind of interesting to see that. And this is what the screen looks like.
1647070	1654290	So you can see here I have one active connection to call in server and I can see the target IP address.
1654290	1660690	And that IP address is obviously not routable because I wanted it to stay in a private VNet.
1661110	1667810	And it's using the SSH protocol. So you can see some of the good details there.
1669290	1673550	And again, with this provider, you get similar support.
1673550	1681470	So Azure makes it really easy. You don't get it quite for free, but it's so easy to set up and send it off that it's pretty much for free.
1681570	1686910	So you just have to tell Azure where you want to send the audit logs from the bastion.
1687350	1692490	And so these are the three choices, log analytics, a storage account or an event hub.
1692670	1697430	And I did not see any facility for setting up full session logs at this time.
1698010	1701510	Because again, it's using SSH, so it's going to be encrypted.
1702350	1706470	So if you sent it off to log analytics, this is kind of what it would look like.
1706470	1715830	Again, you get some useful metadata around the connections themselves and when they're ended and the protocols they're using.
1717770	1721330	So that's it for the deep dive.
1721930	1728610	So just to kind of wrap things up here, running your own bastion hosts may not be necessary.
1728610	1737950	If you're looking at rolling something out, if your organization is migrating to the cloud and you're thinking about rolling out bastion hosts,
1737950	1741490	you should really look at these alternatives. You may not need to.
1743050	1750050	If you need to, for some reason, just at least be aware that these SSH multiplexing attacks exist.
1750050	1753530	And so there's ways of dealing with it.
1753530	1761270	But I think personally, the best way of dealing with it is just not to have to manage your own bastion hosts.
1762090	1768530	Cloud providers and even third party vendors provide some great alternatives for managing this level of access.
1768550	1771390	So it's definitely worth looking at.
1772370	1780690	These solutions that are out there, they can help with basically all aspects of the bastion host issue.
1781270	1786690	If you're running your own bastions, you're running more compute instances that you have to manage.
1786690	1791390	You have to manage the patching schedule, you have to deal with the costs, all of that.
1791450	1796930	And then there's identity management that comes into play and the logging and monitoring.
1796930	1802230	I was surprised at the level of logging and monitoring you can get from these cloud providers right out of the box.
1802390	1805410	So it's definitely something worth considering there.
1806670	1809330	So with that, I'll take any questions.
1809570	1816090	One thing I wanted to plug here is if this sort of content is useful to you,
1816090	1821090	I highly encourage you to follow our blog, the Netscope Threat Labs blog.
1821090	1829110	We post things about threats that we find, but we also post stuff that we just find interesting, things that we dream up.
1829310	1832850	It could be some useful content.
1832850	1838310	So I highly encourage anybody who's interested to check that out and follow us on Twitter.
1838310	1842670	You can reach out to me on Twitter or LinkedIn if you have any more questions.
1842670	1851750	Or even if you're like, hey, I've been looking at these things and what do you think of some other issue I didn't bring up here?
1851750	1856650	I love to nerd out about cloud stuff, cloud security.