start	end	text
4260	7460	Okay. Well, welcome everybody. My name is X-Ray.
7460	12600	This is the DEF CON 30 Altspace VR Village for DEF CON groups.
12720	17220	So, welcome. And our speaker is Jim Shaver.
17580	21800	He's going to be talking about AWS Metadata Privilege Escalation.
21800	25440	Jim is a Pentester, Offensive Cloud Security Researcher,
25440	29060	and public speaker with 13 years of IT and security experience.
29060	31080	So, Jim, go ahead and take it away.
31860	35340	All right. I'm going to stand by the podium so I'm heard.
35720	40480	Today we're going to be talking about AWS Privilege Escalation.
40980	43340	And when I talk about Privilege Escalation,
43340	50300	I'm mostly talking about the API and sort of the back end of AWS.
50300	57280	I'm not specifically talking about, like, necessarily operating system Privilege Escalation,
57280	59800	although operating systems will be involved.
59800	62100	Okay. Next slide, please.
64740	70660	All right. So, some of the things we're going to be talking about today are how authentication works within AWS.
70660	75460	We're going to be talking about the Instant Metadata Service, IMDS.
75600	78600	I will just call it the Metadata Service.
79320	83180	I'm going to be talking about various modes of escalation,
83900	89300	as well as several tools and resources that you can use, some of which I have written
89300	91540	and some of which other people have written.
92060	93500	Next slide, please.
94840	98640	All right. So, this is what IMDS looks like.
98700	108980	If you're on an EC2 virtual machine, you can just curl this IP address 169.254.169.254.
109320	112220	You'll notice it's kind of an unusual IP address.
112220	117400	If you've seen or been on a network that doesn't have DHCP working properly,
117400	119740	you've probably gotten a 169 address.
120020	125280	And you can think of that IP space as an IP space that is non-routable.
125280	131780	And in order to avoid collisions with 10. and 172 and 192 addresses,
131780	140280	Amazon chose a 169 address for this non-routable interaction with the IMDS service.
140620	145400	Just a little bit of background on what the Metadata Service is.
145400	156880	It's a little bit of semi-dynamic data that the operating system uses for its own purposes,
156880	158540	for whatever it needs.
158540	167420	So, things that are included are kind of benign things like the AMI ID and MAC addresses,
167420	171680	network information, even the region that the virtual machine is running in.
171680	174460	So, the US East 1, for example.
174460	180980	But also other more interesting things for attackers in here, things like user data,
180980	186560	which is like the machine startup script, or even the machine's own identity credentials
186560	192200	and role-based credentials that the machine may have been granted.
193660	199180	And so, these credentials you can think of as sort of the machine account, quote-unquote.
199300	201020	Next slide, please.
202100	215040	All right, so this is an example of a simple, I guess, SSRF where you have a proxy parameter
215040	223200	that's injectable with this local IP, this 169 address, and it returns a role.
223200	232880	So, latest metadata, this is actually returning a text response from the metadata service
232880	236280	within the virtual machine that's running this web application, okay?
236620	242220	And what makes this kind of like the machine account, you can think of like in the Windows world,
242220	251000	you can give like an Active Directory machine, like domain admin, for example.
251000	254040	Nobody does it, but it's something that's technically possible.
254040	261020	It's much more common in AWS to give EC2 instances roles, which means they typically have permissions
261020	266560	and policies either attached or assumable or whatever through those roles.
267240	273900	So, if you go into latest metadata IAM security credentials, and then there's a whole bunch of stuff.
273900	278680	In this case, this role name is called EC2 default SSM.
279500	290120	If you then sort of browse to that role name, what will happen is it will show you a JSON response
290680	299120	that is an access key ID, secret access key, and then a token that changes every once in a while.
299120	307880	This is the authentication for this role for this machine, so that when the machine does certain administrative tasks,
307880	316440	it can authenticate using these credentials, and they kind of work like long-lived API keys for AWS,
316440	327460	but they're used via the short token service instead of long-lived API keys.
327460	333100	So, the way that you can tell that that is happening is the first four letters of the key ID are ASIA,
333100	336740	whereas long-lived keys are AKIA.
336880	341760	Okay, so we'll see another example of that later as well.
341760	343220	Next slide, please.
344760	346540	Okay, ways to get at IMDS.
346540	355020	So, obviously, you can imagine a bad guy wants to get at the IMDS service of an EC2 instance.
355020	357480	A CC2 instance is in a VPC.
357700	362700	Maybe it has a public interface where there's some sort of vulnerability,
362700	366380	or the bad guy has somehow gotten into the virtual machine.
366680	371140	Some of the ways that that can happen is you can be on the box.
371140	375900	So, if you get a shell on the box, you can just use curl, like we did in some of the previous examples.
375900	381360	So, in that case, you would just curl this web address, and then you would go from there.
381960	391440	There's also examples of command injection and SSRF, where if you are able to, you know,
391440	396060	render a web page or something like that, the results of a command,
396060	399780	then you can also display the contents of the metadata service.
399780	406680	There are also more novel examples that vendors and service providers haven't always thought about,
406680	419260	in terms of some sort of SSH key for bastion access, or if they give you, you know,
419260	429860	there's some sort of reverse tunnel they use for support or VPN keys that they use to connect to a box in EC2.
430240	435140	Even if they disable, like, SSH access to the EC2 instance,
435140	443560	you may disable the tunnel network traffic over SSH via SOCKS into the EC2 instance,
443560	448640	and you can reach a local address, even though you're routing across the internet,
448640	452020	you can still reach that local address over a SOCKS connection.
453560	459620	So, in that case, you would just, you know, have a browser or whatever tool that supported SOCKS,
459620	465960	and then you would tell it what SOCKS proxy to use, and you would just query the 169 address with curl
465960	472060	or whatever tool you're using to access the metadata service.
472420	473600	Next slide, please.
475280	476580	All right, so using the creds.
476580	479080	So, we've gotten this JSON response.
479340	487020	Basically, what we do is, this is an example of long-lived credentials.
487020	490360	So, you see, like, AKIA, these are obviously example credentials.
491780	499960	So, that's what normal, like, API keys that you might find on, like, your average developer's machine look like.
499960	506100	And this is an AWS credentials file that might exist on their laptop or on your laptop.
507920	517440	You can also, so you can also just use the tools that are on the box to interact with the AWS CLI or pull your tools onto the box, if you can do that.
517440	526080	But if you can get to this, you can just copy all of the data out of here, copy it to, you know, a box that you're using that has all your tools on it,
526080	531160	and then put it in your AWS credentials file and use it as a profile for authenticating.
531160	540140	So, then, when you're using tools like the AWS CLI, you just say, AWS profile, AWS session zero, which is what we have here.
540140	545440	And then we run whatever commands using the CLI that we have.
545440	551760	So, in this case, we're just doing a really basic STS get color identity, which is, if you're familiar with AWS,
551760	555760	this is a thing that tells you, like, information about the account that you're running in.
555760	559840	It's a good way to check to make sure that the credentials are valid.
559840	563460	Okay. Next slide, please.
564780	570120	Okay, so I'm not going to talk really about IAM policies because they are complicated.
570120	579300	There's lots of ways that policies can be associated with a user directly, either through inline policies that are directly attached to the user,
579300	585890	managed policies that are associated with the user, inline policies that are associated with a group,
585890	592230	managed policies that are associated with a group, attached or past roles, etc.
592230	598850	There's also service control policies and permission boundaries that are used in more advanced environments.
599230	609430	And there are some limitations to privilege escalation that I'm not going to get super deep into today because it's really complicated.
609430	614610	Even some of the tools don't even understand a lot of the nuance that's out there.
614610	616730	Next slide, please.
617890	627030	And that's because this is like the decision tree of how policies and that type of thing happen according to AWS.
627030	629290	And even this doesn't include all of the nuance.
629290	641330	So there's a very thorough decision tree around how AWS makes decisions around whether or not you or a resource in your environment has permissions to do something.
642810	644110	Next slide, please.
645850	651190	All right, so some ways that you can use the creds.
652750	657810	Most of the examples we're going to be using today are with the AWS CLI.
659210	662690	But there are also lots of other ways that you can do it.
662690	667410	You can use Boto3, which is the Python SDK for AWS.
667410	681830	It just sort of takes you the next step after the AWS CLI, allows you to chain multiple things together and write like a Python script that understands how to talk to the AWS API.
682310	685510	There's other SDKs as well that you could also use.
686730	703750	There's a pretty good tool written by Rhino Security Labs called AWS Escalate that can just sort of try to brute force and figure out whether or not you have an escalatable path to escalation within AWS.
704310	710770	And we're going to talk through some of those paths manually here in a little bit using the AWS CLI instead.
710770	719910	There's also a really good framework called Paku, also by Rhino Security Labs, that you can dockerize and all of that.
719910	735630	And it's basically a Python application that gives you a menu and you can sort of step it through all of the ways it can escalate in another enumeration.
735630	738330	It's like a Swiss Army knife, basically.
738330	744450	There's a couple of tools that I have written that I've thrown in here as well. One is called RedBoto.
745110	765350	And basically it is a set of tools that either do enumeration or do interesting things with operating systems within AWS or connect to SSM, which is basically like SCCM for AWS and other interesting things.
765350	787610	There's also a tool called FederateMe, which is... I'm not sure if AWS has actually fixed this as a thing, but basically it uses federation to go from credentials to an AWS console, which is the web user interface of AWS.
787610	800050	And sometimes it's just easier to work in the web interface than it is to work via CLI or API or with various tools. So that is just easier to just pop open a browser.
800050	815450	So what FederateMe does is basically you give it your credentials and it creates a signed login federation link that will pop you into the console, even as an EC2 instance or whatever.
815450	821050	And you have the ability to do whatever it is that that machine would be able to do if it could log into the console.
823190	839490	There's also another tool called EnumerateIAM, which is a good enumeration tool of different IAM permissions. IAM is the system in which you do all of your identity and access management within AWS.
839490	853690	A lot of tools obviously revolve around that. Also, the best resource out there on the internet with offensive security, Red Team, AWS, and other cloud information is Hacking the Cloud.
853830	862950	It's maintained by Nick Frechette and it's very high quality and the best resource out there for this type of information.
862950	870770	I have references throughout the rest of the presentation to some articles on that.
871670	872790	Next slide.
875170	885010	All right, so a couple of more easy ways to escalate that I'm not going to demonstrate because I think you can kind of imagine what they might look like.
885010	894830	One of the ways that you can escalate is you can pillage S3 buckets for other creds.
894830	903010	You might get onto a virtual machine that has some role credentials that may have access to some S3 buckets.
903010	927810	If you are doing sort of a gray box assessment on the environment and you have like a scout report or something like that, you can know what the role credentials have as permissions and what S3 buckets they may have access to with that report.
927810	938910	It may not be actually obvious to you or even possible for you to know without brute forcing what permissions that role that you have access to, what it can do.
938910	963730	So if it's more of a gray box assessment and you've got the capability to do a scout report with ScoutSweeter, whatever they're calling it this week, then this can be a good way to find other credentials that developers or other people have left laying around or that might have higher privilege than what you have.
963730	976870	It's very common for EC2 instances to be granted roles that have access to S3 buckets and nothing else because that's one of the ways that they get data in and out of the EC2 instance.
976870	992870	And so it's very common for them to have that access. And then it's common for, you know, them not to lock down to specific S3 buckets and they may have access to like, you know, the Terraform S3 bucket or, you know, what have you.
992870	998290	There's lots of ways that they can get permissive access to the environment's S3 buckets.
999150	1010610	Another area is user data. I talked about that a little bit earlier, and it's basically a Base64 encoded script that the machine runs at startup.
1010610	1031050	And those role credentials that we've been talking about may have the ability to read the user data of other EC2 instances and other virtual machines in the environment.
1031050	1047310	And so sometimes people put things in the startup script, assuming that they as the administrators are the only people that read that stuff. And it actually doesn't take a lot of permission to be able to read an EC2 instances startup script.
1047310	1057350	So you should really treat that stuff as, you know, a sensitive thing and use other means to get secrets and that type of thing onto the box.
1058030	1059410	Next slide, please.
1061050	1071750	All right, so we're going to talk about our first IAM policy or permission that allows you to escalate.
1071750	1078230	So if you have IAM add user to group, this one's like a really basic privilege escalation.
1078230	1095670	So obviously you're not administrator. If you have the ability to add any user to any group, then, you know, you could just run the AWS CLI command, you know, IAM add user to group, user name Alice, group name administrators.
1095670	1103930	And if you're able to run that command, you've just privilege escalated the user Alice to administrator in the AWS count.
1103930	1115130	Okay, so might seem like a super obvious way to privilege escalate, but it's an example of the type of thing that I'm talking about.
1115790	1117670	Next slide, please.
1118870	1127110	One that's pretty straightforward, but also a little bit more nuanced is IAM create access key.
1127110	1139190	So in this case, you are running, you know, IAM create access key for the user name admin Bob.
1139190	1148650	So basically you're saying, I want to create an access key for this very privileged user that I know is privileged, and I want to have their access key, basically.
1148690	1149410	Okay.
1153530	1155290	Okay, next slide, please.
1158720	1160620	All right.
1161540	1164060	Pass rule and run instances.
1164060	1167220	Okay, pass rule is more complicated.
1169020	1176940	Because basically, this gets into resource based privilege escalation.
1176940	1180920	So in this case, you are not doing the privilege escalation.
1180920	1187920	In fact, you are spinning up a virtual machine that is doing the actual escalation.
1187920	1198920	Okay, so the way that pass rule works is you are given the ability to pass a role to a resource that being the virtual machine.
1199000	1205120	And then you run that instance with a role that is very permissioned, very well permissioned.
1205120	1220600	And then you have that resource, the virtual machine, run a user data script that executes some AWS commands that create an administrator user for you, for example.
1220600	1221200	Okay.
1221760	1229200	A lot of people get tripped up on this one, they think that they need to have SSH access to the EC2 instance once they spin it up.
1229200	1237620	In fact, if you specify user data, the code executes and you don't need to be on the box for it to happen.
1237660	1244600	You don't need to have a VPC that you can SSH into, as an example.
1244600	1248480	There's lots of things that make this a lot easier.
1249800	1253900	And so there are some limitations on this one.
1253900	1259440	Obviously, you need to be able to pass a role that matters and exists.
1259440	1263200	You need to be able to run arbitrary instances.
1263660	1271980	And there also needs to be an instance profile that is privileged enough that can do the thing that you want it to do for you to do.
1272060	1275740	So there's a lot of moving parts that are involved with this one.
1275740	1280780	But that is how you do that one.
1281640	1283010	Next slide, please.
1285600	1289010	There are also many other types of privilege escalation.
1289010	1306370	So there are other services within AWS that are not even commonly used by the majority of AWS customers.
1306370	1318430	So like Glue and Data Pipelines and CodeStar are all DevOps-y type things that are used by some very advanced environments.
1318430	1327250	They absolutely have methods, if configured incorrectly, that allow you to privilege escalate.
1327250	1337670	Another one that I didn't include in the slides because it worked very similarly to the EC2 instance, one that we just talked about, is Lambda.
1337670	1341890	Lambda is like the serverless functionality within AWS.
1341890	1351270	And essentially, if you have pass, roll, and create function, invoke function, maybe a couple of other things,
1351270	1367110	you can do the same thing with Lambda that we did with that EC2 instance where you can have the serverless Lambda function do all of the IAM functions that create a user or make you an administrator
1367110	1371610	or whatever it is that you use to privilege escalate.
1373970	1375290	Next slide, please.
1376890	1385810	All right, so another tool that I have not really talked about publicly, but I released about a month ago, is this thing called Red AEMM.
1385810	1401190	And basically what it is, is I wanted to solve the problem of how do you have a cloud section to a CTF, okay?
1401190	1409830	And there are cloud CTFs out there that are like flaws.cloud, and they're just sort of publicly on the internet.
1409830	1416650	And what happens with those CTFs is that you're very limited in scope in terms of what you can do.
1416650	1426510	And there's a lot of risk in setting them up because you're hooking somebody up to a real AWS account that could not only get hacked, but also cost you a lot of money if it gets hacked.
1426670	1434390	And so this incentivizes people from having cloud CTF type things.
1434390	1446650	And so one of the things that I wanted to solve the problem was how do we teach people about the metadata service, but don't hook them up to a real metadata service in ED2.
1447090	1458130	And so what I came up with was AWS actually has a project called the Amazon EC2 Mock Metadata Service.
1458130	1469590	It's a fake metadata service that is a Go application and a Docker container that emulates a real metadata service.
1469590	1491350	And it's used for evops type testing, where you need to test something that needs to interact with the metadata service, but isn't, for whatever reason, you want it to talk to a fake metadata service just because it's easier for you or it doesn't need to be in AWS or whatever the reason is.
1491350	1511370	And so I forked that and basically developed a front end that is very basic SSRF and this is an example challenge of like, hey, if you put proxy and then the domain name, this is how this proxy works.
1511370	1516690	And it's not really a proxy, it's just a thing that gets you to the metadata service as part of the challenge.
1516690	1529670	So the idea here is that this tool can be used in capture the flag exercises so that people can understand a little bit more and interact with something that acts just like a metadata service.
1530270	1537330	It doesn't expose your AWS account, so if somebody gets access to a role within here, it doesn't actually do anything.
1537930	1539370	Next slide, please.
1540450	1546910	So this is what it looks like when you exploit the SSRF that I've built in here.
1546910	1556350	So just like what we've been seeing all along, you know, you hit this address, this is not really a fake address, this is just part of the application.
1556390	1563110	So it just looks like we're exploiting an SSRF in AWS, but this is just a Docker container.
1563950	1573370	And it returns a text response just like a real metadata service, but all of this information is totally bogus.
1573370	1579330	If you pull up this AMI ID, it's an AMI ID that doesn't exist within AWS.
1579870	1584530	These are not the MAC addresses of the machine that this is running on.
1584530	1586510	All of this information is bogus.
1586510	1596950	And you can control these through a JSON configuration file, and you can also control through environment variables as well.
1596950	1608570	So you can imagine hiding a flag in the user data of this MAC metadata service, essentially, on a CTF.
1609950	1614110	And so that can be found on my GitHub. It's called redamn.
1615830	1622430	And if you have any questions about it, feel free to hit me up on LinkedIn or wherever on the internet.
1624030	1625850	And next slide.
1627030	1630250	I think that's it for me. I'm Jim Shaver. You can hit me up on LinkedIn.
1630250	1633430	I'm also iHamburglar on GitHub.
1633830	1637830	I really want to thank the volunteers for their awesome work on this conference.
1638510	1645030	The setup is amazing, and it's one of the best speaker setups that I've had as a speaker.
1645390	1651370	So it's nice to just be able to come in here and talk from my house.
1651410	1659790	So thanks, everybody, and I'm happy to take any of your questions, if any.
1661850	1673030	So I've worked a fair bit in AWS, and what I've noticed is that routinely organizations will have people who are new to AWS,
1673030	1683410	or they've just graduated and they're getting out into their first Java position, and they'll just jump straight into building things.
1683410	1700110	And IAM is sort of this thing that they get as kind of an afterthought, and they don't truly master the service until years into their effort.
1700110	1711710	Have you seen any successful attempts at getting people to train in IAM first, rather than jump straight into building things?
1712950	1718150	So I think, if I could distill down the question for those people that are not in the room,
1718150	1729290	essentially, do you get people to understand IAM and do it securely? Is that a fair representation of the question?
1729450	1733350	Yeah, try to get it right the first time, rather than a retrospective look on it.
1733830	1738250	Yeah, I think, you know, I absolutely agree with what you're saying.
1738250	1744750	I think it's really complicated. And I showed you a slide that demonstrated how stupid it is.
1744750	1750590	And I think it's very difficult to find people who really understand how it is.
1750590	1758730	I think it's the number one problem that's AWS organizations.
1758730	1765330	And I think, you know, it's a, are you training people and investing in them?
1765330	1768290	And are you hiring people that know what they're doing?
1768290	1777270	I think there's probably a lot of opportunity there for Amazon to make it easier.
1778590	1784390	Absolutely, it's one of the number one issues with an AWS account is how complicated IAM is.
1784390	1790510	And it changes. Sorry, I can't really answer that question better.
1791150	1795650	No, no, if you did, that'd be amazing. Right? I'd take that back immediately.
1795850	1801790	Yeah, I would be in a venture capitalist office right now if I had that answer to that.
1803330	1805110	Thank you for the question.
1807470	1808730	Anyone else?
1812580	1817340	All right, well, I'll be hanging out for another 10 or so minutes if people have questions.
1817340	1820360	Fair. I really appreciate the opportunity to talk today.