start	end	text
40340	46200	Hi there! Welcome everyone. We're here with Dylan and Allison, who presented here at DEF CON
46200	51640	SAFEMODE on lateral movement and privilege escalation in GCP. Compromise any organization
51640	57100	without dropping an implant. And I have to say, one of the highest theatrical production values
57100	62240	I think I've seen so far in a video, including the hacker with the hoodie doing the whole, you know,
62240	67940	hey, let's just pound on things, but with a real hacker. So, it was pretty entertaining to say
67940	74360	the least. So, welcome both of you to DEF CON here at SAFEMODE. Thank you. It's great to be here.
74720	81000	Thank you. Yeah, so first thing I want to kind of ask, just to get things started, so really,
81000	86060	right, you like, has anything changed since the DEF CON, since you recorded your DEF CON talk
86060	91160	and it was published? Any major changes or anything you want to share with the DEF CON community?
92140	97560	Yeah, so there are actually a number of changes that Google's been rolling out,
98420	103800	all of which have been super positive. Oh, it looks like Allison's video cut out.
104360	112880	I guess she'll rejoin. I'll be answering the question though. So, I guess most notably,
112880	119480	the service accounts or the roles and permissions which can make use of the default service account
119480	127340	without ACT-AS are being end of life in 2021. And that was the direct response to this talk.
127520	133540	And what that means is you will need the ACT-AS permission in addition to the other permission
133540	138340	in order to do the action of attaching a service account to those services,
138340	142980	even for the default service account. So, that's going to change things a little bit.
142980	148180	In terms of what the overall security story is, I'm not 100% sure because it will lead to the
148180	154980	proliferation of the ACT-AS permission. Ideally, folks will be applying that at the resource level
154980	160280	for least privileged scope service accounts. But like we showed in the talk, people often apply it
160280	165740	at the project level. So, it sort of opens the opportunity up for you to make yourself more
165740	172160	secure, but it also kind of opens up more foot shooting opportunities. Another interesting change
173100	181420	was that the very quick clip that I cut over to GitHub for and showed a bunch of people
181420	186500	uploading their GCP service account credentials, I noticed that when I ran the same search on GitHub,
186500	193180	those keys are no longer there. I assume someone went through a lot of effort to get those removed.
193180	198160	And I was just thinking in the back of my mind, you know, they can go through that effort,
198160	205060	they're still in that Arctic vault. So, those keys are immortalized in some capacity.
205460	210120	But it's good that they took them down. It makes it more tricky to just immediately exploit
210120	213240	lots of people at the gate from the talk. And I'm glad they did that.
214680	220280	Awesome. Awesome. Yeah. So, kind of another kind of thought provoker. So, as I watch your video,
220280	224660	I liked kind of at the beginning how you're like, here's how AWS identity access management works,
224660	229200	right? And kind of here's the GCP model. You know, based on your experience so far,
229200	233480	like, I guess, do you think one is better? Right? Do you prefer one model of the other?
233480	236020	Do you think they both have their place? Kind of what's your take on that?
237620	244280	That's a good question. There are definitely elements of the GCP platform that are unique
244280	251180	and nice to use that AWS doesn't have. But in terms of the IAM story, I would say,
251180	255660	generally speaking, from what I've observed, AWS more follows the principle of least privilege
255660	261460	than GCP. And in that context, I guess it depends on your perspective. If you're a brand new
261460	265760	developer and you want to move really quick and you want to have everything open by default,
267080	271040	GCP is an interesting choice. If you're a security engineer and you want to follow
271040	276640	least privilege and lock things down, AWS, from my observations, tends to do that more.
277540	280840	So, actually, there's a question in the chat that kind of goes into that. So,
280840	288360	go ahead and ask it. It was, for those of us with existing and sprawling GCP environments,
288360	292980	given that the org policies aren't retroactive, what are some good approaches for tackling
292980	299980	remediation for those deployments? That's a really good question. And
300700	305700	I think we wanted to go into a little bit more depth into that subject in the video, but we
305700	312800	didn't have the time to really dig into it. Remediation is really tricky. So, you can enable
312800	317540	org policy when you're first starting an organization or later. And for all new things
317540	321700	going forward, they'll get the benefits of it. But for all the existing projects that are already
321700	329400	using the default service account, it's really hard to wean yourself off of it for a couple of
329400	336060	reasons. First is that if you switch service counts on a live VM, which a lot of services
336060	341100	are powered by, that VM needs to actually be stopped and started again to get the new service
341100	349360	account. The second is that the default service account is kind of, like, it's not just, like,
349480	354220	a set of permissions that's applied to your services, but it's also an IAM policy that's
354220	359760	shared across all of your services. And so, you have two different services using the same policy.
359760	364000	If you want to make modifications to that policy, it's going to affect both services. So, that makes
364000	368720	it a little bit tricky. And then the way service accounts work is you can both use them as keys
368720	372540	and you can use them as attached identities, similar to AWS. And so, you have to kind of
372540	380660	enumerate all the different use cases and kind of systematically knock them out one at a time.
380660	384320	So, you have to see how many keys are exported, you have to see is it attached to any instances,
384320	387080	you have to check the stack driver logs to see whether or not they're in use.
388480	395380	All of that can be really tricky, but some tools that do help with that are the Asset Analyzer API
395380	402820	to see where the default service accounts are in use. And then the other one that's handy is the
402820	410500	role recommender. So, you can get some idea of, based on the last 90-day look back period
410500	415820	for a given project, how many permissions has the default service account actually used.
416280	421840	Now, that's not the whole story for a number of reasons, one of which is the role recommender
421840	427160	will only apply those recommendations sort of, you know, if the role binding is at the project
427160	430500	level, it'll make the recommendation at the project level, it won't make the recommendation down
431140	436480	at the resource level, and it also won't recommend that you split your IAM into two different,
436480	441860	you know, identities, if maybe multiple services are using the default service account, it'll make
442120	446700	a recommendation based on the summation of all those services, it won't say this service should
446700	451440	have this scope of permissions and that service should have that scope of permissions. But it
451440	457500	does kind of go a long way in terms of very quick remediation that you can do for looking backward
458000	460540	for the things that work policy can't cover.
462720	468280	Gotcha. Cool. Yeah, so I kind of had a thought too. So as I was watching this, I was like,
468280	472240	man, this seems like a cool way for, you know, an attacker to maybe gain persistence in one of
472240	475920	these, you know, GCP environments, like, you know, setting up an account, and, you know,
475920	479580	maybe before some of these changes, like, hey, could you set up an account, give it privileges,
479580	483780	and then does it just fly under the hood? I guess, what's your thought around that? Like,
483780	486680	could you see that as a potential attack vector? Or would you see maybe other
486680	491290	kind of attack vectors for hiding in the IAM noise of GCP?
492080	497520	Yeah, I mean, I think a lot of what we showed is that this lateral movement,
497520	504920	privilege escalation, persistence, it can all be done without dropping implants on a server.
505200	510800	Like, you might for act as want to spin some things up, introduce them into the environment.
510800	517140	For servers that have maybe endpoint software, monitoring malware and things like that,
517760	522360	it wouldn't set those things off. So, I think it is good for persistence in that way. We'll
522360	527300	probably need new tooling to look for these types of attacks. One of the services that Google offers
527300	535160	to kind of help with this is the... I forget exactly what it's called, but it's a service
535160	541060	that will monitor your Stackdriver logs and look for bad things and then alert you if it sees bad
541060	550640	things. And so, it's watching those cloud APIs, which will be auditable. And examples of things
550640	558100	they've used in the past are like known bad IP addresses for like crypto miners. If they ever
558100	561160	show up in your Stackdriver logs, they'll send you an alert. But I expect these types of cloud
561160	569180	attacks will become more and more a piece of that story. We also briefly showed a lot fishing.
569180	574520	I think that's also a great way for persistence because if you get somebody to click an allow
574520	581160	link on wide scope open permissions, you can get the permissions of the user without having to get
581360	587100	a service account credential and it functions the same way. So, that's another way that you can
587100	590080	sort of use these same techniques and get persistence.
590880	595120	Something to note on this too is service account keys. So, if someone is able to create a service
595120	599440	account and give it access to a resource or something, a project, that the service account
599440	606220	keys last for 10 years. So, if they're never rotated by the organization or deleted, then
606220	615120	you have that key for a very long time. Wow. Yeah, that's definitely interesting.
615560	623120	Oh, another quick note on that. The long-lived exported credentials last for 10 years,
623120	630020	like Alice mentioned, which is interesting. The short-lived instance credentials, those last,
630020	634740	I think, closer to 10 minutes, but that'll persist after an instance is terminated. So,
634740	639220	if you're an attacker and you're exploiting something like an SSRF or even remote code
639220	645500	execution, grab instance credentials, even after the defensive person deletes the instance,
645500	648120	the attacker can still persist another 10 minutes.
651120	656400	Okay. So, we had another question pop up here in chat and kind of along with your follow-up to
656400	662620	what you're explaining. It says, what about abandoned service accounts? Say they're deleted.
662620	667480	Do you end up with abandoned ACLs that exist on the resources and projects?
669520	673320	That's a good question. Allison, did you want to take that?
673560	680380	So, there's actually something that is kind of interesting about this where we've run into some
680380	688320	bugs, actually, where a service account has been deleted and their IAM bindings were also removed.
688320	693340	So, if a service account is deleted, the IAM binding can still exist there. So, maybe also
693340	698860	if you have a user in your org and you remove them from G Suite, that IAM binding will still exist
698860	706200	in your GCP projects with user at gmail.com or whatever that is. But there's also interesting
706200	711440	things that can happen due to the undelete capabilities for service accounts, or if you
711440	718000	delete a service account, IAM binding, like you asked, it still exists where that IAM binding was
718000	723380	made. And if you do, if you try to undelete your service account, you're like, oh, maybe I actually
723380	729880	need that. And you had, or you try to create a new service account with that same name,
730400	736100	the IAM binding will reference the UID of the service account that was, that was deleted.
736100	741120	So, you can have this weird like caching thing happen where you'll get like permission denied
741120	746940	errors if you're using this. So, if I create service account A and I give it an IAM binding
746940	753160	at my project and I delete it, then I recreate a new service account, service count A, that tries
753160	758800	to reference that existing IAM policy, it doesn't work because it's referencing the old service
758800	762700	account that was deleted by its unique ID. So, yeah, they exist and this can bring up some really
762700	767300	weird bugs if you're trying to create new service accounts with the same name or undelete your
767300	777360	service accounts and you're not updating the IAM policies in parallel. Cool. Yeah, so we kind of
777360	780760	talked about this a little bit and you mentioned this in your talk too, kind of from the defender
780760	786880	perspective, right? Like, there's rolled out like the IAM analyzer, there's the recommender,
786880	791400	you know, I guess, what are, are there any other things or advice you'd give to blue teams looking
791400	796740	to secure kind of the GCP environment, right? Or the IAM offerings, like detection opportunities,
796740	800740	like the tool you mentioned, log analysis, like, are there any, any good things to try to
800740	805940	really dive in? Like, hey, ingest this log feed and, you know, hey, we'll see all your IAM
805940	813600	transactions or anything like that, any advice you give? That's a good question. A lot of
814660	820200	people will recommend that you can pipe your Stackdriver logs over to BigQuery,
820540	825620	so that you get a structured query language that's a little bit easier to work with than
825620	833340	the filters in Stackdriver, easier to write detection rules for and such. I think that
835680	843840	definitely the security API that exists at the org level has some very interesting
843840	848140	introspections that you can do across your whole fleet, and they provide some suggestions and
848140	854320	recommendations around other types of least privilege, like buckets that were left open to
854320	858880	the internet and things like that. There's also a lot of other org policies that we didn't cover
859520	864100	that also will improve your security story, but just aren't necessarily related to the content
864100	873400	we talked about, like allowing your developers to add users outside of your organization
873980	878520	to roles inside your organization. By default, that's allowed, so you can add a random Gmail
878520	884900	account to a project. You can disable that via org policy. So it's definitely worth checking out
884900	893640	those high-level org policies as well. And I think that we mentioned this as well in the talk,
893640	901200	but we had general positive experiences providing feedback to Google, and in some cases,
901200	907140	they actually put us in touch with PMs where we could explain these issues directly to them. And
907140	913280	so in that regard, we strongly recommend people to engage with Google around pain points and
913280	917900	tell them what an ideal workflow would look like. And in our case, they were receptive and worked
917900	922460	with us on those things. Yeah, I was actually going to dive a little into that too. So again,
922460	926920	working with Google that closely, again, it's always an interesting thing when you're
926920	932540	approaching someone with base-level issues like that, how do they respond? Do you have any other
932540	937460	tips for folks who'd be in similar situations, like reaching out to a Google or another company,
937460	942620	like anything you learned out of that that you'd do better next time? Well, I think what was
942620	948200	interesting is there's kind of two ways to engage with Google. There's the customer way,
948200	954100	and then there's the vulnerability way. So you can submit to their bug bounty, and they get a very
954100	958160	binary, this is a problem, this isn't a problem, right? If it's a problem, they'll pay you and
958160	961700	they'll fix it. If it's not a problem, they'll close the ticket, and that's the end of the story.
962200	967440	That's the bug bounty way. And you're interacting with the security engineer when you do that
967440	974220	on the other end of that ticket. When you go the customer route, you're talking with a PM around
974220	980040	them building products and giving them feedback about existing products. And that's a continuous
980040	989160	story that doesn't end, and it's not binary. You have to sort of go through in-depth how security
989160	993100	problems manifest themselves and things like that, which you might not have to do if you're talking
993100	1000160	to a security engineer. And you get to continue that conversation indefinitely. It's not a binary,
1000160	1004760	you know, this is a problem, this isn't a problem, okay, ticket closed. So we pursued both of those
1004760	1018420	paths concurrently. And I think that probably depends on your particular use case. But there
1018420	1023900	were some things which we submitted as bugs through the bug bounty, and we thought that they
1023900	1029080	would just be recognized as bugs and fixed. But they were actually just closed out as working
1029080	1034400	as intended. And for those things, going the customer route actually proved to be more effective.
1036320	1041620	Cool. That's great advice. Great advice for other people. I guess the other question I have, too,
1041620	1046120	so it's kind of what's next, right? You kind of really, I guess, gone into a lot of the problems
1046120	1051840	you've done a lot of improvement on this front with Google. What's your next for the research
1051840	1055620	front? Are you going to continue to refine more attacks here, looking at other things,
1055620	1063260	kind of what's next for the both of you? So recently, I did notice someone sent me some
1063260	1071180	good resources that Rhino Security wrote up on some other GCP privilege escalation techniques,
1071180	1079400	and they released some proof of concept tools. I think it would be cool if sort of the community
1080920	1085760	helped bundle all of those tools into one framework. Of course, I'm selfishly hoping
1085760	1092600	that's gsploit, so that we can kind of have like a metasploit for cloud. So I thought that, you
1092600	1099220	know, that might be fun to consolidate some of that stuff. But beyond that, I don't have a lot
1099220	1104180	of plans to do more extensive work in GCP. What about you, Allison?
1105180	1111980	Yeah, for public research wise, I don't think I'll be elaborating on this topic for a while.
1111980	1115920	But definitely something that I'm personally interested in learning more about and seeing
1115920	1122780	the state of how different cloud providers, Google and AWS tackle IAM, because it's something that's
1122780	1129120	really difficult, and I think is going to be changing very frequently for a long time,
1129120	1135100	because it's such a hard problem to try to tackle. So no public research for the near future,
1135100	1137240	but definitely interested in seeing what happens.
1139160	1143500	And I have a question that kind of came up when I was watching your talk as well.
1143520	1150560	Kind of going back to the tooling, you mentioned that you all developed a framework, gsploit. Is
1150560	1154380	that something that has been or you plan on releasing anytime in the future?
1154800	1159820	So yeah, it's already live. We linked to it in the talk. And you can check it out. It's
1159820	1164380	on GitHub and Docker Hub. So it's just a little Docker container you can run.
1164680	1170100	It was definitely released more on the proof of concept side than production code side. So
1170100	1176460	some people found some bugs in it already and I'm expecting more. But it does the things that
1177000	1182860	we videoed. So those demos, we can reproduce them with the tool.
1183520	1187400	Cool. And then it looks like we have one more question in the chat.
1187760	1193200	From your talk, I got the idea that in AWS, the account holder owns the permissions. But
1193200	1198240	doesn't that cause a problem for research owners who need or want to provide permissions when the
1198240	1204360	account is not under their control? I think this kind of gets at the heart of
1204360	1211920	what we opened the talk with, which is whether or not your IAM model is resource-centric or
1213140	1219800	user-centric. So is it basically the resource owner that controls who can access the resource?
1219800	1225100	Or is it the identity owner that controls what resources that identity can access?
1225100	1229560	And I think that's sort of the main difference between the two IAM stories
1230840	1236840	in AWS versus GCP, at least that we opened with. We talked about some of the challenges that come
1236840	1243540	up with the resource-centric model that's in GCP. We didn't really touch on the challenges
1243540	1252480	of the inverse of that in AWS. And to be honest, I'm not an expert in AWS IAM by any means.
1252480	1255800	And so I'd be curious to know if someone else wanted to do a deep dive on
1255800	1259540	what those cons are.
1261980	1268300	It also depends on how you're managing your infrastructure. It depends on are you
1268300	1272160	allowing engineers to manage different infrastructure configurations, or do you
1272160	1277380	have kind of a more centralized way that you're provisioning access within an account, too?
1277380	1278920	Something to think about.
1281020	1285780	Cool. Yeah, and I guess one thing I didn't really ask there, I don't remember, is why GCP?
1285800	1291580	What kind of drew you to this area for research? Why did you look at this and the weaknesses here?
1292840	1301480	Well, I think some of the defaults were definitely a little eye-opening. I guess
1301480	1310380	the full story of this is someone briefly compressed down in the back of an Uber ride
1310380	1316860	on my way home. I was sharing a car with someone. A bunch of crazy things in GCP that I found hard
1316860	1323580	to believe. Yeah, all these things run as almost full administrator by default. And
1325060	1331440	those things kind of prompted me to poke into it a little more. And then I think my poking
1331440	1336520	got Allison poking, and then we just followed the rabbit hole as far as it would go.
1338640	1343320	Cool. Yeah, there is one other question here that just came in the chat, too. And I don't
1343320	1347800	know if you know the answer, but which model does Azure take? Is it more resource or identity-centric?
1349320	1355500	Actually, not sure. I'm not familiar with Azure very much. I also wouldn't be able to speak to
1355500	1361240	Oracle Cloud, IBM Cloud, or Alibaba Cloud, but I'm very curious to know the case for all of us.
1362240	1364700	Future research for others to look into.
1365780	1366740	Yeah, definitely.
1367360	1373040	Awesome. Yeah, so we're getting near the end of our time here. But really, a couple last final
1373040	1378160	ups. First, what should people look into if they're interested in learning more about the GCP
1378700	1381620	identity access model? Where would you recommend someone start
1381620	1386960	if they're trying to figure out how does this work, and maybe if they want to dive in themselves?
1388380	1394300	Yeah, there's actually... the documentation, I think, is really great to start with. Really,
1394300	1399080	just read it in its entirety. It's a lot of information. But from a hardening perspective,
1399080	1406360	Google also recently released a new blog post that outlines some of the... in one place,
1406360	1411280	not in each individual services, IM docs, some of the different steps you can take to
1411280	1415120	harden the IM of the different services that we brought up in our talk.
1415720	1422360	Oh, yeah. And on that subject, that blog post is also an answer to earlier questions of what
1422360	1426920	they've done since the talk. That blog post was done in response to the talk.
1428840	1435140	Awesome. Yeah, so someone said GCP is like gone cloud pwning, which seems pretty apt.
1436700	1441240	But yeah, so really, I kind of opened up the last user. Anything else you really want to
1441240	1446660	share with anyone? Any other last thoughts? Things that you think folks really should know
1447320	1449900	about GCP before we move on?
1452750	1460410	Oh, I'll just share that I am completely new to Twitter. I made a Twitter account last week,
1460410	1467010	and I shared the handle in the video. But if folks want to connect with me there,
1467010	1471210	and make me feel less lonely, because I don't have a lot of followers, feel free.
1472150	1473850	What Twitter account is that?
1474490	1480970	Oh, it's Insecure Nature. And it's linked in the talk. And I can also link in the chat.
1483390	1487610	Okay, I guess my final thought would be don't trust the documentation. I know I said just read
1487610	1492090	the documentation, but don't trust the documentation. When you're going and assessing
1492090	1497130	things or building things, take kind of like a reverse engineering approach to analyzing things
1497130	1503210	in the cloud. So test and verify the behaviors. Don't maybe just take what's in the documentation
1503210	1507670	or what you think it would work, how you think something should work as the truth.
1508310	1512630	Oh, yeah. And there are also a couple more privilege escalations, like I mentioned,
1512630	1517850	that we just didn't have time to go into. We covered most of the ones that compromise
1519990	1523550	a large set of identities. But there are privilege escalations for getting access
1523550	1529410	to storage and those types of things. They're covered both in our earlier talk that we gave
1529410	1535070	at BSides that I recommend folks check out. And like I mentioned, Rhino Security has done
1535070	1543120	some pretty good write-ups recently on it. Awesome. Well, thank you both, Dylan and Allison.
1543340	1547540	Really what I'd suggest too, if you have any particular links or summarized list of links
1547540	1551560	you want to throw into the Discord chat room, that'll be helpful for everyone. There's lots
1551560	1557260	of folks in there thanking you for the talk. So definitely well received by everyone here.
1558180	1563100	But yeah, just want to say thank you again. Thank you for joining us for the DEF CON Safe Mode talk.
1563100	1567600	Thanks for presenting in these weird times and hopefully see you in the future out in Vegas
1567600	1572720	when we return. Sounds good. Thanks so much for having us. Thank you so much.
1572720	1577580	Yeah, thank you. It's a pleasure. Take care. Yeah, thanks.