start	end	text
1760	3580	Hello, my name is Allison.
7250	8770	Hi, my name is Dylan.
12270	16250	We've been researching GCP for the last year and a half and we've come up with some effective ways
16250	20530	that compromise most GCP organizations, all the ones we've looked at anyway. These techniques
20530	24990	involve moving laterally with service accounts and cloud APIs. We have a lot of ground to cover,
24990	41220	so let's get to it. I think everyone has a slightly different reaction when they first
41220	48060	start learning about GCP IAM. But one common thing I've observed is people often compare it
48060	53020	to AWS IAM, but there are some pretty key differences between GCP and AWS IAM. So let's
53020	58280	talk about some of those differences. First, in AWS, we have user identities and those user
58280	63420	identities can be exported in the form of credentials, which look something like this.
63560	69900	In GCP, the equivalent is a service account and service accounts can also be exported
69900	73200	in credentials that look a little bit differently. They look like this.
74500	79180	And in both cases, we have owners of these identities. But where things get a little
79180	83620	bit different is when you start to ask the question, what can my identity access? In
83620	88180	the world of AWS, you basically have a policy that determines all the different things that
88180	93460	the identity can access. And the person who can create, read, update, delete that policy
93460	98400	is the same person who owns the identity. So here's an example of what that looks like.
99000	102660	In GCP though, this is where things start getting really different. Instead of having
102940	108460	a centralized policy for a given identity, we have policies for resources. So in this case,
108460	113380	we've got a bucket and a bucket owner and that bucket has a policy. The bucket owner controls
113380	118620	who can create, read, update, or delete that policy. And that policy applies to whatever
118620	124420	identity the bucket owner wants it to. So here's an example of what that looks like for storage.
125600	131360	What's interesting here is that the owner of the service account actually doesn't have any
131360	136900	insight into this policy. They can't view it. They can't update it. They don't even know that
136900	141880	their service account was given access to this bucket. Instead, the bucket owner controls this.
141880	146400	And if the owner of the service account doesn't have IAM list on the bucket, then they won't
146400	152660	be able to even see that that policy was granted. And I think this is one of the harder concepts
152660	157660	it was for me personally to wrap my head around in GCP, is if you have an identity in GCP, like a
157660	163080	user or a service account, as the owner of that identity, you can't actually know what that
163080	167740	identity has access to. In fact, no one can answer the question, what does the service account have
167740	172320	access to? Instead, resource owners control what the policy is for a given resource. So you can
172320	176920	answer questions like, who has access to this bucket? But you can't answer questions like,
176920	180360	how many buckets does the service account have access to? Or what buckets does the service
180360	185620	account have access to? Those questions are unanswerable by design in GCP. Likewise, we can
185620	189960	have other resources like BigQuery, for example. And we have an owner for the BigQuery and a policy
189960	195180	for the BigQuery. And the owner of the BigQuery can, again, grant access to any identity they want.
195180	200060	However, the owner of the service account has no control over this and can't see this. One thing
200060	203880	you might be wondering is, before when I mentioned that the owner of the service account can't fully
203880	208120	answer the question, what does the service account have access to? You might be thinking, well, if you
208120	212960	own an organization, can't you just get the IEM policies of all the resources in the organization?
212960	216860	Well, someone outside your organization can still grant your service account access to their
216860	221240	resources. So when I said before, you can't actually answer the question, what does your
221240	225000	service account have access to? It's true. You cannot get the answer to the question because
225000	229640	people outside your organization can also grant access to their resources. We'll come back to that
229640	233920	later and explain what an organization is and how it's structured. But for now, I figured I'd just
233920	237800	mention that in case you were wondering. As you might imagine, not having control or visibility
237800	242640	into what your service accounts can or can't access comes with some challenges. And one particular
242640	246660	place I think those challenges manifest themselves is in a multi-tenant Kubernetes environment.
246660	251280	Google has a managed Kubernetes engine that's often shorthanded GKE. If you're familiar with
251280	256760	Kubernetes, you have a bunch of nodes, and within those nodes you have workloads. In GKE, those nodes
256760	261600	are powered by VMs, and those VMs have service accounts attached to them. If we take a look at
261600	266420	we can see a Kubernetes cluster here that's powered by four underlying nodes.
268060	273140	Then, if we hop over to Compute Engine and click on VM instances, we'll see the same four instances.
273200	276720	And if we click into one of them, we'll see that we have a service account that's attached to it.
276720	280520	We talked a little bit about how workloads interact with the service account in our B-side
280520	285500	San Francisco talk this year, so feel free to check that out. But in short, anything running
285500	289460	on the node can access this identity, so it's a good idea to descope it and not give it many
289460	293440	privileges. But let's think about what that actually means. We have a Kubernetes admin that
293440	297600	runs the nodes and the service account powering the nodes, and then we have a bunch of developers
297600	302260	that run the workloads on the Kubernetes engine. Those developers may be on separate teams and may
302260	306160	be in control of separate resources from each other and from the Kubernetes admin. So you can
306160	311800	imagine in this case we have dev1 and dev2 that have both deployed workloads, workload1 and workload2.
311800	315900	Likewise, they both control resources. In this case, we'll say that they both own a bucket and
315900	320360	they want their workloads to be able to access that bucket. One really easy way to do that is to
320360	325240	just give the service account on the node access to the bucket. But the downside to doing that is
325240	329220	instead of just giving their workload access to the bucket, they'll give every workload access to
329220	333380	the bucket. The developers in this case control this story. They control what the service account
333380	338240	has access to because they're the resource owners. The Kubernetes admin does not control this and
338240	343120	cannot see that this even happened. So you can imagine in a situation where you have a small team
343120	347920	of Kubernetes admins that are trying to maintain a cluster that want to open it up to give many
347920	352560	developers the ability to deploy workloads to this cluster, it can be difficult to secure because they
352560	357240	can't control what the service account attached to the nodes has access to. And some developers
357240	361920	may want to just give that service account access to resources. Rather than using something like
361920	367440	Kubernetes secrets or vault to pass credentials to those workloads, they may choose to just give the
367440	372660	service account attached to the underlying node access to their resources. But when they do that,
372660	378640	they give all workloads access to their resources, not just one workload. And so you end up in a
378640	384460	situation where a Kubernetes admin who wants to onboard developers into their cluster inadvertently
384460	389160	gives every new workload access to a bunch of resources, even though they can't even see that
389160	405440	or have any control over it. Okay, now before we go any further, it's important to point out that
405440	410200	up until this point, I've kind of implied that resources are like these standalone entities that
410200	415220	receive single role bindings directly to them. And while resources definitely can operate that way,
415220	419460	most of the time they're actually bundled together into groupings. And there's a role binding that
419460	423180	goes to the entire group of stuff. So instead of getting a role directly to a bucket, you'd
423180	428200	instead get a role to a collection of buckets. So fundamentally, there are four nodes, resource
428200	433060	nodes within GCP. There's an organization node, which is the parent node. This is generally
433060	438780	associated with your domain or cloud identity service. There are folders and projects. All of
438780	443980	these different top-level nodes are mostly used for IAM. And something that's important to note
443980	449600	within the resource hierarchy is the concept of IAM inheritance, where wherever you have an IAM
449600	455920	binding to one of these top-level nodes, your IAM binding is also going to be inherited to all
455920	461480	subsequent or child resources within this hierarchy. So if I have a role binding at the organization,
461480	466040	I have access to all subsequent resources that are created within the environment. Same at the
466040	471700	folder and project level. Generally, users mostly interact with resources at the project level.
471700	477140	That's where services like buckets, compute, and general services that you may interact with on a
477140	482060	day-to-day basis are. Now, let's dig into a few different IAM workflows and try to understand why
482060	487660	project-level IAM bindings are so prevalent in GCP. If we navigate to IAM and we go to grant a
487660	491300	member access to our resources, what we're doing here is we're actually creating a project-level
491300	495740	role binding, which will give this member access to all resources within our project. If we wanted
495740	500600	to create a resource-level IAM binding, we would actually need to interact with the resource-specific
500600	507960	API and set the IAM binding there. Another IAM workflow where a user is prompted to create a
507960	512000	project-level role binding is when you create a service account. When we create a service account,
512000	517400	we'll actually be prompted to set a project-level IAM binding that will grant this service account
517400	521240	access again to all resources within our project. So we can see that when you go through general
521240	526700	IAM workflows, you are going to always be setting a project-level IAM binding. Your IAM binding is
526700	532100	also going to be inherited to all subsequent or child resources within this project.
533960	538800	Here we can also see something called conditionals. This is an advanced feature within the IAM service
538800	544040	that does enable you to set more granular IAM policies, but also at the project level within
544040	548380	your environment. So I think that's really interesting that the main IAM button in the UI
548380	552780	actually takes you to the project-level bindings. A lot of people don't even realize that you can do
552780	556600	resource-level bindings because you have to actually click through into the resource to
556600	560600	see that there are IAM settings there. And a lot of people just assume that if they go to the IAM
560600	564100	page, they get the whole IAM story. So when we backtrack a little bit to when we were talking
564100	568100	about resource owners and service account owners, realistically what would actually happen is you
568100	572040	would have project owners, and within those projects you would have resources and service
572040	576180	accounts. So before when we were talking about role bindings, we mentioned this idea of Kubernetes
576180	580580	nodes having service accounts attached to them, and developers deploying workloads to the Kubernetes
580580	585180	cluster and granting them access to resources. But more realistically what that would look like is
585180	589420	they would grant that node access to their entire project. Because again, IAM bindings are typically
589420	593260	done at the project level. So instead of granting the node access to a single bucket, they would
593260	597460	grant it access to all buckets within a given project. Now this isn't always the case. Sometimes
597460	602020	developers do resource-level role binding, and it should also be noted that the UI isn't the only
602020	606560	way to apply these role bindings. You can also apply them via Terraform or via CLI. It's pretty
606560	610600	common for developers to do it at the project level, because most of their first exposure
610600	614660	actually comes through the UI. Okay, so if we recap to the earlier section, you might remember
614660	618400	when we talked about service account owners, we mentioned that the service account owner can't
618400	622400	actually know what access their service accounts have. Well, it's not really the whole story. Really
622400	625960	what we have is a project owner, and then that project owner has a bunch of service accounts in
625960	629840	their project, and then those service accounts have access to stuff, and the project owner doesn't
629840	633600	really have the whole story of what their service accounts have access to. And so you might imagine
633600	638060	how that can start to become dangerous, is when I hand over access of my project to someone else,
638060	641720	I don't really know how much access I'm handing over, because again, I don't even know how much
641720	645880	access my service accounts have in my project. So if I hand that over to someone else, I'm handing
645880	650480	over an unknown amount of access. So what does an example of this look like? Well, we have a project
650480	655240	owner that knows the email address of a different project service account. They can then give that
655240	660420	service account access to their project through an IAM binding, and then that service account has
660420	664860	access to their project's resources. Now, one question you might be asking yourself is why
664860	668000	you would even need a cross-project role binding. Well, when you think about it, when you have two
668000	672220	projects, they're not particularly useful unless they have some way to talk to each other. And one
672220	676980	common way to do that is through Google Cloud services like PubSub or Storage. And so it makes
676980	681000	sense that we would want to share some resources between projects. But again, where things sort of
681000	684960	start to become more dangerous is if we share service account access between projects, because
684960	688420	project owners don't really know what access their service accounts have. So we're missing
688420	692640	out on some important existential questions like how interconnected our projects actually are.
692640	696400	Because again, we can answer questions like who has access to a resource, but we really can't
696400	700420	answer questions like what does a service account have access to. So we came up with a solution that
700420	705100	let us sort of answer this question within our own organization. We can't actually get the complete
705100	708900	answer to the question, again, because of these things called cross-organizational bindings. But
708900	713220	at least within our org, we can get the answer for cross-project bindings by just introspecting
713220	717260	every individual project that we own and pulling all the IAM policies. We then took all that
717260	721980	information and we made a graph out of it. We actually went over to GitHub and we crawled a
721980	725960	bunch of IAM policies that people either intentionally or accidentally committed.
725960	745450	And we took all those policies and we kind of compiled this pseudo-org generator.
745450	748670	So let's take a look at one of the graphs we ended up with. As you can see, there are two
748670	753090	different colored nodes. There are red nodes and there are blue nodes. What the red nodes represent
753090	757910	are projects and what the blue nodes represent are service accounts. Those are the only two things
757910	762170	we're graphing here. Then there are two different types of edges. There are contains edges and there
762170	767070	are bindings edges. A contains edge is basically when a service account lives in a given project
767070	771210	and the binding edge is when that service account has a role binding to that project.
771470	775710	Most of the time, a service account has a role binding to the project that it's contained in.
775710	778950	But every once in a while, we'll end up with a service account that has a role binding to a
778950	783050	different project across project binding. So in this case, we have a service account that's
783050	788950	contained in project number 67 and it has a role binding into project number 57. If we zoom out,
788950	792410	we get the view of the entire organization. We have all the different projects here in a map
792410	796290	and we have all the relationships of the service accounts within the org that are contained within
796290	799810	these projects. Now we're not looking at organization-level bindings in this graph
799810	803950	or resource-level bindings. We're only looking at the project-level bindings and we're mapping
803950	808190	these relationships. So because we based this off of real IAM policies that we're able to pull up
808190	813150	GitHub, we can say with some level of confidence that this is an accurate representation of what a
813150	817030	sufficiently large company might look like. We can also say with a little bit of personal experience
817030	821030	based on the large companies that we've looked at, this is an accurate representation of what a
821030	824630	sufficiently large company looks like. Now again, most of the time a developer is just going to be
824630	828310	the owner of a single project and they won't really see the whole picture. All they'll see
828310	832630	is sort of tunnel vision for the one project that they have control over. And so they'll see the
832630	836290	service accounts that belong to their project and they'll see what has role bindings into their
836290	840690	project, but they won't know where their service accounts have role bindings out of their project.
840690	844830	And so when they hand off access to their project to another developer, they won't actually know how
844830	848830	many additional projects they're handing off. And they themselves may not have a good understanding
848830	852470	of how many projects that they have access to through their service accounts. One other
852470	856370	interesting thing that we found here is that there are certain projects that have seemingly
857030	860830	disproportionate large numbers of role bindings and resources in them. We found this was a pretty
860830	865390	consistent story at multiple companies that we looked at. And so in the center here you can see
865390	869250	there are a bunch of projects that have really tightly bound role bindings. And then at the
869250	873310	bottom we have smaller projects that only have one or two service accounts that may not be connected
873310	877170	to the broader web. And then we have a bunch of projects that have no service accounts. What we
877170	881570	found is that most of the sensitive data or sensitive workloads are in the interconnected
881570	885530	projects. That's where most of the development is happening. And so it makes sense that it would have
885530	890110	most of the IAM role bindings connecting it with other projects. So when we first started building
890110	894830	these graphs, we were thinking to ourselves, oh gosh, these projects are supposed to be isolated
894830	899550	boundaries. Are they really this interconnected? And the more we dug into it and the more we thought
899550	904010	about how project owners can't actually see the role bindings their service accounts have been
904010	908330	granted, we realized this is a problem that's really remained invisible. And it really takes
908330	912790	this org-level introspection to even see it. Now there's one final piece here that we haven't talked
912790	916210	about yet, and that's when a developer actually does hand over access to their project to another
916210	920730	developer. Pieces of that might be okay. It might be okay to share storage, for example, but things
920730	924350	start to become really dangerous if you share access to service accounts. Because again, that's where
924350	928210	the problem is hidden, where the owner of the project doesn't actually know what access their
928210	931970	service accounts have. And so if we share that unknown access, that's where things can start to
931970	936490	go astray. So we'll come back to what roles and permissions are actually required to hand over
936490	940610	that level of access. And we'll talk about how common that is. So far we've talked a lot about
940610	945050	roles and mentioned that some are more dangerous than others. So now let's dig into that a little
945050	949610	bit more. Within GCP, there are three distinct types of roles that you can use. Primitive, which
949610	955050	have been around since before Cloud IAM that we know today. Predefined roles, which are curated by
955050	960850	GCP and service specific. And custom roles that enable developers to specify any permission that
960850	965710	they would like that is supported to fit their services needs. Let's focus on primitive roles.
965710	970750	Within primitive roles, there's an owner role, which provides all of the different permissions
970750	976830	that are available within the platform. An editor role, which closely resembles the owner role,
976830	980590	but excludes a few different things for administrative capabilities and that just
980590	985490	might not be supported yet. One thing in particular that's excluded is the ability to set project level
985490	989850	role bindings. And the viewer role, which is exactly what it sounds like. It allows you to
989850	996030	view resources, but not perform any state changing actions. How are these primitive roles used? Well,
996030	1001070	earlier we saw that when you go to Cloud IAM through the UI, these primitive roles are what
1001070	1006350	are first suggested to users. There's another way that these primitive roles manifest themselves
1006350	1011590	within your GCP project. And that's through default IAM role bindings. Whatever identity
1011590	1017710	creates a project will automatically be granted a project level role binding using the owner role.
1017710	1022390	So we know that the owner role contains all of the different permissions that are available,
1022390	1027630	but what about the editor role? Well, the editor role contains, at the time of this talk,
1027630	1031290	2,576 distinct permissions.
1034710	1040250	That's a lot. This means that with the editor role, you can, for example, access all buckets,
1040250	1044410	all databases, all VMs, and much more within your project.
1045950	1050490	Out of these permissions that we talked about earlier, which of these roles actually enable a
1050490	1055210	user to manage service accounts? Well, both the owner and the editor role do. While the editor
1055210	1061610	role does not enable you to create new service accounts or apply project level role bindings
1061610	1067870	for service accounts, it does allow you to manage service accounts in two different ways. And that's
1067870	1073310	by creating keys or credentials for a given service account that already exists, and by
1073310	1079970	associating a service account with a resource. So if a user can create a resource, specify a service
1079970	1085470	account, GCP will go and check, does this user have the IAM service accounts act as permission
1085470	1090170	on the service account that they're attempting to associate with the resource? If so, it gets
1090170	1095690	attached, and then that user can interact with GCP services and APIs as that service account
1095690	1101990	identity and its associated role. So how common are these IAM roles in GCP projects? There's another
1101990	1106850	way that GCP creates automatic role bindings for identities, and this way is a little bit
1106850	1114330	different. When you enable a specific service such as Google Compute Engine, it will go and create a
1114330	1119690	default service account in your project and then grant that service account a project level role
1119690	1125190	binding using the primitive editor role that we chatted about earlier. This also happens with
1125190	1129210	another service. When a user enables the App Engine service, a service account called the App
1129210	1133690	Spot service account is created in your project and also granted a project level editor role binding.
1133690	1138490	Now as we discussed earlier, that editor role contains a permission that enables identities
1138490	1144290	to associate service accounts with resources. So that means that both of these default service
1144290	1149870	accounts, the Compute Engine and the App Spot service account, are able to associate any service
1149870	1155670	account within its GCP project to a resource. So where are these service accounts used? The default
1155910	1162750	compute service account is associated with GCE VMs by default. So whenever you create a VM,
1162750	1166770	the default compute service account is automatically associated with the VM.
1168050	1173030	There's one thing to note about service accounts and the way that they are authorized to interact
1173030	1178710	with services from a VM. There is the role that they have as well as something called scopes.
1178710	1185310	And scopes define, from within the VM, what services and APIs can that identity interact with
1185310	1191250	regardless of the role binding that they have. By default for VMs, scopes are restricted to only a
1191250	1197050	few particular services like storage. But from what we have observed, developers will commonly
1197050	1201550	open these scopes up so that they can interact with many different services. And how are the
1201550	1206370	App Spot service accounts used? Cloud Functions are Google's serverless offerings, which are
1206370	1211570	similar to AWS Lambdas. They're meant to just be lightweight, ephemeral, small chunks of code that
1211570	1217230	run fast and exit once they're done. If the App Engine service is enabled within your project,
1217230	1223290	when a developer goes to create a Cloud Function, that default App Spot service account with the editor role
1223290	1228210	binding will be associated with Cloud Functions by default. When you create a Cloud Function,
1228210	1233190	you actually do not have the ability to specify to set it without an identity. You always have to
1233190	1238250	associate an identity with the Cloud Function. So if you don't know to create a new service account
1238250	1242630	and associate with your Cloud Function, you'll always be using the default App Spot service
1242630	1246450	account that has the editor role binding. Cloud Functions do not have scopes and get all the
1246450	1254070	permissions of the editor role. And coming back to the VMs, a lot of managed services leverage VMs
1254070	1261070	as their underlying infrastructure. Things like GKE, Dataproc, Dataflow, they also get this default
1261070	1265470	compute engine service account associated with them. In AWS, when you spin up something like
1265470	1271030	a VM or a Lambda, it defaults to having no access. There are no roles attached. But in GCP, things
1271390	1275850	default to having thousands of permissions. Earlier, we talked about cross-project role
1275850	1281330	bindings, the ability to leverage service accounts in a project to get access to other service
1281330	1286410	accounts and potentially move laterally across GCP projects. We talked about developers potentially
1286410	1292090	not knowing what access their service accounts have. We mentioned the idea of handing off access.
1292090	1296190	And what this realistically looks like is the developer applies a role binding with the act
1296190	1303170	as permission, like the editor role. Let's look at what that could look like. Now it's demo time.
1308960	1313740	To help understand how to exploit the act as permission, we built a framework called Gsploit.
1313740	1318400	To start out, we need a base identity. There are a number of different ways to get a base identity.
1318400	1323160	One way, for example, is you could OAuth fish a developer that has project level editor access.
1323160	1327560	Another way is maybe a developer accidentally uploaded a service account key to GitHub or
1327560	1332620	and that service account has a editor level role binding because the primitive roles are the first
1332620	1337120	thing that are suggested to you on the IAM page. Or maybe the most likely, maybe the base identity
1337120	1342080	comes from exploiting a service and because most services run with project editor by default, maybe
1342080	1346160	that's how you get your first foothold. In any case, in this case, we've simulated this by just
1346160	1349960	starting with the service account credential. We can see this credential here and we can see that
1349960	1356420	it starts in the project bbs2. Next, we'll run the list operation on the Gsploit tool. We can
1356420	1361360	see that there's no output. The reason there's no output is the list operation lists all the
1361360	1365300	service accounts that you've managed to take control of and it doesn't count the base identity.
1365300	1369820	We'll run the gcloud command projects list on the base identity to see what project we have access
1369820	1377160	to. It looks like we have access to bbs2. Next, we'll try an act as exploit on this project and
1377160	1384500	we'll target every service account in the project. Let's unpack what's happening now.
1384500	1388920	Keep in mind the editor role has two ways to take control of all the service accounts in a project.
1388920	1392860	The first is through creating tokens and the second is through provisioning resources.
1392880	1396580	There's another role that we haven't talked about yet and that's the service account user role.
1396580	1400580	This role is also pretty common and is often given to users and service accounts that are meant to
1400580	1405000	provision resources. Because some things like cloud functions require service accounts be
1405000	1410340	attached to them, this is the role that's usually granted to do that. So because the service account
1410340	1414800	user role and the editor role both have the act as permission, we'll use that to take control of
1414800	1418800	the service accounts because it casts a little bit wider of a net than the token creator permission
1418800	1423100	which is only in the editor role. So what gsploit is actually doing under the hood here is first
1423100	1426860	it's running the service account list operation to get a list of all the service accounts in the
1426860	1431080	project and then it's spinning up a cloud function for each one of those service accounts. Then we
1431080	1435160	keep a database of all the cloud functions we've harvested, each which has its own underlying
1435160	1438960	service account attached to it. This gives us access to all of the service accounts in the
1438960	1442640	project through the cloud functions that we spun up. We control the code that lands on the cloud
1442640	1447400	function, so we'll just deploy code that allows us to run whatever gcloud commands we want. It
1447400	1451000	takes about two minutes to spin up each cloud function, so we'll just speed through this really
1451000	1458020	quick. And now when we run the list command again, we can see all the new service accounts that we
1458020	1462560	now have access to. One of them says owner. That service account actually has an owner level role
1462560	1466040	binding to the project that we started in. That means we now have full control of the project
1466040	1470180	through this service account. If we wanted to, we could use this service account to add ourselves as
1470180	1474500	owners to the project. So just to pause and think about that for a second, what that means is if you
1474500	1478940	ever have a service account that has editor level access and another service account that has owner
1478940	1483840	level access, the editor level service account can always privilege escalate itself to owner.
1483940	1487920	That's the same with developers. If you grant a developer editor level access to a project but you
1487920	1491760	happen to have a service account in that project that has owner level access, that developer can
1491760	1495180	privilege escalate themselves to owner through that service account. Okay, let's take another
1495180	1498480	look at those service accounts that we have. There's one here that's kind of interesting that
1498480	1502900	says do nothing. It might not sound interesting, but keep in mind that the developer that created
1502900	1506840	that service account in this project, although they didn't intend to give it any role bindings,
1506840	1510420	again, the owner of the service accounts can't control the role bindings, and so it's possible
1510420	1514400	that a different project gave it a role binding. So let's run the project list command from this
1514400	1519140	service account that we've compromised. Something interesting here that the base project, bbs2,
1519140	1522960	doesn't actually show up when we run this command. This is because the service account has no role
1522960	1527400	bindings there. The developer didn't want it to have any role bindings. But bbs3, this new project
1527400	1531600	we haven't looked at, shows up. What happened here is the developer in another project granted this
1531600	1535900	service account and added a role binding into their project. So let's run the act as exploit
1535900	1541160	one more time, this time from this do nothing service account into bbs3 and see how many
1541160	1545300	service accounts we can harvest in bbs3. Again, this process takes a little while, so we're going
1545300	1554040	to fast forward through this. And now we can see a whole bunch of new service accounts in the bbs3
1554040	1558320	project. Service accounts that we didn't have access to before that we were able to get strictly
1558320	1561880	through lateral movement. There's a couple service accounts in here that look interesting. There's one
1561880	1565460	that's labeled networking that maybe controls networking for the organization. But probably
1565460	1569520	most interesting is this one that's labeled organizational admin. If while you're moving
1569520	1573680	laterally through projects you happen to land on a service account that has an org level binding,
1573680	1578480	through inheritance you get access to the entire org that way. So to recap, we started with the
1578480	1583420	base identity in bbs2 that had act as on the project. We then used that permission to gain
1583420	1587680	access to every service account in this project via cloud functions. One of the service accounts
1587680	1592880	in bbs2 had an owner role, which allowed us to elevate privileges in the bbs2 project. A different
1592880	1598100	service account had a role binding into bbs3. This allowed us to use act as to take control of all the
1598100	1602320	service accounts in bbs3. One of the service accounts we found in bbs3 had an organizational
1602320	1623870	role binding, and this allowed us to take control of the entire org. So we talked about when you
1623870	1628170	provision resources, when you attach service accounts to those resources, GCP will check to
1628170	1632170	make sure that you have the act as permission first before allowing you to provision the resource.
1632170	1636470	Because some APIs like cloud functions require that you have identities to power them, this leads
1636470	1640630	to a lot of people getting the act as permission. Well it turns out that some APIs will actually
1640630	1644330	allow you to provision resources with the service account attached to them without the need for the
1644330	1649950	act as permission. These include the Dataproc, Dataflow, and Composer APIs. Let's talk about the
1649950	1655910	Dataproc API. The Dataproc API is Google's data processing API. Among other things, it includes
1655910	1661030	managed Apache Hadoop and managed Apache Spark. When you spin up a Dataproc cluster, like a lot
1661030	1665950	of things in GCP, it will default to using the default Compute Engine service account. This means
1665950	1670330	by default your Spark jobs will run as project editor. You only need the permission Dataproc
1670330	1675570	cluster create to spin up this cluster. You don't actually need the act as permission on the default
1675570	1680610	service account. Because of this, anyone who has the Dataproc cluster create permission can actually
1680610	1684690	make use of the default editor service account. Put another way, service accounts that are only
1684690	1688790	intended to be able to spin up Dataproc clusters can privilege escalate themselves to project
1688790	1693290	editor. And then they can make use of that act as permission to take control of the rest of the
1693290	1697330	service accounts in the project. We built support for this into gsploit, so let's take a look and
1697330	1701310	see what it looks like. If we take another look at the list of service accounts we've compromised,
1701310	1705750	there's one here that says Dataproc. Let's run the gcloud projects list on this service account and
1705750	1717400	see what projects it has access to. Looks like it has access to a bbs4. Now let's run the Dataproc
1717400	1722440	exploit on bbs4 through the Dataproc service account. We have to spin up a whole Dataproc
1722440	1726420	cluster to do this, and that takes a little bit of time. And then we delete it afterwards so that
1726420	1730000	we're not filled too long. So we'll fast forward through this part, but at the end of it when we
1730000	1734580	run gcloud list again, we can see a new service account has been added. And that's the default
1734580	1739540	service account in the bbs4 project. We were able to get this service account by spinning up a Dataproc
1739540	1744040	cluster in bbs4 and then running a spark job on that cluster, which enabled us to grab the
1744040	1748740	off the instance. Because our Dataproc service account credential had the ability to spin the
1748740	1752820	cluster up, it also controlled the scopes for that cluster. And so we just defined the scopes
1752820	1757220	for this cluster to be wide open. Next, since we have a default editor service account credential
1757220	1765820	at this point, we can run the act as exploit on this project. We'll again fast forward through
1765820	1774420	this process. And finally, if we run the gcloud list one more time, we can see a whole bunch of
1774420	1779340	service accounts that we now have access to in the bbs4 project. So just to recap, some APIs in
1779340	1783880	GCP do not require the act as permission to spin up resources and attach the default service
1783880	1788300	accounts to them. The identities that can spin those resources up can also control the scopes.
1788300	1792940	So effectively, in cases like Dataproc and Dataflow and Composer, any identity that's
1792940	1797520	granted permissions to those APIs can privilege escalate themselves to project editor. One last
1797520	1801520	thing to note here is because this Dataproc role binding was actually cross-project,
1801520	1806300	this allowed us to again compromise the new project bbs4. And because the default service
1806300	1831140	accounts are in every project, this allows us to take control of all the service accounts in the
1831140	1835440	IAM analyzer. We worked with Google on this feature, and one of the user stories actually
1835440	1841780	came from us. We came up with a list of dangerous roles and permissions that can be used to take
1841780	1846140	control of all the service accounts in the project. These can be seen here. We won't demo Dataflow
1846140	1850780	and Composer, but you can think of them as working the same way as Dataproc. Using the IAM analyzer
1850780	1854980	and the dangerous permissions that we're able to find, we're able to run a BFS search on a base
1854980	1859720	identity without the need to spin up expensive resources exploiting services. As you can see
1859720	1863760	from the output here, we get the same list of service accounts from the same base identity as
1863760	1867940	we did exploiting things, except this was much faster. We didn't have to spin anything up. You
1867940	1873100	need an org binding to use this, so this is a tool for defenders to be able to get quick answers to
1873100	1877080	how an attacker might be able to move laterally through the organization. If we cut back to our
1877080	1882220	mock graph, let's zoom into a seemingly innocent base identity. This service account, gwadvcl,
1882220	1887080	was meant to only run Spark jobs in its base project 14. It only has the Dataproc editor role.
1887080	1890940	Now we'll run a BFS search from this starting identity and highlight the path in blue.
1891020	1894500	As you can see, this allowed us to compromise all the service accounts in the starting project
1894500	1898420	through the default editor role on our Spark job. Most of these service accounts have benign roles
1898420	1903960	like storage or PubSub, and most of them are not cross-project. That said, some, like this one,
1903960	1908240	have dangerous role bindings that are cross-project, in this case the Dataflow developer role.
1908240	1912280	These allow us to take full control of service accounts in different projects. As we start to
1912280	1916360	zoom out, we can begin to see the whole picture. We've managed to compromise most of the service
1916360	1920400	accounts in the org, with a few notable project exceptions that just don't have any dangerous role
1920400	1924500	bindings to them. We're only looking at project-level bindings in this graph, so if any of
1924500	1929760	these service accounts have org-level bindings, it's not reflected here. So we just saw some really
1929760	1934380	interesting techniques around privilege escalation and lateral movement, but we also wanted to
1934380	1938980	provide a way to detect this. So we fingerprinted this tooling, and we'll be releasing monitoring
1938980	1943760	and alerting to detect the behavior of this tool in your environment. I think one of the nice things
1943760	1948040	about doing security work in the cloud is how much help you get from your upstream provider. If we
1948040	1952540	were doing work on a web application or a binary, we'd kind of be left to our own vices to fix the
1952540	1956000	problems we found. But because we had a close partnership with Google, we were able to work
1956000	1959960	with them to roll out some new features that help with the security story of some of these problems.
1960020	1963420	One of the people that helped us do that is a security engineer named Bakunov, and we gave him
1963420	1967240	an invitation to do a brief cameo and talk about some of the new features that we helped Google
1967240	1976680	roll out. Hello, Black Hat. I had the privilege of working with Allison and Dylan. As a result of
1976680	1981540	our collaboration, we at Google Cloud launched several features that help customers address
1981540	1987180	concerns. On the prevention side of things, we now have an org policy constraint that prevents
1987180	1992960	the editor grant to the default service accounts for Compute Engine and App Engine, and we advise
1992960	1998060	customers to turn it on. We recommend that customers create custom service accounts with
1998060	2003620	the necessary set of permissions on the target resources, as opposed to the broad project-level
2003620	2009880	editor grants. Allison and Dylan briefly touched upon the investigative IAM analyzer tool. As you
2009880	2016040	saw, we built IAM analyzer to be able to address the use case of finding resources which are
2016040	2021960	granted access to any specific identity. The other investigative feature available is the IAM
2021960	2027480	recommender. It uses machine learning to evaluate the permissions granted to an identity against
2027480	2032540	actual activities performed by the identity. It's a great way to find overly generous permission
2032540	2037660	grants and to identify the actual permissions that are needed for the workload. I spent many years
2037660	2042600	working with all major cloud providers, and this statement is generally true for all of them.
2042600	2048040	Starter cloud UIs, those flows are optimized for ease of onboarding. And so the fundamental
2048040	2054340	challenge for large organizations is to make available sanely secured defaults, as well as
2054340	2059740	preventative guardrails for their developers. Those are usually in the shape of cloud foundation
2059740	2065060	code bases, such as infrastructures code samples that are built with security in mind, policy as
2065060	2070660	code evaluations as part of the build and release pipelines, and preventative controls exposed by
2070660	2076620	the cloud providers, such as org policy constraints on GCP. If we're leaving developers to the mercy
2076620	2082680	of those starter UI flows on the cloud providers, or stack overflow code snippets, we're going to
2082680	2086960	end up in a bad place. If you're a Google Cloud customer, and they're looking for guidance in this
2086960	2092980	area, you should try to engage your account team so that we can help out. Alison and Dylan, on behalf
2092980	2098720	of everyone at Google Cloud, we are very grateful for the collaboration on improving our customers'
2098720	2103580	experience on the platform, and for the opportunity to have a cameo appearance in this talk. I'm hoping
2103580	2108820	that Black Hat 2021 happens in person, and we can share stories in the face-to-face setting. Until
2108820	2114480	then, please wear face masks, and don't forget to turn on two-factor authentication wherever possible.
2119740	2125000	Thank you. Thank you, Bak. We also wanted to provide some context, and a little bit more
2125000	2130000	insight, and let you know a few of the limitations or constraints to these tools. So one of the main
2130000	2134860	limitations to the organization policy services, you actually have to have an organization resource,
2134860	2140040	either through Cloud Identity or G Suite, to be able to use the organization policy service. If
2140040	2143520	you only have projects, you won't actually be able to leverage any of the constraints that are
2143520	2147840	available. Another limitation within the organization policy service is that constraints are not
2147840	2154040	retroactive. So when the policy is applied, all resources will be affected by the policy or
2154040	2159940	evaluated by the policy after it is enabled, but existing resources will not be modified to match
2159940	2164800	or meet the requirements of a given constraint. Another thing to note is that for the new Disable
2164800	2171380	Automatic IAM Grants for Default Service Accounts, that constraint will only enforce that when the
2171380	2176280	APIs are initially enabled for new projects, that the default service accounts are not granted
2176280	2181080	editor-level role bindings at the time of the APIs being enabled, but there is no enforcement
2181080	2186420	mechanism to ensure that the default service accounts are not granted a project-level binding
2186420	2193120	or other binding. There's also no way to restrict the cross-project bindings that we talked about
2193120	2197440	previously, where if a user knows the name of your service account, they could still give
2197440	2201820	the service account access to their project or project's resources. So the IAM Recommender is
2201820	2207660	very powerful. Some of the limitations on the tool is that you have to have a 90-day period where the
2207660	2212800	policies or behavior of a given identity are evaluated so that you can have a recommendation
2212800	2218020	be made. And there may be more IAM bindings that you may not be aware of, like the cross-project
2218020	2223680	bindings. The IAM Analyzer can provide a lot of context into the IAM bindings or associations for
2223800	2229000	a given identity. And in order to use this auditing tool, you have to have administrative
2229000	2234760	capabilities in an organization to be able to see the policies that are being analyzed. So in order
2234760	2239820	to use this tool to the fullest extent, you need administrative capabilities in your organization,
2239820	2246180	as well as within G Suite, if you want to expand group membership to know who has access to a given
2246180	2251520	service account that is in a group. Earlier, we talked about how Kubernetes workloads have access
2251520	2257820	to the underlying service account that's attached to the node. And the suggested way to harden your
2257820	2262720	clusters is to use a service called Workload Identity. And Workload Identity allows you to
2262720	2268280	associate a Kubernetes service account with a cloud identity. And this is how you can harden
2268280	2272260	your cluster, as well as have a one-to-one mapping of Kubernetes service accounts to
2272260	2276940	cloud identity service accounts. We brought up earlier to ensure default service accounts are not
2276940	2282620	granted project-level role bindings or other role bindings after the disable automatic IM grants for
2282860	2287480	default service accounts constraint is enabled, that you require additional tooling. And to
2287480	2293100	demonstrate a way that you can enforce these configurations, we will be releasing a Terraform
2293100	2298460	Enterprise Sentinel policy set that can demonstrate enforcing that default service accounts are not
2298460	2305020	granted project-level role bindings or that roles that enable service account takeover or management
2305020	2309620	of service accounts cannot be applied at the project level. Sentinel policies are Terraform
2309620	2316500	Enterprise policy configuration language that enable you to either restrict or alert on specific
2316500	2322780	configurations in your runs. So just to recap some of the key takeaways, IAM and GCP is very resource
2322780	2327260	centric, meaning that if you are a service accounts owner, you may not be aware of the configurations
2327260	2332480	that are associated with that service account or how it may be used across a GCP organization. This
2332480	2336880	can lead to projects being interconnected in many different ways that you may not be able to see.
2336880	2342380	Role bindings are often done at the project level and can grant more access than a user intends to.
2342380	2348020	A lot of managed services and general resource provisioning operations can hand off access to
2348020	2353260	your project's resources and IAM that you may not be aware of. This kind of access that's granted
2353260	2358400	through resource creation can lead to privilege escalation and lateral movements within your
2358400	2363300	environments. Another thing we covered is that default service accounts are granted administrative
2363300	2368100	roles within your project automatically and that's with the editor role unless you're using the
2368100	2373120	disable automatic IAM grants for default service accounts organization constraint. Reduce usage of
2373440	2378560	default service accounts where possible by creating service specific service accounts with IAM roles
2378560	2383040	that are scoped to your services needs. There are a few new cool tools in the platform that have been
2383040	2388420	released to assist you in understanding who has access to your service accounts, such as the IAM
2388420	2393880	analyzer, which can also be really helpful for understanding IAM dependencies and access within
2393880	2395320	your environments.