[00:01.760]Hello, my name is Allison. [00:03.580] [00:07.250]Hi, my name is Dylan. [00:08.770] [00:12.270]We've been researching GCP for the last year and a half and we've come up with some effective ways [00:16.250] [00:16.250]that compromise most GCP organizations, all the ones we've looked at anyway. These techniques [00:20.530] [00:20.530]involve moving laterally with service accounts and cloud APIs. We have a lot of ground to cover, [00:24.990] [00:24.990]so let's get to it. I think everyone has a slightly different reaction when they first [00:41.220] [00:41.220]start learning about GCP IAM. But one common thing I've observed is people often compare it [00:48.060] [00:48.060]to AWS IAM, but there are some pretty key differences between GCP and AWS IAM. So let's [00:53.020] [00:53.020]talk about some of those differences. First, in AWS, we have user identities and those user [00:58.280] [00:58.280]identities can be exported in the form of credentials, which look something like this. [01:03.420] [01:03.560]In GCP, the equivalent is a service account and service accounts can also be exported [01:09.900] [01:09.900]in credentials that look a little bit differently. They look like this. [01:13.200] [01:14.500]And in both cases, we have owners of these identities. But where things get a little [01:19.180] [01:19.180]bit different is when you start to ask the question, what can my identity access? In [01:23.620] [01:23.620]the world of AWS, you basically have a policy that determines all the different things that [01:28.180] [01:28.180]the identity can access. And the person who can create, read, update, delete that policy [01:33.460] [01:33.460]is the same person who owns the identity. So here's an example of what that looks like. [01:38.400] [01:39.000]In GCP though, this is where things start getting really different. Instead of having [01:42.660] [01:42.940]a centralized policy for a given identity, we have policies for resources. So in this case, [01:48.460] [01:48.460]we've got a bucket and a bucket owner and that bucket has a policy. The bucket owner controls [01:53.380] [01:53.380]who can create, read, update, or delete that policy. And that policy applies to whatever [01:58.620] [01:58.620]identity the bucket owner wants it to. So here's an example of what that looks like for storage. [02:04.420] [02:05.600]What's interesting here is that the owner of the service account actually doesn't have any [02:11.360] [02:11.360]insight into this policy. They can't view it. They can't update it. They don't even know that [02:16.900] [02:16.900]their service account was given access to this bucket. Instead, the bucket owner controls this. [02:21.880] [02:21.880]And if the owner of the service account doesn't have IAM list on the bucket, then they won't [02:26.400] [02:26.400]be able to even see that that policy was granted. And I think this is one of the harder concepts [02:32.660] [02:32.660]it was for me personally to wrap my head around in GCP, is if you have an identity in GCP, like a [02:37.660] [02:37.660]user or a service account, as the owner of that identity, you can't actually know what that [02:43.080] [02:43.080]identity has access to. In fact, no one can answer the question, what does the service account have [02:47.740] [02:47.740]access to? Instead, resource owners control what the policy is for a given resource. So you can [02:52.320] [02:52.320]answer questions like, who has access to this bucket? But you can't answer questions like, [02:56.920] [02:56.920]how many buckets does the service account have access to? Or what buckets does the service [03:00.360] [03:00.360]account have access to? Those questions are unanswerable by design in GCP. Likewise, we can [03:05.620] [03:05.620]have other resources like BigQuery, for example. And we have an owner for the BigQuery and a policy [03:09.960] [03:09.960]for the BigQuery. And the owner of the BigQuery can, again, grant access to any identity they want. [03:15.180] [03:15.180]However, the owner of the service account has no control over this and can't see this. One thing [03:20.060] [03:20.060]you might be wondering is, before when I mentioned that the owner of the service account can't fully [03:23.880] [03:23.880]answer the question, what does the service account have access to? You might be thinking, well, if you [03:28.120] [03:28.120]own an organization, can't you just get the IEM policies of all the resources in the organization? [03:32.960] [03:32.960]Well, someone outside your organization can still grant your service account access to their [03:36.860] [03:36.860]resources. So when I said before, you can't actually answer the question, what does your [03:41.240] [03:41.240]service account have access to? It's true. You cannot get the answer to the question because [03:45.000] [03:45.000]people outside your organization can also grant access to their resources. We'll come back to that [03:49.640] [03:49.640]later and explain what an organization is and how it's structured. But for now, I figured I'd just [03:53.920] [03:53.920]mention that in case you were wondering. As you might imagine, not having control or visibility [03:57.800] [03:57.800]into what your service accounts can or can't access comes with some challenges. And one particular [04:02.640] [04:02.640]place I think those challenges manifest themselves is in a multi-tenant Kubernetes environment. [04:06.660] [04:06.660]Google has a managed Kubernetes engine that's often shorthanded GKE. If you're familiar with [04:11.280] [04:11.280]Kubernetes, you have a bunch of nodes, and within those nodes you have workloads. In GKE, those nodes [04:16.760] [04:16.760]are powered by VMs, and those VMs have service accounts attached to them. If we take a look at [04:21.600] [04:21.600]we can see a Kubernetes cluster here that's powered by four underlying nodes. [04:26.420] [04:28.060]Then, if we hop over to Compute Engine and click on VM instances, we'll see the same four instances. [04:33.140] [04:33.200]And if we click into one of them, we'll see that we have a service account that's attached to it. [04:36.720] [04:36.720]We talked a little bit about how workloads interact with the service account in our B-side [04:40.520] [04:40.520]San Francisco talk this year, so feel free to check that out. But in short, anything running [04:45.500] [04:45.500]on the node can access this identity, so it's a good idea to descope it and not give it many [04:49.460] [04:49.460]privileges. But let's think about what that actually means. We have a Kubernetes admin that [04:53.440] [04:53.440]runs the nodes and the service account powering the nodes, and then we have a bunch of developers [04:57.600] [04:57.600]that run the workloads on the Kubernetes engine. Those developers may be on separate teams and may [05:02.260] [05:02.260]be in control of separate resources from each other and from the Kubernetes admin. So you can [05:06.160] [05:06.160]imagine in this case we have dev1 and dev2 that have both deployed workloads, workload1 and workload2. [05:11.800] [05:11.800]Likewise, they both control resources. In this case, we'll say that they both own a bucket and [05:15.900] [05:15.900]they want their workloads to be able to access that bucket. One really easy way to do that is to [05:20.360] [05:20.360]just give the service account on the node access to the bucket. But the downside to doing that is [05:25.240] [05:25.240]instead of just giving their workload access to the bucket, they'll give every workload access to [05:29.220] [05:29.220]the bucket. The developers in this case control this story. They control what the service account [05:33.380] [05:33.380]has access to because they're the resource owners. The Kubernetes admin does not control this and [05:38.240] [05:38.240]cannot see that this even happened. So you can imagine in a situation where you have a small team [05:43.120] [05:43.120]of Kubernetes admins that are trying to maintain a cluster that want to open it up to give many [05:47.920] [05:47.920]developers the ability to deploy workloads to this cluster, it can be difficult to secure because they [05:52.560] [05:52.560]can't control what the service account attached to the nodes has access to. And some developers [05:57.240] [05:57.240]may want to just give that service account access to resources. Rather than using something like [06:01.920] [06:01.920]Kubernetes secrets or vault to pass credentials to those workloads, they may choose to just give the [06:07.440] [06:07.440]service account attached to the underlying node access to their resources. But when they do that, [06:12.660] [06:12.660]they give all workloads access to their resources, not just one workload. And so you end up in a [06:18.640] [06:18.640]situation where a Kubernetes admin who wants to onboard developers into their cluster inadvertently [06:24.460] [06:24.460]gives every new workload access to a bunch of resources, even though they can't even see that [06:29.160] [06:29.160]or have any control over it. Okay, now before we go any further, it's important to point out that [06:45.440] [06:45.440]up until this point, I've kind of implied that resources are like these standalone entities that [06:50.200] [06:50.200]receive single role bindings directly to them. And while resources definitely can operate that way, [06:55.220] [06:55.220]most of the time they're actually bundled together into groupings. And there's a role binding that [06:59.460] [06:59.460]goes to the entire group of stuff. So instead of getting a role directly to a bucket, you'd [07:03.180] [07:03.180]instead get a role to a collection of buckets. So fundamentally, there are four nodes, resource [07:08.200] [07:08.200]nodes within GCP. There's an organization node, which is the parent node. This is generally [07:13.060] [07:13.060]associated with your domain or cloud identity service. There are folders and projects. All of [07:18.780] [07:18.780]these different top-level nodes are mostly used for IAM. And something that's important to note [07:23.980] [07:23.980]within the resource hierarchy is the concept of IAM inheritance, where wherever you have an IAM [07:29.600] [07:29.600]binding to one of these top-level nodes, your IAM binding is also going to be inherited to all [07:35.920] [07:35.920]subsequent or child resources within this hierarchy. So if I have a role binding at the organization, [07:41.480] [07:41.480]I have access to all subsequent resources that are created within the environment. Same at the [07:46.040] [07:46.040]folder and project level. Generally, users mostly interact with resources at the project level. [07:51.700] [07:51.700]That's where services like buckets, compute, and general services that you may interact with on a [07:57.140] [07:57.140]day-to-day basis are. Now, let's dig into a few different IAM workflows and try to understand why [08:02.060] [08:02.060]project-level IAM bindings are so prevalent in GCP. If we navigate to IAM and we go to grant a [08:07.660] [08:07.660]member access to our resources, what we're doing here is we're actually creating a project-level [08:11.300] [08:11.300]role binding, which will give this member access to all resources within our project. If we wanted [08:15.740] [08:15.740]to create a resource-level IAM binding, we would actually need to interact with the resource-specific [08:20.600] [08:20.600]API and set the IAM binding there. Another IAM workflow where a user is prompted to create a [08:27.960] [08:27.960]project-level role binding is when you create a service account. When we create a service account, [08:32.000] [08:32.000]we'll actually be prompted to set a project-level IAM binding that will grant this service account [08:37.400] [08:37.400]access again to all resources within our project. So we can see that when you go through general [08:41.240] [08:41.240]IAM workflows, you are going to always be setting a project-level IAM binding. Your IAM binding is [08:46.700] [08:46.700]also going to be inherited to all subsequent or child resources within this project. [08:52.100] [08:53.960]Here we can also see something called conditionals. This is an advanced feature within the IAM service [08:58.800] [08:58.800]that does enable you to set more granular IAM policies, but also at the project level within [09:04.040] [09:04.040]your environment. So I think that's really interesting that the main IAM button in the UI [09:08.380] [09:08.380]actually takes you to the project-level bindings. A lot of people don't even realize that you can do [09:12.780] [09:12.780]resource-level bindings because you have to actually click through into the resource to [09:16.600] [09:16.600]see that there are IAM settings there. And a lot of people just assume that if they go to the IAM [09:20.600] [09:20.600]page, they get the whole IAM story. So when we backtrack a little bit to when we were talking [09:24.100] [09:24.100]about resource owners and service account owners, realistically what would actually happen is you [09:28.100] [09:28.100]would have project owners, and within those projects you would have resources and service [09:32.040] [09:32.040]accounts. So before when we were talking about role bindings, we mentioned this idea of Kubernetes [09:36.180] [09:36.180]nodes having service accounts attached to them, and developers deploying workloads to the Kubernetes [09:40.580] [09:40.580]cluster and granting them access to resources. But more realistically what that would look like is [09:45.180] [09:45.180]they would grant that node access to their entire project. Because again, IAM bindings are typically [09:49.420] [09:49.420]done at the project level. So instead of granting the node access to a single bucket, they would [09:53.260] [09:53.260]grant it access to all buckets within a given project. Now this isn't always the case. Sometimes [09:57.460] [09:57.460]developers do resource-level role binding, and it should also be noted that the UI isn't the only [10:02.020] [10:02.020]way to apply these role bindings. You can also apply them via Terraform or via CLI. It's pretty [10:06.560] [10:06.560]common for developers to do it at the project level, because most of their first exposure [10:10.600] [10:10.600]actually comes through the UI. Okay, so if we recap to the earlier section, you might remember [10:14.660] [10:14.660]when we talked about service account owners, we mentioned that the service account owner can't [10:18.400] [10:18.400]actually know what access their service accounts have. Well, it's not really the whole story. Really [10:22.400] [10:22.400]what we have is a project owner, and then that project owner has a bunch of service accounts in [10:25.960] [10:25.960]their project, and then those service accounts have access to stuff, and the project owner doesn't [10:29.840] [10:29.840]really have the whole story of what their service accounts have access to. And so you might imagine [10:33.600] [10:33.600]how that can start to become dangerous, is when I hand over access of my project to someone else, [10:38.060] [10:38.060]I don't really know how much access I'm handing over, because again, I don't even know how much [10:41.720] [10:41.720]access my service accounts have in my project. So if I hand that over to someone else, I'm handing [10:45.880] [10:45.880]over an unknown amount of access. So what does an example of this look like? Well, we have a project [10:50.480] [10:50.480]owner that knows the email address of a different project service account. They can then give that [10:55.240] [10:55.240]service account access to their project through an IAM binding, and then that service account has [11:00.420] [11:00.420]access to their project's resources. Now, one question you might be asking yourself is why [11:04.860] [11:04.860]you would even need a cross-project role binding. Well, when you think about it, when you have two [11:08.000] [11:08.000]projects, they're not particularly useful unless they have some way to talk to each other. And one [11:12.220] [11:12.220]common way to do that is through Google Cloud services like PubSub or Storage. And so it makes [11:16.980] [11:16.980]sense that we would want to share some resources between projects. But again, where things sort of [11:21.000] [11:21.000]start to become more dangerous is if we share service account access between projects, because [11:24.960] [11:24.960]project owners don't really know what access their service accounts have. So we're missing [11:28.420] [11:28.420]out on some important existential questions like how interconnected our projects actually are. [11:32.640] [11:32.640]Because again, we can answer questions like who has access to a resource, but we really can't [11:36.400] [11:36.400]answer questions like what does a service account have access to. So we came up with a solution that [11:40.420] [11:40.420]let us sort of answer this question within our own organization. We can't actually get the complete [11:45.100] [11:45.100]answer to the question, again, because of these things called cross-organizational bindings. But [11:48.900] [11:48.900]at least within our org, we can get the answer for cross-project bindings by just introspecting [11:53.220] [11:53.220]every individual project that we own and pulling all the IAM policies. We then took all that [11:57.260] [11:57.260]information and we made a graph out of it. We actually went over to GitHub and we crawled a [12:01.980] [12:01.980]bunch of IAM policies that people either intentionally or accidentally committed. [12:05.960] [12:05.960]And we took all those policies and we kind of compiled this pseudo-org generator. [12:25.450] [12:25.450]So let's take a look at one of the graphs we ended up with. As you can see, there are two [12:28.670] [12:28.670]different colored nodes. There are red nodes and there are blue nodes. What the red nodes represent [12:33.090] [12:33.090]are projects and what the blue nodes represent are service accounts. Those are the only two things [12:37.910] [12:37.910]we're graphing here. Then there are two different types of edges. There are contains edges and there [12:42.170] [12:42.170]are bindings edges. A contains edge is basically when a service account lives in a given project [12:47.070] [12:47.070]and the binding edge is when that service account has a role binding to that project. [12:51.210] [12:51.470]Most of the time, a service account has a role binding to the project that it's contained in. [12:55.710] [12:55.710]But every once in a while, we'll end up with a service account that has a role binding to a [12:58.950] [12:58.950]different project across project binding. So in this case, we have a service account that's [13:03.050] [13:03.050]contained in project number 67 and it has a role binding into project number 57. If we zoom out, [13:08.950] [13:08.950]we get the view of the entire organization. We have all the different projects here in a map [13:12.410] [13:12.410]and we have all the relationships of the service accounts within the org that are contained within [13:16.290] [13:16.290]these projects. Now we're not looking at organization-level bindings in this graph [13:19.810] [13:19.810]or resource-level bindings. We're only looking at the project-level bindings and we're mapping [13:23.950] [13:23.950]these relationships. So because we based this off of real IAM policies that we're able to pull up [13:28.190] [13:28.190]GitHub, we can say with some level of confidence that this is an accurate representation of what a [13:33.150] [13:33.150]sufficiently large company might look like. We can also say with a little bit of personal experience [13:37.030] [13:37.030]based on the large companies that we've looked at, this is an accurate representation of what a [13:41.030] [13:41.030]sufficiently large company looks like. Now again, most of the time a developer is just going to be [13:44.630] [13:44.630]the owner of a single project and they won't really see the whole picture. All they'll see [13:48.310] [13:48.310]is sort of tunnel vision for the one project that they have control over. And so they'll see the [13:52.630] [13:52.630]service accounts that belong to their project and they'll see what has role bindings into their [13:56.290] [13:56.290]project, but they won't know where their service accounts have role bindings out of their project. [14:00.690] [14:00.690]And so when they hand off access to their project to another developer, they won't actually know how [14:04.830] [14:04.830]many additional projects they're handing off. And they themselves may not have a good understanding [14:08.830] [14:08.830]of how many projects that they have access to through their service accounts. One other [14:12.470] [14:12.470]interesting thing that we found here is that there are certain projects that have seemingly [14:16.370] [14:17.030]disproportionate large numbers of role bindings and resources in them. We found this was a pretty [14:20.830] [14:20.830]consistent story at multiple companies that we looked at. And so in the center here you can see [14:25.390] [14:25.390]there are a bunch of projects that have really tightly bound role bindings. And then at the [14:29.250] [14:29.250]bottom we have smaller projects that only have one or two service accounts that may not be connected [14:33.310] [14:33.310]to the broader web. And then we have a bunch of projects that have no service accounts. What we [14:37.170] [14:37.170]found is that most of the sensitive data or sensitive workloads are in the interconnected [14:41.570] [14:41.570]projects. That's where most of the development is happening. And so it makes sense that it would have [14:45.530] [14:45.530]most of the IAM role bindings connecting it with other projects. So when we first started building [14:50.110] [14:50.110]these graphs, we were thinking to ourselves, oh gosh, these projects are supposed to be isolated [14:54.830] [14:54.830]boundaries. Are they really this interconnected? And the more we dug into it and the more we thought [14:59.550] [14:59.550]about how project owners can't actually see the role bindings their service accounts have been [15:04.010] [15:04.010]granted, we realized this is a problem that's really remained invisible. And it really takes [15:08.330] [15:08.330]this org-level introspection to even see it. Now there's one final piece here that we haven't talked [15:12.790] [15:12.790]about yet, and that's when a developer actually does hand over access to their project to another [15:16.210] [15:16.210]developer. Pieces of that might be okay. It might be okay to share storage, for example, but things [15:20.730] [15:20.730]start to become really dangerous if you share access to service accounts. Because again, that's where [15:24.350] [15:24.350]the problem is hidden, where the owner of the project doesn't actually know what access their [15:28.210] [15:28.210]service accounts have. And so if we share that unknown access, that's where things can start to [15:31.970] [15:31.970]go astray. So we'll come back to what roles and permissions are actually required to hand over [15:36.490] [15:36.490]that level of access. And we'll talk about how common that is. So far we've talked a lot about [15:40.610] [15:40.610]roles and mentioned that some are more dangerous than others. So now let's dig into that a little [15:45.050] [15:45.050]bit more. Within GCP, there are three distinct types of roles that you can use. Primitive, which [15:49.610] [15:49.610]have been around since before Cloud IAM that we know today. Predefined roles, which are curated by [15:55.050] [15:55.050]GCP and service specific. And custom roles that enable developers to specify any permission that [16:00.850] [16:00.850]they would like that is supported to fit their services needs. Let's focus on primitive roles. [16:05.710] [16:05.710]Within primitive roles, there's an owner role, which provides all of the different permissions [16:10.750] [16:10.750]that are available within the platform. An editor role, which closely resembles the owner role, [16:16.830] [16:16.830]but excludes a few different things for administrative capabilities and that just [16:20.590] [16:20.590]might not be supported yet. One thing in particular that's excluded is the ability to set project level [16:25.490] [16:25.490]role bindings. And the viewer role, which is exactly what it sounds like. It allows you to [16:29.850] [16:29.850]view resources, but not perform any state changing actions. How are these primitive roles used? Well, [16:36.030] [16:36.030]earlier we saw that when you go to Cloud IAM through the UI, these primitive roles are what [16:41.070] [16:41.070]are first suggested to users. There's another way that these primitive roles manifest themselves [16:46.350] [16:46.350]within your GCP project. And that's through default IAM role bindings. Whatever identity [16:51.590] [16:51.590]creates a project will automatically be granted a project level role binding using the owner role. [16:57.710] [16:57.710]So we know that the owner role contains all of the different permissions that are available, [17:02.390] [17:02.390]but what about the editor role? Well, the editor role contains, at the time of this talk, [17:07.630] [17:07.630]2,576 distinct permissions. [17:11.290] [17:14.710]That's a lot. This means that with the editor role, you can, for example, access all buckets, [17:20.250] [17:20.250]all databases, all VMs, and much more within your project. [17:24.410] [17:25.950]Out of these permissions that we talked about earlier, which of these roles actually enable a [17:30.490] [17:30.490]user to manage service accounts? Well, both the owner and the editor role do. While the editor [17:35.210] [17:35.210]role does not enable you to create new service accounts or apply project level role bindings [17:41.610] [17:41.610]for service accounts, it does allow you to manage service accounts in two different ways. And that's [17:47.870] [17:47.870]by creating keys or credentials for a given service account that already exists, and by [17:53.310] [17:53.310]associating a service account with a resource. So if a user can create a resource, specify a service [17:59.970] [17:59.970]account, GCP will go and check, does this user have the IAM service accounts act as permission [18:05.470] [18:05.470]on the service account that they're attempting to associate with the resource? If so, it gets [18:10.170] [18:10.170]attached, and then that user can interact with GCP services and APIs as that service account [18:15.690] [18:15.690]identity and its associated role. So how common are these IAM roles in GCP projects? There's another [18:21.990] [18:21.990]way that GCP creates automatic role bindings for identities, and this way is a little bit [18:26.850] [18:26.850]different. When you enable a specific service such as Google Compute Engine, it will go and create a [18:34.330] [18:34.330]default service account in your project and then grant that service account a project level role [18:39.690] [18:39.690]binding using the primitive editor role that we chatted about earlier. This also happens with [18:45.190] [18:45.190]another service. When a user enables the App Engine service, a service account called the App [18:49.210] [18:49.210]Spot service account is created in your project and also granted a project level editor role binding. [18:53.690] [18:53.690]Now as we discussed earlier, that editor role contains a permission that enables identities [18:58.490] [18:58.490]to associate service accounts with resources. So that means that both of these default service [19:04.290] [19:04.290]accounts, the Compute Engine and the App Spot service account, are able to associate any service [19:09.870] [19:09.870]account within its GCP project to a resource. So where are these service accounts used? The default [19:15.670] [19:15.910]compute service account is associated with GCE VMs by default. So whenever you create a VM, [19:22.750] [19:22.750]the default compute service account is automatically associated with the VM. [19:26.770] [19:28.050]There's one thing to note about service accounts and the way that they are authorized to interact [19:33.030] [19:33.030]with services from a VM. There is the role that they have as well as something called scopes. [19:38.710] [19:38.710]And scopes define, from within the VM, what services and APIs can that identity interact with [19:45.310] [19:45.310]regardless of the role binding that they have. By default for VMs, scopes are restricted to only a [19:51.250] [19:51.250]few particular services like storage. But from what we have observed, developers will commonly [19:57.050] [19:57.050]open these scopes up so that they can interact with many different services. And how are the [20:01.550] [20:01.550]App Spot service accounts used? Cloud Functions are Google's serverless offerings, which are [20:06.370] [20:06.370]similar to AWS Lambdas. They're meant to just be lightweight, ephemeral, small chunks of code that [20:11.570] [20:11.570]run fast and exit once they're done. If the App Engine service is enabled within your project, [20:17.230] [20:17.230]when a developer goes to create a Cloud Function, that default App Spot service account with the editor role [20:23.290] [20:23.290]binding will be associated with Cloud Functions by default. When you create a Cloud Function, [20:28.210] [20:28.210]you actually do not have the ability to specify to set it without an identity. You always have to [20:33.190] [20:33.190]associate an identity with the Cloud Function. So if you don't know to create a new service account [20:38.250] [20:38.250]and associate with your Cloud Function, you'll always be using the default App Spot service [20:42.630] [20:42.630]account that has the editor role binding. Cloud Functions do not have scopes and get all the [20:46.450] [20:46.450]permissions of the editor role. And coming back to the VMs, a lot of managed services leverage VMs [20:54.070] [20:54.070]as their underlying infrastructure. Things like GKE, Dataproc, Dataflow, they also get this default [21:01.070] [21:01.070]compute engine service account associated with them. In AWS, when you spin up something like [21:05.470] [21:05.470]a VM or a Lambda, it defaults to having no access. There are no roles attached. But in GCP, things [21:11.030] [21:11.390]default to having thousands of permissions. Earlier, we talked about cross-project role [21:15.850] [21:15.850]bindings, the ability to leverage service accounts in a project to get access to other service [21:21.330] [21:21.330]accounts and potentially move laterally across GCP projects. We talked about developers potentially [21:26.410] [21:26.410]not knowing what access their service accounts have. We mentioned the idea of handing off access. [21:32.090] [21:32.090]And what this realistically looks like is the developer applies a role binding with the act [21:36.190] [21:36.190]as permission, like the editor role. Let's look at what that could look like. Now it's demo time. [21:43.170] [21:48.960]To help understand how to exploit the act as permission, we built a framework called Gsploit. [21:53.740] [21:53.740]To start out, we need a base identity. There are a number of different ways to get a base identity. [21:58.400] [21:58.400]One way, for example, is you could OAuth fish a developer that has project level editor access. [22:03.160] [22:03.160]Another way is maybe a developer accidentally uploaded a service account key to GitHub or [22:07.560] [22:07.560]and that service account has a editor level role binding because the primitive roles are the first [22:12.620] [22:12.620]thing that are suggested to you on the IAM page. Or maybe the most likely, maybe the base identity [22:17.120] [22:17.120]comes from exploiting a service and because most services run with project editor by default, maybe [22:22.080] [22:22.080]that's how you get your first foothold. In any case, in this case, we've simulated this by just [22:26.160] [22:26.160]starting with the service account credential. We can see this credential here and we can see that [22:29.960] [22:29.960]it starts in the project bbs2. Next, we'll run the list operation on the Gsploit tool. We can [22:36.420] [22:36.420]see that there's no output. The reason there's no output is the list operation lists all the [22:41.360] [22:41.360]service accounts that you've managed to take control of and it doesn't count the base identity. [22:45.300] [22:45.300]We'll run the gcloud command projects list on the base identity to see what project we have access [22:49.820] [22:49.820]to. It looks like we have access to bbs2. Next, we'll try an act as exploit on this project and [22:57.160] [22:57.160]we'll target every service account in the project. Let's unpack what's happening now. [23:04.500] [23:04.500]Keep in mind the editor role has two ways to take control of all the service accounts in a project. [23:08.920] [23:08.920]The first is through creating tokens and the second is through provisioning resources. [23:12.860] [23:12.880]There's another role that we haven't talked about yet and that's the service account user role. [23:16.580] [23:16.580]This role is also pretty common and is often given to users and service accounts that are meant to [23:20.580] [23:20.580]provision resources. Because some things like cloud functions require service accounts be [23:25.000] [23:25.000]attached to them, this is the role that's usually granted to do that. So because the service account [23:30.340] [23:30.340]user role and the editor role both have the act as permission, we'll use that to take control of [23:34.800] [23:34.800]the service accounts because it casts a little bit wider of a net than the token creator permission [23:38.800] [23:38.800]which is only in the editor role. So what gsploit is actually doing under the hood here is first [23:43.100] [23:43.100]it's running the service account list operation to get a list of all the service accounts in the [23:46.860] [23:46.860]project and then it's spinning up a cloud function for each one of those service accounts. Then we [23:51.080] [23:51.080]keep a database of all the cloud functions we've harvested, each which has its own underlying [23:55.160] [23:55.160]service account attached to it. This gives us access to all of the service accounts in the [23:58.960] [23:58.960]project through the cloud functions that we spun up. We control the code that lands on the cloud [24:02.640] [24:02.640]function, so we'll just deploy code that allows us to run whatever gcloud commands we want. It [24:07.400] [24:07.400]takes about two minutes to spin up each cloud function, so we'll just speed through this really [24:11.000] [24:11.000]quick. And now when we run the list command again, we can see all the new service accounts that we [24:18.020] [24:18.020]now have access to. One of them says owner. That service account actually has an owner level role [24:22.560] [24:22.560]binding to the project that we started in. That means we now have full control of the project [24:26.040] [24:26.040]through this service account. If we wanted to, we could use this service account to add ourselves as [24:30.180] [24:30.180]owners to the project. So just to pause and think about that for a second, what that means is if you [24:34.500] [24:34.500]ever have a service account that has editor level access and another service account that has owner [24:38.940] [24:38.940]level access, the editor level service account can always privilege escalate itself to owner. [24:43.840] [24:43.940]That's the same with developers. If you grant a developer editor level access to a project but you [24:47.920] [24:47.920]happen to have a service account in that project that has owner level access, that developer can [24:51.760] [24:51.760]privilege escalate themselves to owner through that service account. Okay, let's take another [24:55.180] [24:55.180]look at those service accounts that we have. There's one here that's kind of interesting that [24:58.480] [24:58.480]says do nothing. It might not sound interesting, but keep in mind that the developer that created [25:02.900] [25:02.900]that service account in this project, although they didn't intend to give it any role bindings, [25:06.840] [25:06.840]again, the owner of the service accounts can't control the role bindings, and so it's possible [25:10.420] [25:10.420]that a different project gave it a role binding. So let's run the project list command from this [25:14.400] [25:14.400]service account that we've compromised. Something interesting here that the base project, bbs2, [25:19.140] [25:19.140]doesn't actually show up when we run this command. This is because the service account has no role [25:22.960] [25:22.960]bindings there. The developer didn't want it to have any role bindings. But bbs3, this new project [25:27.400] [25:27.400]we haven't looked at, shows up. What happened here is the developer in another project granted this [25:31.600] [25:31.600]service account and added a role binding into their project. So let's run the act as exploit [25:35.900] [25:35.900]one more time, this time from this do nothing service account into bbs3 and see how many [25:41.160] [25:41.160]service accounts we can harvest in bbs3. Again, this process takes a little while, so we're going [25:45.300] [25:45.300]to fast forward through this. And now we can see a whole bunch of new service accounts in the bbs3 [25:54.040] [25:54.040]project. Service accounts that we didn't have access to before that we were able to get strictly [25:58.320] [25:58.320]through lateral movement. There's a couple service accounts in here that look interesting. There's one [26:01.880] [26:01.880]that's labeled networking that maybe controls networking for the organization. But probably [26:05.460] [26:05.460]most interesting is this one that's labeled organizational admin. If while you're moving [26:09.520] [26:09.520]laterally through projects you happen to land on a service account that has an org level binding, [26:13.680] [26:13.680]through inheritance you get access to the entire org that way. So to recap, we started with the [26:18.480] [26:18.480]base identity in bbs2 that had act as on the project. We then used that permission to gain [26:23.420] [26:23.420]access to every service account in this project via cloud functions. One of the service accounts [26:27.680] [26:27.680]in bbs2 had an owner role, which allowed us to elevate privileges in the bbs2 project. A different [26:32.880] [26:32.880]service account had a role binding into bbs3. This allowed us to use act as to take control of all the [26:38.100] [26:38.100]service accounts in bbs3. One of the service accounts we found in bbs3 had an organizational [26:42.320] [26:42.320]role binding, and this allowed us to take control of the entire org. So we talked about when you [27:03.870] [27:03.870]provision resources, when you attach service accounts to those resources, GCP will check to [27:08.170] [27:08.170]make sure that you have the act as permission first before allowing you to provision the resource. [27:12.170] [27:12.170]Because some APIs like cloud functions require that you have identities to power them, this leads [27:16.470] [27:16.470]to a lot of people getting the act as permission. Well it turns out that some APIs will actually [27:20.630] [27:20.630]allow you to provision resources with the service account attached to them without the need for the [27:24.330] [27:24.330]act as permission. These include the Dataproc, Dataflow, and Composer APIs. Let's talk about the [27:29.950] [27:29.950]Dataproc API. The Dataproc API is Google's data processing API. Among other things, it includes [27:35.910] [27:35.910]managed Apache Hadoop and managed Apache Spark. When you spin up a Dataproc cluster, like a lot [27:41.030] [27:41.030]of things in GCP, it will default to using the default Compute Engine service account. This means [27:45.950] [27:45.950]by default your Spark jobs will run as project editor. You only need the permission Dataproc [27:50.330] [27:50.330]cluster create to spin up this cluster. You don't actually need the act as permission on the default [27:55.570] [27:55.570]service account. Because of this, anyone who has the Dataproc cluster create permission can actually [28:00.610] [28:00.610]make use of the default editor service account. Put another way, service accounts that are only [28:04.690] [28:04.690]intended to be able to spin up Dataproc clusters can privilege escalate themselves to project [28:08.790] [28:08.790]editor. And then they can make use of that act as permission to take control of the rest of the [28:13.290] [28:13.290]service accounts in the project. We built support for this into gsploit, so let's take a look and [28:17.330] [28:17.330]see what it looks like. If we take another look at the list of service accounts we've compromised, [28:21.310] [28:21.310]there's one here that says Dataproc. Let's run the gcloud projects list on this service account and [28:25.750] [28:25.750]see what projects it has access to. Looks like it has access to a bbs4. Now let's run the Dataproc [28:37.400] [28:37.400]exploit on bbs4 through the Dataproc service account. We have to spin up a whole Dataproc [28:42.440] [28:42.440]cluster to do this, and that takes a little bit of time. And then we delete it afterwards so that [28:46.420] [28:46.420]we're not filled too long. So we'll fast forward through this part, but at the end of it when we [28:50.000] [28:50.000]run gcloud list again, we can see a new service account has been added. And that's the default [28:54.580] [28:54.580]service account in the bbs4 project. We were able to get this service account by spinning up a Dataproc [28:59.540] [28:59.540]cluster in bbs4 and then running a spark job on that cluster, which enabled us to grab the [29:04.040] [29:04.040]off the instance. Because our Dataproc service account credential had the ability to spin the [29:08.740] [29:08.740]cluster up, it also controlled the scopes for that cluster. And so we just defined the scopes [29:12.820] [29:12.820]for this cluster to be wide open. Next, since we have a default editor service account credential [29:17.220] [29:17.220]at this point, we can run the act as exploit on this project. We'll again fast forward through [29:25.820] [29:25.820]this process. And finally, if we run the gcloud list one more time, we can see a whole bunch of [29:34.420] [29:34.420]service accounts that we now have access to in the bbs4 project. So just to recap, some APIs in [29:39.340] [29:39.340]GCP do not require the act as permission to spin up resources and attach the default service [29:43.880] [29:43.880]accounts to them. The identities that can spin those resources up can also control the scopes. [29:48.300] [29:48.300]So effectively, in cases like Dataproc and Dataflow and Composer, any identity that's [29:52.940] [29:52.940]granted permissions to those APIs can privilege escalate themselves to project editor. One last [29:57.520] [29:57.520]thing to note here is because this Dataproc role binding was actually cross-project, [30:01.520] [30:01.520]this allowed us to again compromise the new project bbs4. And because the default service [30:06.300] [30:06.300]accounts are in every project, this allows us to take control of all the service accounts in the [30:31.140] [30:31.140]IAM analyzer. We worked with Google on this feature, and one of the user stories actually [30:35.440] [30:35.440]came from us. We came up with a list of dangerous roles and permissions that can be used to take [30:41.780] [30:41.780]control of all the service accounts in the project. These can be seen here. We won't demo Dataflow [30:46.140] [30:46.140]and Composer, but you can think of them as working the same way as Dataproc. Using the IAM analyzer [30:50.780] [30:50.780]and the dangerous permissions that we're able to find, we're able to run a BFS search on a base [30:54.980] [30:54.980]identity without the need to spin up expensive resources exploiting services. As you can see [30:59.720] [30:59.720]from the output here, we get the same list of service accounts from the same base identity as [31:03.760] [31:03.760]we did exploiting things, except this was much faster. We didn't have to spin anything up. You [31:07.940] [31:07.940]need an org binding to use this, so this is a tool for defenders to be able to get quick answers to [31:13.100] [31:13.100]how an attacker might be able to move laterally through the organization. If we cut back to our [31:17.080] [31:17.080]mock graph, let's zoom into a seemingly innocent base identity. This service account, gwadvcl, [31:22.220] [31:22.220]was meant to only run Spark jobs in its base project 14. It only has the Dataproc editor role. [31:27.080] [31:27.080]Now we'll run a BFS search from this starting identity and highlight the path in blue. [31:30.940] [31:31.020]As you can see, this allowed us to compromise all the service accounts in the starting project [31:34.500] [31:34.500]through the default editor role on our Spark job. Most of these service accounts have benign roles [31:38.420] [31:38.420]like storage or PubSub, and most of them are not cross-project. That said, some, like this one, [31:43.960] [31:43.960]have dangerous role bindings that are cross-project, in this case the Dataflow developer role. [31:48.240] [31:48.240]These allow us to take full control of service accounts in different projects. As we start to [31:52.280] [31:52.280]zoom out, we can begin to see the whole picture. We've managed to compromise most of the service [31:56.360] [31:56.360]accounts in the org, with a few notable project exceptions that just don't have any dangerous role [32:00.400] [32:00.400]bindings to them. We're only looking at project-level bindings in this graph, so if any of [32:04.500] [32:04.500]these service accounts have org-level bindings, it's not reflected here. So we just saw some really [32:09.760] [32:09.760]interesting techniques around privilege escalation and lateral movement, but we also wanted to [32:14.380] [32:14.380]provide a way to detect this. So we fingerprinted this tooling, and we'll be releasing monitoring [32:18.980] [32:18.980]and alerting to detect the behavior of this tool in your environment. I think one of the nice things [32:23.760] [32:23.760]about doing security work in the cloud is how much help you get from your upstream provider. If we [32:28.040] [32:28.040]were doing work on a web application or a binary, we'd kind of be left to our own vices to fix the [32:32.540] [32:32.540]problems we found. But because we had a close partnership with Google, we were able to work [32:36.000] [32:36.000]with them to roll out some new features that help with the security story of some of these problems. [32:39.960] [32:40.020]One of the people that helped us do that is a security engineer named Bakunov, and we gave him [32:43.420] [32:43.420]an invitation to do a brief cameo and talk about some of the new features that we helped Google [32:47.240] [32:47.240]roll out. Hello, Black Hat. I had the privilege of working with Allison and Dylan. As a result of [32:56.680] [32:56.680]our collaboration, we at Google Cloud launched several features that help customers address [33:01.540] [33:01.540]concerns. On the prevention side of things, we now have an org policy constraint that prevents [33:07.180] [33:07.180]the editor grant to the default service accounts for Compute Engine and App Engine, and we advise [33:12.960] [33:12.960]customers to turn it on. We recommend that customers create custom service accounts with [33:18.060] [33:18.060]the necessary set of permissions on the target resources, as opposed to the broad project-level [33:23.620] [33:23.620]editor grants. Allison and Dylan briefly touched upon the investigative IAM analyzer tool. As you [33:29.880] [33:29.880]saw, we built IAM analyzer to be able to address the use case of finding resources which are [33:36.040] [33:36.040]granted access to any specific identity. The other investigative feature available is the IAM [33:41.960] [33:41.960]recommender. It uses machine learning to evaluate the permissions granted to an identity against [33:47.480] [33:47.480]actual activities performed by the identity. It's a great way to find overly generous permission [33:52.540] [33:52.540]grants and to identify the actual permissions that are needed for the workload. I spent many years [33:57.660] [33:57.660]working with all major cloud providers, and this statement is generally true for all of them. [34:02.600] [34:02.600]Starter cloud UIs, those flows are optimized for ease of onboarding. And so the fundamental [34:08.040] [34:08.040]challenge for large organizations is to make available sanely secured defaults, as well as [34:14.340] [34:14.340]preventative guardrails for their developers. Those are usually in the shape of cloud foundation [34:19.740] [34:19.740]code bases, such as infrastructures code samples that are built with security in mind, policy as [34:25.060] [34:25.060]code evaluations as part of the build and release pipelines, and preventative controls exposed by [34:30.660] [34:30.660]the cloud providers, such as org policy constraints on GCP. If we're leaving developers to the mercy [34:36.620] [34:36.620]of those starter UI flows on the cloud providers, or stack overflow code snippets, we're going to [34:42.680] [34:42.680]end up in a bad place. If you're a Google Cloud customer, and they're looking for guidance in this [34:46.960] [34:46.960]area, you should try to engage your account team so that we can help out. Alison and Dylan, on behalf [34:52.980] [34:52.980]of everyone at Google Cloud, we are very grateful for the collaboration on improving our customers' [34:58.720] [34:58.720]experience on the platform, and for the opportunity to have a cameo appearance in this talk. I'm hoping [35:03.580] [35:03.580]that Black Hat 2021 happens in person, and we can share stories in the face-to-face setting. Until [35:08.820] [35:08.820]then, please wear face masks, and don't forget to turn on two-factor authentication wherever possible. [35:14.480] [35:19.740]Thank you. Thank you, Bak. We also wanted to provide some context, and a little bit more [35:25.000] [35:25.000]insight, and let you know a few of the limitations or constraints to these tools. So one of the main [35:30.000] [35:30.000]limitations to the organization policy services, you actually have to have an organization resource, [35:34.860] [35:34.860]either through Cloud Identity or G Suite, to be able to use the organization policy service. If [35:40.040] [35:40.040]you only have projects, you won't actually be able to leverage any of the constraints that are [35:43.520] [35:43.520]available. Another limitation within the organization policy service is that constraints are not [35:47.840] [35:47.840]retroactive. So when the policy is applied, all resources will be affected by the policy or [35:54.040] [35:54.040]evaluated by the policy after it is enabled, but existing resources will not be modified to match [35:59.940] [35:59.940]or meet the requirements of a given constraint. Another thing to note is that for the new Disable [36:04.800] [36:04.800]Automatic IAM Grants for Default Service Accounts, that constraint will only enforce that when the [36:11.380] [36:11.380]APIs are initially enabled for new projects, that the default service accounts are not granted [36:16.280] [36:16.280]editor-level role bindings at the time of the APIs being enabled, but there is no enforcement [36:21.080] [36:21.080]mechanism to ensure that the default service accounts are not granted a project-level binding [36:26.420] [36:26.420]or other binding. There's also no way to restrict the cross-project bindings that we talked about [36:33.120] [36:33.120]previously, where if a user knows the name of your service account, they could still give [36:37.440] [36:37.440]the service account access to their project or project's resources. So the IAM Recommender is [36:41.820] [36:41.820]very powerful. Some of the limitations on the tool is that you have to have a 90-day period where the [36:47.660] [36:47.660]policies or behavior of a given identity are evaluated so that you can have a recommendation [36:52.800] [36:52.800]be made. And there may be more IAM bindings that you may not be aware of, like the cross-project [36:58.020] [36:58.020]bindings. The IAM Analyzer can provide a lot of context into the IAM bindings or associations for [37:03.680] [37:03.800]a given identity. And in order to use this auditing tool, you have to have administrative [37:09.000] [37:09.000]capabilities in an organization to be able to see the policies that are being analyzed. So in order [37:14.760] [37:14.760]to use this tool to the fullest extent, you need administrative capabilities in your organization, [37:19.820] [37:19.820]as well as within G Suite, if you want to expand group membership to know who has access to a given [37:26.180] [37:26.180]service account that is in a group. Earlier, we talked about how Kubernetes workloads have access [37:31.520] [37:31.520]to the underlying service account that's attached to the node. And the suggested way to harden your [37:37.820] [37:37.820]clusters is to use a service called Workload Identity. And Workload Identity allows you to [37:42.720] [37:42.720]associate a Kubernetes service account with a cloud identity. And this is how you can harden [37:48.280] [37:48.280]your cluster, as well as have a one-to-one mapping of Kubernetes service accounts to [37:52.260] [37:52.260]cloud identity service accounts. We brought up earlier to ensure default service accounts are not [37:56.940] [37:56.940]granted project-level role bindings or other role bindings after the disable automatic IM grants for [38:02.620] [38:02.860]default service accounts constraint is enabled, that you require additional tooling. And to [38:07.480] [38:07.480]demonstrate a way that you can enforce these configurations, we will be releasing a Terraform [38:13.100] [38:13.100]Enterprise Sentinel policy set that can demonstrate enforcing that default service accounts are not [38:18.460] [38:18.460]granted project-level role bindings or that roles that enable service account takeover or management [38:25.020] [38:25.020]of service accounts cannot be applied at the project level. Sentinel policies are Terraform [38:29.620] [38:29.620]Enterprise policy configuration language that enable you to either restrict or alert on specific [38:36.500] [38:36.500]configurations in your runs. So just to recap some of the key takeaways, IAM and GCP is very resource [38:42.780] [38:42.780]centric, meaning that if you are a service accounts owner, you may not be aware of the configurations [38:47.260] [38:47.260]that are associated with that service account or how it may be used across a GCP organization. This [38:52.480] [38:52.480]can lead to projects being interconnected in many different ways that you may not be able to see. [38:56.880] [38:56.880]Role bindings are often done at the project level and can grant more access than a user intends to. [39:02.380] [39:02.380]A lot of managed services and general resource provisioning operations can hand off access to [39:08.020] [39:08.020]your project's resources and IAM that you may not be aware of. This kind of access that's granted [39:13.260] [39:13.260]through resource creation can lead to privilege escalation and lateral movements within your [39:18.400] [39:18.400]environments. Another thing we covered is that default service accounts are granted administrative [39:23.300] [39:23.300]roles within your project automatically and that's with the editor role unless you're using the [39:28.100] [39:28.100]disable automatic IAM grants for default service accounts organization constraint. Reduce usage of [39:33.120] [39:33.440]default service accounts where possible by creating service specific service accounts with IAM roles [39:38.560] [39:38.560]that are scoped to your services needs. There are a few new cool tools in the platform that have been [39:43.040] [39:43.040]released to assist you in understanding who has access to your service accounts, such as the IAM [39:48.420] [39:48.420]analyzer, which can also be really helpful for understanding IAM dependencies and access within [39:53.880] [39:53.880]your environments. [39:55.320]