[00:01.760 --> 00:03.580] Hello, my name is Allison. [00:07.250 --> 00:08.770] Hi, my name is Dylan. [00:12.270 --> 00:16.250] We've been researching GCP for the last year and a half and we've come up with some effective ways [00:16.250 --> 00:20.530] that compromise most GCP organizations, all the ones we've looked at anyway. These techniques [00:20.530 --> 00:24.990] involve moving laterally with service accounts and cloud APIs. We have a lot of ground to cover, [00:24.990 --> 00:41.220] so let's get to it. I think everyone has a slightly different reaction when they first [00:41.220 --> 00:48.060] start learning about GCP IAM. But one common thing I've observed is people often compare it [00:48.060 --> 00:53.020] to AWS IAM, but there are some pretty key differences between GCP and AWS IAM. So let's [00:53.020 --> 00:58.280] talk about some of those differences. First, in AWS, we have user identities and those user [00:58.280 --> 01:03.420] identities can be exported in the form of credentials, which look something like this. [01:03.560 --> 01:09.900] In GCP, the equivalent is a service account and service accounts can also be exported [01:09.900 --> 01:13.200] in credentials that look a little bit differently. They look like this. [01:14.500 --> 01:19.180] And in both cases, we have owners of these identities. But where things get a little [01:19.180 --> 01:23.620] bit different is when you start to ask the question, what can my identity access? In [01:23.620 --> 01:28.180] the world of AWS, you basically have a policy that determines all the different things that [01:28.180 --> 01:33.460] the identity can access. And the person who can create, read, update, delete that policy [01:33.460 --> 01:38.400] is the same person who owns the identity. So here's an example of what that looks like. [01:39.000 --> 01:42.660] In GCP though, this is where things start getting really different. Instead of having [01:42.940 --> 01:48.460] a centralized policy for a given identity, we have policies for resources. So in this case, [01:48.460 --> 01:53.380] we've got a bucket and a bucket owner and that bucket has a policy. The bucket owner controls [01:53.380 --> 01:58.620] who can create, read, update, or delete that policy. And that policy applies to whatever [01:58.620 --> 02:04.420] identity the bucket owner wants it to. So here's an example of what that looks like for storage. [02:05.600 --> 02:11.360] What's interesting here is that the owner of the service account actually doesn't have any [02:11.360 --> 02:16.900] insight into this policy. They can't view it. They can't update it. They don't even know that [02:16.900 --> 02:21.880] their service account was given access to this bucket. Instead, the bucket owner controls this. [02:21.880 --> 02:26.400] And if the owner of the service account doesn't have IAM list on the bucket, then they won't [02:26.400 --> 02:32.660] be able to even see that that policy was granted. And I think this is one of the harder concepts [02:32.660 --> 02:37.660] it was for me personally to wrap my head around in GCP, is if you have an identity in GCP, like a [02:37.660 --> 02:43.080] user or a service account, as the owner of that identity, you can't actually know what that [02:43.080 --> 02:47.740] identity has access to. In fact, no one can answer the question, what does the service account have [02:47.740 --> 02:52.320] access to? Instead, resource owners control what the policy is for a given resource. So you can [02:52.320 --> 02:56.920] answer questions like, who has access to this bucket? But you can't answer questions like, [02:56.920 --> 03:00.360] how many buckets does the service account have access to? Or what buckets does the service [03:00.360 --> 03:05.620] account have access to? Those questions are unanswerable by design in GCP. Likewise, we can [03:05.620 --> 03:09.960] have other resources like BigQuery, for example. And we have an owner for the BigQuery and a policy [03:09.960 --> 03:15.180] for the BigQuery. And the owner of the BigQuery can, again, grant access to any identity they want. [03:15.180 --> 03:20.060] However, the owner of the service account has no control over this and can't see this. One thing [03:20.060 --> 03:23.880] you might be wondering is, before when I mentioned that the owner of the service account can't fully [03:23.880 --> 03:28.120] answer the question, what does the service account have access to? You might be thinking, well, if you [03:28.120 --> 03:32.960] own an organization, can't you just get the IEM policies of all the resources in the organization? [03:32.960 --> 03:36.860] Well, someone outside your organization can still grant your service account access to their [03:36.860 --> 03:41.240] resources. So when I said before, you can't actually answer the question, what does your [03:41.240 --> 03:45.000] service account have access to? It's true. You cannot get the answer to the question because [03:45.000 --> 03:49.640] people outside your organization can also grant access to their resources. We'll come back to that [03:49.640 --> 03:53.920] later and explain what an organization is and how it's structured. But for now, I figured I'd just [03:53.920 --> 03:57.800] mention that in case you were wondering. As you might imagine, not having control or visibility [03:57.800 --> 04:02.640] into what your service accounts can or can't access comes with some challenges. And one particular [04:02.640 --> 04:06.660] place I think those challenges manifest themselves is in a multi-tenant Kubernetes environment. [04:06.660 --> 04:11.280] Google has a managed Kubernetes engine that's often shorthanded GKE. If you're familiar with [04:11.280 --> 04:16.760] Kubernetes, you have a bunch of nodes, and within those nodes you have workloads. In GKE, those nodes [04:16.760 --> 04:21.600] are powered by VMs, and those VMs have service accounts attached to them. If we take a look at [04:21.600 --> 04:26.420] we can see a Kubernetes cluster here that's powered by four underlying nodes. [04:28.060 --> 04:33.140] Then, if we hop over to Compute Engine and click on VM instances, we'll see the same four instances. [04:33.200 --> 04:36.720] And if we click into one of them, we'll see that we have a service account that's attached to it. [04:36.720 --> 04:40.520] We talked a little bit about how workloads interact with the service account in our B-side [04:40.520 --> 04:45.500] San Francisco talk this year, so feel free to check that out. But in short, anything running [04:45.500 --> 04:49.460] on the node can access this identity, so it's a good idea to descope it and not give it many [04:49.460 --> 04:53.440] privileges. But let's think about what that actually means. We have a Kubernetes admin that [04:53.440 --> 04:57.600] runs the nodes and the service account powering the nodes, and then we have a bunch of developers [04:57.600 --> 05:02.260] that run the workloads on the Kubernetes engine. Those developers may be on separate teams and may [05:02.260 --> 05:06.160] be in control of separate resources from each other and from the Kubernetes admin. So you can [05:06.160 --> 05:11.800] imagine in this case we have dev1 and dev2 that have both deployed workloads, workload1 and workload2. [05:11.800 --> 05:15.900] Likewise, they both control resources. In this case, we'll say that they both own a bucket and [05:15.900 --> 05:20.360] they want their workloads to be able to access that bucket. One really easy way to do that is to [05:20.360 --> 05:25.240] just give the service account on the node access to the bucket. But the downside to doing that is [05:25.240 --> 05:29.220] instead of just giving their workload access to the bucket, they'll give every workload access to [05:29.220 --> 05:33.380] the bucket. The developers in this case control this story. They control what the service account [05:33.380 --> 05:38.240] has access to because they're the resource owners. The Kubernetes admin does not control this and [05:38.240 --> 05:43.120] cannot see that this even happened. So you can imagine in a situation where you have a small team [05:43.120 --> 05:47.920] of Kubernetes admins that are trying to maintain a cluster that want to open it up to give many [05:47.920 --> 05:52.560] developers the ability to deploy workloads to this cluster, it can be difficult to secure because they [05:52.560 --> 05:57.240] can't control what the service account attached to the nodes has access to. And some developers [05:57.240 --> 06:01.920] may want to just give that service account access to resources. Rather than using something like [06:01.920 --> 06:07.440] Kubernetes secrets or vault to pass credentials to those workloads, they may choose to just give the [06:07.440 --> 06:12.660] service account attached to the underlying node access to their resources. But when they do that, [06:12.660 --> 06:18.640] they give all workloads access to their resources, not just one workload. And so you end up in a [06:18.640 --> 06:24.460] situation where a Kubernetes admin who wants to onboard developers into their cluster inadvertently [06:24.460 --> 06:29.160] gives every new workload access to a bunch of resources, even though they can't even see that [06:29.160 --> 06:45.440] or have any control over it. Okay, now before we go any further, it's important to point out that [06:45.440 --> 06:50.200] up until this point, I've kind of implied that resources are like these standalone entities that [06:50.200 --> 06:55.220] receive single role bindings directly to them. And while resources definitely can operate that way, [06:55.220 --> 06:59.460] most of the time they're actually bundled together into groupings. And there's a role binding that [06:59.460 --> 07:03.180] goes to the entire group of stuff. So instead of getting a role directly to a bucket, you'd [07:03.180 --> 07:08.200] instead get a role to a collection of buckets. So fundamentally, there are four nodes, resource [07:08.200 --> 07:13.060] nodes within GCP. There's an organization node, which is the parent node. This is generally [07:13.060 --> 07:18.780] associated with your domain or cloud identity service. There are folders and projects. All of [07:18.780 --> 07:23.980] these different top-level nodes are mostly used for IAM. And something that's important to note [07:23.980 --> 07:29.600] within the resource hierarchy is the concept of IAM inheritance, where wherever you have an IAM [07:29.600 --> 07:35.920] binding to one of these top-level nodes, your IAM binding is also going to be inherited to all [07:35.920 --> 07:41.480] subsequent or child resources within this hierarchy. So if I have a role binding at the organization, [07:41.480 --> 07:46.040] I have access to all subsequent resources that are created within the environment. Same at the [07:46.040 --> 07:51.700] folder and project level. Generally, users mostly interact with resources at the project level. [07:51.700 --> 07:57.140] That's where services like buckets, compute, and general services that you may interact with on a [07:57.140 --> 08:02.060] day-to-day basis are. Now, let's dig into a few different IAM workflows and try to understand why [08:02.060 --> 08:07.660] project-level IAM bindings are so prevalent in GCP. If we navigate to IAM and we go to grant a [08:07.660 --> 08:11.300] member access to our resources, what we're doing here is we're actually creating a project-level [08:11.300 --> 08:15.740] role binding, which will give this member access to all resources within our project. If we wanted [08:15.740 --> 08:20.600] to create a resource-level IAM binding, we would actually need to interact with the resource-specific [08:20.600 --> 08:27.960] API and set the IAM binding there. Another IAM workflow where a user is prompted to create a [08:27.960 --> 08:32.000] project-level role binding is when you create a service account. When we create a service account, [08:32.000 --> 08:37.400] we'll actually be prompted to set a project-level IAM binding that will grant this service account [08:37.400 --> 08:41.240] access again to all resources within our project. So we can see that when you go through general [08:41.240 --> 08:46.700] IAM workflows, you are going to always be setting a project-level IAM binding. Your IAM binding is [08:46.700 --> 08:52.100] also going to be inherited to all subsequent or child resources within this project. [08:53.960 --> 08:58.800] Here we can also see something called conditionals. This is an advanced feature within the IAM service [08:58.800 --> 09:04.040] that does enable you to set more granular IAM policies, but also at the project level within [09:04.040 --> 09:08.380] your environment. So I think that's really interesting that the main IAM button in the UI [09:08.380 --> 09:12.780] actually takes you to the project-level bindings. A lot of people don't even realize that you can do [09:12.780 --> 09:16.600] resource-level bindings because you have to actually click through into the resource to [09:16.600 --> 09:20.600] see that there are IAM settings there. And a lot of people just assume that if they go to the IAM [09:20.600 --> 09:24.100] page, they get the whole IAM story. So when we backtrack a little bit to when we were talking [09:24.100 --> 09:28.100] about resource owners and service account owners, realistically what would actually happen is you [09:28.100 --> 09:32.040] would have project owners, and within those projects you would have resources and service [09:32.040 --> 09:36.180] accounts. So before when we were talking about role bindings, we mentioned this idea of Kubernetes [09:36.180 --> 09:40.580] nodes having service accounts attached to them, and developers deploying workloads to the Kubernetes [09:40.580 --> 09:45.180] cluster and granting them access to resources. But more realistically what that would look like is [09:45.180 --> 09:49.420] they would grant that node access to their entire project. Because again, IAM bindings are typically [09:49.420 --> 09:53.260] done at the project level. So instead of granting the node access to a single bucket, they would [09:53.260 --> 09:57.460] grant it access to all buckets within a given project. Now this isn't always the case. Sometimes [09:57.460 --> 10:02.020] developers do resource-level role binding, and it should also be noted that the UI isn't the only [10:02.020 --> 10:06.560] way to apply these role bindings. You can also apply them via Terraform or via CLI. It's pretty [10:06.560 --> 10:10.600] common for developers to do it at the project level, because most of their first exposure [10:10.600 --> 10:14.660] actually comes through the UI. Okay, so if we recap to the earlier section, you might remember [10:14.660 --> 10:18.400] when we talked about service account owners, we mentioned that the service account owner can't [10:18.400 --> 10:22.400] actually know what access their service accounts have. Well, it's not really the whole story. Really [10:22.400 --> 10:25.960] what we have is a project owner, and then that project owner has a bunch of service accounts in [10:25.960 --> 10:29.840] their project, and then those service accounts have access to stuff, and the project owner doesn't [10:29.840 --> 10:33.600] really have the whole story of what their service accounts have access to. And so you might imagine [10:33.600 --> 10:38.060] how that can start to become dangerous, is when I hand over access of my project to someone else, [10:38.060 --> 10:41.720] I don't really know how much access I'm handing over, because again, I don't even know how much [10:41.720 --> 10:45.880] access my service accounts have in my project. So if I hand that over to someone else, I'm handing [10:45.880 --> 10:50.480] over an unknown amount of access. So what does an example of this look like? Well, we have a project [10:50.480 --> 10:55.240] owner that knows the email address of a different project service account. They can then give that [10:55.240 --> 11:00.420] service account access to their project through an IAM binding, and then that service account has [11:00.420 --> 11:04.860] access to their project's resources. Now, one question you might be asking yourself is why [11:04.860 --> 11:08.000] you would even need a cross-project role binding. Well, when you think about it, when you have two [11:08.000 --> 11:12.220] projects, they're not particularly useful unless they have some way to talk to each other. And one [11:12.220 --> 11:16.980] common way to do that is through Google Cloud services like PubSub or Storage. And so it makes [11:16.980 --> 11:21.000] sense that we would want to share some resources between projects. But again, where things sort of [11:21.000 --> 11:24.960] start to become more dangerous is if we share service account access between projects, because [11:24.960 --> 11:28.420] project owners don't really know what access their service accounts have. So we're missing [11:28.420 --> 11:32.640] out on some important existential questions like how interconnected our projects actually are. [11:32.640 --> 11:36.400] Because again, we can answer questions like who has access to a resource, but we really can't [11:36.400 --> 11:40.420] answer questions like what does a service account have access to. So we came up with a solution that [11:40.420 --> 11:45.100] let us sort of answer this question within our own organization. We can't actually get the complete [11:45.100 --> 11:48.900] answer to the question, again, because of these things called cross-organizational bindings. But [11:48.900 --> 11:53.220] at least within our org, we can get the answer for cross-project bindings by just introspecting [11:53.220 --> 11:57.260] every individual project that we own and pulling all the IAM policies. We then took all that [11:57.260 --> 12:01.980] information and we made a graph out of it. We actually went over to GitHub and we crawled a [12:01.980 --> 12:05.960] bunch of IAM policies that people either intentionally or accidentally committed. [12:05.960 --> 12:25.450] And we took all those policies and we kind of compiled this pseudo-org generator. [12:25.450 --> 12:28.670] So let's take a look at one of the graphs we ended up with. As you can see, there are two [12:28.670 --> 12:33.090] different colored nodes. There are red nodes and there are blue nodes. What the red nodes represent [12:33.090 --> 12:37.910] are projects and what the blue nodes represent are service accounts. Those are the only two things [12:37.910 --> 12:42.170] we're graphing here. Then there are two different types of edges. There are contains edges and there [12:42.170 --> 12:47.070] are bindings edges. A contains edge is basically when a service account lives in a given project [12:47.070 --> 12:51.210] and the binding edge is when that service account has a role binding to that project. [12:51.470 --> 12:55.710] Most of the time, a service account has a role binding to the project that it's contained in. [12:55.710 --> 12:58.950] But every once in a while, we'll end up with a service account that has a role binding to a [12:58.950 --> 13:03.050] different project across project binding. So in this case, we have a service account that's [13:03.050 --> 13:08.950] contained in project number 67 and it has a role binding into project number 57. If we zoom out, [13:08.950 --> 13:12.410] we get the view of the entire organization. We have all the different projects here in a map [13:12.410 --> 13:16.290] and we have all the relationships of the service accounts within the org that are contained within [13:16.290 --> 13:19.810] these projects. Now we're not looking at organization-level bindings in this graph [13:19.810 --> 13:23.950] or resource-level bindings. We're only looking at the project-level bindings and we're mapping [13:23.950 --> 13:28.190] these relationships. So because we based this off of real IAM policies that we're able to pull up [13:28.190 --> 13:33.150] GitHub, we can say with some level of confidence that this is an accurate representation of what a [13:33.150 --> 13:37.030] sufficiently large company might look like. We can also say with a little bit of personal experience [13:37.030 --> 13:41.030] based on the large companies that we've looked at, this is an accurate representation of what a [13:41.030 --> 13:44.630] sufficiently large company looks like. Now again, most of the time a developer is just going to be [13:44.630 --> 13:48.310] the owner of a single project and they won't really see the whole picture. All they'll see [13:48.310 --> 13:52.630] is sort of tunnel vision for the one project that they have control over. And so they'll see the [13:52.630 --> 13:56.290] service accounts that belong to their project and they'll see what has role bindings into their [13:56.290 --> 14:00.690] project, but they won't know where their service accounts have role bindings out of their project. [14:00.690 --> 14:04.830] And so when they hand off access to their project to another developer, they won't actually know how [14:04.830 --> 14:08.830] many additional projects they're handing off. And they themselves may not have a good understanding [14:08.830 --> 14:12.470] of how many projects that they have access to through their service accounts. One other [14:12.470 --> 14:16.370] interesting thing that we found here is that there are certain projects that have seemingly [14:17.030 --> 14:20.830] disproportionate large numbers of role bindings and resources in them. We found this was a pretty [14:20.830 --> 14:25.390] consistent story at multiple companies that we looked at. And so in the center here you can see [14:25.390 --> 14:29.250] there are a bunch of projects that have really tightly bound role bindings. And then at the [14:29.250 --> 14:33.310] bottom we have smaller projects that only have one or two service accounts that may not be connected [14:33.310 --> 14:37.170] to the broader web. And then we have a bunch of projects that have no service accounts. What we [14:37.170 --> 14:41.570] found is that most of the sensitive data or sensitive workloads are in the interconnected [14:41.570 --> 14:45.530] projects. That's where most of the development is happening. And so it makes sense that it would have [14:45.530 --> 14:50.110] most of the IAM role bindings connecting it with other projects. So when we first started building [14:50.110 --> 14:54.830] these graphs, we were thinking to ourselves, oh gosh, these projects are supposed to be isolated [14:54.830 --> 14:59.550] boundaries. Are they really this interconnected? And the more we dug into it and the more we thought [14:59.550 --> 15:04.010] about how project owners can't actually see the role bindings their service accounts have been [15:04.010 --> 15:08.330] granted, we realized this is a problem that's really remained invisible. And it really takes [15:08.330 --> 15:12.790] this org-level introspection to even see it. Now there's one final piece here that we haven't talked [15:12.790 --> 15:16.210] about yet, and that's when a developer actually does hand over access to their project to another [15:16.210 --> 15:20.730] developer. Pieces of that might be okay. It might be okay to share storage, for example, but things [15:20.730 --> 15:24.350] start to become really dangerous if you share access to service accounts. Because again, that's where [15:24.350 --> 15:28.210] the problem is hidden, where the owner of the project doesn't actually know what access their [15:28.210 --> 15:31.970] service accounts have. And so if we share that unknown access, that's where things can start to [15:31.970 --> 15:36.490] go astray. So we'll come back to what roles and permissions are actually required to hand over [15:36.490 --> 15:40.610] that level of access. And we'll talk about how common that is. So far we've talked a lot about [15:40.610 --> 15:45.050] roles and mentioned that some are more dangerous than others. So now let's dig into that a little [15:45.050 --> 15:49.610] bit more. Within GCP, there are three distinct types of roles that you can use. Primitive, which [15:49.610 --> 15:55.050] have been around since before Cloud IAM that we know today. Predefined roles, which are curated by [15:55.050 --> 16:00.850] GCP and service specific. And custom roles that enable developers to specify any permission that [16:00.850 --> 16:05.710] they would like that is supported to fit their services needs. Let's focus on primitive roles. [16:05.710 --> 16:10.750] Within primitive roles, there's an owner role, which provides all of the different permissions [16:10.750 --> 16:16.830] that are available within the platform. An editor role, which closely resembles the owner role, [16:16.830 --> 16:20.590] but excludes a few different things for administrative capabilities and that just [16:20.590 --> 16:25.490] might not be supported yet. One thing in particular that's excluded is the ability to set project level [16:25.490 --> 16:29.850] role bindings. And the viewer role, which is exactly what it sounds like. It allows you to [16:29.850 --> 16:36.030] view resources, but not perform any state changing actions. How are these primitive roles used? Well, [16:36.030 --> 16:41.070] earlier we saw that when you go to Cloud IAM through the UI, these primitive roles are what [16:41.070 --> 16:46.350] are first suggested to users. There's another way that these primitive roles manifest themselves [16:46.350 --> 16:51.590] within your GCP project. And that's through default IAM role bindings. Whatever identity [16:51.590 --> 16:57.710] creates a project will automatically be granted a project level role binding using the owner role. [16:57.710 --> 17:02.390] So we know that the owner role contains all of the different permissions that are available, [17:02.390 --> 17:07.630] but what about the editor role? Well, the editor role contains, at the time of this talk, [17:07.630 --> 17:11.290] 2,576 distinct permissions. [17:14.710 --> 17:20.250] That's a lot. This means that with the editor role, you can, for example, access all buckets, [17:20.250 --> 17:24.410] all databases, all VMs, and much more within your project. [17:25.950 --> 17:30.490] Out of these permissions that we talked about earlier, which of these roles actually enable a [17:30.490 --> 17:35.210] user to manage service accounts? Well, both the owner and the editor role do. While the editor [17:35.210 --> 17:41.610] role does not enable you to create new service accounts or apply project level role bindings [17:41.610 --> 17:47.870] for service accounts, it does allow you to manage service accounts in two different ways. And that's [17:47.870 --> 17:53.310] by creating keys or credentials for a given service account that already exists, and by [17:53.310 --> 17:59.970] associating a service account with a resource. So if a user can create a resource, specify a service [17:59.970 --> 18:05.470] account, GCP will go and check, does this user have the IAM service accounts act as permission [18:05.470 --> 18:10.170] on the service account that they're attempting to associate with the resource? If so, it gets [18:10.170 --> 18:15.690] attached, and then that user can interact with GCP services and APIs as that service account [18:15.690 --> 18:21.990] identity and its associated role. So how common are these IAM roles in GCP projects? There's another [18:21.990 --> 18:26.850] way that GCP creates automatic role bindings for identities, and this way is a little bit [18:26.850 --> 18:34.330] different. When you enable a specific service such as Google Compute Engine, it will go and create a [18:34.330 --> 18:39.690] default service account in your project and then grant that service account a project level role [18:39.690 --> 18:45.190] binding using the primitive editor role that we chatted about earlier. This also happens with [18:45.190 --> 18:49.210] another service. When a user enables the App Engine service, a service account called the App [18:49.210 --> 18:53.690] Spot service account is created in your project and also granted a project level editor role binding. [18:53.690 --> 18:58.490] Now as we discussed earlier, that editor role contains a permission that enables identities [18:58.490 --> 19:04.290] to associate service accounts with resources. So that means that both of these default service [19:04.290 --> 19:09.870] accounts, the Compute Engine and the App Spot service account, are able to associate any service [19:09.870 --> 19:15.670] account within its GCP project to a resource. So where are these service accounts used? The default [19:15.910 --> 19:22.750] compute service account is associated with GCE VMs by default. So whenever you create a VM, [19:22.750 --> 19:26.770] the default compute service account is automatically associated with the VM. [19:28.050 --> 19:33.030] There's one thing to note about service accounts and the way that they are authorized to interact [19:33.030 --> 19:38.710] with services from a VM. There is the role that they have as well as something called scopes. [19:38.710 --> 19:45.310] And scopes define, from within the VM, what services and APIs can that identity interact with [19:45.310 --> 19:51.250] regardless of the role binding that they have. By default for VMs, scopes are restricted to only a [19:51.250 --> 19:57.050] few particular services like storage. But from what we have observed, developers will commonly [19:57.050 --> 20:01.550] open these scopes up so that they can interact with many different services. And how are the [20:01.550 --> 20:06.370] App Spot service accounts used? Cloud Functions are Google's serverless offerings, which are [20:06.370 --> 20:11.570] similar to AWS Lambdas. They're meant to just be lightweight, ephemeral, small chunks of code that [20:11.570 --> 20:17.230] run fast and exit once they're done. If the App Engine service is enabled within your project, [20:17.230 --> 20:23.290] when a developer goes to create a Cloud Function, that default App Spot service account with the editor role [20:23.290 --> 20:28.210] binding will be associated with Cloud Functions by default. When you create a Cloud Function, [20:28.210 --> 20:33.190] you actually do not have the ability to specify to set it without an identity. You always have to [20:33.190 --> 20:38.250] associate an identity with the Cloud Function. So if you don't know to create a new service account [20:38.250 --> 20:42.630] and associate with your Cloud Function, you'll always be using the default App Spot service [20:42.630 --> 20:46.450] account that has the editor role binding. Cloud Functions do not have scopes and get all the [20:46.450 --> 20:54.070] permissions of the editor role. And coming back to the VMs, a lot of managed services leverage VMs [20:54.070 --> 21:01.070] as their underlying infrastructure. Things like GKE, Dataproc, Dataflow, they also get this default [21:01.070 --> 21:05.470] compute engine service account associated with them. In AWS, when you spin up something like [21:05.470 --> 21:11.030] a VM or a Lambda, it defaults to having no access. There are no roles attached. But in GCP, things [21:11.390 --> 21:15.850] default to having thousands of permissions. Earlier, we talked about cross-project role [21:15.850 --> 21:21.330] bindings, the ability to leverage service accounts in a project to get access to other service [21:21.330 --> 21:26.410] accounts and potentially move laterally across GCP projects. We talked about developers potentially [21:26.410 --> 21:32.090] not knowing what access their service accounts have. We mentioned the idea of handing off access. [21:32.090 --> 21:36.190] And what this realistically looks like is the developer applies a role binding with the act [21:36.190 --> 21:43.170] as permission, like the editor role. Let's look at what that could look like. Now it's demo time. [21:48.960 --> 21:53.740] To help understand how to exploit the act as permission, we built a framework called Gsploit. [21:53.740 --> 21:58.400] To start out, we need a base identity. There are a number of different ways to get a base identity. [21:58.400 --> 22:03.160] One way, for example, is you could OAuth fish a developer that has project level editor access. [22:03.160 --> 22:07.560] Another way is maybe a developer accidentally uploaded a service account key to GitHub or [22:07.560 --> 22:12.620] and that service account has a editor level role binding because the primitive roles are the first [22:12.620 --> 22:17.120] thing that are suggested to you on the IAM page. Or maybe the most likely, maybe the base identity [22:17.120 --> 22:22.080] comes from exploiting a service and because most services run with project editor by default, maybe [22:22.080 --> 22:26.160] that's how you get your first foothold. In any case, in this case, we've simulated this by just [22:26.160 --> 22:29.960] starting with the service account credential. We can see this credential here and we can see that [22:29.960 --> 22:36.420] it starts in the project bbs2. Next, we'll run the list operation on the Gsploit tool. We can [22:36.420 --> 22:41.360] see that there's no output. The reason there's no output is the list operation lists all the [22:41.360 --> 22:45.300] service accounts that you've managed to take control of and it doesn't count the base identity. [22:45.300 --> 22:49.820] We'll run the gcloud command projects list on the base identity to see what project we have access [22:49.820 --> 22:57.160] to. It looks like we have access to bbs2. Next, we'll try an act as exploit on this project and [22:57.160 --> 23:04.500] we'll target every service account in the project. Let's unpack what's happening now. [23:04.500 --> 23:08.920] Keep in mind the editor role has two ways to take control of all the service accounts in a project. [23:08.920 --> 23:12.860] The first is through creating tokens and the second is through provisioning resources. [23:12.880 --> 23:16.580] There's another role that we haven't talked about yet and that's the service account user role. [23:16.580 --> 23:20.580] This role is also pretty common and is often given to users and service accounts that are meant to [23:20.580 --> 23:25.000] provision resources. Because some things like cloud functions require service accounts be [23:25.000 --> 23:30.340] attached to them, this is the role that's usually granted to do that. So because the service account [23:30.340 --> 23:34.800] user role and the editor role both have the act as permission, we'll use that to take control of [23:34.800 --> 23:38.800] the service accounts because it casts a little bit wider of a net than the token creator permission [23:38.800 --> 23:43.100] which is only in the editor role. So what gsploit is actually doing under the hood here is first [23:43.100 --> 23:46.860] it's running the service account list operation to get a list of all the service accounts in the [23:46.860 --> 23:51.080] project and then it's spinning up a cloud function for each one of those service accounts. Then we [23:51.080 --> 23:55.160] keep a database of all the cloud functions we've harvested, each which has its own underlying [23:55.160 --> 23:58.960] service account attached to it. This gives us access to all of the service accounts in the [23:58.960 --> 24:02.640] project through the cloud functions that we spun up. We control the code that lands on the cloud [24:02.640 --> 24:07.400] function, so we'll just deploy code that allows us to run whatever gcloud commands we want. It [24:07.400 --> 24:11.000] takes about two minutes to spin up each cloud function, so we'll just speed through this really [24:11.000 --> 24:18.020] quick. And now when we run the list command again, we can see all the new service accounts that we [24:18.020 --> 24:22.560] now have access to. One of them says owner. That service account actually has an owner level role [24:22.560 --> 24:26.040] binding to the project that we started in. That means we now have full control of the project [24:26.040 --> 24:30.180] through this service account. If we wanted to, we could use this service account to add ourselves as [24:30.180 --> 24:34.500] owners to the project. So just to pause and think about that for a second, what that means is if you [24:34.500 --> 24:38.940] ever have a service account that has editor level access and another service account that has owner [24:38.940 --> 24:43.840] level access, the editor level service account can always privilege escalate itself to owner. [24:43.940 --> 24:47.920] That's the same with developers. If you grant a developer editor level access to a project but you [24:47.920 --> 24:51.760] happen to have a service account in that project that has owner level access, that developer can [24:51.760 --> 24:55.180] privilege escalate themselves to owner through that service account. Okay, let's take another [24:55.180 --> 24:58.480] look at those service accounts that we have. There's one here that's kind of interesting that [24:58.480 --> 25:02.900] says do nothing. It might not sound interesting, but keep in mind that the developer that created [25:02.900 --> 25:06.840] that service account in this project, although they didn't intend to give it any role bindings, [25:06.840 --> 25:10.420] again, the owner of the service accounts can't control the role bindings, and so it's possible [25:10.420 --> 25:14.400] that a different project gave it a role binding. So let's run the project list command from this [25:14.400 --> 25:19.140] service account that we've compromised. Something interesting here that the base project, bbs2, [25:19.140 --> 25:22.960] doesn't actually show up when we run this command. This is because the service account has no role [25:22.960 --> 25:27.400] bindings there. The developer didn't want it to have any role bindings. But bbs3, this new project [25:27.400 --> 25:31.600] we haven't looked at, shows up. What happened here is the developer in another project granted this [25:31.600 --> 25:35.900] service account and added a role binding into their project. So let's run the act as exploit [25:35.900 --> 25:41.160] one more time, this time from this do nothing service account into bbs3 and see how many [25:41.160 --> 25:45.300] service accounts we can harvest in bbs3. Again, this process takes a little while, so we're going [25:45.300 --> 25:54.040] to fast forward through this. And now we can see a whole bunch of new service accounts in the bbs3 [25:54.040 --> 25:58.320] project. Service accounts that we didn't have access to before that we were able to get strictly [25:58.320 --> 26:01.880] through lateral movement. There's a couple service accounts in here that look interesting. There's one [26:01.880 --> 26:05.460] that's labeled networking that maybe controls networking for the organization. But probably [26:05.460 --> 26:09.520] most interesting is this one that's labeled organizational admin. If while you're moving [26:09.520 --> 26:13.680] laterally through projects you happen to land on a service account that has an org level binding, [26:13.680 --> 26:18.480] through inheritance you get access to the entire org that way. So to recap, we started with the [26:18.480 --> 26:23.420] base identity in bbs2 that had act as on the project. We then used that permission to gain [26:23.420 --> 26:27.680] access to every service account in this project via cloud functions. One of the service accounts [26:27.680 --> 26:32.880] in bbs2 had an owner role, which allowed us to elevate privileges in the bbs2 project. A different [26:32.880 --> 26:38.100] service account had a role binding into bbs3. This allowed us to use act as to take control of all the [26:38.100 --> 26:42.320] service accounts in bbs3. One of the service accounts we found in bbs3 had an organizational [26:42.320 --> 27:03.870] role binding, and this allowed us to take control of the entire org. So we talked about when you [27:03.870 --> 27:08.170] provision resources, when you attach service accounts to those resources, GCP will check to [27:08.170 --> 27:12.170] make sure that you have the act as permission first before allowing you to provision the resource. [27:12.170 --> 27:16.470] Because some APIs like cloud functions require that you have identities to power them, this leads [27:16.470 --> 27:20.630] to a lot of people getting the act as permission. Well it turns out that some APIs will actually [27:20.630 --> 27:24.330] allow you to provision resources with the service account attached to them without the need for the [27:24.330 --> 27:29.950] act as permission. These include the Dataproc, Dataflow, and Composer APIs. Let's talk about the [27:29.950 --> 27:35.910] Dataproc API. The Dataproc API is Google's data processing API. Among other things, it includes [27:35.910 --> 27:41.030] managed Apache Hadoop and managed Apache Spark. When you spin up a Dataproc cluster, like a lot [27:41.030 --> 27:45.950] of things in GCP, it will default to using the default Compute Engine service account. This means [27:45.950 --> 27:50.330] by default your Spark jobs will run as project editor. You only need the permission Dataproc [27:50.330 --> 27:55.570] cluster create to spin up this cluster. You don't actually need the act as permission on the default [27:55.570 --> 28:00.610] service account. Because of this, anyone who has the Dataproc cluster create permission can actually [28:00.610 --> 28:04.690] make use of the default editor service account. Put another way, service accounts that are only [28:04.690 --> 28:08.790] intended to be able to spin up Dataproc clusters can privilege escalate themselves to project [28:08.790 --> 28:13.290] editor. And then they can make use of that act as permission to take control of the rest of the [28:13.290 --> 28:17.330] service accounts in the project. We built support for this into gsploit, so let's take a look and [28:17.330 --> 28:21.310] see what it looks like. If we take another look at the list of service accounts we've compromised, [28:21.310 --> 28:25.750] there's one here that says Dataproc. Let's run the gcloud projects list on this service account and [28:25.750 --> 28:37.400] see what projects it has access to. Looks like it has access to a bbs4. Now let's run the Dataproc [28:37.400 --> 28:42.440] exploit on bbs4 through the Dataproc service account. We have to spin up a whole Dataproc [28:42.440 --> 28:46.420] cluster to do this, and that takes a little bit of time. And then we delete it afterwards so that [28:46.420 --> 28:50.000] we're not filled too long. So we'll fast forward through this part, but at the end of it when we [28:50.000 --> 28:54.580] run gcloud list again, we can see a new service account has been added. And that's the default [28:54.580 --> 28:59.540] service account in the bbs4 project. We were able to get this service account by spinning up a Dataproc [28:59.540 --> 29:04.040] cluster in bbs4 and then running a spark job on that cluster, which enabled us to grab the [29:04.040 --> 29:08.740] off the instance. Because our Dataproc service account credential had the ability to spin the [29:08.740 --> 29:12.820] cluster up, it also controlled the scopes for that cluster. And so we just defined the scopes [29:12.820 --> 29:17.220] for this cluster to be wide open. Next, since we have a default editor service account credential [29:17.220 --> 29:25.820] at this point, we can run the act as exploit on this project. We'll again fast forward through [29:25.820 --> 29:34.420] this process. And finally, if we run the gcloud list one more time, we can see a whole bunch of [29:34.420 --> 29:39.340] service accounts that we now have access to in the bbs4 project. So just to recap, some APIs in [29:39.340 --> 29:43.880] GCP do not require the act as permission to spin up resources and attach the default service [29:43.880 --> 29:48.300] accounts to them. The identities that can spin those resources up can also control the scopes. [29:48.300 --> 29:52.940] So effectively, in cases like Dataproc and Dataflow and Composer, any identity that's [29:52.940 --> 29:57.520] granted permissions to those APIs can privilege escalate themselves to project editor. One last [29:57.520 --> 30:01.520] thing to note here is because this Dataproc role binding was actually cross-project, [30:01.520 --> 30:06.300] this allowed us to again compromise the new project bbs4. And because the default service [30:06.300 --> 30:31.140] accounts are in every project, this allows us to take control of all the service accounts in the [30:31.140 --> 30:35.440] IAM analyzer. We worked with Google on this feature, and one of the user stories actually [30:35.440 --> 30:41.780] came from us. We came up with a list of dangerous roles and permissions that can be used to take [30:41.780 --> 30:46.140] control of all the service accounts in the project. These can be seen here. We won't demo Dataflow [30:46.140 --> 30:50.780] and Composer, but you can think of them as working the same way as Dataproc. Using the IAM analyzer [30:50.780 --> 30:54.980] and the dangerous permissions that we're able to find, we're able to run a BFS search on a base [30:54.980 --> 30:59.720] identity without the need to spin up expensive resources exploiting services. As you can see [30:59.720 --> 31:03.760] from the output here, we get the same list of service accounts from the same base identity as [31:03.760 --> 31:07.940] we did exploiting things, except this was much faster. We didn't have to spin anything up. You [31:07.940 --> 31:13.100] need an org binding to use this, so this is a tool for defenders to be able to get quick answers to [31:13.100 --> 31:17.080] how an attacker might be able to move laterally through the organization. If we cut back to our [31:17.080 --> 31:22.220] mock graph, let's zoom into a seemingly innocent base identity. This service account, gwadvcl, [31:22.220 --> 31:27.080] was meant to only run Spark jobs in its base project 14. It only has the Dataproc editor role. [31:27.080 --> 31:30.940] Now we'll run a BFS search from this starting identity and highlight the path in blue. [31:31.020 --> 31:34.500] As you can see, this allowed us to compromise all the service accounts in the starting project [31:34.500 --> 31:38.420] through the default editor role on our Spark job. Most of these service accounts have benign roles [31:38.420 --> 31:43.960] like storage or PubSub, and most of them are not cross-project. That said, some, like this one, [31:43.960 --> 31:48.240] have dangerous role bindings that are cross-project, in this case the Dataflow developer role. [31:48.240 --> 31:52.280] These allow us to take full control of service accounts in different projects. As we start to [31:52.280 --> 31:56.360] zoom out, we can begin to see the whole picture. We've managed to compromise most of the service [31:56.360 --> 32:00.400] accounts in the org, with a few notable project exceptions that just don't have any dangerous role [32:00.400 --> 32:04.500] bindings to them. We're only looking at project-level bindings in this graph, so if any of [32:04.500 --> 32:09.760] these service accounts have org-level bindings, it's not reflected here. So we just saw some really [32:09.760 --> 32:14.380] interesting techniques around privilege escalation and lateral movement, but we also wanted to [32:14.380 --> 32:18.980] provide a way to detect this. So we fingerprinted this tooling, and we'll be releasing monitoring [32:18.980 --> 32:23.760] and alerting to detect the behavior of this tool in your environment. I think one of the nice things [32:23.760 --> 32:28.040] about doing security work in the cloud is how much help you get from your upstream provider. If we [32:28.040 --> 32:32.540] were doing work on a web application or a binary, we'd kind of be left to our own vices to fix the [32:32.540 --> 32:36.000] problems we found. But because we had a close partnership with Google, we were able to work [32:36.000 --> 32:39.960] with them to roll out some new features that help with the security story of some of these problems. [32:40.020 --> 32:43.420] One of the people that helped us do that is a security engineer named Bakunov, and we gave him [32:43.420 --> 32:47.240] an invitation to do a brief cameo and talk about some of the new features that we helped Google [32:47.240 --> 32:56.680] roll out. Hello, Black Hat. I had the privilege of working with Allison and Dylan. As a result of [32:56.680 --> 33:01.540] our collaboration, we at Google Cloud launched several features that help customers address [33:01.540 --> 33:07.180] concerns. On the prevention side of things, we now have an org policy constraint that prevents [33:07.180 --> 33:12.960] the editor grant to the default service accounts for Compute Engine and App Engine, and we advise [33:12.960 --> 33:18.060] customers to turn it on. We recommend that customers create custom service accounts with [33:18.060 --> 33:23.620] the necessary set of permissions on the target resources, as opposed to the broad project-level [33:23.620 --> 33:29.880] editor grants. Allison and Dylan briefly touched upon the investigative IAM analyzer tool. As you [33:29.880 --> 33:36.040] saw, we built IAM analyzer to be able to address the use case of finding resources which are [33:36.040 --> 33:41.960] granted access to any specific identity. The other investigative feature available is the IAM [33:41.960 --> 33:47.480] recommender. It uses machine learning to evaluate the permissions granted to an identity against [33:47.480 --> 33:52.540] actual activities performed by the identity. It's a great way to find overly generous permission [33:52.540 --> 33:57.660] grants and to identify the actual permissions that are needed for the workload. I spent many years [33:57.660 --> 34:02.600] working with all major cloud providers, and this statement is generally true for all of them. [34:02.600 --> 34:08.040] Starter cloud UIs, those flows are optimized for ease of onboarding. And so the fundamental [34:08.040 --> 34:14.340] challenge for large organizations is to make available sanely secured defaults, as well as [34:14.340 --> 34:19.740] preventative guardrails for their developers. Those are usually in the shape of cloud foundation [34:19.740 --> 34:25.060] code bases, such as infrastructures code samples that are built with security in mind, policy as [34:25.060 --> 34:30.660] code evaluations as part of the build and release pipelines, and preventative controls exposed by [34:30.660 --> 34:36.620] the cloud providers, such as org policy constraints on GCP. If we're leaving developers to the mercy [34:36.620 --> 34:42.680] of those starter UI flows on the cloud providers, or stack overflow code snippets, we're going to [34:42.680 --> 34:46.960] end up in a bad place. If you're a Google Cloud customer, and they're looking for guidance in this [34:46.960 --> 34:52.980] area, you should try to engage your account team so that we can help out. Alison and Dylan, on behalf [34:52.980 --> 34:58.720] of everyone at Google Cloud, we are very grateful for the collaboration on improving our customers' [34:58.720 --> 35:03.580] experience on the platform, and for the opportunity to have a cameo appearance in this talk. I'm hoping [35:03.580 --> 35:08.820] that Black Hat 2021 happens in person, and we can share stories in the face-to-face setting. Until [35:08.820 --> 35:14.480] then, please wear face masks, and don't forget to turn on two-factor authentication wherever possible. [35:19.740 --> 35:25.000] Thank you. Thank you, Bak. We also wanted to provide some context, and a little bit more [35:25.000 --> 35:30.000] insight, and let you know a few of the limitations or constraints to these tools. So one of the main [35:30.000 --> 35:34.860] limitations to the organization policy services, you actually have to have an organization resource, [35:34.860 --> 35:40.040] either through Cloud Identity or G Suite, to be able to use the organization policy service. If [35:40.040 --> 35:43.520] you only have projects, you won't actually be able to leverage any of the constraints that are [35:43.520 --> 35:47.840] available. Another limitation within the organization policy service is that constraints are not [35:47.840 --> 35:54.040] retroactive. So when the policy is applied, all resources will be affected by the policy or [35:54.040 --> 35:59.940] evaluated by the policy after it is enabled, but existing resources will not be modified to match [35:59.940 --> 36:04.800] or meet the requirements of a given constraint. Another thing to note is that for the new Disable [36:04.800 --> 36:11.380] Automatic IAM Grants for Default Service Accounts, that constraint will only enforce that when the [36:11.380 --> 36:16.280] APIs are initially enabled for new projects, that the default service accounts are not granted [36:16.280 --> 36:21.080] editor-level role bindings at the time of the APIs being enabled, but there is no enforcement [36:21.080 --> 36:26.420] mechanism to ensure that the default service accounts are not granted a project-level binding [36:26.420 --> 36:33.120] or other binding. There's also no way to restrict the cross-project bindings that we talked about [36:33.120 --> 36:37.440] previously, where if a user knows the name of your service account, they could still give [36:37.440 --> 36:41.820] the service account access to their project or project's resources. So the IAM Recommender is [36:41.820 --> 36:47.660] very powerful. Some of the limitations on the tool is that you have to have a 90-day period where the [36:47.660 --> 36:52.800] policies or behavior of a given identity are evaluated so that you can have a recommendation [36:52.800 --> 36:58.020] be made. And there may be more IAM bindings that you may not be aware of, like the cross-project [36:58.020 --> 37:03.680] bindings. The IAM Analyzer can provide a lot of context into the IAM bindings or associations for [37:03.800 --> 37:09.000] a given identity. And in order to use this auditing tool, you have to have administrative [37:09.000 --> 37:14.760] capabilities in an organization to be able to see the policies that are being analyzed. So in order [37:14.760 --> 37:19.820] to use this tool to the fullest extent, you need administrative capabilities in your organization, [37:19.820 --> 37:26.180] as well as within G Suite, if you want to expand group membership to know who has access to a given [37:26.180 --> 37:31.520] service account that is in a group. Earlier, we talked about how Kubernetes workloads have access [37:31.520 --> 37:37.820] to the underlying service account that's attached to the node. And the suggested way to harden your [37:37.820 --> 37:42.720] clusters is to use a service called Workload Identity. And Workload Identity allows you to [37:42.720 --> 37:48.280] associate a Kubernetes service account with a cloud identity. And this is how you can harden [37:48.280 --> 37:52.260] your cluster, as well as have a one-to-one mapping of Kubernetes service accounts to [37:52.260 --> 37:56.940] cloud identity service accounts. We brought up earlier to ensure default service accounts are not [37:56.940 --> 38:02.620] granted project-level role bindings or other role bindings after the disable automatic IM grants for [38:02.860 --> 38:07.480] default service accounts constraint is enabled, that you require additional tooling. And to [38:07.480 --> 38:13.100] demonstrate a way that you can enforce these configurations, we will be releasing a Terraform [38:13.100 --> 38:18.460] Enterprise Sentinel policy set that can demonstrate enforcing that default service accounts are not [38:18.460 --> 38:25.020] granted project-level role bindings or that roles that enable service account takeover or management [38:25.020 --> 38:29.620] of service accounts cannot be applied at the project level. Sentinel policies are Terraform [38:29.620 --> 38:36.500] Enterprise policy configuration language that enable you to either restrict or alert on specific [38:36.500 --> 38:42.780] configurations in your runs. So just to recap some of the key takeaways, IAM and GCP is very resource [38:42.780 --> 38:47.260] centric, meaning that if you are a service accounts owner, you may not be aware of the configurations [38:47.260 --> 38:52.480] that are associated with that service account or how it may be used across a GCP organization. This [38:52.480 --> 38:56.880] can lead to projects being interconnected in many different ways that you may not be able to see. [38:56.880 --> 39:02.380] Role bindings are often done at the project level and can grant more access than a user intends to. [39:02.380 --> 39:08.020] A lot of managed services and general resource provisioning operations can hand off access to [39:08.020 --> 39:13.260] your project's resources and IAM that you may not be aware of. This kind of access that's granted [39:13.260 --> 39:18.400] through resource creation can lead to privilege escalation and lateral movements within your [39:18.400 --> 39:23.300] environments. Another thing we covered is that default service accounts are granted administrative [39:23.300 --> 39:28.100] roles within your project automatically and that's with the editor role unless you're using the [39:28.100 --> 39:33.120] disable automatic IAM grants for default service accounts organization constraint. Reduce usage of [39:33.440 --> 39:38.560] default service accounts where possible by creating service specific service accounts with IAM roles [39:38.560 --> 39:43.040] that are scoped to your services needs. There are a few new cool tools in the platform that have been [39:43.040 --> 39:48.420] released to assist you in understanding who has access to your service accounts, such as the IAM [39:48.420 --> 39:53.880] analyzer, which can also be really helpful for understanding IAM dependencies and access within [39:53.880 --> 39:55.320] your environments.