WEBVTT 00:01.760 --> 00:03.580 Hello, my name is Allison. 00:07.250 --> 00:08.770 Hi, my name is Dylan. 00:12.270 --> 00:20.010 We've been researching GCP for the last year and a half and we've come up with some effective ways that compromise most GCP organizations, all the ones we've looked at anyway. 00:20.010 --> 00:23.790 These techniques involve moving laterally with service accounts and cloud APIs. 00:23.790 --> 00:26.250 We have a lot of ground to cover, so let's get to it. 00:38.120 --> 00:43.440 I think everyone has a slightly different reaction when they first start learning about GCP IAM. 00:44.100 --> 00:52.560 But one common thing I've observed is people often compare it to AWS IAM, but there are some pretty key differences between GCP and AWS IAM. 00:52.560 --> 00:54.500 So let's talk about some of those differences. 00:54.500 --> 01:03.420 First, in AWS, we have user identities and those user identities can be exported in the form of credentials, which look something like this. 01:03.560 --> 01:11.900 In GCP, the equivalent is a service account and service accounts can also be exported in credentials that look a little bit differently. 01:11.900 --> 01:13.200 They look like this. 01:14.500 --> 01:17.900 And in both cases, we have owners of these identities. 01:18.080 --> 01:23.140 But where things get a little bit different is when you start to ask the question, what can my identity access? 01:23.400 --> 01:29.860 In the world of AWS, you basically have a policy that determines all the different things that the identity can access. 01:29.860 --> 01:36.340 And the person who can create, read, update, delete that policy is the same person who owns the identity. 01:36.340 --> 01:38.400 So here's an example of what that looks like. 01:39.000 --> 01:41.960 In GCP though, this is where things start getting really different. 01:41.960 --> 01:47.580 Instead of having a centralized policy for a given identity, we have policies for resources. 01:47.720 --> 01:52.180 So in this case, we've got a bucket and a bucket owner and that bucket has a policy. 01:52.180 --> 01:56.700 The bucket owner controls who can create, read, update, or delete that policy. 01:56.700 --> 02:00.640 And that policy applies to whatever identity the bucket owner wants it to. 02:00.640 --> 02:04.420 So here's an example of what that looks like for storage. 02:05.600 --> 02:13.280 What's interesting here is that the owner of the service account actually doesn't have any insight into this policy. 02:13.280 --> 02:14.380 They can't view it. 02:14.380 --> 02:15.580 They can't update it. 02:15.580 --> 02:19.580 They don't even know that their service account was given access to this bucket. 02:19.580 --> 02:21.880 Instead, the bucket owner controls this. 02:21.880 --> 02:29.360 And if the owner of the service account doesn't have IAM list on the bucket, then they won't be able to even see that that policy was granted. 02:30.500 --> 02:44.080 And I think this is one of the harder concepts it was for me personally to wrap my head around in GCP, is if you have an identity in GCP, like a user or a service account, as the owner of that identity, you can't actually know what that identity has access to. 02:44.080 --> 02:48.160 In fact, no one can answer the question, what does the service account have access to? 02:48.160 --> 02:51.880 Instead, resource owners control what the policy is for a given resource. 02:51.880 --> 02:55.060 So you can answer questions like, who has access to this bucket? 02:55.060 --> 02:58.980 But you can't answer questions like, how many buckets does the service account have access to? 02:58.980 --> 03:01.600 Or what buckets does the service account have access to? 03:01.600 --> 03:04.760 Those questions are unanswerable by design in GCP. 03:04.760 --> 03:07.880 Likewise, we can have other resources like BigQuery, for example. 03:07.880 --> 03:10.960 And we have an owner for the BigQuery and a policy for the BigQuery. 03:10.960 --> 03:15.180 And the owner of the BigQuery can, again, grant access to any identity they want. 03:15.180 --> 03:19.760 However, the owner of the service account has no control over this and can't see this. 03:19.760 --> 03:26.640 One thing you might be wondering is, before when I mentioned that the owner of the service account can't fully answer the question, what does the service account have access to? 03:26.640 --> 03:32.960 You might be thinking, well, if you own an organization, can't you just get the IEM policies of all the resources in the organization? 03:32.960 --> 03:37.680 Well, someone outside your organization can still grant your service account access to their resources. 03:37.680 --> 03:42.460 So when I said before, you can't actually answer the question, what does your service account have access to? 03:42.460 --> 03:43.000 It's true. 03:43.000 --> 03:48.840 You cannot get the answer to the question because people outside your organization can also grant access to their resources. 03:48.920 --> 03:52.940 We'll come back to that later and explain what an organization is and how it's structured. 03:52.940 --> 03:55.340 But for now, I figured I'd just mention that in case you were wondering. 03:55.340 --> 04:02.100 As you might imagine, not having control or visibility into what your service accounts can or can't access comes with some challenges. 04:02.100 --> 04:06.660 And one particular place I think those challenges manifest themselves is in a multi-tenant Kubernetes environment. 04:06.660 --> 04:10.260 Google has a managed Kubernetes engine that's often shorthanded GKE. 04:10.260 --> 04:15.580 If you're familiar with Kubernetes, you have a bunch of nodes, and within those nodes you have workloads. 04:15.580 --> 04:20.780 In GKE, those nodes are powered by VMs, and those VMs have service accounts attached to them. 04:20.780 --> 04:26.420 If we take a look at we can see a Kubernetes cluster here that's powered by four underlying nodes. 04:28.060 --> 04:33.140 Then, if we hop over to Compute Engine and click on VM instances, we'll see the same four instances. 04:33.200 --> 04:36.720 And if we click into one of them, we'll see that we have a service account that's attached to it. 04:36.720 --> 04:43.340 We talked a little bit about how workloads interact with the service account in our B-side San Francisco talk this year, so feel free to check that out. 04:44.160 --> 04:50.060 But in short, anything running on the node can access this identity, so it's a good idea to descope it and not give it many privileges. 04:50.060 --> 04:52.100 But let's think about what that actually means. 04:52.100 --> 05:00.200 We have a Kubernetes admin that runs the nodes and the service account powering the nodes, and then we have a bunch of developers that run the workloads on the Kubernetes engine. 05:00.200 --> 05:05.820 Those developers may be on separate teams and may be in control of separate resources from each other and from the Kubernetes admin. 05:05.820 --> 05:11.800 So you can imagine in this case we have dev1 and dev2 that have both deployed workloads, workload1 and workload2. 05:11.800 --> 05:13.660 Likewise, they both control resources. 05:13.660 --> 05:18.220 In this case, we'll say that they both own a bucket and they want their workloads to be able to access that bucket. 05:18.220 --> 05:23.540 One really easy way to do that is to just give the service account on the node access to the bucket. 05:23.540 --> 05:29.700 But the downside to doing that is instead of just giving their workload access to the bucket, they'll give every workload access to the bucket. 05:29.700 --> 05:32.100 The developers in this case control this story. 05:32.100 --> 05:36.020 They control what the service account has access to because they're the resource owners. 05:36.020 --> 05:39.660 The Kubernetes admin does not control this and cannot see that this even happened. 05:39.660 --> 05:56.400 So you can imagine in a situation where you have a small team of Kubernetes admins that are trying to maintain a cluster that want to open it up to give many developers the ability to deploy workloads to this cluster, it can be difficult to secure because they can't control what the service account attached to the nodes has access to. 05:56.400 --> 06:00.520 And some developers may want to just give that service account access to resources. 06:00.520 --> 06:11.640 Rather than using something like Kubernetes secrets or vault to pass credentials to those workloads, they may choose to just give the service account attached to the underlying node access to their resources. 06:11.640 --> 06:17.020 But when they do that, they give all workloads access to their resources, not just one workload. 06:17.020 --> 06:30.840 And so you end up in a situation where a Kubernetes admin who wants to onboard developers into their cluster inadvertently gives every new workload access to a bunch of resources, even though they can't even see that or have any control over it. 06:42.680 --> 06:52.680 Okay, now before we go any further, it's important to point out that up until this point, I've kind of implied that resources are like these standalone entities that receive single role bindings directly to them. 06:52.680 --> 06:58.420 And while resources definitely can operate that way, most of the time they're actually bundled together into groupings. 06:58.420 --> 07:01.120 And there's a role binding that goes to the entire group of stuff. 07:01.120 --> 07:05.480 So instead of getting a role directly to a bucket, you'd instead get a role to a collection of buckets. 07:05.480 --> 07:09.440 So fundamentally, there are four nodes, resource nodes within GCP. 07:09.440 --> 07:12.260 There's an organization node, which is the parent node. 07:12.260 --> 07:16.020 This is generally associated with your domain or cloud identity service. 07:16.020 --> 07:17.900 There are folders and projects. 07:17.940 --> 07:22.660 All of these different top-level nodes are mostly used for IAM. 07:22.660 --> 07:39.480 And something that's important to note within the resource hierarchy is the concept of IAM inheritance, where wherever you have an IAM binding to one of these top-level nodes, your IAM binding is also going to be inherited to all subsequent or child resources within this hierarchy. 07:39.480 --> 07:45.660 So if I have a role binding at the organization, I have access to all subsequent resources that are created within the environment. 07:45.660 --> 07:47.720 Same at the folder and project level. 07:47.800 --> 07:51.700 Generally, users mostly interact with resources at the project level. 07:51.700 --> 07:58.340 That's where services like buckets, compute, and general services that you may interact with on a day -to-day basis are. 07:58.340 --> 08:05.000 Now, let's dig into a few different IAM workflows and try to understand why project -level IAM bindings are so prevalent in GCP. 08:05.220 --> 08:15.320 If we navigate to IAM and we go to grant a member access to our resources, what we're doing here is we're actually creating a project-level role binding, which will give this member access to all resources within our project. 08:15.320 --> 08:23.160 If we wanted to create a resource-level IAM binding, we would actually need to interact with the resource-specific API and set the IAM binding there. 08:24.820 --> 08:30.560 Another IAM workflow where a user is prompted to create a project -level role binding is when you create a service account. 08:30.560 --> 08:39.080 When we create a service account, we'll actually be prompted to set a project-level IAM binding that will grant this service account access again to all resources within our project. 08:39.080 --> 08:45.620 So we can see that when you go through general IAM workflows, you are going to always be setting a project-level IAM binding. 08:45.620 --> 08:52.100 Your IAM binding is also going to be inherited to all subsequent or child resources within this project. 08:53.960 --> 08:56.320 Here we can also see something called conditionals. 08:56.320 --> 09:05.240 This is an advanced feature within the IAM service that does enable you to set more granular IAM policies, but also at the project level within your environment. 09:05.440 --> 09:10.980 So I think that's really interesting that the main IAM button in the UI actually takes you to the project-level bindings. 09:10.980 --> 09:18.380 A lot of people don't even realize that you can do resource-level bindings because you have to actually click through into the resource to see that there are IAM settings there. 09:18.380 --> 09:22.360 And a lot of people just assume that if they go to the IAM page, they get the whole IAM story. 09:22.360 --> 09:32.640 So when we backtrack a little bit to when we were talking about resource owners and service account owners, realistically what would actually happen is you would have project owners, and within those projects you would have resources and service accounts. 09:32.640 --> 09:43.260 So before when we were talking about role bindings, we mentioned this idea of Kubernetes nodes having service accounts attached to them, and developers deploying workloads to the Kubernetes cluster and granting them access to resources. 09:43.260 --> 09:47.720 But more realistically what that would look like is they would grant that node access to their entire project. 09:47.720 --> 09:50.540 Because again, IAM bindings are typically done at the project level. 09:50.540 --> 09:55.800 So instead of granting the node access to a single bucket, they would grant it access to all buckets within a given project. 09:55.800 --> 09:57.140 Now this isn't always the case. 09:57.140 --> 10:03.440 Sometimes developers do resource-level role binding, and it should also be noted that the UI isn't the only way to apply these role bindings. 10:03.440 --> 10:06.200 You can also apply them via Terraform or via CLI. 10:06.200 --> 10:12.340 It's pretty common for developers to do it at the project level, because most of their first exposure actually comes through the UI. 10:12.340 --> 10:20.840 Okay, so if we recap to the earlier section, you might remember when we talked about service account owners, we mentioned that the service account owner can't actually know what access their service accounts have. 10:20.840 --> 10:22.200 Well, it's not really the whole story. 10:22.200 --> 10:32.800 Really what we have is a project owner, and then that project owner has a bunch of service accounts in their project, and then those service accounts have access to stuff, and the project owner doesn't really have the whole story of what their service accounts have access to. 10:32.800 --> 10:43.740 And so you might imagine how that can start to become dangerous, is when I hand over access of my project to someone else, I don't really know how much access I'm handing over, because again, I don't even know how much access my service accounts have in my project. 10:43.740 --> 10:47.600 So if I hand that over to someone else, I'm handing over an unknown amount of access. 10:47.600 --> 10:49.300 So what does an example of this look like? 10:49.300 --> 10:54.020 Well, we have a project owner that knows the email address of a different project service account. 10:54.020 --> 11:02.220 They can then give that service account access to their project through an IAM binding, and then that service account has access to their project's resources. 11:02.820 --> 11:06.700 Now, one question you might be asking yourself is why you would even need a cross-project role binding. 11:06.700 --> 11:11.980 Well, when you think about it, when you have two projects, they're not particularly useful unless they have some way to talk to each other. 11:11.980 --> 11:16.380 And one common way to do that is through Google Cloud services like PubSub or Storage. 11:16.520 --> 11:19.600 And so it makes sense that we would want to share some resources between projects. 11:19.600 --> 11:27.960 But again, where things sort of start to become more dangerous is if we share service account access between projects, because project owners don't really know what access their service accounts have. 11:27.960 --> 11:32.640 So we're missing out on some important existential questions like how interconnected our projects actually are. 11:32.640 --> 11:39.040 Because again, we can answer questions like who has access to a resource, but we really can't answer questions like what does a service account have access to. 11:39.040 --> 11:43.660 So we came up with a solution that let us sort of answer this question within our own organization. 11:43.660 --> 11:48.780 We can't actually get the complete answer to the question, again, because of these things called cross-organizational bindings. 11:48.780 --> 11:56.500 But at least within our org, we can get the answer for cross-project bindings by just introspecting every individual project that we own and pulling all the IAM policies. 11:56.500 --> 11:59.480 We then took all that information and we made a graph out of it. 11:59.480 --> 12:05.760 We actually went over to GitHub and we crawled a bunch of IAM policies that people either intentionally or accidentally committed. 12:05.760 --> 12:10.420 And we took all those policies and we kind of compiled this pseudo-org generator. 12:25.050 --> 12:27.410 So let's take a look at one of the graphs we ended up with. 12:27.410 --> 12:29.710 As you can see, there are two different colored nodes. 12:29.710 --> 12:32.110 There are red nodes and there are blue nodes. 12:32.110 --> 12:36.870 What the red nodes represent are projects and what the blue nodes represent are service accounts. 12:36.870 --> 12:38.970 Those are the only two things we're graphing here. 12:38.970 --> 12:40.710 Then there are two different types of edges. 12:40.750 --> 12:43.550 There are contains edges and there are bindings edges. 12:43.550 --> 12:51.210 A contains edge is basically when a service account lives in a given project and the binding edge is when that service account has a role binding to that project. 12:51.470 --> 12:55.710 Most of the time, a service account has a role binding to the project that it's contained in. 12:55.710 --> 13:00.930 But every once in a while, we'll end up with a service account that has a role binding to a different project across project binding. 13:00.950 --> 13:08.370 So in this case, we have a service account that's contained in project number 67 and it has a role binding into project number 57. 13:08.370 --> 13:10.670 If we zoom out, we get the view of the entire organization. 13:10.690 --> 13:17.230 We have all the different projects here in a map and we have all the relationships of the service accounts within the org that are contained within these projects. 13:17.230 --> 13:21.370 Now we're not looking at organization-level bindings in this graph or resource-level bindings. 13:21.370 --> 13:25.170 We're only looking at the project-level bindings and we're mapping these relationships. 13:25.170 --> 13:34.490 So because we based this off of real IAM policies that we're able to pull up GitHub, we can say with some level of confidence that this is an accurate representation of what a sufficiently large company might look like. 13:34.490 --> 13:42.290 We can also say with a little bit of personal experience based on the large companies that we've looked at, this is an accurate representation of what a sufficiently large company looks like. 13:42.290 --> 13:47.610 Now again, most of the time a developer is just going to be the owner of a single project and they won't really see the whole picture. 13:47.610 --> 13:51.450 All they'll see is sort of tunnel vision for the one project that they have control over. 13:51.450 --> 14:00.690 And so they'll see the service accounts that belong to their project and they'll see what has role bindings into their project, but they won't know where their service accounts have role bindings out of their project. 14:00.690 --> 14:06.790 And so when they hand off access to their project to another developer, they won't actually know how many additional projects they're handing off. 14:06.790 --> 14:12.150 And they themselves may not have a good understanding of how many projects that they have access to through their service accounts. 14:12.150 --> 14:19.950 One other interesting thing that we found here is that there are certain projects that have seemingly disproportionate large numbers of role bindings and resources in them. 14:19.950 --> 14:23.810 We found this was a pretty consistent story at multiple companies that we looked at. 14:23.810 --> 14:28.810 And so in the center here you can see there are a bunch of projects that have really tightly bound role bindings. 14:28.810 --> 14:34.530 And then at the bottom we have smaller projects that only have one or two service accounts that may not be connected to the broader web. 14:34.530 --> 14:36.930 And then we have a bunch of projects that have no service accounts. 14:36.930 --> 14:42.430 What we found is that most of the sensitive data or sensitive workloads are in the interconnected projects. 14:42.430 --> 14:44.210 That's where most of the development is happening. 14:44.210 --> 14:49.090 And so it makes sense that it would have most of the IAM role bindings connecting it with other projects. 14:49.090 --> 14:55.670 So when we first started building these graphs, we were thinking to ourselves, oh gosh, these projects are supposed to be isolated boundaries. 14:55.670 --> 14:57.310 Are they really this interconnected? 14:57.310 --> 15:07.550 And the more we dug into it and the more we thought about how project owners can't actually see the role bindings their service accounts have been granted, we realized this is a problem that's really remained invisible. 15:07.550 --> 15:10.850 And it really takes this org-level introspection to even see it. 15:10.850 --> 15:16.810 Now there's one final piece here that we haven't talked about yet, and that's when a developer actually does hand over access to their project to another developer. 15:16.810 --> 15:18.510 Pieces of that might be okay. 15:18.510 --> 15:23.450 It might be okay to share storage, for example, but things start to become really dangerous if you share access to service accounts. 15:23.450 --> 15:29.330 Because again, that's where the problem is hidden, where the owner of the project doesn't actually know what access their service accounts have. 15:29.330 --> 15:32.630 And so if we share that unknown access, that's where things can start to go astray. 15:32.630 --> 15:37.730 So we'll come back to what roles and permissions are actually required to hand over that level of access. 15:37.730 --> 15:39.450 And we'll talk about how common that is. 15:39.450 --> 15:43.430 So far we've talked a lot about roles and mentioned that some are more dangerous than others. 15:43.490 --> 15:45.570 So now let's dig into that a little bit more. 15:45.590 --> 15:49.090 Within GCP, there are three distinct types of roles that you can use. 15:49.090 --> 15:53.150 Primitive, which have been around since before Cloud IAM that we know today. 15:53.150 --> 15:57.170 Predefined roles, which are curated by GCP and service specific. 15:57.410 --> 16:03.830 And custom roles that enable developers to specify any permission that they would like that is supported to fit their services needs. 16:03.830 --> 16:05.710 Let's focus on primitive roles. 16:05.710 --> 16:12.810 Within primitive roles, there's an owner role, which provides all of the different permissions that are available within the platform. 16:12.830 --> 16:22.150 An editor role, which closely resembles the owner role, but excludes a few different things for administrative capabilities and that just might not be supported yet. 16:22.170 --> 16:26.530 One thing in particular that's excluded is the ability to set project level role bindings. 16:26.530 --> 16:29.270 And the viewer role, which is exactly what it sounds like. 16:29.270 --> 16:33.170 It allows you to view resources, but not perform any state changing actions. 16:33.710 --> 16:35.830 How are these primitive roles used? 16:35.830 --> 16:43.290 Well, earlier we saw that when you go to Cloud IAM through the UI, these primitive roles are what are first suggested to users. 16:43.550 --> 16:47.750 There's another way that these primitive roles manifest themselves within your GCP project. 16:47.750 --> 16:50.870 And that's through default IAM role bindings. 16:50.870 --> 16:57.710 Whatever identity creates a project will automatically be granted a project level role binding using the owner role. 16:57.710 --> 17:04.110 So we know that the owner role contains all of the different permissions that are available, but what about the editor role? 17:04.170 --> 17:11.290 Well, the editor role contains, at the time of this talk, 2,576 distinct permissions. 17:14.710 --> 17:15.750 That's a lot. 17:15.750 --> 17:24.410 This means that with the editor role, you can, for example, access all buckets, all databases, all VMs, and much more within your project. 17:25.950 --> 17:32.310 Out of these permissions that we talked about earlier, which of these roles actually enable a user to manage service accounts? 17:32.310 --> 17:34.870 Well, both the owner and the editor role do. 17:34.890 --> 17:47.270 While the editor role does not enable you to create new service accounts or apply project level role bindings for service accounts, it does allow you to manage service accounts in two different ways. 17:47.270 --> 17:56.550 And that's by creating keys or credentials for a given service account that already exists, and by associating a service account with a resource. 17:56.550 --> 18:08.790 So if a user can create a resource, specify a service account, GCP will go and check, does this user have the IAM service accounts act as permission on the service account that they're attempting to associate with the resource? 18:08.790 --> 18:17.670 If so, it gets attached, and then that user can interact with GCP services and APIs as that service account identity and its associated role. 18:17.670 --> 18:21.430 So how common are these IAM roles in GCP projects? 18:21.430 --> 18:27.270 There's another way that GCP creates automatic role bindings for identities, and this way is a little bit different. 18:27.270 --> 18:43.050 When you enable a specific service such as Google Compute Engine, it will go and create a default service account in your project and then grant that service account a project level role binding using the primitive editor role that we chatted about earlier. 18:43.230 --> 18:45.310 This also happens with another service . 18:45.310 --> 18:53.690 When a user enables the App Engine service, a service account called the App Spot service account is created in your project and also granted a project level editor role binding. 18:53.690 --> 19:01.710 Now as we discussed earlier, that editor role contains a permission that enables identities to associate service accounts with resources. 19:01.710 --> 19:12.910 So that means that both of these default service accounts, the Compute Engine and the App Spot service account, are able to associate any service account within its GCP project to a resource. 19:12.910 --> 19:14.810 So where are these service accounts used? 19:14.810 --> 19:21.330 The default compute service account is associated with GCE VMs by default. 19:21.330 --> 19:26.770 So whenever you create a VM, the default compute service account is automatically associated with the VM. 19:28.050 --> 19:34.830 There's one thing to note about service accounts and the way that they are authorized to interact with services from a VM. 19:34.830 --> 19:38.710 There is the role that they have as well as something called scopes. 19:38.710 --> 19:47.570 And scopes define, from within the VM, what services and APIs can that identity interact with regardless of the role binding that they have. 19:47.570 --> 19:54.390 By default for VMs, scopes are restricted to only a few particular services like storage. 19:54.390 --> 20:00.910 But from what we have observed, developers will commonly open these scopes up so that they can interact with many different services. 20:00.910 --> 20:03.350 And how are the App Spot service accounts used? 20:03.350 --> 20:08.190 Cloud Functions are Google's serverless offerings, which are similar to AWS Lambdas. 20:08.190 --> 20:13.730 They're meant to just be lightweight, ephemeral, small chunks of code that run fast and exit once they're done. 20:13.950 --> 20:26.830 If the App Engine service is enabled within your project, when a developer goes to create a Cloud Function, that default App Spot service account with the editor role binding will be associated with Cloud Functions by default. 20:26.830 --> 20:32.170 When you create a Cloud Function, you actually do not have the ability to specify to set it without an identity. 20:32.170 --> 20:35.450 You always have to associate an identity with the Cloud Function. 20:35.450 --> 20:44.470 So if you don't know to create a new service account and associate with your Cloud Function, you'll always be using the default App Spot service account that has the editor role binding. 20:44.470 --> 20:48.270 Cloud Functions do not have scopes and get all the permissions of the editor role. 20:49.030 --> 20:56.230 And coming back to the VMs, a lot of managed services leverage VMs as their underlying infrastructure. 20:56.230 --> 21:03.270 Things like GKE, Dataproc, Dataflow, they also get this default compute engine service account associated with them. 21:03.270 --> 21:08.390 In AWS, when you spin up something like a VM or a Lambda, it defaults to having no access. 21:08.390 --> 21:09.770 There are no roles attached. 21:09.770 --> 21:13.790 But in GCP, things default to having thousands of permissions. 21:13.790 --> 21:24.530 Earlier, we talked about cross-project role bindings, the ability to leverage service accounts in a project to get access to other service accounts and potentially move laterally across GCP projects. 21:24.530 --> 21:28.870 We talked about developers potentially not knowing what access their service accounts have. 21:28.870 --> 21:32.090 We mentioned the idea of handing off access. 21:32.090 --> 21:38.230 And what this realistically looks like is the developer applies a role binding with the act as permission, like the editor role. 21:38.230 --> 21:40.770 Let's look at what that could look like. 21:40.770 --> 21:43.170 Now it's demo time. 21:48.960 --> 21:53.740 To help understand how to exploit the act as permission, we built a framework called Gsploit. 21:53.740 --> 21:55.900 To start out, we need a base identity. 21:56.000 --> 21:58.400 There are a number of different ways to get a base identity. 21:58.400 --> 22:03.160 One way, for example, is you could OAuth fish a developer that has project level editor access. 22:03.160 --> 22:14.880 Another way is maybe a developer accidentally uploaded a service account key to GitHub or and that service account has a editor level role binding because the primitive roles are the first thing that are suggested to you on the IAM page. 22:14.880 --> 22:23.440 Or maybe the most likely, maybe the base identity comes from exploiting a service and because most services run with project editor by default, maybe that's how you get your first foothold. 22:23.440 --> 22:27.960 In any case, in this case, we've simulated this by just starting with the service account credential. 22:27.960 --> 22:32.700 We can see this credential here and we can see that it starts in the project bbs2. 22:33.200 --> 22:36.160 Next, we'll run the list operation on the Gsploit tool. 22:36.160 --> 22:37.520 We can see that there's no output. 22:37.520 --> 22:45.300 The reason there's no output is the list operation lists all the service accounts that you've managed to take control of and it doesn't count the base identity. 22:45.300 --> 22:50.620 We'll run the gcloud command projects list on the base identity to see what project we have access to. 22:51.660 --> 22:54.240 It looks like we have access to bbs2. 22:54.240 --> 22:59.620 Next, we'll try an act as exploit on this project and we'll target every service account in the project. 23:02.680 --> 23:04.500 Let's unpack what's happening now. 23:04.500 --> 23:08.920 Keep in mind the editor role has two ways to take control of all the service accounts in a project. 23:08.920 --> 23:12.860 The first is through creating tokens and the second is through provisioning resources. 23:12.880 --> 23:16.580 There's another role that we haven't talked about yet and that's the service account user role. 23:16.580 --> 23:22.000 This role is also pretty common and is often given to users and service accounts that are meant to provision resources. 23:22.020 --> 23:28.380 Because some things like cloud functions require service accounts be attached to them, this is the role that's usually granted to do that. 23:29.080 --> 23:40.900 So because the service account user role and the editor role both have the act as permission, we'll use that to take control of the service accounts because it casts a little bit wider of a net than the token creator permission which is only in the editor role. 23:40.900 --> 23:50.800 So what gsploit is actually doing under the hood here is first it's running the service account list operation to get a list of all the service accounts in the project and then it's spinning up a cloud function for each one of those service accounts. 23:50.800 --> 23:56.580 Then we keep a database of all the cloud functions we've harvested, each which has its own underlying service account attached to it. 23:56.580 --> 24:01.100 This gives us access to all of the service accounts in the project through the cloud functions that we spun up. 24:01.100 --> 24:07.300 We control the code that lands on the cloud function, so we'll just deploy code that allows us to run whatever gcloud commands we want. 24:07.300 --> 24:11.660 It takes about two minutes to spin up each cloud function, so we'll just speed through this really quick. 24:14.060 --> 24:19.140 And now when we run the list command again, we can see all the new service accounts that we now have access to. 24:19.140 --> 24:20.340 One of them says owner. 24:20.340 --> 24:24.300 That service account actually has an owner level role binding to the project that we started in. 24:24.300 --> 24:27.340 That means we now have full control of the project through this service account. 24:27.340 --> 24:31.260 If we wanted to, we could use this service account to add ourselves as owners to the project. 24:31.260 --> 24:43.840 So just to pause and think about that for a second, what that means is if you ever have a service account that has editor level access and another service account that has owner level access, the editor level service account can always privilege escalate itself to owner. 24:43.940 --> 24:45.060 That's the same with developers. 24:45.060 --> 24:54.400 If you grant a developer editor level access to a project but you happen to have a service account in that project that has owner level access, that developer can privilege escalate themselves to owner through that service account. 24:54.400 --> 24:57.120 Okay, let's take another look at those service accounts that we have. 24:57.140 --> 24:59.660 There's one here that's kind of interesting that says do nothing. 24:59.740 --> 25:12.520 It might not sound interesting, but keep in mind that the developer that created that service account in this project, although they didn't intend to give it any role bindings, again, the owner of the service accounts can't control the role bindings, and so it's possible that a different project gave it a role binding. 25:12.520 --> 25:15.940 So let's run the project list command from this service account that we've compromised. 25:16.560 --> 25:21.160 Something interesting here that the base project, bbs2, doesn't actually show up when we run this command. 25:21.280 --> 25:23.500 This is because the service account has no role bindings there. 25:23.500 --> 25:25.520 The developer didn't want it to have any role bindings. 25:25.520 --> 25:28.920 But bbs3, this new project we haven't looked at, shows up. 25:28.920 --> 25:34.400 What happened here is the developer in another project granted this service account and added a role binding into their project. 25:34.400 --> 25:43.440 So let's run the act as exploit one more time, this time from this do nothing service account into bbs3 and see how many service accounts we can harvest in bbs3. 25:43.440 --> 25:46.700 Again, this process takes a little while, so we're going to fast forward through this. 25:50.900 --> 25:54.960 And now we can see a whole bunch of new service accounts in the bbs3 project. 25:54.960 --> 25:59.480 Service accounts that we didn't have access to before that we were able to get strictly through lateral movement. 25:59.480 --> 26:01.600 There's a couple service accounts in here that look interesting. 26:01.600 --> 26:05.000 There's one that's labeled networking that maybe controls networking for the organization. 26:05.000 --> 26:08.780 But probably most interesting is this one that's labeled organizational admin. 26:08.780 --> 26:16.900 If while you're moving laterally through projects you happen to land on a service account that has an org level binding, through inheritance you get access to the entire org that way. 26:17.160 --> 26:21.660 So to recap, we started with the base identity in bbs2 that had act as on the project. 26:21.660 --> 26:26.940 We then used that permission to gain access to every service account in this project via cloud functions. 26:26.940 --> 26:32.560 One of the service accounts in bbs2 had an owner role, which allowed us to elevate privileges in the bbs2 project. 26:32.560 --> 26:35.380 A different service account had a role binding into bbs3. 26:35.600 --> 26:39.500 This allowed us to use act as to take control of all the service accounts in bbs3. 26:39.500 --> 26:45.880 One of the service accounts we found in bbs3 had an organizational role binding, and this allowed us to take control of the entire org. 27:02.650 --> 27:12.170 So we talked about when you provision resources, when you attach service accounts to those resources, GCP will check to make sure that you have the act as permission first before allowing you to provision the resource. 27:12.170 --> 27:18.650 Because some APIs like cloud functions require that you have identities to power them, this leads to a lot of people getting the act as permission. 27:18.650 --> 27:25.330 Well it turns out that some APIs will actually allow you to provision resources with the service account attached to them without the need for the act as permission. 27:25.330 --> 27:29.130 These include the Dataproc, Dataflow, and Composer APIs. 27:29.270 --> 27:31.130 Let's talk about the Dataproc API. 27:31.130 --> 27:34.590 The Dataproc API is Google's data processing API. 27:34.590 --> 27:38.850 Among other things, it includes managed Apache Hadoop and managed Apache Spark. 27:38.850 --> 27:45.570 When you spin up a Dataproc cluster, like a lot of things in GCP, it will default to using the default Compute Engine service account. 27:45.570 --> 27:48.810 This means by default your Spark jobs will run as project editor. 27:48.810 --> 27:52.410 You only need the permission Dataproc cluster create to spin up this cluster. 27:52.410 --> 27:56.130 You don't actually need the act as permission on the default service account. 27:56.130 --> 28:02.890 Because of this, anyone who has the Dataproc cluster create permission can actually make use of the default editor service account. 28:02.890 --> 28:09.510 Put another way, service accounts that are only intended to be able to spin up Dataproc clusters can privilege escalate themselves to project editor. 28:09.510 --> 28:14.550 And then they can make use of that act as permission to take control of the rest of the service accounts in the project. 28:14.550 --> 28:18.370 We built support for this into gsploit, so let's take a look and see what it looks like. 28:18.370 --> 28:22.990 If we take another look at the list of service accounts we've compromised, there's one here that says Dataproc. 28:22.990 --> 28:27.550 Let's run the gcloud projects list on this service account and see what projects it has access to. 28:34.060 --> 28:36.300 Looks like it has access to a bbs4. 28:36.300 --> 28:40.840 Now let's run the Dataproc exploit on bbs4 through the Dataproc service account. 28:40.840 --> 28:44.880 We have to spin up a whole Dataproc cluster to do this, and that takes a little bit of time. 28:44.880 --> 28:47.620 And then we delete it afterwards so that we're not filled too long. 28:47.620 --> 28:53.460 So we'll fast forward through this part, but at the end of it when we run gcloud list again, we can see a new service account has been added. 28:53.460 --> 28:56.400 And that's the default service account in the bbs4 project. 28:56.400 --> 29:05.580 We were able to get this service account by spinning up a Dataproc cluster in bbs4 and then running a spark job on that cluster, which enabled us to grab the off the instance. 29:05.580 --> 29:11.740 Because our Dataproc service account credential had the ability to spin the cluster up, it also controlled the scopes for that cluster. 29:11.740 --> 29:14.560 And so we just defined the scopes for this cluster to be wide open. 29:14.560 --> 29:20.260 Next, since we have a default editor service account credential at this point, we can run the act as exploit on this project. 29:24.680 --> 29:26.860 We'll again fast forward through this process. 29:30.660 --> 29:37.440 And finally, if we run the gcloud list one more time, we can see a whole bunch of service accounts that we now have access to in the bbs4 project. 29:37.440 --> 29:44.700 So just to recap, some APIs in GCP do not require the act as permission to spin up resources and attach the default service accounts to them. 29:44.700 --> 29:48.300 The identities that can spin those resources up can also control the scopes. 29:48.300 --> 29:57.180 So effectively, in cases like Dataproc and Dataflow and Composer, any identity that's granted permissions to those APIs can privilege escalate themselves to project editor. 29:57.180 --> 30:05.100 One last thing to note here is because this Dataproc role binding was actually cross-project, this allowed us to again compromise the new project bbs4. 30:05.100 --> 30:32.340 And because the default service accounts are in every project, this allows us to take control of all the service accounts in the IAM analyzer. 30:32.360 --> 30:36.560 We worked with Google on this feature, and one of the user stories actually came from us. 30:38.260 --> 30:43.760 We came up with a list of dangerous roles and permissions that can be used to take control of all the service accounts in the project. 30:43.760 --> 30:44.920 These can be seen here. 30:44.920 --> 30:49.460 We won't demo Dataflow and Composer, but you can think of them as working the same way as Dataproc. 30:49.460 --> 30:59.020 Using the IAM analyzer and the dangerous permissions that we're able to find, we're able to run a BFS search on a base identity without the need to spin up expensive resources exploiting services. 30:59.200 --> 31:06.400 As you can see from the output here, we get the same list of service accounts from the same base identity as we did exploiting things, except this was much faster. 31:06.400 --> 31:07.900 We didn't have to spin anything up. 31:07.900 --> 31:16.160 You need an org binding to use this, so this is a tool for defenders to be able to get quick answers to how an attacker might be able to move laterally through the organization. 31:16.220 --> 31:20.240 If we cut back to our mock graph, let's zoom into a seemingly innocent base identity. 31:20.260 --> 31:25.200 This service account, gwadvcl, was meant to only run Spark jobs in its base project 14. 31:25.200 --> 31:27.080 It only has the Dataproc editor role. 31:27.080 --> 31:30.940 Now we'll run a BFS search from this starting identity and highlight the path in blue. 31:31.020 --> 31:36.620 As you can see, this allowed us to compromise all the service accounts in the starting project through the default editor role on our Spark job. 31:36.620 --> 31:42.040 Most of these service accounts have benign roles like storage or PubSub, and most of them are not cross-project. 31:42.040 --> 31:48.240 That said, some, like this one, have dangerous role bindings that are cross-project, in this case the Dataflow developer role. 31:48.240 --> 31:51.420 These allow us to take full control of service accounts in different projects. 31:51.420 --> 31:54.120 As we start to zoom out, we can begin to see the whole picture. 31:54.120 --> 32:01.220 We've managed to compromise most of the service accounts in the org, with a few notable project exceptions that just don't have any dangerous role bindings to them. 32:01.220 --> 32:08.240 We're only looking at project-level bindings in this graph, so if any of these service accounts have org-level bindings, it's not reflected here. 32:08.640 --> 32:16.220 So we just saw some really interesting techniques around privilege escalation and lateral movement, but we also wanted to provide a way to detect this. 32:16.220 --> 32:22.820 So we fingerprinted this tooling, and we'll be releasing monitoring and alerting to detect the behavior of this tool in your environment. 32:22.820 --> 32:27.800 I think one of the nice things about doing security work in the cloud is how much help you get from your upstream provider. 32:27.800 --> 32:33.480 If we were doing work on a web application or a binary, we'd kind of be left to our own vices to fix the problems we found. 32:33.480 --> 32:39.960 But because we had a close partnership with Google, we were able to work with them to roll out some new features that help with the security story of some of these problems. 32:40.020 --> 32:47.880 One of the people that helped us do that is a security engineer named Bakunov, and we gave him an invitation to do a brief cameo and talk about some of the new features that we helped Google roll out. 32:47.880 --> 32:49.460 Hello, Black Hat. 32:52.740 --> 32:56.040 I had the privilege of working with Allison and Dylan. 32:56.040 --> 33:02.300 As a result of our collaboration, we at Google Cloud launched several features that help customers address concerns. 33:02.300 --> 33:14.260 On the prevention side of things, we now have an org policy constraint that prevents the editor grant to the default service accounts for Compute Engine and App Engine, and we advise customers to turn it on. 33:14.260 --> 33:24.600 We recommend that customers create custom service accounts with the necessary set of permissions on the target resources, as opposed to the broad project-level editor grants. 33:24.600 --> 33:29.580 Allison and Dylan briefly touched upon the investigative IAM analyzer tool. 33:29.580 --> 33:38.940 As you saw, we built IAM analyzer to be able to address the use case of finding resources which are granted access to any specific identity. 33:38.940 --> 33:42.940 The other investigative feature available is the IAM recommender. 33:42.940 --> 33:50.140 It uses machine learning to evaluate the permissions granted to an identity against actual activities performed by the identity. 33:50.140 --> 33:56.720 It's a great way to find overly generous permission grants and to identify the actual permissions that are needed for the workload. 33:56.720 --> 34:02.600 I spent many years working with all major cloud providers, and this statement is generally true for all of them. 34:02.600 --> 34:07.440 Starter cloud UIs, those flows are optimized for ease of onboarding. 34:07.440 --> 34:17.080 And so the fundamental challenge for large organizations is to make available sanely secured defaults, as well as preventative guardrails for their developers. 34:17.080 --> 34:31.880 Those are usually in the shape of cloud foundation code bases, such as infrastructures code samples that are built with security in mind, policy as code evaluations as part of the build and release pipelines, and preventative controls exposed by the cloud providers, 34:31.880 --> 34:34.720 such as org policy constraints on GCP. 34:34.720 --> 34:43.960 If we're leaving developers to the mercy of those starter UI flows on the cloud providers, or stack overflow code snippets, we're going to end up in a bad place. 34:43.960 --> 34:51.340 If you're a Google Cloud customer, and they're looking for guidance in this area, you should try to engage your account team so that we can help out. 34:51.340 --> 35:03.160 Alison and Dylan, on behalf of everyone at Google Cloud, we are very grateful for the collaboration on improving our customers' experience on the platform, and for the opportunity to have a cameo appearance in this talk. 35:03.160 --> 35:08.560 I'm hoping that Black Hat 2021 happens in person, and we can share stories in the face-to-face setting. 35:08.560 --> 35:14.480 Until then, please wear face masks, and don't forget to turn on two-factor authentication wherever possible. 35:19.740 --> 35:20.680 Thank you. 35:20.840 --> 35:21.740 Thank you, Bak. 35:21.740 --> 35:28.880 We also wanted to provide some context, and a little bit more insight, and let you know a few of the limitations or constraints to these tools. 35:28.880 --> 35:39.840 So one of the main limitations to the organization policy services, you actually have to have an organization resource, either through Cloud Identity or G Suite, to be able to use the organization policy service. 35:39.840 --> 35:44.240 If you only have projects, you won't actually be able to leverage any of the constraints that are available. 35:44.240 --> 35:48.920 Another limitation within the organization policy service is that constraints are not retroactive. 35:48.920 --> 36:02.420 So when the policy is applied, all resources will be affected by the policy or evaluated by the policy after it is enabled, but existing resources will not be modified to match or meet the requirements of a given constraint. 36:02.420 --> 36:19.880 Another thing to note is that for the new Disable Automatic IAM Grants for Default Service Accounts, that constraint will only enforce that when the APIs are initially enabled for new projects, that the default service accounts are not granted editor-level role bindings at the time of the APIs being enabled, 36:19.880 --> 36:28.020 but there is no enforcement mechanism to ensure that the default service accounts are not granted a project-level binding or other binding. 36:28.020 --> 36:40.660 There's also no way to restrict the cross-project bindings that we talked about previously, where if a user knows the name of your service account, they could still give the service account access to their project or project's resources. 36:40.660 --> 36:43.000 So the IAM Recommender is very powerful. 36:43.000 --> 36:53.500 Some of the limitations on the tool is that you have to have a 90-day period where the policies or behavior of a given identity are evaluated so that you can have a recommendation be made. 36:53.500 --> 36:58.980 And there may be more IAM bindings that you may not be aware of, like the cross-project bindings. 36:58.980 --> 37:04.740 The IAM Analyzer can provide a lot of context into the IAM bindings or associations for a given identity. 37:04.740 --> 37:14.380 And in order to use this auditing tool, you have to have administrative capabilities in an organization to be able to see the policies that are being analyzed. 37:14.380 --> 37:28.180 So in order to use this tool to the fullest extent, you need administrative capabilities in your organization, as well as within G Suite, if you want to expand group membership to know who has access to a given service account that is in a group. 37:28.180 --> 37:34.820 Earlier, we talked about how Kubernetes workloads have access to the underlying service account that's attached to the node. 37:34.820 --> 37:40.980 And the suggested way to harden your clusters is to use a service called Workload Identity. 37:40.980 --> 37:46.760 And Workload Identity allows you to associate a Kubernetes service account with a cloud identity. 37:46.760 --> 37:53.960 And this is how you can harden your cluster, as well as have a one-to-one mapping of Kubernetes service accounts to cloud identity service accounts. 37:53.960 --> 38:07.200 We brought up earlier to ensure default service accounts are not granted project-level role bindings or other role bindings after the disable automatic IM grants for default service accounts constraint is enabled, that you require additional tooling. 38:07.200 --> 38:28.100 And to demonstrate a way that you can enforce these configurations, we will be releasing a Terraform Enterprise Sentinel policy set that can demonstrate enforcing that default service accounts are not granted project-level role bindings or that roles that enable service account takeover or management of service accounts cannot be applied at the project level. 38:28.100 --> 38:38.540 Sentinel policies are Terraform Enterprise policy configuration language that enable you to either restrict or alert on specific configurations in your runs. 38:38.540 --> 38:52.320 So just to recap some of the key takeaways, IAM and GCP is very resource centric, meaning that if you are a service accounts owner, you may not be aware of the configurations that are associated with that service account or how it may be used across a GCP organization. 38:52.320 --> 38:56.880 This can lead to projects being interconnected in many different ways that you may not be able to see. 38:56.880 --> 39:02.380 Role bindings are often done at the project level and can grant more access than a user intends to. 39:02.380 --> 39:11.700 A lot of managed services and general resource provisioning operations can hand off access to your project's resources and IAM that you may not be aware of. 39:11.700 --> 39:19.260 This kind of access that's granted through resource creation can lead to privilege escalation and lateral movements within your environments. 39:19.260 --> 39:32.380 Another thing we covered is that default service accounts are granted administrative roles within your project automatically and that's with the editor role unless you're using the disable automatic IAM grants for default service accounts organization constraint. 39:32.380 --> 39:40.320 Reduce usage of default service accounts where possible by creating service specific service accounts with IAM roles that are scoped to your services needs. 39:40.320 --> 39:55.320 There are a few new cool tools in the platform that have been released to assist you in understanding who has access to your service accounts, such as the IAM analyzer, which can also be really helpful for understanding IAM dependencies and access within your environments.