WEBVTT 00:00.000 --> 00:04.220 As you mentioned, we're going to talk about hijacking OAuth tokens in Google Cloud. 00:05.060 --> 00:10.840 Good thing about Safe Mode this year is if this is the wrong talk, you can quietly leave and I won't even see you. 00:10.840 --> 00:14.460 All right, let's get into it here. 00:14.880 --> 00:18.960 So, if you need to get hold of me, just DM me at Twitter. 00:18.960 --> 00:21.820 You can usually find me on most of the social platforms. 00:22.060 --> 00:34.340 What we're going to cover in this talk is hijacking of OAuth tokens, session tokens, for CLI access, API access, and it's really the ease of attack that we're going to focus on. 00:34.340 --> 00:41.740 There's additional use cases related to service accounts and compute instances that we'll look at as well. 00:41.740 --> 00:48.240 And from a defensive perspective, we're definitely going to drill into some of the challenges in securing the environment. 00:48.820 --> 00:51.240 Really, who cares, right? 00:51.240 --> 00:54.380 Well, OAuth is fundamental to Google authentication. 00:54.380 --> 00:56.140 It's underneath everywhere. 00:56.140 --> 01:13.720 There are some nuances depending on if you are authenticating with user accounts versus service accounts, whether your sessions are from browsers versus SDK access, whether you're external to the GCP environment versus internal, say, on a compute VM. 01:13.720 --> 01:21.200 And that is part of what leads to a lot of the opportunity from the tech perspective, as well as the challenges from the defensive perspective. 01:21.320 --> 01:24.780 Essentially, the session tokens are what the name implies. 01:24.780 --> 01:28.860 They're post-authentication representing a valid authentication session. 01:28.860 --> 01:35.380 And it's really an opportunity for us to hijack and use them and gain access to existing authenticated sessions. 01:35.740 --> 01:45.860 And if I, as an attacker, have access to an endpoint, say a GCP administrator's device, those cached accounts and environments are easily accessible to me. 01:45.860 --> 01:47.680 And that's what we're going to go through. 01:48.380 --> 01:56.500 At the end of the day, it's all about persistence, as well as evading detection from the attacker's perspective. 01:56.560 --> 02:02.220 Now, the defensive side, what is interesting are some of the challenges. 02:02.220 --> 02:11.500 Some of the controls such as MFA don't quite help, although you might counterintuitively, you might think they do, right? 02:11.500 --> 02:12.820 And we'll go through that. 02:12.820 --> 02:23.220 And in terms of securing the environment, there's a lot of options that are confusing, easy to misconfigure, and also easily misunderstood. 02:23.380 --> 02:27.360 So at the end of the day, we're talking about how do you handle incident response? 02:27.360 --> 02:29.420 What's your playbook for ops? 02:29.460 --> 02:34.920 Is it fully up to date with the most optimal measures or controls? 02:37.680 --> 02:42.440 So let's start with just a brief overview of OAuth. 02:42.440 --> 02:47.980 Okay, just really briefly, there's some verbiage on this slide, but focus on the right hand diagram. 02:47.980 --> 02:52.380 At a very high level, there's a simplified flow, no matter how you authenticate. 02:54.120 --> 03:01.020 First, you'll request a token, it'll be signed with your credentials, you'll request scopes or permissions. 03:01.020 --> 03:06.960 And as a user, you often see this when you first authenticate with gcloud or CLI. 03:07.120 --> 03:15.660 By default, a browser is launched, you have to go type in your Google credentials, you also have to approve the actual scopes being requested. 03:15.900 --> 03:31.300 And if that goes properly, then what is returned ultimately to the requester, in this case, it'd be the CLI, would be two important things, the access session token, that's an OAuth session token, as well as a refresh token. 03:31.340 --> 03:34.280 The access token is good for an hour. 03:34.360 --> 03:39.580 By default, you use it directly in API calls, they all take OAuth tokens. 03:40.480 --> 03:43.940 After the hour is up, you can refresh and get a new one. 03:44.180 --> 03:51.540 So gcloud, gsutil, CLI tools do this automatically, you usually don't see this level of the protocol in the handshake. 03:52.100 --> 03:57.940 The local language SDK libraries like Python, etc, also do this transparently. 03:57.940 --> 04:00.320 So generally, your code is not at this level. 04:00.320 --> 04:07.160 If you do direct REST API calls, you will see this handshake and have to go through two or three back and forths. 04:07.160 --> 04:17.540 The other important thing to remember about OAuth, besides this handshake and the access and refresh tokens, is that, well, you don't authenticate every time you use the CLI, right? 04:17.540 --> 04:18.780 It's cached. 04:18.780 --> 04:23.240 All session tokens almost everywhere in every model and architecture are cached. 04:23.240 --> 04:28.020 In this case, it's in the SQLite database and has big implications for this talk. 04:28.240 --> 04:35.600 So let's continue on and let's actually start talking about an attack scenario and we'll jump to the command line to demonstrate this. 04:35.940 --> 04:37.940 Okay, a bit of a busy diagram. 04:37.940 --> 04:50.980 Focus on what we're assuming is we have access to an endpoint device, a client device, where a GCP admin has been using the CLI, gcloud gsutil. 04:50.980 --> 04:53.660 What we're going to do is access that. 04:53.660 --> 05:08.040 Once we have access, we'll access that and look at the cached accounts, utilize it, access the GCP environment that the admin that user has previously accessed. 05:08.040 --> 05:16.380 Then we're going to copy the credentials, move it off board to a different host under my control, me, the attacker, and then we'll show how to access it from there. 05:17.320 --> 05:22.260 Okay, so I've got a split screen shell environment. 05:22.260 --> 05:23.820 Left is the target host. 05:24.100 --> 05:26.560 The right will be an additional attacker host. 05:26.560 --> 05:27.820 Let me explain a few things. 05:27.820 --> 05:30.300 This is video just recorded in the past day. 05:31.760 --> 05:38.160 I have obfuscated host names, IDs, a lot of unique things, but this was run against a live GCP environment. 05:38.480 --> 05:44.740 We're going through the attack scenario, but just a few notes on the defensive side, which will come into play in the second half of this presentation. 05:45.000 --> 05:49.920 MFA is enabled for the environments that we'll be looking at. 05:49.920 --> 05:52.520 Okay, just keep that in mind. 05:52.740 --> 06:01.220 And initially, it was a number of weeks since access was initially configured for these environments. 06:01.220 --> 06:02.740 So let's jump into it. 06:02.740 --> 06:11.040 If I have access to my target host, I can, of course, run gcloud commands as that user. 06:11.220 --> 06:15.140 And I can check with auth list what accounts are configured. 06:15.140 --> 06:21.260 Two accounts, user dev MFA, admin at prod MFA, named to give us a hint. 06:21.260 --> 06:22.620 One's a production environment. 06:22.740 --> 06:23.440 Others are dev. 06:23.440 --> 06:24.820 Both have MFA enabled. 06:24.820 --> 06:27.580 In fact, there's a hardware key on the prod environment. 06:27.580 --> 06:30.120 There's software MFA on the dev one. 06:31.300 --> 06:38.000 So I've got some comments along the way , but I'm going to highlight the bold white commands. 06:38.000 --> 06:40.940 All right, I can switch accounts with gcloud config set account. 06:40.940 --> 06:43.340 Let's switch the admin prod account. 06:43.340 --> 06:44.680 That's interesting. 06:44.820 --> 06:46.720 All right, switch is over. 06:46.720 --> 06:48.580 Let's see if I can actually access it. 06:48.580 --> 06:50.020 Let me list the projects. 06:50.020 --> 06:50.660 Boom. 06:50.660 --> 06:52.520 Let me list the buckets in the project. 06:52.520 --> 06:53.360 I get a bucket. 06:53.360 --> 06:55.120 Let me enumerate the bucket. 06:55.280 --> 06:57.320 Okay, let's pause here. 06:57.320 --> 07:05.840 Okay, if you're an admin, normally using gcloud, gsutil, this is boring. 07:05.840 --> 07:07.240 This is what you do every day. 07:07.240 --> 07:11.600 What makes this interesting is the attacker perspective within the context of the actual environments. 07:11.600 --> 07:15.440 As I mentioned, the environments have MFA enabled. 07:15.540 --> 07:22.100 And as we'll see as we get further along, these accounts were actually configured quite a while ago. 07:22.740 --> 07:27.820 The production account was added back in June, maybe six, seven weeks ago, right? 07:27.820 --> 07:34.600 So essentially, in this environment, this live environment, I have cached access to a production account. 07:34.600 --> 07:37.000 And as an attacker, it was easy, right? 07:37.000 --> 07:43.000 Just no authentication required six weeks later, seven weeks later, despite things like MFA. 07:43.000 --> 07:44.620 Let's continue on. 07:44.700 --> 07:52.500 All right, so what I want to do now is I want to exfiltrate this whole cache, move it off host, right? 07:52.500 --> 07:56.880 So what I'm going to do is go to the location of cache. 07:56.880 --> 08:03.140 It's in your home directory dot config in a gcloud subdirectory on Unix. 08:03.140 --> 08:06.060 Windows has its own corresponding directory. 08:06.420 --> 08:12.380 I'm going to tar it up, I'll SCP it off board, and then I'm going to switch to the right side. 08:12.380 --> 08:17.840 Okay, so focus on this other host, where I've SCP the tarball. 08:17.840 --> 08:22.160 And what I'm going to do is, so far, I'm just checking my environment. 08:22.160 --> 08:23.780 I haven't done anything yet. 08:23.820 --> 08:24.660 All right. 08:25.800 --> 08:29.820 I'm going to show that there's no accounts configured, right? 08:29.820 --> 08:31.420 It's empty, no credential accounts. 08:31.420 --> 08:36.500 I'm going to take that tarball, unpack it in my own dot config directory. 08:37.580 --> 08:40.720 And boom, right off. 08:40.760 --> 08:43.100 At least the configuration matches, right? 08:43.100 --> 08:45.420 Shows exactly the state. 08:45.420 --> 08:50.120 Now, the note to self right here is, that was easy. 08:50.120 --> 08:50.840 Way too easy. 08:50.840 --> 08:58.220 There's no device-specific encryption, obviously, or any kind of tagging in this data. 08:58.220 --> 09:03.440 So it transparently carries over, and I don't have to do anything except place it in the same directory. 09:03.540 --> 09:05.780 Let's see if it's actually useful and valid. 09:05.780 --> 09:07.020 I can switch accounts. 09:07.020 --> 09:10.100 Again, different host, same credential file copied over. 09:10.100 --> 09:12.480 I can switch accounts to that prod account. 09:12.480 --> 09:17.920 I can list the same production bucket, right? 09:17.920 --> 09:19.820 Way too easy, all right? 09:19.820 --> 09:23.900 The commands, nothing special in the commands. 09:23.900 --> 09:35.740 This is really a lesson in how easy it is to access and exfiltrate and reuse the credentials in bulk for CLI access, which is pretty much gives me everything, all right? 09:36.840 --> 09:47.460 So just to deepen our understanding, a variation of this attack actually makes use of the OAuth tokens in the gcloud subdirectory, all right? 09:48.000 --> 09:54.760 What we're going to do is go through a flow where we actually extract something from the SQLite databases holding the cached tokens. 09:54.760 --> 10:04.720 We're going to use that and perform some of the OAuth flow, which is request a new access token, because most likely that the last access token used has expired. 10:04.720 --> 10:06.620 It only lasts an hour. 10:07.080 --> 10:11.900 Get a new access token, which gives me another hour, and then start making API calls. 10:13.300 --> 10:17.840 Okay, so let's continue on, right? 10:18.640 --> 10:25.640 What I'm going to do, and this is going to get a bear with me as we go through this video and I highlight things, all right? 10:25.640 --> 10:31.600 In the gcloud subdirectory where my token cache is stored, there are two .db files. 10:31.600 --> 10:36.100 They're SQLite database files, unencrypted, nothing special. 10:36.100 --> 10:39.780 What I can do is do a SQLite command, all right? 10:39.780 --> 10:42.980 I'm not going over the schema particularly. 10:43.400 --> 10:51.880 There are sources for that, but there's a simple table that has the last used or last retrieved access token by account. 10:51.880 --> 10:56.300 So I'm pulling the one with associated with the production account. 10:56.300 --> 10:58.160 There it is, the YA-29. 10:58.380 --> 11:00.360 All right, we'll continue on. 11:00.360 --> 11:07.940 Now that I have that, I want to look at the other database, which is credentials.db. 11:07.940 --> 11:10.360 This has more information in a big JSON blob. 11:10.360 --> 11:19.020 I'm also going to look at what's associated with the production admin account here and I have more information. 11:19.020 --> 11:27.800 I have a client ID and secret, which is related to OAuth and the protocol, but it's needed if I want to perform the whole OAuth flow myself. 11:27.800 --> 11:30.140 I have the refresh token, so that's useful. 11:30.140 --> 11:31.200 I need that. 11:31.200 --> 11:33.360 Also, I have access token repeated. 11:33.440 --> 11:38.260 Just note that this information is from my initial authentication configuration of this production account. 11:38.260 --> 11:48.360 So technically, if you take this epoch time with this EXP field, you run it through date and convert it, you'll get a June 17th date. 11:48.440 --> 12:02.040 So what I said is this is a live environment configured six, seven weeks ago and I haven't changed it yet and had default security settings and seven weeks later still have access to it easily. 12:02.380 --> 12:03.980 All right, so let's move on. 12:04.000 --> 12:07.160 Basically, I want to get that refresh token, right? 12:07.160 --> 12:14.820 So I just did a little sedgrep foo on that same SQLite entry with the JSON above and pulled out that access token. 12:15.300 --> 12:21.320 Now I'm just going to do a little cURL REST API calls to go through the OAuth flow, all right? 12:21.820 --> 12:26.660 And the actual cURL command will be shown, but I'm using the refresh token. 12:26.660 --> 12:29.620 So let's see if I can pause this. 12:29.980 --> 12:33.060 Here's this cURL command right here in the screen. 12:33.060 --> 12:40.980 You can see that the client ID and secret and that refresh token are passed in as post parameters. 12:41.240 --> 12:44.160 I've got scope permissions I'm requesting. 12:44.160 --> 12:50.580 The cloud platform one is key, but some other IAM type scope as well. 12:50.580 --> 12:52.580 The endpoint is OAuth. 12:52.680 --> 12:57.360 And what I get back is a fresh new access token right here, okay? 12:57.360 --> 12:59.800 Now, good for an hour. 13:00.760 --> 13:03.060 Now, what can I do with that new access token? 13:03.060 --> 13:05.040 I can do real API calls. 13:05.040 --> 13:06.740 So let's continue on. 13:07.200 --> 13:09.340 I'm going to do another cURL command. 13:09.840 --> 13:13.720 Let's pause it once the output comes back. 13:15.000 --> 13:17.000 All right, here we go. 13:17.000 --> 13:19.740 So what did I do? 13:19.740 --> 13:23.960 Here's the cURL command at the top, right, with that fresh new access token. 13:23.960 --> 13:36.200 I happen to choose a fairly easy API, the storage API, and I'm using the same requesting listing of the same bucket that I had done in the first scenario. 13:36.280 --> 13:39.640 And the JSON that's returned shows that I'm accessing it, right? 13:39.640 --> 13:42.660 Same pass, credit card, sensitive bucket, so on. 13:43.000 --> 13:53.340 So all in all, it just shows that I can do a variation at a deeper level, actually perform the OAuth flow. 13:53.420 --> 14:05.120 And although functionally, this is, in terms of an ATT&CK perspective, it's not that different from the CLI since they're pretty much equivalent to doing direct API calls. 14:05.120 --> 14:14.980 This is more flexible in that I can spin up my own script and code and have somewhat tighter ATT&CK opportunities here. 14:14.980 --> 14:29.560 So the lesson from this is, either way, it's really easy to access the token cache and make use of it, and in a lot of cases, generate new tokens and access what environments have been cached. 14:29.640 --> 14:37.340 So if that's all there were to this, that might be more limiting. 14:37.340 --> 14:46.120 And really, though, there's some related areas with service accounts and compute instances that I want to talk about, right? 14:46.120 --> 14:49.980 More opportunities on the ATT&CK side, more hassle from the defensive viewpoint. 14:49.980 --> 15:03.540 We've been talking a lot about user accounts in the cache, and one might normally think that service accounts wouldn't really be that valuable to look at on a compromised host. 15:03.540 --> 15:04.020 Why? 15:04.020 --> 15:18.280 Because if the target host is external to GCP, most likely what's happened is a service account key file has been generated when you generated a key, and that's been downloaded. 15:18.400 --> 15:24.640 And that JSON file can then be used within API code, but it can also be used within the CLI. 15:24.640 --> 15:33.120 We can activate and add that to our gcloud config and then switch to it as an account and start executing commands as that service account. 15:33.120 --> 15:39.440 But because a key file is very likely still on that client, that's a better target for me as an attacker, right? 15:39.440 --> 15:46.220 Because it's a full-blown credential and I can do anything with that, as opposed to having a temporary OAuth token that expires in an hour. 15:46.220 --> 16:00.140 However, as we'll see later on the defensive side, it is useful to grab this whole cache, including what would be the OAuth session tokens for the service accounts, if they've been configured, because it gives us persistence. 16:00.140 --> 16:19.100 We'll find that a lot of remediation commands that we try with service accounts disable the service account, but they leave the tokens valid, meaning it's a way to have additional access, even though the defender is remediating and revoking the service account access. 16:19.100 --> 16:28.940 All right, so for that reason alone, there's this variation where OAuth tokens for service accounts on an external client are still valuable to grab from an attack perspective. 16:29.980 --> 16:44.120 Speaking of service accounts, well, if they, you know, compute instances run as a service account, to be more secure, there's no key file there if you want to run code on that VM as that service account. 16:44.120 --> 16:55.680 What you do is you query the metadata service and you get an OAuth token that your scripts or code can use, and that way you have no key file that's lying around to be compromised. 16:55.680 --> 16:59.400 So that's an easy REST call. 16:59.720 --> 17:04.860 The URL is internal, resolves to a non-routable local IP. 17:04.860 --> 17:12.620 What we get back is an access token at which we can use in a second API call, as an example of a storage call again. 17:12.620 --> 17:14.180 What does that look like? 17:17.420 --> 17:20.720 Sorry, it's weird. 17:21.860 --> 17:25.900 Okay, sorry, getting a little click problems here. 17:25.900 --> 17:27.060 What does that look like? 17:27.060 --> 17:30.240 Well, you can see the curl command, okay? 17:30.620 --> 17:35.680 What it returns is this access token highlighted in blue, looking at the right side, right? 17:35.800 --> 17:38.060 The expiration date is a little bit less than an hour, basically. 17:38.060 --> 17:44.020 I'm sort of midstream, 400 or so seconds, and I'm getting the cache token. 17:44.180 --> 17:49.180 But at the end of the hour, it automatically refreshes and I can query it again and get a new token. 17:49.520 --> 17:56.060 I can then use that token in a second API call in the storage bucket, boom, it lists it, okay? 17:56.140 --> 17:59.100 So I don't even have to worry about caches, etc. 17:59.100 --> 18:04.360 I can always access the service account OAuth token if I have compromised the compute instance. 18:04.360 --> 18:06.040 And this is really common, right? 18:06.040 --> 18:07.580 Compute instances are targets. 18:07.580 --> 18:14.680 Once you have access, this is one of the first things that we should look at as attackers and worry about as defenders. 18:14.920 --> 18:25.180 Speaking of compute instances, if you have your own compute instances inside the GCP environment, and you install Google Cloud SDK there, everything we've talked about applies there, right? 18:25.180 --> 18:26.660 Makes perfect sense. 18:29.240 --> 18:32.040 Because you have cached tokens as well. 18:32.040 --> 18:34.320 So that becomes a target on the tech side. 18:34.320 --> 18:42.500 But what's maybe not so obvious is that our GCP-managed compute instances, if you run Cloud Shell, the whole purpose of that is sort of ease of use. 18:42.500 --> 18:44.720 You don't have to worry about a VM or the SDK. 18:44.720 --> 18:47.700 Well, the SDK, the latest version is always installed for you. 18:47.700 --> 18:54.520 You now have a nice up-to-date environment to do your CLI commands and manage your environment as a defender . 18:54.520 --> 18:59.180 Well, from an attacker perspective, that becomes a target as well, because the SDK is installed. 18:59.360 --> 19:10.080 There's been some good work, and I recommend if you're interested in this particular area, read Juan Berner's post a year and some ago on backdoors with Google Cloud Shell. 19:10.160 --> 19:20.140 Basically, you can install a backdoor shell, you know, one-liner, you can exfiltrate the token cache automatically. 19:20.140 --> 19:24.400 So every time someone starts up the Cloud Shell, you get the fresh sort of token dump. 19:25.820 --> 19:27.920 And you can move on your merry way. 19:27.920 --> 19:29.480 One note about Cloud Shell. 19:30.000 --> 19:30.920 Well, how do you access that? 19:30.920 --> 19:34.820 You can do it through the Google console, but you can also do it through a gcloud command, right? 19:34.820 --> 19:36.880 So you can think that this could be a chain command. 19:36.880 --> 19:53.260 If I compromise your endpoint, that's my target outside of GCP, and you use Cloud Shell, I may be able to get into your Cloud Shell environment, install a backdoor, and then continue to have access from this compute instance sort of environment. 19:53.940 --> 19:55.980 So this complicates things, right? 19:55.980 --> 19:58.500 These are all sort of variations. 19:58.620 --> 20:04.580 We're not just talking about your endpoint where you normally do Google Cloud Admin. 20:04.740 --> 20:07.820 All right, we're about halfway through. 20:07.820 --> 20:10.980 Let's focus to the defensive side. 20:10.980 --> 20:12.860 We have a lot to talk about here. 20:13.400 --> 20:17.120 Basically, from a prevention viewpoint, put on your blue hats. 20:17.120 --> 20:21.160 There are three basic things that I think will help. 20:21.160 --> 20:24.200 It is hard, right, to prevent credential compromise. 20:24.200 --> 20:25.400 You really can't. 20:25.400 --> 20:28.280 How do you know it's compromised versus the normal user? 20:28.320 --> 20:29.620 But we can mitigate it. 20:29.620 --> 20:32.480 So the number one thing to do is expiration time. 20:32.480 --> 20:37.480 IP whitelisting, if you can manage that, is helpful, as well as MFA. 20:38.120 --> 20:43.240 And there are some additional topics regarding IP whitelisting that we need to touch base on. 20:43.240 --> 20:45.900 Okay, so let's talk about cloud session duration. 20:45.900 --> 20:51.260 On the G Suite admin side, which is, you know, the flag already. 20:51.260 --> 20:57.120 Anytime I say G Suite admin, and you're a GCP admin, your ear should sort of perk up. 20:57.340 --> 20:58.820 It's a different console. 20:58.820 --> 21:01.600 If you're a bigger organization, that may be a different team. 21:01.600 --> 21:04.460 If you have access to it as a GCP admin, great. 21:04.460 --> 21:05.740 This will be familiar to you. 21:05.740 --> 21:09.640 But it's a beta feature, about a year old, if I'm correct on that. 21:09.640 --> 21:11.720 So people may not know about it. 21:11.800 --> 21:13.740 This is one of the most useful things. 21:13.740 --> 21:20.540 By default, sessions for SDK access do not expire, never expires. 21:20.540 --> 21:26.060 In fact, in the earlier attack scenario, in the prod environment we went into, guess what the default was set to? 21:26.200 --> 21:28.120 The default was never expires. 21:28.120 --> 21:29.380 It was never changed. 21:29.380 --> 21:31.700 You can set it to one to 24 hours. 21:32.060 --> 21:39.040 Recommend really short, like one hour for prod environments, maybe eight plus hours for your dev environments, right? 21:39.040 --> 21:54.640 When it expires, all access, both console and programmatic, all those sessions are revoked, and anyone accessing the environment has to actually re-authenticate, all right? 21:54.640 --> 21:58.760 So this shrinks the window during which you can be compromised. 22:00.020 --> 22:02.080 Now, there is a re-authentication method. 22:02.080 --> 22:03.120 You have two choices. 22:03.120 --> 22:09.460 When they're re-authenticating after the expiration, a password or security key. 22:09.460 --> 22:10.900 This is hardware security key. 22:10.900 --> 22:23.860 The only thing to note here is if you use something in the middle, like a software MFA, which I have on one of my dev environments, you have to pick a lesser, a less strong method, which is password. 22:24.280 --> 22:37.180 Just a note, it means your re-authentication is less secure because it's no longer MFA, and opens up to keyboard logging and losing a password credential. 22:38.840 --> 22:43.380 Let's talk about IP whitelisting, the second thing on my list, all right? 22:43.380 --> 22:51.980 So still a relatively new feature, but you can do VPC service controls to implement IP whitelist. 22:51.980 --> 23:01.080 In the Access Context Manager in Google Cloud, you create a whitelist, specify a bunch of ciders and set up a policy. 23:01.120 --> 23:03.900 Please don't be thrown that there's a 1918 address in here. 23:03.900 --> 23:06.260 That's my obfuscation in play. 23:06.260 --> 23:11.640 This is normally your public IP or cider that you want in your whitelist. 23:12.380 --> 23:27.340 Switching over to the VPC service control section, you can specify the APIs and services, everything as an example, attach and reference the actual access level with the whitelist IPs, put it in place. 23:27.500 --> 23:36.840 And the next time that someone tries to do a command to the resources inside that perimeter, VPC perimeter, they'll get an error, right? 23:36.840 --> 23:40.720 So a different host, different IP, it's not in the whitelist. 23:40.960 --> 23:42.540 All good, right? 23:42.540 --> 23:43.640 Really good. 23:46.680 --> 23:50.540 What's of interest to note is you get sort of a verbose message. 23:50.540 --> 23:56.240 Not sure what I think about that from a defensive side, sort of unsettles me, but it's clear that it works. 23:56.860 --> 24:01.800 There is more complication when you think about IP whitelisting. 24:01.800 --> 24:06.180 It's not just users and what devices they're coming from or what IPs rather. 24:06.800 --> 24:15.700 You can manage that, try to enforce VPNs, space that you know, proxies, but you also have to worry about VM instances. 24:15.700 --> 24:19.160 And here's where it's harder lifting. 24:19.160 --> 24:29.600 Essentially, if you go IP whitelisting, you want to make sure that your VM IPs are also whitelisted so that any API calls that they do are allowed, right? 24:29.600 --> 24:31.840 And so there's a couple things to worry about here. 24:31.840 --> 24:33.700 It's just the maintenance issue. 24:33.860 --> 24:46.340 You could hook into the startup and provisioning of your VMs to make sure that the VMs IP addresses are included automatically in your IP list, especially if you have a lot of VMs or dynamic IPs. 24:48.100 --> 24:55.240 Detection, it's not prevention, but a side note, could be done based on logs instead of an explicit whitelist, right? 24:55.240 --> 24:59.520 And some of the references below sort of talk about that as a technique, right? 24:59.520 --> 25:13.880 So IP whitelisting sort of alternatives are cull from logs, the IPs of VMs that have been started and see if there's anything abnormal in it compared to a whitelist or instead of a whitelist. 25:14.440 --> 25:26.100 You can hook in and put in place a man-in-the-middle proxy for metadata, a metadata service, which would give you visibility to anyone querying it for OAuth tokens. 25:26.140 --> 25:30.920 And that might be useful to track and determine and see if that's worth monitoring and learning on. 25:30.920 --> 25:34.400 Okay, so this is also a bit of a detection piece. 25:34.500 --> 25:44.020 But the main point here from a prevention perspective is you have to think about VM instances and ensure that they are whitelisted properly if you go down that route. 25:44.020 --> 25:56.460 So there is some work to be done, but if you can accomplish it, then IP whitelisting is very effective at helping you mitigate the issues with any kind of compromised credentials. 25:56.600 --> 26:02.440 And I'm going to do a little shout out to the Netflix security team, and I have some references there. 26:02.440 --> 26:05.000 And they've blogged quite a while about the work they've done in AWS. 26:05.000 --> 26:10.740 A lot of their concepts apply over to GCP and are worth thinking about as we go. 26:10.820 --> 26:14.660 Once you have the whitelist in place, of course, you have to make sure it's enforced. 26:14.660 --> 26:16.640 So you can do a little CLI foo. 26:16.640 --> 26:25.280 And on the right, you can see the definitions of your access levels and policies and check the IPs and make sure they're the right ones. 26:25.440 --> 26:32.900 You can check the VPC service parameters and make sure that they're accessing access levels as well. 26:32.900 --> 26:38.640 So it's all well and good to have a whitelist, just make sure it's actually enforced and put in place. 26:38.920 --> 26:42.460 Let's move to the third area of prevention, MFA. 26:42.500 --> 26:48.920 On the G Suite admin side, yellow flag, if you don't have access, is where MFA is controlled. 26:48.960 --> 26:53.200 Set it as much and enforce it with your admin users. 26:53.200 --> 27:00.140 It'll help mitigate that compromise of credentials during any re-authentication. 27:01.020 --> 27:05.020 Okay, let's move to detection. 27:05.300 --> 27:08.440 This is a shorter discussion, unfortunately. 27:08.440 --> 27:14.660 Behavioral detection is really the problem for compromised credentials. 27:14.660 --> 27:26.160 It's difficult, discern differences from valid user actions and behavior unless they do something egregious, like exfiltrate some terabytes. 27:26.920 --> 27:28.940 I'm not going to talk about that. 27:28.940 --> 27:42.500 I want to talk about if you do or are able to implement IP whitelisting effectively, effectively with low false positives, detecting failed authentications is worth doing, right? 27:42.500 --> 27:49.920 And Stackdriver on the right will get a nice entry that we can key off of and monitor. 27:49.980 --> 28:00.700 And if we have low enough false positives and it's maintained, it's a good way to get a heads up that there may be a compromised credential being used from an IP. 28:00.700 --> 28:08.880 This would cover the scenario where I exfiltrate credentials or tokens and try to use this from an IP that I control, right? 28:08.880 --> 28:12.780 The old style way of tacking would be to always do that. 28:12.780 --> 28:24.920 If people focus on whitelisting from a defensive viewpoint, it would be interesting to see if, from a tech perspective, we did more follow-up work from the compromised host, because that would still be allowed in this scenario. 28:24.920 --> 28:27.340 So we're not dealing with perfection. 28:27.340 --> 28:30.040 We're just trying to improve our ability to detect. 28:31.280 --> 28:35.260 Okay, we've got to spend some time on remediation, okay? 28:35.260 --> 28:37.540 This is complicated. 28:37.540 --> 28:39.120 Let me see if I can get through this. 28:39.120 --> 28:42.240 There's a basically remediation. 28:42.240 --> 28:44.000 I think there's two major things we have to worry about. 28:44.000 --> 28:46.520 How do we lock out the account, prevent future access? 28:46.660 --> 28:53.840 How do we revoke current access, the current sessions, which are the OAuth access tokens and refresh tokens? 28:54.260 --> 28:56.160 User and service accounts are different. 28:56.920 --> 29:01.860 So here are some of the commonly available or known options. 29:01.860 --> 29:03.380 Let's go through them one by one. 29:03.420 --> 29:06.440 It's going to be a little bit of remediation bingo, all right? 29:06.440 --> 29:12.680 As we go through this, see if you can pick out or suggest the best option, and I'll tell you mine. 29:13.860 --> 29:14.940 All right, user accounts. 29:14.940 --> 29:16.340 How do we lock out user accounts? 29:16.340 --> 29:24.180 Well, G Suite Admin, okay, G Suite Admin, we can reset password, suspend the user, and reactivate it later, delete the user. 29:24.180 --> 29:25.140 They're all effective. 29:25.140 --> 29:27.860 They all lock out the user or attacker. 29:27.960 --> 29:31.100 We'll pick the lowest impact one. 29:31.620 --> 29:33.900 Resetting password, pretty common, works. 29:33.900 --> 29:35.000 That's the answer. 29:35.880 --> 29:37.720 How about service accounts? 29:37.720 --> 29:39.880 Similar, parallel set of options. 29:40.260 --> 29:43.620 I can delete the key here. 29:43.620 --> 29:46.620 I'm in Google Cloud Console for service accounts, right? 29:46.620 --> 29:50.540 G Suite Admin for user accounts, Google Cloud for service accounts. 29:50.540 --> 29:56.120 I can disable the service account itself, and I can delete the service account. 29:56.180 --> 29:57.420 Which one? 29:57.420 --> 29:58.700 They all work. 29:58.740 --> 30:00.320 We'll pick the lowest one. 30:00.320 --> 30:05.840 Rotate, right? 30:05.840 --> 30:07.780 Now, let's get to the tough part. 30:07.780 --> 30:12.760 The tough part is revoking current access, and that's the topic of this talk, right? 30:13.000 --> 30:21.240 So, let's see of these six suggested options, the last three are the lockout account measures. 30:21.240 --> 30:26.340 The first three are potentially common ones that people try or think about. 30:26.340 --> 30:28.240 Let's go through them one by one. 30:28.240 --> 30:36.100 In G Suite Admin, for a particular user, their security section, there is a sign-in cookies and a reset action. 30:36.100 --> 30:37.060 What does this do? 30:37.060 --> 30:41.000 It revokes all web sessions immediately. 30:41.000 --> 30:43.840 This includes Google Cloud Console. 30:43.840 --> 30:47.040 It would also include G Suite, okay? 30:47.040 --> 30:48.000 This works. 30:48.000 --> 30:48.880 This is good. 30:48.920 --> 30:51.880 Include any Google app and service. 30:52.300 --> 30:56.840 This is great, except it doesn't do anything about SDK sessions. 30:56.840 --> 30:58.860 So, actually, it's only half of a solution. 30:58.920 --> 31:00.400 So, we lose. 31:00.400 --> 31:01.620 Let's keep going. 31:01.620 --> 31:06.480 There's an interesting G Cloud CLI command that you have probably seen or used, auth revoke. 31:06.480 --> 31:14.360 It takes an account name, and it will revoke both the access and refresh tokens associated for that user account. 31:14.360 --> 31:22.080 It, underneath, calls an API call with this endpoint to revoke the specific token. 31:22.080 --> 31:24.360 That also works if you want to call it directly. 31:24.360 --> 31:26.140 Sounds great, doesn't it? 31:26.140 --> 31:30.740 But it's in the middle of my list, so you know that it actually doesn't work. 31:31.160 --> 31:32.540 Here's the problem. 31:33.140 --> 31:35.520 It can only be run on the G Cloud client machine. 31:35.520 --> 31:35.900 Why? 31:35.900 --> 31:41.440 Because you pass in an account name, and it looks in the SQLite database cache to find the token. 31:41.860 --> 31:48.100 And that is easily manipulated by the attacker, at a minimum, deleted, right? 31:48.100 --> 31:51.640 So, it can't be run anywhere else. 31:51.640 --> 31:57.160 Frankly, I don't think any incident responder would trust a compromised client machine for anything, right? 31:58.260 --> 32:03.880 Similarly, calling the API itself doesn't help. 32:03.880 --> 32:04.680 Why? 32:04.800 --> 32:10.780 You need to pass in an OAuth access token or revoke, sorry, refresh token. 32:11.940 --> 32:13.340 Those are not logged. 32:13.340 --> 32:14.620 They're not accessible. 32:14.620 --> 32:21.600 They don't exist anywhere except on the client machine, which is compromised, which is easily manipulated by the attacker. 32:21.600 --> 32:23.840 So, it's useless. 32:23.840 --> 32:28.400 These are for self-help, for you to clean up your cache and config, nothing else. 32:28.560 --> 32:33.560 A one side note, which is a diversion, but the call to revoke token doesn't require authentication. 32:33.560 --> 32:37.200 You just pass in this long access and refresh token. 32:37.200 --> 32:42.380 It's long enough that you probably can't brute force it, but it's an interesting note that you don't need OAuth to call. 32:42.720 --> 32:53.000 Okay, let's cover that lockout service account, sorry, user lockout options, all right? 32:53.000 --> 32:54.840 Changing the user password. 32:55.020 --> 32:56.480 These all don't work. 32:56.480 --> 32:58.800 They don't revoke SDK sessions. 32:58.800 --> 33:07.340 There's a narrow case where if you have requested scope for Gmail, then resetting the password would work, but that doesn't apply to us. 33:07.340 --> 33:13.080 We would have requested scope probably for just gcloud or SDK access. 33:13.280 --> 33:15.800 Suspending user account works temporarily. 33:15.800 --> 33:19.840 While the user account is suspended, the tokens will not work. 33:19.840 --> 33:25.380 Once, though, it's re-enabled, the tokens will start working again, assuming they have not expired. 33:25.380 --> 33:27.640 So, that's not a great solution. 33:27.640 --> 33:30.720 Unless you suspend the account for an hour, that would work. 33:32.600 --> 33:35.340 Sorry, let me backtrack on that. 33:36.060 --> 33:39.280 Deleting user account works, but has high impact, okay? 33:39.280 --> 33:41.020 So, these options aren't great. 33:41.020 --> 33:42.520 So, this is all yellow. 33:42.520 --> 33:45.060 The question is, what's our best option? 33:45.060 --> 33:58.200 So, one thing I did not list in the table was there's actually the best option that works and is low impact is to delete the connected application. 33:58.200 --> 34:01.800 This is the connected OAuth application in the G Suite admin side. 34:01.840 --> 34:14.680 Basically, every time you authenticate and request access, the OAuth application, in this case, we're talking about the Google Cloud SDK, is listed here, as well as this access level, and you can delete it. 34:14.680 --> 34:16.680 And deleting it has no big ramifications. 34:16.680 --> 34:23.560 It just means next time someone accesses Google Cloud SDK, like through CLI, they're forced to re-authenticate. 34:23.560 --> 34:27.240 But once you delete it, at that point in time, all session tokens are revoked. 34:27.240 --> 34:29.120 So, it's the answer. 34:29.120 --> 34:29.960 That's the bingo. 34:29.960 --> 34:33.400 That's the bonus right there, okay? 34:33.400 --> 34:40.100 And if you haven't worked with G Suite admin, you wouldn't know about it, or maybe think about it, or if you're not an OAuth jockey, right? 34:40.400 --> 34:43.600 In OAuth land, this is pretty common to think about these connections. 34:43.600 --> 34:44.620 It just means it's trusted. 34:44.620 --> 34:46.940 And once you delete it, it revokes the sessions. 34:47.180 --> 34:47.940 All right. 34:48.260 --> 34:50.220 Let's talk about the service account. 34:51.540 --> 34:52.960 How do we revoke that? 34:52.960 --> 34:54.500 Well, do we revoke it? 34:54.500 --> 34:58.140 Do we do some of these other lockout measures? 34:58.300 --> 35:00.020 Let's jump into it. 35:00.040 --> 35:06.620 Well, you might have guessed that the revoke CLI and the API call won't work, because we just went through it for the user account. 35:07.280 --> 35:08.440 It definitely doesn't work. 35:08.440 --> 35:12.360 You actually get errors here, right, when you try to do the CLI or the API. 35:12.360 --> 35:12.980 Outright errors. 35:12.980 --> 35:14.420 It just says it doesn't apply. 35:14.420 --> 35:20.600 But what's of note is, look in the error, sorry for the verbiage, but look at the error you get back from the CLI from gcloud. 35:20.660 --> 35:23.200 It actually hints in the last sentence that there's hope. 35:23.960 --> 35:31.180 If you revoke the parent service account, it's supposed to revoke the session, the actual sessions. 35:31.180 --> 35:31.980 What does that mean? 35:31.980 --> 35:35.540 Well, if you suspend it or delete it, that's true. 35:35.640 --> 35:40.900 But the suspend is sort of a half answer, because once you re-enable it, the tokens actually still work. 35:40.900 --> 35:42.340 What about the service account key? 35:42.340 --> 35:43.260 Revoke it? 35:43.280 --> 35:43.920 What's that mean? 35:43.920 --> 35:46.400 Well, it means delete the API key. 35:46.760 --> 35:48.020 That definitely doesn't work. 35:48.020 --> 35:48.620 That's a bug. 35:48.620 --> 35:50.080 Bug has been filed. 35:50.080 --> 35:51.520 Don't believe it, right? 35:52.720 --> 35:54.040 We've already gone through this. 35:54.040 --> 35:55.360 This doesn't work. 35:56.020 --> 35:59.920 How about the measures to rotate and delete the API key? 35:59.920 --> 36:10.600 Look, if you rotate, delete the API key, you can still use OAuth tokens and still access the service login and do actions as the service account. 36:10.800 --> 36:15.680 If you disable it, at that point in time, the tokens don't work, but they still exist. 36:15.680 --> 36:20.860 So once you re-enable, reactivate the service account, they'll work again. 36:20.860 --> 36:27.040 So the solution there is you could suspend for an hour, then re-enable it, because in that hour, tokens will expire. 36:27.200 --> 36:28.440 How about deleting service account? 36:28.440 --> 36:31.520 Works, just like deleting user account, high impact. 36:31.980 --> 36:40.180 So the summary is actually disable for an hour or delete and reprovision. 36:40.240 --> 36:43.740 You will choose whichever you're more comfortable with. 36:43.920 --> 36:50.480 Whichever gets you back up and running quicker, that you can perform with confidence, without making errors. 36:50.560 --> 36:54.140 This isn't great, but there are ways forward, right? 36:54.140 --> 37:03.440 And the main thing of this whole slide and the time spent on all the paths that weren't great paths or were dead ends is to make sure we don't waste time on that, especially in the middle of an incident, right? 37:03.440 --> 37:15.020 Is to feed this and review our operations playbooks and make sure that we have the right actions for remediation for the different types of accounts. 37:15.780 --> 37:23.620 So to wrap up in the next minute or so and at the end, we went over a bunch of attack scenarios, right? 37:23.620 --> 37:26.560 Definitely user accounts and the cache on an external client. 37:26.560 --> 37:28.300 That's really what we focused on. 37:28.300 --> 37:41.400 But don't forget that service accounts also cache the access token and that an attacker would want to grab that because it helps them maintain persistence in the light of all those remediation actions that don't quite work. 37:41.740 --> 37:43.720 But it's not just external clients. 37:43.720 --> 37:50.780 If you switch to a Compute Engine environment and you've installed SDK, all the same things apply. 37:50.780 --> 37:53.320 And it's not just things that you explicitly start. 37:53.320 --> 38:00.300 Remember to think about Cloud Shell, which starts up a VM in the back with the latest SDK installed, right? 38:00.300 --> 38:07.420 These are all having the same cache token kinds of exposure and opportunity. 38:07.700 --> 38:15.040 And fifth, last but not least, the fifth area is even if SDK is not installed, right? 38:15.040 --> 38:27.560 Compute Engine compromise, OAuth tokens for that service account are easily queried from the metadata service and can allow the attacker access as well as persistence for that access. 38:27.560 --> 38:35.360 And from a defensive measure, real quick summary after that long sort of part of the presentation, do three things on the prevention side. 38:35.360 --> 38:38.420 The no-brainer is set your session timeout in G Suite. 38:38.460 --> 38:48.980 If you can, implement IP whitelisting with service controls and set your MFA so that these all mitigate credential compromise in some way. 38:49.100 --> 38:55.120 And if you do do that IP whitelisting, then you can improve your detection by detecting the failed authorizations. 38:55.120 --> 39:02.240 From a remediation perspective for user accounts, reset the password to lock it out and delete the OAuth connected app in G Suite. 39:02.240 --> 39:05.040 That's the key one that may not be obvious to everyone. 39:05.040 --> 39:07.000 For service accounts, pick your poison. 39:07.000 --> 39:12.860 One of two, disable reactivate within our time period or delete, recreate, reprovision. 39:12.860 --> 39:17.360 If you have that automated, just make sure and want to get up and running faster. 39:17.360 --> 39:22.500 Just have to worry about VMs, compute instances, anything using those service accounts. 39:23.120 --> 39:30.760 So that is the end and I will sort of ask Jaron to see if we have any time for questions. 39:30.760 --> 39:33.980 Otherwise, you can always reach me directly as well. 39:33.980 --> 39:39.500 Yeah, we have one question from Eric. 39:39.560 --> 39:49.400 Do you know of any blogs or white papers that people can use to replicate the OAuth token hijacking that you showed earlier? 39:51.880 --> 39:59.240 Yes, so there are some resources for sure. 39:59.240 --> 40:05.900 We will be blogging over the next week or two a lot of the details from this presentation. 40:05.900 --> 40:12.720 So you'll actually be able to recreate what's seen in the video with actual commands. 40:12.720 --> 40:14.800 But there's been prior work as well. 40:14.800 --> 40:24.560 So if you search on G Cloud, I want to say if I have the name right, I didn't reference in the presentation, I'll have to add that later. 40:24.560 --> 40:34.320 But John Haney has, I believe I have his name right, has done some early work on just peeling apart the SQL databases and some of the REST API calls. 40:34.620 --> 40:36.580 And that's a useful resource. 40:37.140 --> 40:42.220 It's not documented well, particularly, but it's also not hard to delve into it. 40:42.220 --> 40:57.640 So both our blogs should be released in part, I think today, as well as over the next two weeks, as well as there's some prior work if you search on some of the keywords like the G Cloud, the dot config, the SQL light database names.