1 00:00:00,000 --> 00:00:04,220 As you mentioned, we're going to talk about hijacking OAuth tokens in Google Cloud. 2 00:00:05,060 --> 00:00:10,840 Good thing about Safe Mode this year is if this is the wrong talk, you can quietly leave and I won't even see you. 3 00:00:10,840 --> 00:00:14,460 All right, let's get into it here. 4 00:00:14,880 --> 00:00:18,960 So, if you need to get hold of me, just DM me at Twitter. 5 00:00:18,960 --> 00:00:21,820 You can usually find me on most of the social platforms. 6 00:00:22,060 --> 00:00:34,340 What we're going to cover in this talk is hijacking of OAuth tokens, session tokens, for CLI access, API access, and it's really the ease of attack that we're going to focus on. 7 00:00:34,340 --> 00:00:41,740 There's additional use cases related to service accounts and compute instances that we'll look at as well. 8 00:00:41,740 --> 00:00:48,240 And from a defensive perspective, we're definitely going to drill into some of the challenges in securing the environment. 9 00:00:48,820 --> 00:00:51,240 Really, who cares, right? 10 00:00:51,240 --> 00:00:54,380 Well, OAuth is fundamental to Google authentication. 11 00:00:54,380 --> 00:00:56,140 It's underneath everywhere. 12 00:00:56,140 --> 00:01:13,720 There are some nuances depending on if you are authenticating with user accounts versus service accounts, whether your sessions are from browsers versus SDK access, whether you're external to the GCP environment versus internal, say, on a compute VM. 13 00:01:13,720 --> 00:01:21,200 And that is part of what leads to a lot of the opportunity from the tech perspective, as well as the challenges from the defensive perspective. 14 00:01:21,320 --> 00:01:24,780 Essentially, the session tokens are what the name implies. 15 00:01:24,780 --> 00:01:28,860 They're post-authentication representing a valid authentication session. 16 00:01:28,860 --> 00:01:35,380 And it's really an opportunity for us to hijack and use them and gain access to existing authenticated sessions. 17 00:01:35,740 --> 00:01:45,860 And if I, as an attacker, have access to an endpoint, say a GCP administrator's device, those cached accounts and environments are easily accessible to me. 18 00:01:45,860 --> 00:01:47,680 And that's what we're going to go through. 19 00:01:48,380 --> 00:01:56,500 At the end of the day, it's all about persistence, as well as evading detection from the attacker's perspective. 20 00:01:56,560 --> 00:02:02,220 Now, the defensive side, what is interesting are some of the challenges. 21 00:02:02,220 --> 00:02:11,500 Some of the controls such as MFA don't quite help, although you might counterintuitively, you might think they do, right? 22 00:02:11,500 --> 00:02:12,820 And we'll go through that. 23 00:02:12,820 --> 00:02:23,220 And in terms of securing the environment, there's a lot of options that are confusing, easy to misconfigure, and also easily misunderstood. 24 00:02:23,380 --> 00:02:27,360 So at the end of the day, we're talking about how do you handle incident response? 25 00:02:27,360 --> 00:02:29,420 What's your playbook for ops? 26 00:02:29,460 --> 00:02:34,920 Is it fully up to date with the most optimal measures or controls? 27 00:02:37,680 --> 00:02:42,440 So let's start with just a brief overview of OAuth. 28 00:02:42,440 --> 00:02:47,980 Okay, just really briefly, there's some verbiage on this slide, but focus on the right hand diagram. 29 00:02:47,980 --> 00:02:52,380 At a very high level, there's a simplified flow, no matter how you authenticate. 30 00:02:54,120 --> 00:03:01,020 First, you'll request a token, it'll be signed with your credentials, you'll request scopes or permissions. 31 00:03:01,020 --> 00:03:06,960 And as a user, you often see this when you first authenticate with gcloud or CLI. 32 00:03:07,120 --> 00:03:15,660 By default, a browser is launched, you have to go type in your Google credentials, you also have to approve the actual scopes being requested. 33 00:03:15,900 --> 00:03:31,300 And if that goes properly, then what is returned ultimately to the requester, in this case, it'd be the CLI, would be two important things, the access session token, that's an OAuth session token, as well as a refresh token. 34 00:03:31,340 --> 00:03:34,280 The access token is good for an hour. 35 00:03:34,360 --> 00:03:39,580 By default, you use it directly in API calls, they all take OAuth tokens. 36 00:03:40,480 --> 00:03:43,940 After the hour is up, you can refresh and get a new one. 37 00:03:44,180 --> 00:03:51,540 So gcloud, gsutil, CLI tools do this automatically, you usually don't see this level of the protocol in the handshake. 38 00:03:52,100 --> 00:03:57,940 The local language SDK libraries like Python, etc, also do this transparently. 39 00:03:57,940 --> 00:04:00,320 So generally, your code is not at this level. 40 00:04:00,320 --> 00:04:07,160 If you do direct REST API calls, you will see this handshake and have to go through two or three back and forths. 41 00:04:07,160 --> 00:04:17,540 The other important thing to remember about OAuth, besides this handshake and the access and refresh tokens, is that, well, you don't authenticate every time you use the CLI, right? 42 00:04:17,540 --> 00:04:18,780 It's cached. 43 00:04:18,780 --> 00:04:23,240 All session tokens almost everywhere in every model and architecture are cached. 44 00:04:23,240 --> 00:04:28,020 In this case, it's in the SQLite database and has big implications for this talk. 45 00:04:28,240 --> 00:04:35,600 So let's continue on and let's actually start talking about an attack scenario and we'll jump to the command line to demonstrate this. 46 00:04:35,940 --> 00:04:37,940 Okay, a bit of a busy diagram. 47 00:04:37,940 --> 00:04:50,980 Focus on what we're assuming is we have access to an endpoint device, a client device, where a GCP admin has been using the CLI, gcloud gsutil. 48 00:04:50,980 --> 00:04:53,660 What we're going to do is access that. 49 00:04:53,660 --> 00:05:08,040 Once we have access, we'll access that and look at the cached accounts, utilize it, access the GCP environment that the admin that user has previously accessed. 50 00:05:08,040 --> 00:05:16,380 Then we're going to copy the credentials, move it off board to a different host under my control, me, the attacker, and then we'll show how to access it from there. 51 00:05:17,320 --> 00:05:22,260 Okay, so I've got a split screen shell environment. 52 00:05:22,260 --> 00:05:23,820 Left is the target host. 53 00:05:24,100 --> 00:05:26,560 The right will be an additional attacker host. 54 00:05:26,560 --> 00:05:27,820 Let me explain a few things. 55 00:05:27,820 --> 00:05:30,300 This is video just recorded in the past day. 56 00:05:31,760 --> 00:05:38,160 I have obfuscated host names, IDs, a lot of unique things, but this was run against a live GCP environment. 57 00:05:38,480 --> 00:05:44,740 We're going through the attack scenario, but just a few notes on the defensive side, which will come into play in the second half of this presentation. 58 00:05:45,000 --> 00:05:49,920 MFA is enabled for the environments that we'll be looking at. 59 00:05:49,920 --> 00:05:52,520 Okay, just keep that in mind. 60 00:05:52,740 --> 00:06:01,220 And initially, it was a number of weeks since access was initially configured for these environments. 61 00:06:01,220 --> 00:06:02,740 So let's jump into it. 62 00:06:02,740 --> 00:06:11,040 If I have access to my target host, I can, of course, run gcloud commands as that user. 63 00:06:11,220 --> 00:06:15,140 And I can check with auth list what accounts are configured. 64 00:06:15,140 --> 00:06:21,260 Two accounts, user dev MFA, admin at prod MFA, named to give us a hint. 65 00:06:21,260 --> 00:06:22,620 One's a production environment. 66 00:06:22,740 --> 00:06:23,440 Others are dev. 67 00:06:23,440 --> 00:06:24,820 Both have MFA enabled. 68 00:06:24,820 --> 00:06:27,580 In fact, there's a hardware key on the prod environment. 69 00:06:27,580 --> 00:06:30,120 There's software MFA on the dev one. 70 00:06:31,300 --> 00:06:38,000 So I've got some comments along the way , but I'm going to highlight the bold white commands. 71 00:06:38,000 --> 00:06:40,940 All right, I can switch accounts with gcloud config set account. 72 00:06:40,940 --> 00:06:43,340 Let's switch the admin prod account. 73 00:06:43,340 --> 00:06:44,680 That's interesting. 74 00:06:44,820 --> 00:06:46,720 All right, switch is over. 75 00:06:46,720 --> 00:06:48,580 Let's see if I can actually access it. 76 00:06:48,580 --> 00:06:50,020 Let me list the projects. 77 00:06:50,020 --> 00:06:50,660 Boom. 78 00:06:50,660 --> 00:06:52,520 Let me list the buckets in the project. 79 00:06:52,520 --> 00:06:53,360 I get a bucket. 80 00:06:53,360 --> 00:06:55,120 Let me enumerate the bucket. 81 00:06:55,280 --> 00:06:57,320 Okay, let's pause here. 82 00:06:57,320 --> 00:07:05,840 Okay, if you're an admin, normally using gcloud, gsutil, this is boring. 83 00:07:05,840 --> 00:07:07,240 This is what you do every day. 84 00:07:07,240 --> 00:07:11,600 What makes this interesting is the attacker perspective within the context of the actual environments. 85 00:07:11,600 --> 00:07:15,440 As I mentioned, the environments have MFA enabled. 86 00:07:15,540 --> 00:07:22,100 And as we'll see as we get further along, these accounts were actually configured quite a while ago. 87 00:07:22,740 --> 00:07:27,820 The production account was added back in June, maybe six, seven weeks ago, right? 88 00:07:27,820 --> 00:07:34,600 So essentially, in this environment, this live environment, I have cached access to a production account. 89 00:07:34,600 --> 00:07:37,000 And as an attacker, it was easy, right? 90 00:07:37,000 --> 00:07:43,000 Just no authentication required six weeks later, seven weeks later, despite things like MFA. 91 00:07:43,000 --> 00:07:44,620 Let's continue on. 92 00:07:44,700 --> 00:07:52,500 All right, so what I want to do now is I want to exfiltrate this whole cache, move it off host, right? 93 00:07:52,500 --> 00:07:56,880 So what I'm going to do is go to the location of cache. 94 00:07:56,880 --> 00:08:03,140 It's in your home directory dot config in a gcloud subdirectory on Unix. 95 00:08:03,140 --> 00:08:06,060 Windows has its own corresponding directory. 96 00:08:06,420 --> 00:08:12,380 I'm going to tar it up, I'll SCP it off board, and then I'm going to switch to the right side. 97 00:08:12,380 --> 00:08:17,840 Okay, so focus on this other host, where I've SCP the tarball. 98 00:08:17,840 --> 00:08:22,160 And what I'm going to do is, so far, I'm just checking my environment. 99 00:08:22,160 --> 00:08:23,780 I haven't done anything yet. 100 00:08:23,820 --> 00:08:24,660 All right. 101 00:08:25,800 --> 00:08:29,820 I'm going to show that there's no accounts configured, right? 102 00:08:29,820 --> 00:08:31,420 It's empty, no credential accounts. 103 00:08:31,420 --> 00:08:36,500 I'm going to take that tarball, unpack it in my own dot config directory. 104 00:08:37,580 --> 00:08:40,720 And boom, right off. 105 00:08:40,760 --> 00:08:43,100 At least the configuration matches, right? 106 00:08:43,100 --> 00:08:45,420 Shows exactly the state. 107 00:08:45,420 --> 00:08:50,120 Now, the note to self right here is, that was easy. 108 00:08:50,120 --> 00:08:50,840 Way too easy. 109 00:08:50,840 --> 00:08:58,220 There's no device-specific encryption, obviously, or any kind of tagging in this data. 110 00:08:58,220 --> 00:09:03,440 So it transparently carries over, and I don't have to do anything except place it in the same directory. 111 00:09:03,540 --> 00:09:05,780 Let's see if it's actually useful and valid. 112 00:09:05,780 --> 00:09:07,020 I can switch accounts. 113 00:09:07,020 --> 00:09:10,100 Again, different host, same credential file copied over. 114 00:09:10,100 --> 00:09:12,480 I can switch accounts to that prod account. 115 00:09:12,480 --> 00:09:17,920 I can list the same production bucket, right? 116 00:09:17,920 --> 00:09:19,820 Way too easy, all right? 117 00:09:19,820 --> 00:09:23,900 The commands, nothing special in the commands. 118 00:09:23,900 --> 00:09:35,740 This is really a lesson in how easy it is to access and exfiltrate and reuse the credentials in bulk for CLI access, which is pretty much gives me everything, all right? 119 00:09:36,840 --> 00:09:47,460 So just to deepen our understanding, a variation of this attack actually makes use of the OAuth tokens in the gcloud subdirectory, all right? 120 00:09:48,000 --> 00:09:54,760 What we're going to do is go through a flow where we actually extract something from the SQLite databases holding the cached tokens. 121 00:09:54,760 --> 00:10:04,720 We're going to use that and perform some of the OAuth flow, which is request a new access token, because most likely that the last access token used has expired. 122 00:10:04,720 --> 00:10:06,620 It only lasts an hour. 123 00:10:07,080 --> 00:10:11,900 Get a new access token, which gives me another hour, and then start making API calls. 124 00:10:13,300 --> 00:10:17,840 Okay, so let's continue on, right? 125 00:10:18,640 --> 00:10:25,640 What I'm going to do, and this is going to get a bear with me as we go through this video and I highlight things, all right? 126 00:10:25,640 --> 00:10:31,600 In the gcloud subdirectory where my token cache is stored, there are two .db files. 127 00:10:31,600 --> 00:10:36,100 They're SQLite database files, unencrypted, nothing special. 128 00:10:36,100 --> 00:10:39,780 What I can do is do a SQLite command, all right? 129 00:10:39,780 --> 00:10:42,980 I'm not going over the schema particularly. 130 00:10:43,400 --> 00:10:51,880 There are sources for that, but there's a simple table that has the last used or last retrieved access token by account. 131 00:10:51,880 --> 00:10:56,300 So I'm pulling the one with associated with the production account. 132 00:10:56,300 --> 00:10:58,160 There it is, the YA-29. 133 00:10:58,380 --> 00:11:00,360 All right, we'll continue on. 134 00:11:00,360 --> 00:11:07,940 Now that I have that, I want to look at the other database, which is credentials.db. 135 00:11:07,940 --> 00:11:10,360 This has more information in a big JSON blob. 136 00:11:10,360 --> 00:11:19,020 I'm also going to look at what's associated with the production admin account here and I have more information. 137 00:11:19,020 --> 00:11:27,800 I have a client ID and secret, which is related to OAuth and the protocol, but it's needed if I want to perform the whole OAuth flow myself. 138 00:11:27,800 --> 00:11:30,140 I have the refresh token, so that's useful. 139 00:11:30,140 --> 00:11:31,200 I need that. 140 00:11:31,200 --> 00:11:33,360 Also, I have access token repeated. 141 00:11:33,440 --> 00:11:38,260 Just note that this information is from my initial authentication configuration of this production account. 142 00:11:38,260 --> 00:11:48,360 So technically, if you take this epoch time with this EXP field, you run it through date and convert it, you'll get a June 17th date. 143 00:11:48,440 --> 00:12:02,040 So what I said is this is a live environment configured six, seven weeks ago and I haven't changed it yet and had default security settings and seven weeks later still have access to it easily. 144 00:12:02,380 --> 00:12:03,980 All right, so let's move on. 145 00:12:04,000 --> 00:12:07,160 Basically, I want to get that refresh token, right? 146 00:12:07,160 --> 00:12:14,820 So I just did a little sedgrep foo on that same SQLite entry with the JSON above and pulled out that access token. 147 00:12:15,300 --> 00:12:21,320 Now I'm just going to do a little cURL REST API calls to go through the OAuth flow, all right? 148 00:12:21,820 --> 00:12:26,660 And the actual cURL command will be shown, but I'm using the refresh token. 149 00:12:26,660 --> 00:12:29,620 So let's see if I can pause this. 150 00:12:29,980 --> 00:12:33,060 Here's this cURL command right here in the screen. 151 00:12:33,060 --> 00:12:40,980 You can see that the client ID and secret and that refresh token are passed in as post parameters. 152 00:12:41,240 --> 00:12:44,160 I've got scope permissions I'm requesting. 153 00:12:44,160 --> 00:12:50,580 The cloud platform one is key, but some other IAM type scope as well. 154 00:12:50,580 --> 00:12:52,580 The endpoint is OAuth. 155 00:12:52,680 --> 00:12:57,360 And what I get back is a fresh new access token right here, okay? 156 00:12:57,360 --> 00:12:59,800 Now, good for an hour. 157 00:13:00,760 --> 00:13:03,060 Now, what can I do with that new access token? 158 00:13:03,060 --> 00:13:05,040 I can do real API calls. 159 00:13:05,040 --> 00:13:06,740 So let's continue on. 160 00:13:07,200 --> 00:13:09,340 I'm going to do another cURL command. 161 00:13:09,840 --> 00:13:13,720 Let's pause it once the output comes back. 162 00:13:15,000 --> 00:13:17,000 All right, here we go. 163 00:13:17,000 --> 00:13:19,740 So what did I do? 164 00:13:19,740 --> 00:13:23,960 Here's the cURL command at the top, right, with that fresh new access token. 165 00:13:23,960 --> 00:13:36,200 I happen to choose a fairly easy API, the storage API, and I'm using the same requesting listing of the same bucket that I had done in the first scenario. 166 00:13:36,280 --> 00:13:39,640 And the JSON that's returned shows that I'm accessing it, right? 167 00:13:39,640 --> 00:13:42,660 Same pass, credit card, sensitive bucket, so on. 168 00:13:43,000 --> 00:13:53,340 So all in all, it just shows that I can do a variation at a deeper level, actually perform the OAuth flow. 169 00:13:53,420 --> 00:14:05,120 And although functionally, this is, in terms of an ATT&CK perspective, it's not that different from the CLI since they're pretty much equivalent to doing direct API calls. 170 00:14:05,120 --> 00:14:14,980 This is more flexible in that I can spin up my own script and code and have somewhat tighter ATT&CK opportunities here. 171 00:14:14,980 --> 00:14:29,560 So the lesson from this is, either way, it's really easy to access the token cache and make use of it, and in a lot of cases, generate new tokens and access what environments have been cached. 172 00:14:29,640 --> 00:14:37,340 So if that's all there were to this, that might be more limiting. 173 00:14:37,340 --> 00:14:46,120 And really, though, there's some related areas with service accounts and compute instances that I want to talk about, right? 174 00:14:46,120 --> 00:14:49,980 More opportunities on the ATT&CK side, more hassle from the defensive viewpoint. 175 00:14:49,980 --> 00:15:03,540 We've been talking a lot about user accounts in the cache, and one might normally think that service accounts wouldn't really be that valuable to look at on a compromised host. 176 00:15:03,540 --> 00:15:04,020 Why? 177 00:15:04,020 --> 00:15:18,280 Because if the target host is external to GCP, most likely what's happened is a service account key file has been generated when you generated a key, and that's been downloaded. 178 00:15:18,400 --> 00:15:24,640 And that JSON file can then be used within API code, but it can also be used within the CLI. 179 00:15:24,640 --> 00:15:33,120 We can activate and add that to our gcloud config and then switch to it as an account and start executing commands as that service account. 180 00:15:33,120 --> 00:15:39,440 But because a key file is very likely still on that client, that's a better target for me as an attacker, right? 181 00:15:39,440 --> 00:15:46,220 Because it's a full-blown credential and I can do anything with that, as opposed to having a temporary OAuth token that expires in an hour. 182 00:15:46,220 --> 00:16:00,140 However, as we'll see later on the defensive side, it is useful to grab this whole cache, including what would be the OAuth session tokens for the service accounts, if they've been configured, because it gives us persistence. 183 00:16:00,140 --> 00:16:19,100 We'll find that a lot of remediation commands that we try with service accounts disable the service account, but they leave the tokens valid, meaning it's a way to have additional access, even though the defender is remediating and revoking the service account access. 184 00:16:19,100 --> 00:16:28,940 All right, so for that reason alone, there's this variation where OAuth tokens for service accounts on an external client are still valuable to grab from an attack perspective. 185 00:16:29,980 --> 00:16:44,120 Speaking of service accounts, well, if they, you know, compute instances run as a service account, to be more secure, there's no key file there if you want to run code on that VM as that service account. 186 00:16:44,120 --> 00:16:55,680 What you do is you query the metadata service and you get an OAuth token that your scripts or code can use, and that way you have no key file that's lying around to be compromised. 187 00:16:55,680 --> 00:16:59,400 So that's an easy REST call. 188 00:16:59,720 --> 00:17:04,860 The URL is internal, resolves to a non-routable local IP. 189 00:17:04,860 --> 00:17:12,620 What we get back is an access token at which we can use in a second API call, as an example of a storage call again. 190 00:17:12,620 --> 00:17:14,180 What does that look like? 191 00:17:17,420 --> 00:17:20,720 Sorry, it's weird. 192 00:17:21,860 --> 00:17:25,900 Okay, sorry, getting a little click problems here. 193 00:17:25,900 --> 00:17:27,060 What does that look like? 194 00:17:27,060 --> 00:17:30,240 Well, you can see the curl command, okay? 195 00:17:30,620 --> 00:17:35,680 What it returns is this access token highlighted in blue, looking at the right side, right? 196 00:17:35,800 --> 00:17:38,060 The expiration date is a little bit less than an hour, basically. 197 00:17:38,060 --> 00:17:44,020 I'm sort of midstream, 400 or so seconds, and I'm getting the cache token. 198 00:17:44,180 --> 00:17:49,180 But at the end of the hour, it automatically refreshes and I can query it again and get a new token. 199 00:17:49,520 --> 00:17:56,060 I can then use that token in a second API call in the storage bucket, boom, it lists it, okay? 200 00:17:56,140 --> 00:17:59,100 So I don't even have to worry about caches, etc. 201 00:17:59,100 --> 00:18:04,360 I can always access the service account OAuth token if I have compromised the compute instance. 202 00:18:04,360 --> 00:18:06,040 And this is really common, right? 203 00:18:06,040 --> 00:18:07,580 Compute instances are targets. 204 00:18:07,580 --> 00:18:14,680 Once you have access, this is one of the first things that we should look at as attackers and worry about as defenders. 205 00:18:14,920 --> 00:18:25,180 Speaking of compute instances, if you have your own compute instances inside the GCP environment, and you install Google Cloud SDK there, everything we've talked about applies there, right? 206 00:18:25,180 --> 00:18:26,660 Makes perfect sense. 207 00:18:29,240 --> 00:18:32,040 Because you have cached tokens as well. 208 00:18:32,040 --> 00:18:34,320 So that becomes a target on the tech side. 209 00:18:34,320 --> 00:18:42,500 But what's maybe not so obvious is that our GCP-managed compute instances, if you run Cloud Shell, the whole purpose of that is sort of ease of use. 210 00:18:42,500 --> 00:18:44,720 You don't have to worry about a VM or the SDK. 211 00:18:44,720 --> 00:18:47,700 Well, the SDK, the latest version is always installed for you. 212 00:18:47,700 --> 00:18:54,520 You now have a nice up-to-date environment to do your CLI commands and manage your environment as a defender . 213 00:18:54,520 --> 00:18:59,180 Well, from an attacker perspective, that becomes a target as well, because the SDK is installed. 214 00:18:59,360 --> 00:19:10,080 There's been some good work, and I recommend if you're interested in this particular area, read Juan Berner's post a year and some ago on backdoors with Google Cloud Shell. 215 00:19:10,160 --> 00:19:20,140 Basically, you can install a backdoor shell, you know, one-liner, you can exfiltrate the token cache automatically. 216 00:19:20,140 --> 00:19:24,400 So every time someone starts up the Cloud Shell, you get the fresh sort of token dump. 217 00:19:25,820 --> 00:19:27,920 And you can move on your merry way. 218 00:19:27,920 --> 00:19:29,480 One note about Cloud Shell. 219 00:19:30,000 --> 00:19:30,920 Well, how do you access that? 220 00:19:30,920 --> 00:19:34,820 You can do it through the Google console, but you can also do it through a gcloud command, right? 221 00:19:34,820 --> 00:19:36,880 So you can think that this could be a chain command. 222 00:19:36,880 --> 00:19:53,260 If I compromise your endpoint, that's my target outside of GCP, and you use Cloud Shell, I may be able to get into your Cloud Shell environment, install a backdoor, and then continue to have access from this compute instance sort of environment. 223 00:19:53,940 --> 00:19:55,980 So this complicates things, right? 224 00:19:55,980 --> 00:19:58,500 These are all sort of variations. 225 00:19:58,620 --> 00:20:04,580 We're not just talking about your endpoint where you normally do Google Cloud Admin. 226 00:20:04,740 --> 00:20:07,820 All right, we're about halfway through. 227 00:20:07,820 --> 00:20:10,980 Let's focus to the defensive side. 228 00:20:10,980 --> 00:20:12,860 We have a lot to talk about here. 229 00:20:13,400 --> 00:20:17,120 Basically, from a prevention viewpoint, put on your blue hats. 230 00:20:17,120 --> 00:20:21,160 There are three basic things that I think will help. 231 00:20:21,160 --> 00:20:24,200 It is hard, right, to prevent credential compromise. 232 00:20:24,200 --> 00:20:25,400 You really can't. 233 00:20:25,400 --> 00:20:28,280 How do you know it's compromised versus the normal user? 234 00:20:28,320 --> 00:20:29,620 But we can mitigate it. 235 00:20:29,620 --> 00:20:32,480 So the number one thing to do is expiration time. 236 00:20:32,480 --> 00:20:37,480 IP whitelisting, if you can manage that, is helpful, as well as MFA. 237 00:20:38,120 --> 00:20:43,240 And there are some additional topics regarding IP whitelisting that we need to touch base on. 238 00:20:43,240 --> 00:20:45,900 Okay, so let's talk about cloud session duration. 239 00:20:45,900 --> 00:20:51,260 On the G Suite admin side, which is, you know, the flag already. 240 00:20:51,260 --> 00:20:57,120 Anytime I say G Suite admin, and you're a GCP admin, your ear should sort of perk up. 241 00:20:57,340 --> 00:20:58,820 It's a different console. 242 00:20:58,820 --> 00:21:01,600 If you're a bigger organization, that may be a different team. 243 00:21:01,600 --> 00:21:04,460 If you have access to it as a GCP admin, great. 244 00:21:04,460 --> 00:21:05,740 This will be familiar to you. 245 00:21:05,740 --> 00:21:09,640 But it's a beta feature, about a year old, if I'm correct on that. 246 00:21:09,640 --> 00:21:11,720 So people may not know about it. 247 00:21:11,800 --> 00:21:13,740 This is one of the most useful things. 248 00:21:13,740 --> 00:21:20,540 By default, sessions for SDK access do not expire, never expires. 249 00:21:20,540 --> 00:21:26,060 In fact, in the earlier attack scenario, in the prod environment we went into, guess what the default was set to? 250 00:21:26,200 --> 00:21:28,120 The default was never expires. 251 00:21:28,120 --> 00:21:29,380 It was never changed. 252 00:21:29,380 --> 00:21:31,700 You can set it to one to 24 hours. 253 00:21:32,060 --> 00:21:39,040 Recommend really short, like one hour for prod environments, maybe eight plus hours for your dev environments, right? 254 00:21:39,040 --> 00:21:54,640 When it expires, all access, both console and programmatic, all those sessions are revoked, and anyone accessing the environment has to actually re-authenticate, all right? 255 00:21:54,640 --> 00:21:58,760 So this shrinks the window during which you can be compromised. 256 00:22:00,020 --> 00:22:02,080 Now, there is a re-authentication method. 257 00:22:02,080 --> 00:22:03,120 You have two choices. 258 00:22:03,120 --> 00:22:09,460 When they're re-authenticating after the expiration, a password or security key. 259 00:22:09,460 --> 00:22:10,900 This is hardware security key. 260 00:22:10,900 --> 00:22:23,860 The only thing to note here is if you use something in the middle, like a software MFA, which I have on one of my dev environments, you have to pick a lesser, a less strong method, which is password. 261 00:22:24,280 --> 00:22:37,180 Just a note, it means your re-authentication is less secure because it's no longer MFA, and opens up to keyboard logging and losing a password credential. 262 00:22:38,840 --> 00:22:43,380 Let's talk about IP whitelisting, the second thing on my list, all right? 263 00:22:43,380 --> 00:22:51,980 So still a relatively new feature, but you can do VPC service controls to implement IP whitelist. 264 00:22:51,980 --> 00:23:01,080 In the Access Context Manager in Google Cloud, you create a whitelist, specify a bunch of ciders and set up a policy. 265 00:23:01,120 --> 00:23:03,900 Please don't be thrown that there's a 1918 address in here. 266 00:23:03,900 --> 00:23:06,260 That's my obfuscation in play. 267 00:23:06,260 --> 00:23:11,640 This is normally your public IP or cider that you want in your whitelist. 268 00:23:12,380 --> 00:23:27,340 Switching over to the VPC service control section, you can specify the APIs and services, everything as an example, attach and reference the actual access level with the whitelist IPs, put it in place. 269 00:23:27,500 --> 00:23:36,840 And the next time that someone tries to do a command to the resources inside that perimeter, VPC perimeter, they'll get an error, right? 270 00:23:36,840 --> 00:23:40,720 So a different host, different IP, it's not in the whitelist. 271 00:23:40,960 --> 00:23:42,540 All good, right? 272 00:23:42,540 --> 00:23:43,640 Really good. 273 00:23:46,680 --> 00:23:50,540 What's of interest to note is you get sort of a verbose message. 274 00:23:50,540 --> 00:23:56,240 Not sure what I think about that from a defensive side, sort of unsettles me, but it's clear that it works. 275 00:23:56,860 --> 00:24:01,800 There is more complication when you think about IP whitelisting. 276 00:24:01,800 --> 00:24:06,180 It's not just users and what devices they're coming from or what IPs rather. 277 00:24:06,800 --> 00:24:15,700 You can manage that, try to enforce VPNs, space that you know, proxies, but you also have to worry about VM instances. 278 00:24:15,700 --> 00:24:19,160 And here's where it's harder lifting. 279 00:24:19,160 --> 00:24:29,600 Essentially, if you go IP whitelisting, you want to make sure that your VM IPs are also whitelisted so that any API calls that they do are allowed, right? 280 00:24:29,600 --> 00:24:31,840 And so there's a couple things to worry about here. 281 00:24:31,840 --> 00:24:33,700 It's just the maintenance issue. 282 00:24:33,860 --> 00:24:46,340 You could hook into the startup and provisioning of your VMs to make sure that the VMs IP addresses are included automatically in your IP list, especially if you have a lot of VMs or dynamic IPs. 283 00:24:48,100 --> 00:24:55,240 Detection, it's not prevention, but a side note, could be done based on logs instead of an explicit whitelist, right? 284 00:24:55,240 --> 00:24:59,520 And some of the references below sort of talk about that as a technique, right? 285 00:24:59,520 --> 00:25:13,880 So IP whitelisting sort of alternatives are cull from logs, the IPs of VMs that have been started and see if there's anything abnormal in it compared to a whitelist or instead of a whitelist. 286 00:25:14,440 --> 00:25:26,100 You can hook in and put in place a man-in-the-middle proxy for metadata, a metadata service, which would give you visibility to anyone querying it for OAuth tokens. 287 00:25:26,140 --> 00:25:30,920 And that might be useful to track and determine and see if that's worth monitoring and learning on. 288 00:25:30,920 --> 00:25:34,400 Okay, so this is also a bit of a detection piece. 289 00:25:34,500 --> 00:25:44,020 But the main point here from a prevention perspective is you have to think about VM instances and ensure that they are whitelisted properly if you go down that route. 290 00:25:44,020 --> 00:25:56,460 So there is some work to be done, but if you can accomplish it, then IP whitelisting is very effective at helping you mitigate the issues with any kind of compromised credentials. 291 00:25:56,600 --> 00:26:02,440 And I'm going to do a little shout out to the Netflix security team, and I have some references there. 292 00:26:02,440 --> 00:26:05,000 And they've blogged quite a while about the work they've done in AWS. 293 00:26:05,000 --> 00:26:10,740 A lot of their concepts apply over to GCP and are worth thinking about as we go. 294 00:26:10,820 --> 00:26:14,660 Once you have the whitelist in place, of course, you have to make sure it's enforced. 295 00:26:14,660 --> 00:26:16,640 So you can do a little CLI foo. 296 00:26:16,640 --> 00:26:25,280 And on the right, you can see the definitions of your access levels and policies and check the IPs and make sure they're the right ones. 297 00:26:25,440 --> 00:26:32,900 You can check the VPC service parameters and make sure that they're accessing access levels as well. 298 00:26:32,900 --> 00:26:38,640 So it's all well and good to have a whitelist, just make sure it's actually enforced and put in place. 299 00:26:38,920 --> 00:26:42,460 Let's move to the third area of prevention, MFA. 300 00:26:42,500 --> 00:26:48,920 On the G Suite admin side, yellow flag, if you don't have access, is where MFA is controlled. 301 00:26:48,960 --> 00:26:53,200 Set it as much and enforce it with your admin users. 302 00:26:53,200 --> 00:27:00,140 It'll help mitigate that compromise of credentials during any re-authentication. 303 00:27:01,020 --> 00:27:05,020 Okay, let's move to detection. 304 00:27:05,300 --> 00:27:08,440 This is a shorter discussion, unfortunately. 305 00:27:08,440 --> 00:27:14,660 Behavioral detection is really the problem for compromised credentials. 306 00:27:14,660 --> 00:27:26,160 It's difficult, discern differences from valid user actions and behavior unless they do something egregious, like exfiltrate some terabytes. 307 00:27:26,920 --> 00:27:28,940 I'm not going to talk about that. 308 00:27:28,940 --> 00:27:42,500 I want to talk about if you do or are able to implement IP whitelisting effectively, effectively with low false positives, detecting failed authentications is worth doing, right? 309 00:27:42,500 --> 00:27:49,920 And Stackdriver on the right will get a nice entry that we can key off of and monitor. 310 00:27:49,980 --> 00:28:00,700 And if we have low enough false positives and it's maintained, it's a good way to get a heads up that there may be a compromised credential being used from an IP. 311 00:28:00,700 --> 00:28:08,880 This would cover the scenario where I exfiltrate credentials or tokens and try to use this from an IP that I control, right? 312 00:28:08,880 --> 00:28:12,780 The old style way of tacking would be to always do that. 313 00:28:12,780 --> 00:28:24,920 If people focus on whitelisting from a defensive viewpoint, it would be interesting to see if, from a tech perspective, we did more follow-up work from the compromised host, because that would still be allowed in this scenario. 314 00:28:24,920 --> 00:28:27,340 So we're not dealing with perfection. 315 00:28:27,340 --> 00:28:30,040 We're just trying to improve our ability to detect. 316 00:28:31,280 --> 00:28:35,260 Okay, we've got to spend some time on remediation, okay? 317 00:28:35,260 --> 00:28:37,540 This is complicated. 318 00:28:37,540 --> 00:28:39,120 Let me see if I can get through this. 319 00:28:39,120 --> 00:28:42,240 There's a basically remediation. 320 00:28:42,240 --> 00:28:44,000 I think there's two major things we have to worry about. 321 00:28:44,000 --> 00:28:46,520 How do we lock out the account, prevent future access? 322 00:28:46,660 --> 00:28:53,840 How do we revoke current access, the current sessions, which are the OAuth access tokens and refresh tokens? 323 00:28:54,260 --> 00:28:56,160 User and service accounts are different. 324 00:28:56,920 --> 00:29:01,860 So here are some of the commonly available or known options. 325 00:29:01,860 --> 00:29:03,380 Let's go through them one by one. 326 00:29:03,420 --> 00:29:06,440 It's going to be a little bit of remediation bingo, all right? 327 00:29:06,440 --> 00:29:12,680 As we go through this, see if you can pick out or suggest the best option, and I'll tell you mine. 328 00:29:13,860 --> 00:29:14,940 All right, user accounts. 329 00:29:14,940 --> 00:29:16,340 How do we lock out user accounts? 330 00:29:16,340 --> 00:29:24,180 Well, G Suite Admin, okay, G Suite Admin, we can reset password, suspend the user, and reactivate it later, delete the user. 331 00:29:24,180 --> 00:29:25,140 They're all effective. 332 00:29:25,140 --> 00:29:27,860 They all lock out the user or attacker. 333 00:29:27,960 --> 00:29:31,100 We'll pick the lowest impact one. 334 00:29:31,620 --> 00:29:33,900 Resetting password, pretty common, works. 335 00:29:33,900 --> 00:29:35,000 That's the answer. 336 00:29:35,880 --> 00:29:37,720 How about service accounts? 337 00:29:37,720 --> 00:29:39,880 Similar, parallel set of options. 338 00:29:40,260 --> 00:29:43,620 I can delete the key here. 339 00:29:43,620 --> 00:29:46,620 I'm in Google Cloud Console for service accounts, right? 340 00:29:46,620 --> 00:29:50,540 G Suite Admin for user accounts, Google Cloud for service accounts. 341 00:29:50,540 --> 00:29:56,120 I can disable the service account itself, and I can delete the service account. 342 00:29:56,180 --> 00:29:57,420 Which one? 343 00:29:57,420 --> 00:29:58,700 They all work. 344 00:29:58,740 --> 00:30:00,320 We'll pick the lowest one. 345 00:30:00,320 --> 00:30:05,840 Rotate, right? 346 00:30:05,840 --> 00:30:07,780 Now, let's get to the tough part. 347 00:30:07,780 --> 00:30:12,760 The tough part is revoking current access, and that's the topic of this talk, right? 348 00:30:13,000 --> 00:30:21,240 So, let's see of these six suggested options, the last three are the lockout account measures. 349 00:30:21,240 --> 00:30:26,340 The first three are potentially common ones that people try or think about. 350 00:30:26,340 --> 00:30:28,240 Let's go through them one by one. 351 00:30:28,240 --> 00:30:36,100 In G Suite Admin, for a particular user, their security section, there is a sign-in cookies and a reset action. 352 00:30:36,100 --> 00:30:37,060 What does this do? 353 00:30:37,060 --> 00:30:41,000 It revokes all web sessions immediately. 354 00:30:41,000 --> 00:30:43,840 This includes Google Cloud Console. 355 00:30:43,840 --> 00:30:47,040 It would also include G Suite, okay? 356 00:30:47,040 --> 00:30:48,000 This works. 357 00:30:48,000 --> 00:30:48,880 This is good. 358 00:30:48,920 --> 00:30:51,880 Include any Google app and service. 359 00:30:52,300 --> 00:30:56,840 This is great, except it doesn't do anything about SDK sessions. 360 00:30:56,840 --> 00:30:58,860 So, actually, it's only half of a solution. 361 00:30:58,920 --> 00:31:00,400 So, we lose. 362 00:31:00,400 --> 00:31:01,620 Let's keep going. 363 00:31:01,620 --> 00:31:06,480 There's an interesting G Cloud CLI command that you have probably seen or used, auth revoke. 364 00:31:06,480 --> 00:31:14,360 It takes an account name, and it will revoke both the access and refresh tokens associated for that user account. 365 00:31:14,360 --> 00:31:22,080 It, underneath, calls an API call with this endpoint to revoke the specific token. 366 00:31:22,080 --> 00:31:24,360 That also works if you want to call it directly. 367 00:31:24,360 --> 00:31:26,140 Sounds great, doesn't it? 368 00:31:26,140 --> 00:31:30,740 But it's in the middle of my list, so you know that it actually doesn't work. 369 00:31:31,160 --> 00:31:32,540 Here's the problem. 370 00:31:33,140 --> 00:31:35,520 It can only be run on the G Cloud client machine. 371 00:31:35,520 --> 00:31:35,900 Why? 372 00:31:35,900 --> 00:31:41,440 Because you pass in an account name, and it looks in the SQLite database cache to find the token. 373 00:31:41,860 --> 00:31:48,100 And that is easily manipulated by the attacker, at a minimum, deleted, right? 374 00:31:48,100 --> 00:31:51,640 So, it can't be run anywhere else. 375 00:31:51,640 --> 00:31:57,160 Frankly, I don't think any incident responder would trust a compromised client machine for anything, right? 376 00:31:58,260 --> 00:32:03,880 Similarly, calling the API itself doesn't help. 377 00:32:03,880 --> 00:32:04,680 Why? 378 00:32:04,800 --> 00:32:10,780 You need to pass in an OAuth access token or revoke, sorry, refresh token. 379 00:32:11,940 --> 00:32:13,340 Those are not logged. 380 00:32:13,340 --> 00:32:14,620 They're not accessible. 381 00:32:14,620 --> 00:32:21,600 They don't exist anywhere except on the client machine, which is compromised, which is easily manipulated by the attacker. 382 00:32:21,600 --> 00:32:23,840 So, it's useless. 383 00:32:23,840 --> 00:32:28,400 These are for self-help, for you to clean up your cache and config, nothing else. 384 00:32:28,560 --> 00:32:33,560 A one side note, which is a diversion, but the call to revoke token doesn't require authentication. 385 00:32:33,560 --> 00:32:37,200 You just pass in this long access and refresh token. 386 00:32:37,200 --> 00:32:42,380 It's long enough that you probably can't brute force it, but it's an interesting note that you don't need OAuth to call. 387 00:32:42,720 --> 00:32:53,000 Okay, let's cover that lockout service account, sorry, user lockout options, all right? 388 00:32:53,000 --> 00:32:54,840 Changing the user password. 389 00:32:55,020 --> 00:32:56,480 These all don't work. 390 00:32:56,480 --> 00:32:58,800 They don't revoke SDK sessions. 391 00:32:58,800 --> 00:33:07,340 There's a narrow case where if you have requested scope for Gmail, then resetting the password would work, but that doesn't apply to us. 392 00:33:07,340 --> 00:33:13,080 We would have requested scope probably for just gcloud or SDK access. 393 00:33:13,280 --> 00:33:15,800 Suspending user account works temporarily. 394 00:33:15,800 --> 00:33:19,840 While the user account is suspended, the tokens will not work. 395 00:33:19,840 --> 00:33:25,380 Once, though, it's re-enabled, the tokens will start working again, assuming they have not expired. 396 00:33:25,380 --> 00:33:27,640 So, that's not a great solution. 397 00:33:27,640 --> 00:33:30,720 Unless you suspend the account for an hour, that would work. 398 00:33:32,600 --> 00:33:35,340 Sorry, let me backtrack on that. 399 00:33:36,060 --> 00:33:39,280 Deleting user account works, but has high impact, okay? 400 00:33:39,280 --> 00:33:41,020 So, these options aren't great. 401 00:33:41,020 --> 00:33:42,520 So, this is all yellow. 402 00:33:42,520 --> 00:33:45,060 The question is, what's our best option? 403 00:33:45,060 --> 00:33:58,200 So, one thing I did not list in the table was there's actually the best option that works and is low impact is to delete the connected application. 404 00:33:58,200 --> 00:34:01,800 This is the connected OAuth application in the G Suite admin side. 405 00:34:01,840 --> 00:34:14,680 Basically, every time you authenticate and request access, the OAuth application, in this case, we're talking about the Google Cloud SDK, is listed here, as well as this access level, and you can delete it. 406 00:34:14,680 --> 00:34:16,680 And deleting it has no big ramifications. 407 00:34:16,680 --> 00:34:23,560 It just means next time someone accesses Google Cloud SDK, like through CLI, they're forced to re-authenticate. 408 00:34:23,560 --> 00:34:27,240 But once you delete it, at that point in time, all session tokens are revoked. 409 00:34:27,240 --> 00:34:29,120 So, it's the answer. 410 00:34:29,120 --> 00:34:29,960 That's the bingo. 411 00:34:29,960 --> 00:34:33,400 That's the bonus right there, okay? 412 00:34:33,400 --> 00:34:40,100 And if you haven't worked with G Suite admin, you wouldn't know about it, or maybe think about it, or if you're not an OAuth jockey, right? 413 00:34:40,400 --> 00:34:43,600 In OAuth land, this is pretty common to think about these connections. 414 00:34:43,600 --> 00:34:44,620 It just means it's trusted. 415 00:34:44,620 --> 00:34:46,940 And once you delete it, it revokes the sessions. 416 00:34:47,180 --> 00:34:47,940 All right. 417 00:34:48,260 --> 00:34:50,220 Let's talk about the service account. 418 00:34:51,540 --> 00:34:52,960 How do we revoke that? 419 00:34:52,960 --> 00:34:54,500 Well, do we revoke it? 420 00:34:54,500 --> 00:34:58,140 Do we do some of these other lockout measures? 421 00:34:58,300 --> 00:35:00,020 Let's jump into it. 422 00:35:00,040 --> 00:35:06,620 Well, you might have guessed that the revoke CLI and the API call won't work, because we just went through it for the user account. 423 00:35:07,280 --> 00:35:08,440 It definitely doesn't work. 424 00:35:08,440 --> 00:35:12,360 You actually get errors here, right, when you try to do the CLI or the API. 425 00:35:12,360 --> 00:35:12,980 Outright errors. 426 00:35:12,980 --> 00:35:14,420 It just says it doesn't apply. 427 00:35:14,420 --> 00:35:20,600 But what's of note is, look in the error, sorry for the verbiage, but look at the error you get back from the CLI from gcloud. 428 00:35:20,660 --> 00:35:23,200 It actually hints in the last sentence that there's hope. 429 00:35:23,960 --> 00:35:31,180 If you revoke the parent service account, it's supposed to revoke the session, the actual sessions. 430 00:35:31,180 --> 00:35:31,980 What does that mean? 431 00:35:31,980 --> 00:35:35,540 Well, if you suspend it or delete it, that's true. 432 00:35:35,640 --> 00:35:40,900 But the suspend is sort of a half answer, because once you re-enable it, the tokens actually still work. 433 00:35:40,900 --> 00:35:42,340 What about the service account key? 434 00:35:42,340 --> 00:35:43,260 Revoke it? 435 00:35:43,280 --> 00:35:43,920 What's that mean? 436 00:35:43,920 --> 00:35:46,400 Well, it means delete the API key. 437 00:35:46,760 --> 00:35:48,020 That definitely doesn't work. 438 00:35:48,020 --> 00:35:48,620 That's a bug. 439 00:35:48,620 --> 00:35:50,080 Bug has been filed. 440 00:35:50,080 --> 00:35:51,520 Don't believe it, right? 441 00:35:52,720 --> 00:35:54,040 We've already gone through this. 442 00:35:54,040 --> 00:35:55,360 This doesn't work. 443 00:35:56,020 --> 00:35:59,920 How about the measures to rotate and delete the API key? 444 00:35:59,920 --> 00:36:10,600 Look, if you rotate, delete the API key, you can still use OAuth tokens and still access the service login and do actions as the service account. 445 00:36:10,800 --> 00:36:15,680 If you disable it, at that point in time, the tokens don't work, but they still exist. 446 00:36:15,680 --> 00:36:20,860 So once you re-enable, reactivate the service account, they'll work again. 447 00:36:20,860 --> 00:36:27,040 So the solution there is you could suspend for an hour, then re-enable it, because in that hour, tokens will expire. 448 00:36:27,200 --> 00:36:28,440 How about deleting service account? 449 00:36:28,440 --> 00:36:31,520 Works, just like deleting user account, high impact. 450 00:36:31,980 --> 00:36:40,180 So the summary is actually disable for an hour or delete and reprovision. 451 00:36:40,240 --> 00:36:43,740 You will choose whichever you're more comfortable with. 452 00:36:43,920 --> 00:36:50,480 Whichever gets you back up and running quicker, that you can perform with confidence, without making errors. 453 00:36:50,560 --> 00:36:54,140 This isn't great, but there are ways forward, right? 454 00:36:54,140 --> 00:37:03,440 And the main thing of this whole slide and the time spent on all the paths that weren't great paths or were dead ends is to make sure we don't waste time on that, especially in the middle of an incident, right? 455 00:37:03,440 --> 00:37:15,020 Is to feed this and review our operations playbooks and make sure that we have the right actions for remediation for the different types of accounts. 456 00:37:15,780 --> 00:37:23,620 So to wrap up in the next minute or so and at the end, we went over a bunch of attack scenarios, right? 457 00:37:23,620 --> 00:37:26,560 Definitely user accounts and the cache on an external client. 458 00:37:26,560 --> 00:37:28,300 That's really what we focused on. 459 00:37:28,300 --> 00:37:41,400 But don't forget that service accounts also cache the access token and that an attacker would want to grab that because it helps them maintain persistence in the light of all those remediation actions that don't quite work. 460 00:37:41,740 --> 00:37:43,720 But it's not just external clients. 461 00:37:43,720 --> 00:37:50,780 If you switch to a Compute Engine environment and you've installed SDK, all the same things apply. 462 00:37:50,780 --> 00:37:53,320 And it's not just things that you explicitly start. 463 00:37:53,320 --> 00:38:00,300 Remember to think about Cloud Shell, which starts up a VM in the back with the latest SDK installed, right? 464 00:38:00,300 --> 00:38:07,420 These are all having the same cache token kinds of exposure and opportunity. 465 00:38:07,700 --> 00:38:15,040 And fifth, last but not least, the fifth area is even if SDK is not installed, right? 466 00:38:15,040 --> 00:38:27,560 Compute Engine compromise, OAuth tokens for that service account are easily queried from the metadata service and can allow the attacker access as well as persistence for that access. 467 00:38:27,560 --> 00:38:35,360 And from a defensive measure, real quick summary after that long sort of part of the presentation, do three things on the prevention side. 468 00:38:35,360 --> 00:38:38,420 The no-brainer is set your session timeout in G Suite. 469 00:38:38,460 --> 00:38:48,980 If you can, implement IP whitelisting with service controls and set your MFA so that these all mitigate credential compromise in some way. 470 00:38:49,100 --> 00:38:55,120 And if you do do that IP whitelisting, then you can improve your detection by detecting the failed authorizations. 471 00:38:55,120 --> 00:39:02,240 From a remediation perspective for user accounts, reset the password to lock it out and delete the OAuth connected app in G Suite. 472 00:39:02,240 --> 00:39:05,040 That's the key one that may not be obvious to everyone. 473 00:39:05,040 --> 00:39:07,000 For service accounts, pick your poison. 474 00:39:07,000 --> 00:39:12,860 One of two, disable reactivate within our time period or delete, recreate, reprovision. 475 00:39:12,860 --> 00:39:17,360 If you have that automated, just make sure and want to get up and running faster. 476 00:39:17,360 --> 00:39:22,500 Just have to worry about VMs, compute instances, anything using those service accounts. 477 00:39:23,120 --> 00:39:30,760 So that is the end and I will sort of ask Jaron to see if we have any time for questions. 478 00:39:30,760 --> 00:39:33,980 Otherwise, you can always reach me directly as well. 479 00:39:33,980 --> 00:39:39,500 Yeah, we have one question from Eric. 480 00:39:39,560 --> 00:39:49,400 Do you know of any blogs or white papers that people can use to replicate the OAuth token hijacking that you showed earlier? 481 00:39:51,880 --> 00:39:59,240 Yes, so there are some resources for sure. 482 00:39:59,240 --> 00:40:05,900 We will be blogging over the next week or two a lot of the details from this presentation. 483 00:40:05,900 --> 00:40:12,720 So you'll actually be able to recreate what's seen in the video with actual commands. 484 00:40:12,720 --> 00:40:14,800 But there's been prior work as well. 485 00:40:14,800 --> 00:40:24,560 So if you search on G Cloud, I want to say if I have the name right, I didn't reference in the presentation, I'll have to add that later. 486 00:40:24,560 --> 00:40:34,320 But John Haney has, I believe I have his name right, has done some early work on just peeling apart the SQL databases and some of the REST API calls. 487 00:40:34,620 --> 00:40:36,580 And that's a useful resource. 488 00:40:37,140 --> 00:40:42,220 It's not documented well, particularly, but it's also not hard to delve into it. 489 00:40:42,220 --> 00:40:57,640 So both our blogs should be released in part, I think today, as well as over the next two weeks, as well as there's some prior work if you search on some of the keywords like the G Cloud, the dot config, the SQL light database names.