start	end	text
0	4220	As you mentioned, we're going to talk about hijacking OAuth tokens in Google Cloud.
5060	9660	Good thing about Safe Mode this year is if this is the wrong talk, you can quietly leave and I
9660	17660	won't even see you. All right, let's get into it here. So, if you need to get hold of me,
17660	21820	just DM me at Twitter. You can usually find me on most of the social platforms.
22060	28100	What we're going to cover in this talk is hijacking of OAuth tokens, session tokens,
28100	34340	for CLI access, API access, and it's really the ease of attack that we're going to focus on.
34340	39440	There's additional use cases related to service accounts and compute instances
40060	43640	that we'll look at as well. And from a defensive perspective,
43640	48240	we're definitely going to drill into some of the challenges in securing the environment.
48820	54540	Really, who cares, right? Well, OAuth is fundamental to Google authentication. It's
54540	60020	underneath everywhere. There are some nuances depending on if you are authenticating with
60020	68100	user accounts versus service accounts, whether your sessions are from browsers versus SDK access,
68100	73720	whether you're external to the GCP environment versus internal, say, on a compute VM.
73720	78360	And that is part of what leads to a lot of the opportunity from the tech perspective,
78360	82920	as well as the challenges from the defensive perspective. Essentially, the session tokens are
82920	88860	what the name implies. They're post-authentication representing a valid authentication session.
88860	93680	And it's really an opportunity for us to hijack and use them and gain access to existing
93680	100960	authenticated sessions. And if I, as an attacker, have access to an endpoint, say a GCP administrator's
100960	106460	device, those cached accounts and environments are easily accessible to me. And that's what
106460	111940	we're going to go through. At the end of the day, it's all about persistence, as well as
111940	118840	evading detection from the attacker's perspective. Now, the defensive side,
118840	128040	what is interesting are some of the challenges. Some of the controls such as MFA don't quite help,
128040	132820	although you might counterintuitively, you might think they do, right? And we'll go through that.
132820	138780	And in terms of securing the environment, there's a lot of options that are confusing,
138780	144240	easy to misconfigure, and also easily misunderstood. So at the end of the day,
144240	149920	we're talking about how do you handle incident response? What's your playbook for ops? Is it
149920	160100	fully up to date with the most optimal measures or controls? So let's start with just a brief
160100	166660	overview of OAuth. Okay, just really briefly, there's some verbiage on this slide, but focus
166660	171340	on the right hand diagram. At a very high level, there's a simplified flow, no matter how you
171340	179500	authenticate. First, you'll request a token, it'll be signed with your credentials, you'll request
179500	185700	scopes or permissions. And as a user, you often see this when you first authenticate with gcloud
185700	192240	or CLI. By default, a browser is launched, you have to go type in your Google credentials,
192240	197820	you also have to approve the actual scopes being requested. And if that goes properly,
197820	203500	then what is returned ultimately to the requester, in this case, it'd be the CLI,
203500	209420	would be two important things, the access session token, that's an OAuth session token,
209420	215940	as well as a refresh token. The access token is good for an hour. By default,
215940	222040	you use it directly in API calls, they all take OAuth tokens. After the hour is up,
222040	228000	you can refresh and get a new one. So gcloud, gsutil, CLI tools do this automatically,
228000	234860	you usually don't see this level of the protocol in the handshake. The local language SDK libraries
234860	240320	like Python, etc, also do this transparently. So generally, your code is not at this level.
240320	244540	If you do direct REST API calls, you will see this handshake and have to go through
244540	249440	two or three back and forths. The other important thing to remember about OAuth,
249440	255540	besides this handshake and the access and refresh tokens, is that, well, you don't authenticate
255540	261340	every time you use the CLI, right? It's cached. All session tokens almost everywhere in every
261340	265420	model and architecture are cached. In this case, it's in the SQLite database and has
265420	272080	big implications for this talk. So let's continue on and let's actually start talking about an
272080	277320	attack scenario and we'll jump to the command line to demonstrate this. Okay, a bit of a busy
277320	284660	diagram. Focus on what we're assuming is we have access to an endpoint device, a client device,
285360	293660	where a GCP admin has been using the CLI, gcloud gsutil. What we're going to do is access that.
293660	300180	Once we have access, we'll access that and look at the cached accounts, utilize it,
300180	308700	access the GCP environment that the admin that user has previously accessed. Then we're going to
308700	313680	copy the credentials, move it off board to a different host under my control, me, the attacker,
313680	322260	and then we'll show how to access it from there. Okay, so I've got a split screen shell environment.
322260	327300	Left is the target host. The right will be an additional attacker host. Let me explain a few
327300	335360	things. This is video just recorded in the past day. I have obfuscated host names, IDs, a lot of unique
335360	339360	things, but this was run against a live GCP environment. We're going through the attack
339360	343480	scenario, but just a few notes on the defensive side, which will come into play in the second half
343480	350660	of this presentation. MFA is enabled for the environments that we'll be looking at. Okay,
350660	359000	just keep that in mind. And initially, it was a number of weeks since access was initially
359000	366180	configured for these environments. So let's jump into it. If I have access to my target host,
366180	373800	I can, of course, run gcloud commands as that user. And I can check with auth list what accounts
373800	381720	are configured. Two accounts, user dev MFA, admin at prod MFA, named to give us a hint. One's a
381720	386580	production environment. Others are dev. Both have MFA enabled. In fact, there's a hardware key
386580	394400	on the prod environment. There's software MFA on the dev one. So I've got some comments along the
394400	399220	way, but I'm going to highlight the bold white commands. All right, I can switch accounts with
399220	405440	gcloud config set account. Let's switch the admin prod account. That's interesting. All right,
405440	411140	switch is over. Let's see if I can actually access it. Let me list the projects. Boom. Let me list
411140	417320	the buckets in the project. I get a bucket. Let me enumerate the bucket. Okay, let's pause here.
417320	426520	Okay, if you're an admin, normally using gcloud, gsutil, this is boring. This is what you do every
426520	430900	day. What makes this interesting is the attacker perspective within the context of the actual
430900	436600	environments. As I mentioned, the environments have MFA enabled. And as we'll see as we get
436600	443380	further along, these accounts were actually configured quite a while ago. The production
443380	449200	account was added back in June, maybe six, seven weeks ago, right? So essentially, in this
449200	455640	environment, this live environment, I have cached access to a production account. And as an attacker,
455640	461220	it was easy, right? Just no authentication required six weeks later, seven weeks later,
461220	468260	despite things like MFA. Let's continue on. All right, so what I want to do now is I want to
468920	476080	exfiltrate this whole cache, move it off host, right? So what I'm going to do is go to the
476080	483140	location of cache. It's in your home directory dot config in a gcloud subdirectory on Unix.
483140	490480	Windows has its own corresponding directory. I'm going to tar it up, I'll SCP it off board,
490480	496140	and then I'm going to switch to the right side. Okay, so focus on this other host, where I've
496140	502160	SCP the tarball. And what I'm going to do is, so far, I'm just checking my environment.
502160	507900	I haven't done anything yet. All right. I'm going to show that there's no accounts
508660	512840	configured, right? It's empty, no credential accounts. I'm going to take that tarball,
512840	521620	unpack it in my own dot config directory. And boom, right off. At least the configuration
521620	529260	matches, right? Shows exactly the state. Now, the note to self right here is,
529260	535500	that was easy. Way too easy. There's no device-specific encryption, obviously, or any kind
535500	541300	of tagging in this data. So it transparently carries over, and I don't have to do anything
541300	546240	except place it in the same directory. Let's see if it's actually useful and valid. I can switch
546240	551360	accounts. Again, different host, same credential file copied over. I can switch accounts to that
551360	559820	prod account. I can list the same production bucket, right? Way too easy, all right?
559820	566400	The commands, nothing special in the commands. This is really a lesson in how easy it is to
566400	573240	access and exfiltrate and reuse the credentials in bulk for CLI access, which is pretty much
573840	580900	gives me everything, all right? So just to deepen our understanding, a variation of this attack
581460	587460	actually makes use of the OAuth tokens in the gcloud subdirectory, all right?
588000	592260	What we're going to do is go through a flow where we actually extract something from the SQLite
592260	598480	databases holding the cached tokens. We're going to use that and perform some of the OAuth flow,
598480	604020	which is request a new access token, because most likely that the last access token used has
604020	610320	expired. It only lasts an hour. Get a new access token, which gives me another hour, and then start
610320	621940	making API calls. Okay, so let's continue on, right? What I'm going to do, and this is going to get a
621940	627100	bear with me as we go through this video and I highlight things, all right? In the gcloud subdirectory
627100	633460	where my token cache is stored, there are two .db files. They're SQLite database files,
634260	640320	unencrypted, nothing special. What I can do is do a SQLite command, all right? I'm not going over
640320	647640	the schema particularly. There are sources for that, but there's a simple table that has the last
647640	655000	used or last retrieved access token by account. So I'm pulling the one with associated with
655000	661220	the production account. There it is, the YA-29. All right, we'll continue on. Now that I have
663360	668960	that, I want to look at the other database, which is credentials.db. This has more information
668960	675180	in a big JSON blob. I'm also going to look at what's associated with the production admin account
675180	683700	here and I have more information. I have a client ID and secret, which is related to OAuth and the
683700	689100	protocol, but it's needed if I want to perform the whole OAuth flow myself. I have the refresh token,
689100	694680	so that's useful. I need that. Also, I have access token repeated. Just note that this information is
694680	699480	from my initial authentication configuration of this production account. So technically,
699480	706280	if you take this epoch time with this EXP field, you run it through date and convert it,
706280	712840	you'll get a June 17th date. So what I said is this is a live environment configured six, seven
712840	718560	weeks ago and I haven't changed it yet and had default security settings and seven weeks later
719080	725420	still have access to it easily. All right, so let's move on. Basically, I want to get that
725420	732120	refresh token, right? So I just did a little sedgrep foo on that same SQLite entry with the
732120	738580	JSON above and pulled out that access token. Now I'm just going to do a little cURL REST API calls
738580	745320	to go through the OAuth flow, all right? And the actual cURL command will be shown, but I'm using
745320	752060	the refresh token. So let's see if I can pause this. Here's this cURL command right here in the
752060	758160	screen. You can see that the client ID and secret and that refresh token are passed in as
759020	766140	post parameters. I've got scope permissions I'm requesting. The cloud platform one is key,
766140	774240	but some other IAM type scope as well. The endpoint is OAuth. And what I get back is a
774240	782220	fresh new access token right here, okay? Now, good for an hour. Now, what can I do with that
782220	788420	new access token? I can do real API calls. So let's continue on. I'm going to do another cURL
788420	799740	command. Let's pause it once the output comes back. All right, here we go. So what did I do?
799740	805040	Here's the cURL command at the top, right, with that fresh new access token. I happen to choose
805740	812200	a fairly easy API, the storage API, and I'm using the same requesting listing of the same bucket
812980	819360	that I had done in the first scenario. And the JSON that's returned shows that I'm accessing it,
819360	827740	right? Same pass, credit card, sensitive bucket, so on. So all in all, it just shows that I can
828310	835120	do a variation at a deeper level, actually perform the OAuth flow. And although functionally,
835120	841050	this is, in terms of an ATT&CK perspective, it's not that different from the CLI since they're
841600	847420	pretty much equivalent to doing direct API calls. This is more flexible in that I can
848540	854980	spin up my own script and code and have somewhat tighter ATT&CK opportunities here.
854980	861740	So the lesson from this is, either way, it's really easy to access the token cache and make
861740	868620	use of it, and in a lot of cases, generate new tokens and access what environments have been
868620	879640	cached. So if that's all there were to this, that might be more limiting. And really, though,
879640	885940	there's some related areas with service accounts and compute instances that I want to talk about,
885940	889980	right? More opportunities on the ATT&CK side, more hassle from the defensive viewpoint.
889980	896200	We've been talking a lot about user accounts in the cache, and one might normally think that
896200	904460	service accounts wouldn't really be that valuable to look at on a compromised host. Why? Because
904460	912760	if the target host is external to GCP, most likely what's happened is a service account
912760	918820	key file has been generated when you generated a key, and that's been downloaded. And that JSON
918820	925800	file can then be used within API code, but it can also be used within the CLI. We can activate and
925800	931200	add that to our gcloud config and then switch to it as an account and start executing
931200	937180	commands as that service account. But because a key file is very likely still on that client,
937180	940820	that's a better target for me as an attacker, right? Because it's a full-blown credential and
940820	946220	I can do anything with that, as opposed to having a temporary OAuth token that expires in an hour.
946220	952060	However, as we'll see later on the defensive side, it is useful to grab this whole cache,
952060	957080	including what would be the OAuth session tokens for the service accounts,
957080	961440	if they've been configured, because it gives us persistence. We'll find that a lot of
962040	967980	remediation commands that we try with service accounts disable the service account, but they
967980	974640	leave the tokens valid, meaning it's a way to have additional access, even though the defender is
975420	980680	remediating and revoking the service account access. All right, so for that reason alone,
980680	985840	there's this variation where OAuth tokens for service accounts on an external client are still
985840	993280	valuable to grab from an attack perspective. Speaking of service accounts, well, if they,
993280	999620	you know, compute instances run as a service account, to be more secure, there's no key file
999620	1005820	there if you want to run code on that VM as that service account. What you do is you query the
1005820	1012920	metadata service and you get an OAuth token that your scripts or code can use, and that way you
1012920	1019840	have no key file that's lying around to be compromised. So that's an easy REST call. The
1019840	1026840	URL is internal, resolves to a non-routable local IP. What we get back is an access token
1027560	1032620	at which we can use in a second API call, as an example of a storage call again.
1032620	1043840	What does that look like? Sorry, it's weird. Okay, sorry, getting a little click
1045220	1050240	problems here. What does that look like? Well, you can see the curl command, okay?
1050620	1055680	What it returns is this access token highlighted in blue, looking at the right side, right?
1055800	1058540	The expiration date is a little bit less than an hour, basically. I'm sort of
1058540	1065260	midstream, 400 or so seconds, and I'm getting the cache token. But at the end of the hour,
1065260	1070640	it automatically refreshes and I can query it again and get a new token. I can then use that
1070640	1077680	token in a second API call in the storage bucket, boom, it lists it, okay? So I don't even have to
1077680	1083500	worry about caches, etc. I can always access the service account OAuth token if I have compromised
1083500	1087880	the compute instance. And this is really common, right? Compute instances are targets. Once you
1087880	1093840	have access, this is one of the first things that we should look at as attackers and worry about as
1093840	1099200	defenders. Speaking of compute instances, if you have your own compute instances inside the GCP
1099200	1104520	environment, and you install Google Cloud SDK there, everything we've talked about applies
1104520	1112740	there, right? Makes perfect sense. Because you have cached tokens as well. So that becomes a
1112740	1119360	target on the tech side. But what's maybe not so obvious is that our GCP-managed compute instances,
1119360	1123060	if you run Cloud Shell, the whole purpose of that is sort of ease of use. You don't have to worry
1123060	1128420	about a VM or the SDK. Well, the SDK, the latest version is always installed for you. You now have
1129360	1134360	a nice up-to-date environment to do your CLI commands and manage your environment as a
1134360	1138460	defender. Well, from an attacker perspective, that becomes a target as well, because the SDK is
1138460	1143060	installed. There's been some good work, and I recommend if you're interested in this particular
1143060	1151080	area, read Juan Berner's post a year and some ago on backdoors with Google Cloud Shell. Basically,
1151080	1158720	you can install a backdoor shell, you know, one-liner, you can exfiltrate the token cache
1159480	1164400	automatically. So every time someone starts up the Cloud Shell, you get the fresh sort of token dump.
1165820	1170920	And you can move on your merry way. One note about Cloud Shell. Well, how do you access that?
1170920	1174540	You can do it through the Google console, but you can also do it through a gcloud command,
1174540	1178740	right? So you can think that this could be a chain command. If I compromise your endpoint,
1178740	1183740	that's my target outside of GCP, and you use Cloud Shell, I may be able to get into your
1183740	1189320	Cloud Shell environment, install a backdoor, and then continue to have access from this
1189880	1196320	compute instance sort of environment. So this complicates things, right? These are
1196320	1202300	all sort of variations. We're not just talking about your endpoint where you normally do Google
1202960	1210980	Cloud Admin. All right, we're about halfway through. Let's focus to the defensive side.
1210980	1217120	We have a lot to talk about here. Basically, from a prevention viewpoint, put on your blue hats.
1217120	1223520	There are three basic things that I think will help. It is hard, right, to prevent credential
1223520	1228860	compromise. You really can't. How do you know it's compromised versus the normal user? But we can
1228860	1234000	mitigate it. So the number one thing to do is expiration time. IP whitelisting, if you can
1234000	1240820	manage that, is helpful, as well as MFA. And there are some additional topics regarding IP
1240820	1245900	whitelisting that we need to touch base on. Okay, so let's talk about cloud session duration.
1245900	1253360	On the G Suite admin side, which is, you know, the flag already. Anytime I say G Suite admin,
1253360	1258980	and you're a GCP admin, your ear should sort of perk up. It's a different console. If you're
1259080	1263820	a bigger organization, that may be a different team. If you have access to it as a GCP admin,
1263820	1268900	great. This will be familiar to you. But it's a beta feature, about a year old, if I'm correct
1268900	1274840	on that. So people may not know about it. This is one of the most useful things. By default,
1274840	1282180	sessions for SDK access do not expire, never expires. In fact, in the earlier attack scenario,
1282180	1287480	in the prod environment we went into, guess what the default was set to? The default was never
1287480	1293060	expires. It was never changed. You can set it to one to 24 hours. Recommend really short,
1293060	1299040	like one hour for prod environments, maybe eight plus hours for your dev environments, right?
1299040	1308120	When it expires, all access, both console and programmatic, all those sessions are revoked,
1308120	1314940	and anyone accessing the environment has to actually re-authenticate, all right? So this
1314940	1321680	shrinks the window during which you can be compromised. Now, there is a re-authentication
1321680	1326100	method. You have two choices. When they're re-authenticating after the expiration,
1328000	1331920	a password or security key. This is hardware security key. The only thing to note here is
1331920	1338780	if you use something in the middle, like a software MFA, which I have on one of my dev
1338780	1343860	environments, you have to pick a lesser, a less strong method, which is password.
1344280	1350320	Just a note, it means your re-authentication is less secure because it's no longer MFA,
1350880	1357180	and opens up to keyboard logging and losing a password credential.
1358840	1362440	Let's talk about IP whitelisting, the second thing on my list,
1362440	1368660	all right? So still a relatively new feature, but you can do VPC service controls
1369560	1377440	to implement IP whitelist. In the Access Context Manager in Google Cloud, you create a whitelist,
1377440	1383000	specify a bunch of ciders and set up a policy. Please don't be thrown that there's a 1918
1383000	1388180	address in here. That's my obfuscation in play. This is normally your public IP
1388940	1394640	or cider that you want in your whitelist. Switching over to the VPC service control
1394640	1402540	section, you can specify the APIs and services, everything as an example, attach and reference
1402540	1408760	the actual access level with the whitelist IPs, put it in place. And the next time that someone
1408760	1416640	tries to do a command to the resources inside that perimeter, VPC perimeter, they'll get an error,
1416640	1423640	right? So a different host, different IP, it's not in the whitelist. All good, right? Really good.
1426680	1431380	What's of interest to note is you get sort of a verbose message. Not sure what I think about
1431380	1438040	that from a defensive side, sort of unsettles me, but it's clear that it works. There is more
1439140	1444020	complication when you think about IP whitelisting. It's not just users and what devices they're
1444020	1451800	coming from or what IPs rather. You can manage that, try to enforce VPNs, space that you know,
1451800	1459160	proxies, but you also have to worry about VM instances. And here's where it's harder lifting.
1459160	1464900	Essentially, if you go IP whitelisting, you want to make sure that your VM IPs are also whitelisted
1464900	1471200	so that any API calls that they do are allowed, right? And so there's a couple things to worry
1471200	1476320	about here. It's just the maintenance issue. You could hook into the startup and provisioning of
1476320	1482560	your VMs to make sure that the VMs IP addresses are included automatically in your IP list,
1482560	1490660	especially if you have a lot of VMs or dynamic IPs. Detection, it's not prevention, but a side
1490660	1495860	note, could be done based on logs instead of an explicit whitelist, right? And some of the
1495860	1501700	references below sort of talk about that as a technique, right? So IP whitelisting sort of
1501700	1509360	alternatives are cull from logs, the IPs of VMs that have been started and see if there's anything
1509360	1516800	abnormal in it compared to a whitelist or instead of a whitelist. You can hook in and put in place
1517000	1523400	a man-in-the-middle proxy for metadata, a metadata service, which would give you visibility to anyone
1523400	1528660	querying it for OAuth tokens. And that might be useful to track and determine and see if that's
1528660	1535740	worth monitoring and learning on. Okay, so this is also a bit of a detection piece. But the main
1535740	1541680	point here from a prevention perspective is you have to think about VM instances and ensure that
1541680	1545600	they are whitelisted properly if you go down that route. So there is some work to be done, but if
1545600	1554320	you can accomplish it, then IP whitelisting is very effective at helping you mitigate the issues
1554320	1558600	with any kind of compromised credentials. And I'm going to do a little shout out to
1559440	1563280	the Netflix security team, and I have some references there. And they've blogged quite a
1563280	1568180	while about the work they've done in AWS. A lot of their concepts apply over to GCP and are worth
1568180	1574260	thinking about as we go. Once you have the whitelist in place, of course, you have to make sure it's
1574260	1580920	enforced. So you can do a little CLI foo. And on the right, you can see the definitions of your
1580920	1586440	access levels and policies and check the IPs and make sure they're the right ones. You can check
1586440	1593260	the VPC service parameters and make sure that they're accessing access levels as well. So
1593820	1598640	it's all well and good to have a whitelist, just make sure it's actually enforced and put in place.
1598920	1605720	Let's move to the third area of prevention, MFA. On the G Suite admin side, yellow flag,
1605720	1611520	if you don't have access, is where MFA is controlled. Set it as much and enforce it
1611520	1618080	with your admin users. It'll help mitigate that compromise of credentials during any
1618080	1628440	re-authentication. Okay, let's move to detection. This is a shorter discussion, unfortunately.
1628440	1636260	Behavioral detection is really the problem for compromised credentials. It's difficult,
1636260	1643880	discern differences from valid user actions and behavior unless they do something egregious,
1643880	1651320	like exfiltrate some terabytes. I'm not going to talk about that. I want to talk about if you do
1651920	1657820	or are able to implement IP whitelisting effectively, effectively with low false positives,
1658740	1664620	detecting failed authentications is worth doing, right? And Stackdriver on the right will get a
1664620	1672240	nice entry that we can key off of and monitor. And if we have low enough false positives and
1672240	1678360	it's maintained, it's a good way to get a heads up that there may be a compromised credential
1678360	1684000	being used from an IP. This would cover the scenario where I exfiltrate credentials or
1684000	1690960	tokens and try to use this from an IP that I control, right? The old style way of tacking
1690960	1695440	would be to always do that. If people focus on whitelisting from a defensive viewpoint,
1695440	1699640	it would be interesting to see if, from a tech perspective, we did more follow-up work from the
1699640	1706580	compromised host, because that would still be allowed in this scenario. So we're not dealing
1706580	1713020	with perfection. We're just trying to improve our ability to detect. Okay, we've got to spend some
1713020	1720900	time on remediation, okay? This is complicated. Let me see if I can get through this. There's a
1720900	1724640	basically remediation. I think there's two major things we have to worry about. How do we lock out
1724640	1729420	the account, prevent future access? How do we revoke current access, the current sessions,
1729420	1736160	which are the OAuth access tokens and refresh tokens? User and service accounts are different.
1736920	1743380	So here are some of the commonly available or known options. Let's go through them one by one.
1743420	1747940	It's going to be a little bit of remediation bingo, all right? As we go through this,
1747940	1752680	see if you can pick out or suggest the best option, and I'll tell you mine.
1753860	1757740	All right, user accounts. How do we lock out user accounts? Well, G Suite Admin,
1757740	1763400	okay, G Suite Admin, we can reset password, suspend the user, and reactivate it later,
1763400	1768340	delete the user. They're all effective. They all lock out the user or attacker. We'll pick
1768340	1775000	the lowest impact one. Resetting password, pretty common, works. That's the answer.
1775880	1783720	How about service accounts? Similar, parallel set of options. I can delete the key here. I'm
1783720	1788120	in Google Cloud Console for service accounts, right? G Suite Admin for user accounts,
1788120	1793020	Google Cloud for service accounts. I can disable the service account itself,
1793020	1804980	and I can delete the service account. Which one? They all work. We'll pick the lowest one. Rotate,
1804980	1809940	right? Now, let's get to the tough part. The tough part is revoking current access,
1809940	1817660	and that's the topic of this talk, right? So, let's see of these six suggested options,
1817660	1824500	the last three are the lockout account measures. The first three are potentially common ones that
1824500	1831100	people try or think about. Let's go through them one by one. In G Suite Admin, for a particular
1831100	1837500	user, their security section, there is a sign-in cookies and a reset action. What does this do? It
1837500	1845040	revokes all web sessions immediately. This includes Google Cloud Console. It would also include G
1845040	1853840	Suite, okay? This works. This is good. Include any Google app and service. This is great,
1853840	1858180	except it doesn't do anything about SDK sessions. So, actually, it's only half of a
1858180	1864320	solution. So, we lose. Let's keep going. There's an interesting G Cloud CLI command that you have
1864320	1871080	probably seen or used, auth revoke. It takes an account name, and it will revoke both the access
1871080	1878040	and refresh tokens associated for that user account. It, underneath, calls an API call with
1878040	1884360	this endpoint to revoke the specific token. That also works if you want to call it directly.
1884360	1889940	Sounds great, doesn't it? But it's in the middle of my list, so you know that it actually doesn't
1889940	1896620	work. Here's the problem. It can only be run on the G Cloud client machine. Why? Because you pass
1896620	1902820	in an account name, and it looks in the SQLite database cache to find the token. And that is
1902820	1911640	easily manipulated by the attacker, at a minimum, deleted, right? So, it can't be run anywhere else.
1911640	1915780	Frankly, I don't think any incident responder would trust a compromised client machine for
1915780	1925940	anything, right? Similarly, calling the API itself doesn't help. Why? You need to pass in
1926520	1934620	an OAuth access token or revoke, sorry, refresh token. Those are not logged. They're not accessible.
1934620	1938940	They don't exist anywhere except on the client machine, which is compromised, which is easily
1938940	1946140	manipulated by the attacker. So, it's useless. These are for self-help, for you to clean up your
1946140	1952140	cache and config, nothing else. A one side note, which is a diversion, but the call to revoke token
1952140	1957760	doesn't require authentication. You just pass in this long access and refresh token. It's long
1957760	1961180	enough that you probably can't brute force it, but it's an interesting note that you don't need
1961180	1972500	OAuth to call. Okay, let's cover that lockout service account, sorry, user lockout options,
1972500	1978800	all right? Changing the user password. These all don't work. They don't revoke SDK sessions.
1978800	1984680	There's a narrow case where if you have requested scope for Gmail, then resetting the password would
1984680	1990900	work, but that doesn't apply to us. We would have requested scope probably for just gcloud
1990900	1998020	or SDK access. Suspending user account works temporarily. While the user account is suspended,
1998020	2003780	the tokens will not work. Once, though, it's re-enabled, the tokens will start working again,
2003780	2008900	assuming they have not expired. So, that's not a great solution. Unless you suspend the account
2008900	2017660	for an hour, that would work. Sorry, let me backtrack on that. Deleting user account works,
2017660	2022860	but has high impact, okay? So, these options aren't great. So, this is all yellow. The question
2022860	2031880	is, what's our best option? So, one thing I did not list in the table was there's actually the best
2031880	2038940	option that works and is low impact is to delete the connected application. This is the connected
2038940	2044160	OAuth application in the G Suite admin side. Basically, every time you authenticate and
2044160	2050320	request access, the OAuth application, in this case, we're talking about the Google Cloud SDK,
2050320	2055720	is listed here, as well as this access level, and you can delete it. And deleting it has no
2055720	2062080	big ramifications. It just means next time someone accesses Google Cloud SDK, like through CLI,
2062080	2066180	they're forced to re-authenticate. But once you delete it, at that point in time, all session
2066180	2073400	tokens are revoked. So, it's the answer. That's the bingo. That's the bonus right there, okay?
2073400	2077680	And if you haven't worked with G Suite admin, you wouldn't know about it, or maybe think about
2077680	2082780	it, or if you're not an OAuth jockey, right? In OAuth land, this is pretty common to think about
2082780	2086940	these connections. It just means it's trusted. And once you delete it, it revokes the sessions.
2087180	2094580	All right. Let's talk about the service account. How do we revoke that? Well, do we revoke it? Do
2094580	2101100	we do some of these other lockout measures? Let's jump into it. Well, you might have guessed that
2101100	2105900	the revoke CLI and the API call won't work, because we just went through it for the user
2105900	2111340	account. It definitely doesn't work. You actually get errors here, right, when you try to do the
2111340	2115820	CLI or the API. Outright errors. It just says it doesn't apply. But what's of note is, look in the
2115820	2120600	error, sorry for the verbiage, but look at the error you get back from the CLI from gcloud.
2120660	2125380	It actually hints in the last sentence that there's hope. If you revoke the parent service
2125380	2132120	account, it's supposed to revoke the session, the actual sessions. What does that mean? Well,
2132120	2137400	if you suspend it or delete it, that's true. But the suspend is sort of a half answer,
2137400	2141580	because once you re-enable it, the tokens actually still work. What about the service
2141580	2147500	account key? Revoke it? What's that mean? Well, it means delete the API key. That definitely
2147500	2153560	doesn't work. That's a bug. Bug has been filed. Don't believe it, right? We've already gone
2153560	2160540	through this. This doesn't work. How about the measures to rotate and delete the API key? Look,
2160540	2166840	if you rotate, delete the API key, you can still use OAuth tokens and still access the service
2167480	2173600	login and do actions as the service account. If you disable it, at that point in time,
2173600	2179180	the tokens don't work, but they still exist. So once you re-enable, reactivate the service account,
2179180	2184600	they'll work again. So the solution there is you could suspend for an hour, then re-enable it,
2184600	2188840	because in that hour, tokens will expire. How about deleting service account? Works,
2188840	2197600	just like deleting user account, high impact. So the summary is actually disable for an hour
2198140	2203740	or delete and reprovision. You will choose whichever you're more comfortable with.
2203920	2208820	Whichever gets you back up and running quicker, that you can perform with confidence,
2208820	2215240	without making errors. This isn't great, but there are ways forward, right? And the main thing of
2215240	2219940	this whole slide and the time spent on all the paths that weren't great paths or were dead ends
2219940	2223440	is to make sure we don't waste time on that, especially in the middle of an incident, right?
2223440	2231740	Is to feed this and review our operations playbooks and make sure that we have the right actions for
2231740	2240420	remediation for the different types of accounts. So to wrap up in the next minute or so and at the
2240420	2245260	end, we went over a bunch of attack scenarios, right? Definitely user accounts and the cache
2245260	2249780	on an external client. That's really what we focused on. But don't forget that service accounts
2249780	2255980	also cache the access token and that an attacker would want to grab that because it helps them
2255980	2261400	maintain persistence in the light of all those remediation actions that don't quite work.
2261740	2268280	But it's not just external clients. If you switch to a Compute Engine environment and you've installed
2268280	2273800	SDK, all the same things apply. And it's not just things that you explicitly start. Remember to
2273800	2280300	think about Cloud Shell, which starts up a VM in the back with the latest SDK installed, right?
2280300	2289600	These are all having the same cache token kinds of exposure and opportunity. And fifth, last but
2289600	2297600	not least, the fifth area is even if SDK is not installed, right? Compute Engine compromise,
2297600	2303480	OAuth tokens for that service account are easily queried from the metadata service and can allow
2303480	2308980	the attacker access as well as persistence for that access. And from a defensive measure,
2308980	2314660	real quick summary after that long sort of part of the presentation, do three things on the
2314660	2320200	prevention side. The no-brainer is set your session timeout in G Suite. If you can, implement
2320200	2326900	IP whitelisting with service controls and set your MFA so that these all mitigate credential
2326900	2332160	compromise in some way. And if you do do that IP whitelisting, then you can improve your detection
2332900	2337380	by detecting the failed authorizations. From a remediation perspective for user accounts,
2337380	2342820	reset the password to lock it out and delete the OAuth connected app in G Suite. That's the key
2342820	2347800	one that may not be obvious to everyone. For service accounts, pick your poison. One of two,
2347800	2353480	disable reactivate within our time period or delete, recreate, reprovision. If you have that
2353480	2359860	automated, just make sure and want to get up and running faster. Just have to worry about VMs,
2359860	2366600	compute instances, anything using those service accounts. So that is the end and I will sort of
2366600	2372380	ask Jaron to see if we have any time for questions. Otherwise, you can always reach me
2372380	2382300	directly as well. Yeah, we have one question from Eric. Do you know of any blogs or white papers
2382300	2389400	that people can use to replicate the OAuth token hijacking that you showed earlier?
2391880	2404180	Yes, so there are some resources for sure. We will be blogging over the next week or two a lot of the
2404180	2410120	details from this presentation. So you'll actually be able to recreate what's seen in the video
2410840	2417120	with actual commands. But there's been prior work as well. So if you search on G Cloud,
2417510	2422530	I want to say if I have the name right, I didn't reference in the presentation,
2422980	2428540	I'll have to add that later. But John Haney has, I believe I have his name right, has done some
2428540	2434320	early work on just peeling apart the SQL databases and some of the REST API calls.
2434620	2441180	And that's a useful resource. It's not documented well, particularly, but it's also not hard to
2441180	2448740	delve into it. So both our blogs should be released in part, I think today, as well as
2448740	2452920	over the next two weeks, as well as there's some prior work if you search on some of the keywords
2452920	2457640	like the G Cloud, the dot config, the SQL light database names.