start	end	text
0	5140	Hi, everyone. I'm Alex Sierra. I'm going to be talking about possible SaaSpocalypse,
5140	10720	the complexity and power of AWS cross-account access. Just a little very quick tidbit about
10720	17160	myself. I am the founder of Tenshi Security, as was just described. I'm based out of Sao Paulo,
17160	24200	Brazil, and I'm currently focusing exclusively on cloud security as my day job on the new company.
24200	30800	This is where my interest in this topic came along because of engagements with customers
31340	36300	that had a lot of SaaS providers that were representing an interesting level of risk
36300	45720	to them. So let's get started. Why should you care? Why should you be interested in learning
45720	52180	more about cross-account access in AWS? The thing is, as the ATT&CK framework mentions,
52180	57960	abusing a relationship that already exists between a vendor and a customer, or what's also
57960	63480	known as supply chain attacks, can be a very effective way for an attacker to gain a foothold
63480	68940	into an organization. There are very well-known APT groups that have used that tactic in the past.
68960	75180	There's very well-known incidents that were brought about by this sort of technique.
75320	80420	And especially when you talk about the cloud, a lot of companies are already either all in
80420	85540	the cloud, startups, new projects, even at large companies, and existing infrastructures being
85540	93920	migrated to the cloud at a frantic pace. And it becomes a lot easier to grant access to your cloud
93920	100160	accounts to all sorts of SaaS providers, CSPM solutions or cloud security partial management
100160	106700	solutions, a single pane of glass, multi-cloud goodness for managing all the things,
106700	113740	identity providers, backup solutions, IT management, all sorts of products are taking
113740	121300	advantage of the fact that using the cloud service provider APIs to gain access to a customer's
121300	128680	account allows you to do a lot more automation and allows you to scale a lot better those services
128680	133820	that would have been a lot more complex to integrate with on-premises environments.
133820	138660	And so the tendency is that those SaaS providers are going to be connected to a lot of customers,
138660	142840	gaining access to a lot of accounts, and that's a huge concentrator of risk.
142940	149460	If you have a single provider that has access to thousands of different AWS accounts around the
149460	157120	world with a decent amount of privileges, that'll be a jackpot for any attacker. So that's a huge
157120	163800	risk concentrator, which is why we need to get granting those accesses right as defenders.
164380	172620	And so I'm going to start with a very quick, you know, a thousand feet high overview of AWS IAM,
172620	179080	just for people that are not very familiar with the concepts. But AWS has a few very interesting
179820	186260	concepts that we need to cover. First of all, on AWS, AWS accounts are self-contained.
186260	191020	The identities exist within the account, the privileges that they have exist within the
191020	196540	account, and the resources they access, the APIs, the actions they execute, all live within an
196540	204560	account. It's not possible directly for one, say, IAM user to action something on another account,
204560	211420	right? Everything that happens typically happens inside a single account. It's very different from,
211420	217320	for example, Azure, where you have an Active Directory user and it has access to several
217320	223560	subscriptions, which kind of would be the equivalence of AWS accounts. It's the same user,
223560	229700	they're just being given privileges on different subscriptions. That's not what happens on AWS,
229700	236260	right? So there are some resources on AWS that you can share across accounts explicitly,
237320	243080	most infamously S3 buckets, which people, you know, since they can share it with the world,
243080	247880	tend to do when they shouldn't. But other things like EBS snapshots,
247880	256440	AMIs, and if you say AMIs, I hate you, but AMIs are other kinds of resources that you can
257420	263860	share either publicly with the whole world or with specific AWS accounts. But more generally,
263860	268440	if you are doing some sort of work, if you are a SaaS provider and need to have some sort of access
268440	275160	into a customer AWS account, you need to have some sort of identity inside that account to be
275160	279960	able to be given privileges and access those actions and resources inside the customer account.
280580	288120	And so the way that this has been done typically is either you create something like an IAM user,
288120	294200	which is one sort of identity that has long lasting credentials. So if you have,
294200	298460	you can have a password to access a console. So for managed services provided,
298460	302760	it's typically what you probably get. Or if you're doing automated access, which is more
302760	308600	typical in the SaaS providers, you would have an access key ID and a secret access key, right?
308940	315300	But AWS also provides another way of doing this, which is using the STS or security token service,
315300	321140	where you create a role, which is a sort of a user that can't access the console, say.
321780	329360	And the way this works is you assign privileges to a role, you say who can assume that role or
329360	335260	become that role. And then the security token service will provide temporary credentials.
335260	338840	And then there will be an extra thing. So it will be the access key ID and the secret access key
338840	348900	and the session token that you can use to call APIs, AWS APIs, limited by those privileges,
348900	352300	specific period of time. So there are temporary credentials.
353700	359200	And so what you need to do to be able to do this is you have to configure the trust policy
359200	363260	on the role, which says who can assume that role. You assign it the proper permissions on
363260	367220	the permission policy. When you are the administrator and you are creating the
367220	372040	role, it can set a maximum session duration. But that's not enough. The identity that's
372040	377980	going to assume that role also needs to have privileges to call the STS assume role action,
377980	387660	which is very aptly named. And so this is what option one of how a SaaS provider can access
388300	394920	AWS accounts for customers looks like. And this looks like what you would do on Azure as well,
394920	402840	and what has been done over time for a variety of solutions. Essentially, you create an IAM user,
402840	409420	you assign it the necessary privileges with identity policies, you create long lived access
409420	414440	keys, and then the customer somehow uses a trusted channel, don't you love those,
414440	420920	to send those credentials to the MSP. It can be just going on a web portal to do the self
421680	428380	onboarding, right? And then what happens is inside the vendor, that's where the trick is,
428380	434820	any machine, any lambda, any container, or any person that needs to perform actions on that
434820	439540	customer's account, will need to be able to retrieve those credentials. So those credentials
439540	445800	need to be stored long term. If they are encrypted, they need to use reversible encryption,
445800	450920	because every time that vendor needs to call API's on behalf of that customer to access
450920	457820	their AWS account, those credentials need to be there. And that's kind of the problem,
457820	464420	right? Credentials are secret, right? If they aren't leaked, that's basically everything that
464420	469920	an attacker needs to be able to access that customer's account. So what happens is there's
470080	476480	a lot of burden associated with maintaining secrets. You need to rotate them regularly,
476480	480360	right? The customer will probably mandate that. So that's a lot of work for the customer,
480360	484880	that's a lot of work for the vendor, the SaaS vendor, especially if a lot of custom
484880	492300	accounts are involved, right? They're very easy to lose, right? I mean, all sorts of ways.
492300	497980	And since every time they need to be used, they will be accessed, that's an opportunity to lose
497980	503660	them, to leak them, right? So someone can break into the worker machines. If there are human
503660	510800	beings involved, their machines can be hacked. Backups can be accidentally published somewhere.
510800	516460	All sorts of things can happen where those secrets leak. And if that happens, if the
516460	523360	secrets that a SaaS vendor uses to access 1000 AWS accounts from customers are lost,
523360	530360	the only way to get back to a secure state is to ask all of those customers
531040	536360	to rotate those credentials, to invalidate the old credentials, generate new ones and
536360	540540	reconfigure them on the MSP. So that generates a lot of work, a lot of downtime, and it has to
540540	546300	be done in a hurry to minimize the exposure window, right? And so if one of your customers
546300	552700	didn't get the email, is on vacation, whatever, that environment will continue to be exposed,
552700	558920	and there's nothing that the SaaS vendor can do unless plead with their customers to rotate the
558920	565600	secrets and be exposed to the liability of any losses the customers incur because of their
565600	573060	breach, because of their leaking of those secrets. So yeah, that's not ideal, right?
574040	579660	Option B looks more interesting. That's where we use roles to do cross-account access.
579660	585200	And the way this works is you create a role on the customer environment, and the trust policy says
585840	592440	the vendor AWS account can assume that role. And it's important to keep in mind this is being
592440	598800	assigned to the entire account. It's up to the vendor to control who inside their account
600100	605080	gets the STS assume role privileges to assume the role. As far as the customer is concerned,
605080	610600	they're trusting the entire SaaS vendor account. They have to trust an account ID, right? So they
610600	615460	create a role, they set up a trust policy saying, I trust this vendor's account, and then they
615460	621380	assign the privileges to that role using the permissions policy that the vendor needs to
621380	627560	perform their work. And now on the vendor side, what they do is they need to control essentially
627560	635000	who gets to call STS assume role on the customer's role, right? That's it. Whenever a machine,
635000	640140	an instance, a container, Lambda, or a person needs to work on that customer's account,
640140	646780	they need to know what the unique identifier of that role is, or the ARN in Amazon parlance,
646780	654320	the ARN. And so they assume that role, they get temporary credentials, they don't store it
654320	659240	anywhere, those are ephemeral credentials, they just keep it in memory, use it for the few API
659240	664740	calls they need to do, and then they throw them away. And they repeat that process each time,
664740	671420	right? And so a lot of risk is already being minimized here, because those credentials are
671420	678320	temporary. Even if they do leak, they will last a short amount of time. And if you have thousands
678320	682920	of accounts, but you're only accessing a few at a time, even if an attacker compromises all of
682920	690960	your worker instances, they probably can read secrets in memory, say. If they can't also have
690960	695240	the ability to call assume role, they will only get secrets for a few customers, and they will
695790	700660	not last long. So there's a lot of mitigation already taking place. But the most important one
700660	706000	is, this is the only state that the vendor needs to keep. They only need to know what
706000	712680	ARN they need to assume, the ARN of the role that they need to assume. This is not a secret.
712680	718360	Even if this leaks, if the database containing all the customer ARNs that they can
718360	724280	connect to is leaked, there's nothing an attacker can do with it, because they still need to break
724280	731320	into the SaaS vendor account and call STS assume role from that account. Because this is what's
731320	737600	happening. AWS, because any identities on the customer account are being authenticated against
737600	743640	AWS. Any identities inside the vendor account are being authenticated against AWS. And then AWS
743640	751220	acts as a middleman to say, if a request comes to assume that role, AWS is assuring that this
751220	757840	is coming from a principle, from an identity that really belongs to the SaaS vendor account
757840	763940	and has been properly authenticated. You don't need another secret. AWS is kind of making the
763940	770960	trust transitive in that sense. And so an attacker that just has the state the vendor keeps here,
770960	777760	still can't do anything and just hack into the vendor. But it's not perfect. This is still naive.
777760	783580	as I described, because there's a problem. We have what's called the confused deputy problem.
783680	792360	So here's what happens. Imagine you go into G Suite today. And you say, look, I own,
792360	799040	I don't know, Capital One.com, right, or Google.com. And I want to create a G Suite domain,
799040	803000	you know, and create email addresses and send emails on behalf of that domain.
803500	809220	What they will tell you is, first of all, you need to prove to me that you own that domain.
809220	816080	Otherwise, you're just someone that's trying to do something evil here and impersonate people
816080	821680	from that place. You need to prove first, somehow that you own that DNS domain. Same if you're
821680	831080	issuing certificates on Let's Encrypt, they'll ask you to post like a public TXT record saying,
831080	836560	proving that you are the admin of that domain. Otherwise, people could just issue certificates
836560	841300	on behalf of domains they don't own and send up phishing sites and things like that.
841540	845720	This is the same problem that's going to happen here. Someone could log into one of the SaaS tools
845720	852700	and find out a target's account ID and pretend to own that account and say, look,
853230	856960	I own this account ID. Can you please do a vulnerability scan? Or can you do a backup
856960	863440	of the data there? And then if that account was set up previously by the customer,
863440	869720	by the actual owner to work with that same SaaS vendor, the account does trust the SaaS vendor
869720	875720	account. So it will work. The SaaS vendor will, let's say, scan, if it's a CSPM solution,
875720	880060	it's going to scan that account twice, once on behalf of the actual owner and another time
880060	884980	on behalf of the attacker. And so the attacker will have access to whatever information
884980	890940	that SaaS vendor or functionality that SaaS vendor provides to an account they don't own.
890940	899400	So that's a serious problem. And so what AWS did to solve this is to create something similar
899400	906060	to what Let's Encrypt does or G Suite does, which is we're going to issue a challenge
906060	914200	that proves that you are an admin of that target account. And actually, that you are an admin of
914200	921640	that role that you're asking me to assume, to let you assume. And so they created what's called
921640	928480	an external ID. This is a non-secret value, which is simply a challenge. The same way that
928480	934740	the value that you're adding to a TXT record, to prove to Let's Encrypt or G Suite,
934740	944700	that you are the admin of that DNS domain. This is simply a nonce, a token that the SaaS vendor
944700	951860	will choose and say to the customer or to the prospective customer saying, if you really own
951860	958280	this account, you're going to configure the trust policy on the role to only allow me in
958840	966100	if this exact nonce is used, this exact external ID is used. And so on the left here, you see what
966100	973020	that trust policy would look like. So you would have the STS assume role allow, you would have
973020	977840	on the customer side, you would configure that on the role, you would put the ID of the
977840	984920	SaaS vendor AWS account, and then you would add a condition, which is something that's
984920	993000	standard on IAM policy syntax, saying that STS external ID needs to match exactly the
993560	999000	value that was created by the SaaS vendor. And so what happens on the other side is that the
999000	1005840	vendor, when they do call assume role, besides telling AWS which role they want to assume,
1005840	1013020	they will also pass an optional external ID. And so if the two things match, then the activity will
1013020	1018640	be successful, the role, the temporary credential will be returned, and everything will be fine.
1018940	1024120	Right? And so in a nutshell, this is what this looks like. The customer creates the role as
1024120	1030300	before, but they add this extra condition to make sure that they're using an external ID
1030860	1035760	that the SaaS vendor has created, right? And they again, assign the correct privileges.
1035840	1042800	The SaaS vendor now has to do more work, though. What they have to do is to make sure that
1042800	1048920	they are choosing the challenge, and that the challenge is unique, in the sense that they don't
1048920	1056680	repeat or it's very unlikely that they will repeat that challenge, right? To make sure that
1058040	1063940	anyone that can change the trust policy on the role they're assuming can be assumed to be an
1063940	1069880	admin of that account and of that role, right? Once that happens, everything is the same.
1069880	1074500	The thing that changed is now there's an additional piece of state as well that the
1074500	1080520	vendors need to keep around, which is not only they have to know what the customer role arm is,
1080520	1086060	but they also need to know what that challenge value is. But let's keep in mind, neither of
1086060	1093280	those are secrets. Again, even if an attacker got a hold of all of that information, if a database
1093280	1100980	dump was leaked, they still don't have any access into customer accounts unless they also were able
1100980	1108680	to break into somehow the SaaS vendor account and can issue STS assume role calls from there,
1108680	1117540	which is the key part, right? Where they need to be authenticated into the SaaS vendor account,
1117540	1123080	right? So this is vastly superior to the scenario we had with the shared secret.
1123280	1130980	Right? So a few caveats here that are important to keep in mind, right? The first one is
1131740	1139600	you can't just check if the operation as a vendor, you need to confirm that in order to
1139600	1144780	confirm that that account belongs to the person that's claiming that account, you can't just
1144780	1151560	do an assume role passing the external ID and seeing if it succeeds. Because here's the problem.
1152190	1159640	If the customer incorrectly sets the trust policy to not have that condition,
1159640	1163860	it just says, I trust that account ID, but they don't add the condition block
1163860	1169780	that requires the external ID, that operation will succeed. Here's the thing. The way that
1169780	1178540	condition keys work on IAM policies is that they're like environment variables, right? You've
1178540	1184420	added a value that can be used or not. But if it's not used, it's not going to have an effect,
1184420	1192160	right? So basically, when the vendor calls STS assume role and adds the external ID on their end,
1192160	1197680	unless the trust policy is checking for it, it won't have an impact. AWS is not treating this
1197680	1204380	as a special value, that if it's not checked on the policy, that the evaluation is going to fail.
1204380	1209060	So essentially, you need to check twice as a vendor, you need to check that you
1209060	1215600	can do the assume role with the external ID. But you also need to check that you cannot that it
1215600	1221680	fails if you do not provide the external ID. That's the only way that you can be sure that
1221680	1228020	the customer really wrote the trust policy correctly with the condition there. So you need
1228020	1233480	to check those two things. And that's very important. The other thing that's really important
1234030	1241820	is what privileges the SaaS vendor will give to the worker. I'm using this general term to
1241820	1246600	describe the instances, the lambdas, the containers, or the human beings that will
1246600	1255560	be able to use those credentials to assume the roles and do API calls on the customer accounts.
1255620	1261840	There are no conditions involved, I'm sorry. And so, since you do not know in advance
1262280	1270420	what all of the account IDs of all the customers are, you might be tempted to do the first version
1270420	1277360	here on the top that has Drake pretty sad. And so asterisks are bad, especially if they're by
1277360	1282720	themselves, they get really angry when they're alone. And so having a resource asterisk there
1282720	1288580	is a horrible idea, please do not do that. Basically, what this is doing is allowing that
1288580	1296620	worker to have unrestricted STS assume role to any role on any account that has a trust policy,
1296620	1303180	trust in your account, but also any role inside the SaaS vendor account as well.
1303180	1308140	So if you have a role lying around that allows for administrative privileges,
1308140	1313870	and that will be the case if your account is part of an AWS organization,
1316270	1321310	then whoever gets those privileges, or if an attacker breaks into your worker,
1321310	1326970	if that human being is a malicious insider of their laptop, their credentials are stolen or
1326970	1332170	something like that, then an attacker can easily use this to escalate privileges and take over
1332170	1337790	the vendor account. So that's a really bad idea. You want to do something that looks like
1337790	1344530	the second version there, where you have standardized the role names on the customers.
1344630	1351010	So that already mitigates a lot of risk. Because even if you just had the allow statement there,
1351010	1358930	it would only allow a role with that name on the SaaS vendor account to be executed. But I would
1358930	1365810	also add that deny as well, that then covers that possibility. So that second version is allowing you
1365810	1371770	to do STS assume roles, only two roles, in that case called SaaS cross account role,
1371770	1379430	you can choose whatever name you want, but on any account other than your own account.
1380270	1387050	And so that would prevent you from overprivileging your workers and allowing an attacker or a
1387050	1392810	malicious insider to do a privilege escalation. So that's really important to keep in mind.
1394790	1401890	It would have been pretty nice if you were able to use SCPs to do something similar,
1401890	1407430	to create an SCP that says the only identity within the SaaS vendor account that can do assume
1407430	1413690	roles for other accounts is this lambda or this container task or whatever.
1413930	1419690	But I haven't been able to do it. And the reason is this code here on the left would have worked
1419810	1425750	a treat, I think, and probably a lot of people that know a lot more than I do will correct me
1425750	1434190	soon on this. But here's the problem. For some weird reason that I have no idea what they are,
1434190	1441670	SCPs do not support the not resource syntax. They only support resource. They support not action,
1441670	1448390	but they do not support not resource. So I can't say I'll deny for anything that's not on my
1448390	1459330	account. That can't be done on an SCP. So why? I mean, why? In any case, that means that you need
1459330	1463890	to protect, as you should anyway, your root account privileges, for example, or anyone with
1463890	1469250	the administrator privileges, because they will still be able to assume roles on all your customers
1469250	1477770	as well if they have those privileges on the SaaS vendor account. CloudTrail is always your friend.
1477770	1485690	You can see those STS assume role activities happening on the customer account as well.
1485690	1491170	This will be logged there. So if you are a vendor or if you are a customer, you need to monitor
1491170	1496910	those activities and make sure that you're not seeing something weird. So for example, any
1496910	1504610	attempts that fail, that are trying to use a different external ID, or once you know
1505250	1511930	what the identity, what the temporary credentials were assigned to a vendor are, if you look for
1511930	1519150	what API calls they're doing with those temporary credentials later on, you can try to find
1519150	1524070	patterns, especially for things that are blocked. If you have a vendor that asks you for
1524070	1531190	the privileges they need, and they have automated tasks running, it shouldn't, I think, have a lot
1531190	1535430	of blocked activities. They shouldn't be trying to do things that they didn't ask
1535430	1539970	you the privileges to do. So there are lots of things that you can very easily monitor that should
1539970	1546450	be low volume, high impact events that you can monitor using CloudTrail. So not only the STS
1546450	1551270	assume role operations themselves, and you can see that the external ID is not a secret because it's
1551270	1558250	explicitly included on the request parameters there. AWS does not consider that to be secret.
1559190	1564390	And so monitoring the STS assume role operations themselves, maybe you find out about cross
1564390	1569930	account access that you didn't know existed, because you see that on the log on your scene,
1569930	1576030	or you keep track of the identities assigned to the vendor, and then you see what other
1576030	1580750	activities they performed. And especially you pay attention to things that might indicate
1580750	1589390	enumeration, like reconnaissance, and denied activities, which again, for automated workloads,
1589390	1594670	probably shouldn't happen that often, right? And probably worth your time investigating.
1595630	1602330	And so I want to highlight the research from Kasten Broughton, I hope I'm pronouncing that
1602330	1610930	right, which is amazing. He looked at this, and he said, how many vendors are doing this right?
1610930	1617910	And so he tested about 200 vendors, which is awesome. This is a lot of vendors. And to my
1617910	1622350	surprise, he found that even though all of the advantages we just described to using the assume
1622350	1630430	role method, about 50% of the vendors he tested chose to work exclusively with IAM user credentials,
1630430	1636710	long-lived credentials, they chose to continue having to deal in secrets with all of the
1636710	1643230	disadvantages that we mentioned, which is to me like a wasted opportunity to do better, to be
1643230	1653310	quite frank. And of those 50% or 100 user vendors that did use the assume role way of getting access
1653310	1663410	to customer accounts, 98% of them, so just two vendors, did this right. So 98% of them got this
1663410	1671510	wrong. Actually did that second check to see if the assume role failed, if they didn't provide the
1671510	1679150	external ID. So only two of them, two out of 100, actually checked if the customer configured the
1679150	1686390	policy correctly, which is kind of depressing if you ask me. And so the thing here is that
1686990	1693030	they would be amenable to a customer configuring things incorrectly, and then essentially the
1693630	1700130	confused deputy problem is still there. 37% of them don't understand what external IDs are,
1700130	1706390	and they explicitly, by design, allow customers to pick their own external IDs on the UI
1706390	1713670	when they're onboarding their integration into the SaaS vendor. And so this defeats
1713670	1719970	the whole purpose, the whole purpose of having the external ID in the first place. It's as if
1719970	1727150	Let's Encrypt said, so configure any TXT record you want to prove you own this domain. That's
1727150	1734530	not how it works. Another 15% unwittingly allowed it to happen this way because of failures,
1734530	1739570	there was a way to bypass the UI, go straight to the API and do this, or some other vulnerability
1739570	1748470	allowed Kasten to do it. So over half of the vendors, 52% of the vendors that used assumed
1748470	1754870	roles had a way for the customer, or the attacker in this case, to just pick whatever external ID
1754870	1759970	they wanted and essentially force the confused deputy problems to happen. So that's a bit
1759970	1765570	depressing. That's amazing research. Check out his talk on Forward Cloud SAC where he delves into
1765570	1772630	this into more detail. And privileges. Whenever you're giving people privileges, you need to be
1772630	1778970	very wary of what those privileges are. You need to make sure you are using this privilege,
1778970	1787110	otherwise you're going to be in a world of pain, right? And so cloud is pretty new, right? And IAM
1787790	1795790	for AWS is really complicated. I'm pretty sure if someone, you know, works the mathematical
1795790	1801490	proofs, it's going to be Turing complete at this point, which makes me particularly mad they didn't
1801490	1810310	do, you know, not resource on SCPs because, you know, anyway. But you should, of course, minimize
1810310	1819290	the privileges that you are assigning to this vendor-customer relationship. And it's an interest
1819290	1823390	of both parties involved, right? As a customer, of course, you want to minimize the amount of
1823390	1827770	privileges anyone has from outside into your environment. But as a vendor, you don't want
1827770	1834170	that liability either, right? If you accidentally ask for more privileges than you need and you can
1834170	1844110	be potentially blamed for illegal access to, I don't know, credit card data, PII, trade secrets,
1844110	1850730	all sorts of things, that's a lot of problems that can cause you as far as legal liability,
1850730	1858270	fines, and all sorts of fun things. So it's in the interest of both parties to keep those
1858270	1863890	privileges to a minimum, right? And so, of course, everyone does that, right?
1866070	1874330	No, not really, no. And so Ben Reeser, and again, I hope I'm saying this right, published this
1874330	1881070	tweet recently, I think last week, and it was very timely for my talk. Thank you very much, Ben.
1881070	1887010	Essentially, he's a user of Okta, which is a publicly traded company.
1887550	1894970	It's a SaaS identity provider, right? And a lot of large corporations use Okta for identity
1895550	1903310	management. And they were asking for very, very overbroad privileges there, like they were asking
1903310	1910410	that they were able to assume any role inside of your account. You can see how that's problematic,
1910410	1918710	right? I mean, an IDP would need to have pretty high privileges, right? Anyway, but this is just
1918710	1926090	too much, right? They can very easily be abused to gain admin level on any AWS account pretty
1926090	1933150	easily, especially again, if you are using AWS organizations, you should write an SSL.
1933150	1941710	And so it's a pretty serious problem. And Ben, I have not been involved in the disclosure process,
1941710	1947110	or I haven't talked directly to Okta about this. What I hear from Ben, and I asked for his
1947110	1954590	permission before sharing this and his opinion on this, is that Okta has been very slow to respond.
1954950	1960310	He's been six months trying to convince them that this is serious. So far, they have updated their
1960310	1967030	documentation. But as far as Ben knows, or from what he told me, they have not notified existing
1967030	1973990	customers to change and reduce those overly permissive roles that have been assigned.
1973990	1979230	So this is the SaaSpocalypse scenario right here. This is a large company with thousands of
1979230	1986610	customers, potentially tens of thousands of AWS accounts connected to them, and an attacker that's
1986610	1992150	able to break into the Okta AWS account that's trusted by all those companies,
1992770	1999130	will very easily be able to find a role, because they have list roles, by the way,
1999130	2005110	and then assume a role with as many privileges as they need, and then take over Okta's customers'
2005110	2011390	accounts. So they did get the external ID right, though, but the privileges, not so much.
2011930	2017650	If you are an Okta customer, look at their updated documentation, update your environment.
2017990	2024090	If you are from Okta, and I'm wrong here, and you are working on this, and they might be just
2024090	2029190	waiting for legal approval or going through some extra hoops internally to be able to notify
2029190	2034870	customers, etc., which I know nothing about, I can only assume they have very talented,
2034870	2042950	competent security and legal people at Okta. So, talk to them, if you are that person,
2042950	2047450	and you have not been involved in this, and get this moving, because there's a lot of potential
2047450	2055530	risk involved. And something that, I mean, Chris Ferris has essentially the same thing,
2055530	2060490	only for natural control, so a vendor that didn't know how to specify less permissive
2061290	2066950	roles essentially asked for, you know, let me manage all of your VPC, what could possibly go
2066950	2075810	wrong? What I have found really, really often, though, is that a lot of vendors ask for the read
2075810	2082810	only AWS managed access policy. And it seems harmless enough, and it's a start, because at
2082810	2087950	least they're not asking to be able to change things in your environment, so yay, I guess.
2087950	2093690	But the problem is that read only access is very misleading. If you actually look at the
2093690	2099390	privileges it's providing, it allows you to do things like read all S3 objects on every bucket
2099390	2105240	on your environment, and list all buckets, perform queries on databases like DynamoDB,
2105870	2111370	and really dangerous things that would give a vendor access to, like, company data.
2111470	2116900	And I've seen this be requested by people, by vendors like, you know, cloud cost optimization,
2117540	2122380	or CSPM providers. They have no business getting access to your data. They only need to see,
2122380	2127620	like, your billing, or even how things are configured, right? But not have access to your
2127620	2134960	actual confidential data, right? So one quick replacement here would be to use the view only
2134960	2140800	access, which is a much better policy in that regard. But, you know, apply list privilege as
2140800	2149460	much as possible, and not necessarily the existing policies are going to be best suited to
2149460	2157240	what you need, right? So in closing, what do I recommend? If you're a customer, you know,
2157240	2162220	you need to be aware of the third party access to your AWS accounts. You need to have governance
2162220	2167820	over it. You need a process to do a regular review, not only on initial onboarding, because those
2168320	2172060	things... privileges only grow, they never diminish. So you need to have regular reviews
2172060	2177660	of what actual privileges are being given to third parties. Ensure there's still business need. We,
2177660	2183440	on our consulting practice, have seen many times the companies that had no longer a contract with
2183440	2189920	our clients, but the access they were given was never revoked. So people need to go account by
2189920	2193800	account and revoke that. So you need to be looking for that, and you can automate that to look for
2193800	2200340	trust policies and so on and so forth. There's an amazing tool written by Kinnard McQuaid called
2200340	2207180	Cloud Splaining that will go through your accounts, and it will look for overprivilege and things that
2207180	2213060	can be abused to exfiltrate data or to do privilege escalation. It's automated. It's awesome. Use it
2213060	2219040	in Parliament by Scott Piper. It can also help you do some QA of policies to make sure required
2219040	2223740	things are not missing, like you're accidentally using something that's not working well. You're
2223740	2230260	an external ID, things like that, and monitor using CloudTrail. If you're a vendor, right,
2230840	2234760	you have the brunt of the work. You have the biggest responsibility here, right? And you
2234760	2239440	have the biggest liability, probably, because you have the sun's liability of all your customers.
2240100	2244080	Minimize as much as you can the privileges asked of customers. And I know this is a balancing act,
2244080	2247500	because if you ask for too few privileges, then when you do need more privileges,
2247500	2251180	you have to go back to customers and ask them to change things, which is bad.
2251180	2256380	But I would err in the side of caution. But then again, that's why I work in security.
2257060	2263220	Make sure you're using external IDs correctly. If you can't easily enforce that the external
2263220	2270220	IDs are unique, use a random UUID. It's such a big key space that statistically you probably
2270220	2275940	won't be in trouble. Ensure customers are correctly implementing the trust policy.
2275940	2282360	I would say standardize the role names as well and use CloudFormation. So the customers don't
2282360	2287620	have to manually set up anything. They just drop CloudFormation stack and everything is done
2287620	2293800	beforehand. There are plenty of examples online if you want to look it up. Or I can provide one
2293800	2299540	later if anyone requests. Limit the attack surface of that account that everyone trusts.
2299540	2303900	You need to run as few things as possible. You need to have as few humans as possible
2303900	2309840	that have access to that one account that every customer is trusting. You need to reduce that
2309840	2318440	attack surface as much as you can. Because if that one account gets broken into, then that's
2318540	2322940	a very bad day for you. The upside is if you're using cross-account access, as soon as you kick
2322940	2327680	the attacker out, they can no longer access your customers. And your customers don't need to do
2327680	2333780	anything themselves. But still, the value for an attacker of breaking into that one trusted
2333780	2339800	account is huge because of all that concentrated access. So you need to be extra careful.
2339920	2345380	Separate it from everything else that you can. And monitor extensively what's happening in that
2345380	2351180	account using CloudTrail. Monitoring is going to be a lot easier if you have less things running on it.