Hi, everyone. I'm Alex Sierra. I'm going to be talking about possible SaaSpocalypse,
the complexity and power of AWS cross-account access. Just a little very quick tidbit about
myself. I am the founder of Tenshi Security, as was just described. I'm based out of Sao Paulo,
Brazil, and I'm currently focusing exclusively on cloud security as my day job on the new company.
This is where my interest in this topic came along because of engagements with customers
that had a lot of SaaS providers that were representing an interesting level of risk
to them. So let's get started. Why should you care? Why should you be interested in learning
more about cross-account access in AWS? The thing is, as the ATT&CK framework mentions,
abusing a relationship that already exists between a vendor and a customer, or what's also
known as supply chain attacks, can be a very effective way for an attacker to gain a foothold
into an organization. There are very well-known APT groups that have used that tactic in the past.
There's very well-known incidents that were brought about by this sort of technique.
And especially when you talk about the cloud, a lot of companies are already either all in
the cloud, startups, new projects, even at large companies, and existing infrastructures being
migrated to the cloud at a frantic pace. And it becomes a lot easier to grant access to your cloud
accounts to all sorts of SaaS providers, CSPM solutions or cloud security partial management
solutions, a single pane of glass, multi-cloud goodness for managing all the things,
identity providers, backup solutions, IT management, all sorts of products are taking
advantage of the fact that using the cloud service provider APIs to gain access to a customer's
account allows you to do a lot more automation and allows you to scale a lot better those services
that would have been a lot more complex to integrate with on-premises environments.
And so the tendency is that those SaaS providers are going to be connected to a lot of customers,
gaining access to a lot of accounts, and that's a huge concentrator of risk.
If you have a single provider that has access to thousands of different AWS accounts around the
world with a decent amount of privileges, that'll be a jackpot for any attacker. So that's a huge
risk concentrator, which is why we need to get granting those accesses right as defenders.
And so I'm going to start with a very quick, you know, a thousand feet high overview of AWS IAM,
just for people that are not very familiar with the concepts. But AWS has a few very interesting
concepts that we need to cover. First of all, on AWS, AWS accounts are self-contained.
The identities exist within the account, the privileges that they have exist within the
account, and the resources they access, the APIs, the actions they execute, all live within an
account. It's not possible directly for one, say, IAM user to action something on another account,
right? Everything that happens typically happens inside a single account. It's very different from,
for example, Azure, where you have an Active Directory user and it has access to several
subscriptions, which kind of would be the equivalence of AWS accounts. It's the same user,
they're just being given privileges on different subscriptions. That's not what happens on AWS,
right? So there are some resources on AWS that you can share across accounts explicitly,
most infamously S3 buckets, which people, you know, since they can share it with the world,
tend to do when they shouldn't. But other things like EBS snapshots,
AMIs, and if you say AMIs, I hate you, but AMIs are other kinds of resources that you can
share either publicly with the whole world or with specific AWS accounts. But more generally,
if you are doing some sort of work, if you are a SaaS provider and need to have some sort of access
into a customer AWS account, you need to have some sort of identity inside that account to be
able to be given privileges and access those actions and resources inside the customer account.
And so the way that this has been done typically is either you create something like an IAM user,
which is one sort of identity that has long lasting credentials. So if you have,
you can have a password to access a console. So for managed services provided,
it's typically what you probably get. Or if you're doing automated access, which is more
typical in the SaaS providers, you would have an access key ID and a secret access key, right?
But AWS also provides another way of doing this, which is using the STS or security token service,
where you create a role, which is a sort of a user that can't access the console, say.
And the way this works is you assign privileges to a role, you say who can assume that role or
become that role. And then the security token service will provide temporary credentials.
And then there will be an extra thing. So it will be the access key ID and the secret access key
and the session token that you can use to call APIs, AWS APIs, limited by those privileges,
specific period of time. So there are temporary credentials.
And so what you need to do to be able to do this is you have to configure the trust policy
on the role, which says who can assume that role. You assign it the proper permissions on
the permission policy. When you are the administrator and you are creating the
role, it can set a maximum session duration. But that's not enough. The identity that's
going to assume that role also needs to have privileges to call the STS assume role action,
which is very aptly named. And so this is what option one of how a SaaS provider can access
AWS accounts for customers looks like. And this looks like what you would do on Azure as well,
and what has been done over time for a variety of solutions. Essentially, you create an IAM user,
you assign it the necessary privileges with identity policies, you create long lived access
keys, and then the customer somehow uses a trusted channel, don't you love those,
to send those credentials to the MSP. It can be just going on a web portal to do the self
onboarding, right? And then what happens is inside the vendor, that's where the trick is,
any machine, any lambda, any container, or any person that needs to perform actions on that
customer's account, will need to be able to retrieve those credentials. So those credentials
need to be stored long term. If they are encrypted, they need to use reversible encryption,
because every time that vendor needs to call API's on behalf of that customer to access
their AWS account, those credentials need to be there. And that's kind of the problem,
right? Credentials are secret, right? If they aren't leaked, that's basically everything that
an attacker needs to be able to access that customer's account. So what happens is there's
a lot of burden associated with maintaining secrets. You need to rotate them regularly,
right? The customer will probably mandate that. So that's a lot of work for the customer,
that's a lot of work for the vendor, the SaaS vendor, especially if a lot of custom
accounts are involved, right? They're very easy to lose, right? I mean, all sorts of ways.
And since every time they need to be used, they will be accessed, that's an opportunity to lose
them, to leak them, right? So someone can break into the worker machines. If there are human
beings involved, their machines can be hacked. Backups can be accidentally published somewhere.
All sorts of things can happen where those secrets leak. And if that happens, if the
secrets that a SaaS vendor uses to access 1000 AWS accounts from customers are lost,
the only way to get back to a secure state is to ask all of those customers
to rotate those credentials, to invalidate the old credentials, generate new ones and
reconfigure them on the MSP. So that generates a lot of work, a lot of downtime, and it has to
be done in a hurry to minimize the exposure window, right? And so if one of your customers
didn't get the email, is on vacation, whatever, that environment will continue to be exposed,
and there's nothing that the SaaS vendor can do unless plead with their customers to rotate the
secrets and be exposed to the liability of any losses the customers incur because of their
breach, because of their leaking of those secrets. So yeah, that's not ideal, right?
Option B looks more interesting. That's where we use roles to do cross-account access.
And the way this works is you create a role on the customer environment, and the trust policy says
the vendor AWS account can assume that role. And it's important to keep in mind this is being
assigned to the entire account. It's up to the vendor to control who inside their account
gets the STS assume role privileges to assume the role. As far as the customer is concerned,
they're trusting the entire SaaS vendor account. They have to trust an account ID, right? So they
create a role, they set up a trust policy saying, I trust this vendor's account, and then they
assign the privileges to that role using the permissions policy that the vendor needs to
perform their work. And now on the vendor side, what they do is they need to control essentially
who gets to call STS assume role on the customer's role, right? That's it. Whenever a machine,
an instance, a container, Lambda, or a person needs to work on that customer's account,
they need to know what the unique identifier of that role is, or the ARN in Amazon parlance,
the ARN. And so they assume that role, they get temporary credentials, they don't store it
anywhere, those are ephemeral credentials, they just keep it in memory, use it for the few API
calls they need to do, and then they throw them away. And they repeat that process each time,
right? And so a lot of risk is already being minimized here, because those credentials are
temporary. Even if they do leak, they will last a short amount of time. And if you have thousands
of accounts, but you're only accessing a few at a time, even if an attacker compromises all of
your worker instances, they probably can read secrets in memory, say. If they can't also have
the ability to call assume role, they will only get secrets for a few customers, and they will
not last long. So there's a lot of mitigation already taking place. But the most important one
is, this is the only state that the vendor needs to keep. They only need to know what
ARN they need to assume, the ARN of the role that they need to assume. This is not a secret.
Even if this leaks, if the database containing all the customer ARNs that they can
connect to is leaked, there's nothing an attacker can do with it, because they still need to break
into the SaaS vendor account and call STS assume role from that account. Because this is what's
happening. AWS, because any identities on the customer account are being authenticated against
AWS. Any identities inside the vendor account are being authenticated against AWS. And then AWS
acts as a middleman to say, if a request comes to assume that role, AWS is assuring that this
is coming from a principle, from an identity that really belongs to the SaaS vendor account
and has been properly authenticated. You don't need another secret. AWS is kind of making the
trust transitive in that sense. And so an attacker that just has the state the vendor keeps here,
still can't do anything and just hack into the vendor. But it's not perfect. This is still naive.
as I described, because there's a problem. We have what's called the confused deputy problem.
So here's what happens. Imagine you go into G Suite today. And you say, look, I own,
I don't know, Capital One.com, right, or Google.com. And I want to create a G Suite domain,
you know, and create email addresses and send emails on behalf of that domain.
What they will tell you is, first of all, you need to prove to me that you own that domain.
Otherwise, you're just someone that's trying to do something evil here and impersonate people
from that place. You need to prove first, somehow that you own that DNS domain. Same if you're
issuing certificates on Let's Encrypt, they'll ask you to post like a public TXT record saying,
proving that you are the admin of that domain. Otherwise, people could just issue certificates
on behalf of domains they don't own and send up phishing sites and things like that.
This is the same problem that's going to happen here. Someone could log into one of the SaaS tools
and find out a target's account ID and pretend to own that account and say, look,
I own this account ID. Can you please do a vulnerability scan? Or can you do a backup
of the data there? And then if that account was set up previously by the customer,
by the actual owner to work with that same SaaS vendor, the account does trust the SaaS vendor
account. So it will work. The SaaS vendor will, let's say, scan, if it's a CSPM solution,
it's going to scan that account twice, once on behalf of the actual owner and another time
on behalf of the attacker. And so the attacker will have access to whatever information
that SaaS vendor or functionality that SaaS vendor provides to an account they don't own.
So that's a serious problem. And so what AWS did to solve this is to create something similar
to what Let's Encrypt does or G Suite does, which is we're going to issue a challenge
that proves that you are an admin of that target account. And actually, that you are an admin of
that role that you're asking me to assume, to let you assume. And so they created what's called
an external ID. This is a non-secret value, which is simply a challenge. The same way that
the value that you're adding to a TXT record, to prove to Let's Encrypt or G Suite,
that you are the admin of that DNS domain. This is simply a nonce, a token that the SaaS vendor
will choose and say to the customer or to the prospective customer saying, if you really own
this account, you're going to configure the trust policy on the role to only allow me in
if this exact nonce is used, this exact external ID is used. And so on the left here, you see what
that trust policy would look like. So you would have the STS assume role allow, you would have
on the customer side, you would configure that on the role, you would put the ID of the
SaaS vendor AWS account, and then you would add a condition, which is something that's
standard on IAM policy syntax, saying that STS external ID needs to match exactly the
value that was created by the SaaS vendor. And so what happens on the other side is that the
vendor, when they do call assume role, besides telling AWS which role they want to assume,
they will also pass an optional external ID. And so if the two things match, then the activity will
be successful, the role, the temporary credential will be returned, and everything will be fine.
Right? And so in a nutshell, this is what this looks like. The customer creates the role as
before, but they add this extra condition to make sure that they're using an external ID
that the SaaS vendor has created, right? And they again, assign the correct privileges.
The SaaS vendor now has to do more work, though. What they have to do is to make sure that
they are choosing the challenge, and that the challenge is unique, in the sense that they don't
repeat or it's very unlikely that they will repeat that challenge, right? To make sure that
anyone that can change the trust policy on the role they're assuming can be assumed to be an
admin of that account and of that role, right? Once that happens, everything is the same.
The thing that changed is now there's an additional piece of state as well that the
vendors need to keep around, which is not only they have to know what the customer role arm is,
but they also need to know what that challenge value is. But let's keep in mind, neither of
those are secrets. Again, even if an attacker got a hold of all of that information, if a database
dump was leaked, they still don't have any access into customer accounts unless they also were able
to break into somehow the SaaS vendor account and can issue STS assume role calls from there,
which is the key part, right? Where they need to be authenticated into the SaaS vendor account,
right? So this is vastly superior to the scenario we had with the shared secret.
Right? So a few caveats here that are important to keep in mind, right? The first one is
you can't just check if the operation as a vendor, you need to confirm that in order to
confirm that that account belongs to the person that's claiming that account, you can't just
do an assume role passing the external ID and seeing if it succeeds. Because here's the problem.
If the customer incorrectly sets the trust policy to not have that condition,
it just says, I trust that account ID, but they don't add the condition block
that requires the external ID, that operation will succeed. Here's the thing. The way that
condition keys work on IAM policies is that they're like environment variables, right? You've
added a value that can be used or not. But if it's not used, it's not going to have an effect,
right? So basically, when the vendor calls STS assume role and adds the external ID on their end,
unless the trust policy is checking for it, it won't have an impact. AWS is not treating this
as a special value, that if it's not checked on the policy, that the evaluation is going to fail.
So essentially, you need to check twice as a vendor, you need to check that you
can do the assume role with the external ID. But you also need to check that you cannot that it
fails if you do not provide the external ID. That's the only way that you can be sure that
the customer really wrote the trust policy correctly with the condition there. So you need
to check those two things. And that's very important. The other thing that's really important
is what privileges the SaaS vendor will give to the worker. I'm using this general term to
describe the instances, the lambdas, the containers, or the human beings that will
be able to use those credentials to assume the roles and do API calls on the customer accounts.
There are no conditions involved, I'm sorry. And so, since you do not know in advance
what all of the account IDs of all the customers are, you might be tempted to do the first version
here on the top that has Drake pretty sad. And so asterisks are bad, especially if they're by
themselves, they get really angry when they're alone. And so having a resource asterisk there
is a horrible idea, please do not do that. Basically, what this is doing is allowing that
worker to have unrestricted STS assume role to any role on any account that has a trust policy,
trust in your account, but also any role inside the SaaS vendor account as well.
So if you have a role lying around that allows for administrative privileges,
and that will be the case if your account is part of an AWS organization,
then whoever gets those privileges, or if an attacker breaks into your worker,
if that human being is a malicious insider of their laptop, their credentials are stolen or
something like that, then an attacker can easily use this to escalate privileges and take over
the vendor account. So that's a really bad idea. You want to do something that looks like
the second version there, where you have standardized the role names on the customers.
So that already mitigates a lot of risk. Because even if you just had the allow statement there,
it would only allow a role with that name on the SaaS vendor account to be executed. But I would
also add that deny as well, that then covers that possibility. So that second version is allowing you
to do STS assume roles, only two roles, in that case called SaaS cross account role,
you can choose whatever name you want, but on any account other than your own account.
And so that would prevent you from overprivileging your workers and allowing an attacker or a
malicious insider to do a privilege escalation. So that's really important to keep in mind.
It would have been pretty nice if you were able to use SCPs to do something similar,
to create an SCP that says the only identity within the SaaS vendor account that can do assume
roles for other accounts is this lambda or this container task or whatever.
But I haven't been able to do it. And the reason is this code here on the left would have worked
a treat, I think, and probably a lot of people that know a lot more than I do will correct me
soon on this. But here's the problem. For some weird reason that I have no idea what they are,
SCPs do not support the not resource syntax. They only support resource. They support not action,
but they do not support not resource. So I can't say I'll deny for anything that's not on my
account. That can't be done on an SCP. So why? I mean, why? In any case, that means that you need
to protect, as you should anyway, your root account privileges, for example, or anyone with
the administrator privileges, because they will still be able to assume roles on all your customers
as well if they have those privileges on the SaaS vendor account. CloudTrail is always your friend.
You can see those STS assume role activities happening on the customer account as well.
This will be logged there. So if you are a vendor or if you are a customer, you need to monitor
those activities and make sure that you're not seeing something weird. So for example, any
attempts that fail, that are trying to use a different external ID, or once you know
what the identity, what the temporary credentials were assigned to a vendor are, if you look for
what API calls they're doing with those temporary credentials later on, you can try to find
patterns, especially for things that are blocked. If you have a vendor that asks you for
the privileges they need, and they have automated tasks running, it shouldn't, I think, have a lot
of blocked activities. They shouldn't be trying to do things that they didn't ask
you the privileges to do. So there are lots of things that you can very easily monitor that should
be low volume, high impact events that you can monitor using CloudTrail. So not only the STS
assume role operations themselves, and you can see that the external ID is not a secret because it's
explicitly included on the request parameters there. AWS does not consider that to be secret.
And so monitoring the STS assume role operations themselves, maybe you find out about cross
account access that you didn't know existed, because you see that on the log on your scene,
or you keep track of the identities assigned to the vendor, and then you see what other
activities they performed. And especially you pay attention to things that might indicate
enumeration, like reconnaissance, and denied activities, which again, for automated workloads,
probably shouldn't happen that often, right? And probably worth your time investigating.
And so I want to highlight the research from Kasten Broughton, I hope I'm pronouncing that
right, which is amazing. He looked at this, and he said, how many vendors are doing this right?
And so he tested about 200 vendors, which is awesome. This is a lot of vendors. And to my
surprise, he found that even though all of the advantages we just described to using the assume
role method, about 50% of the vendors he tested chose to work exclusively with IAM user credentials,
long-lived credentials, they chose to continue having to deal in secrets with all of the
disadvantages that we mentioned, which is to me like a wasted opportunity to do better, to be
quite frank. And of those 50% or 100 user vendors that did use the assume role way of getting access
to customer accounts, 98% of them, so just two vendors, did this right. So 98% of them got this
wrong. Actually did that second check to see if the assume role failed, if they didn't provide the
external ID. So only two of them, two out of 100, actually checked if the customer configured the
policy correctly, which is kind of depressing if you ask me. And so the thing here is that
they would be amenable to a customer configuring things incorrectly, and then essentially the
confused deputy problem is still there. 37% of them don't understand what external IDs are,
and they explicitly, by design, allow customers to pick their own external IDs on the UI
when they're onboarding their integration into the SaaS vendor. And so this defeats
the whole purpose, the whole purpose of having the external ID in the first place. It's as if
Let's Encrypt said, so configure any TXT record you want to prove you own this domain. That's
not how it works. Another 15% unwittingly allowed it to happen this way because of failures,
there was a way to bypass the UI, go straight to the API and do this, or some other vulnerability
allowed Kasten to do it. So over half of the vendors, 52% of the vendors that used assumed
roles had a way for the customer, or the attacker in this case, to just pick whatever external ID
they wanted and essentially force the confused deputy problems to happen. So that's a bit
depressing. That's amazing research. Check out his talk on Forward Cloud SAC where he delves into
this into more detail. And privileges. Whenever you're giving people privileges, you need to be
very wary of what those privileges are. You need to make sure you are using this privilege,
otherwise you're going to be in a world of pain, right? And so cloud is pretty new, right? And IAM
for AWS is really complicated. I'm pretty sure if someone, you know, works the mathematical
proofs, it's going to be Turing complete at this point, which makes me particularly mad they didn't
do, you know, not resource on SCPs because, you know, anyway. But you should, of course, minimize
the privileges that you are assigning to this vendor-customer relationship. And it's an interest
of both parties involved, right? As a customer, of course, you want to minimize the amount of
privileges anyone has from outside into your environment. But as a vendor, you don't want
that liability either, right? If you accidentally ask for more privileges than you need and you can
be potentially blamed for illegal access to, I don't know, credit card data, PII, trade secrets,
all sorts of things, that's a lot of problems that can cause you as far as legal liability,
fines, and all sorts of fun things. So it's in the interest of both parties to keep those
privileges to a minimum, right? And so, of course, everyone does that, right?
No, not really, no. And so Ben Reeser, and again, I hope I'm saying this right, published this
tweet recently, I think last week, and it was very timely for my talk. Thank you very much, Ben.
Essentially, he's a user of Okta, which is a publicly traded company.
It's a SaaS identity provider, right? And a lot of large corporations use Okta for identity
management. And they were asking for very, very overbroad privileges there, like they were asking
that they were able to assume any role inside of your account. You can see how that's problematic,
right? I mean, an IDP would need to have pretty high privileges, right? Anyway, but this is just
too much, right? They can very easily be abused to gain admin level on any AWS account pretty
easily, especially again, if you are using AWS organizations, you should write an SSL.
And so it's a pretty serious problem. And Ben, I have not been involved in the disclosure process,
or I haven't talked directly to Okta about this. What I hear from Ben, and I asked for his
permission before sharing this and his opinion on this, is that Okta has been very slow to respond.
He's been six months trying to convince them that this is serious. So far, they have updated their
documentation. But as far as Ben knows, or from what he told me, they have not notified existing
customers to change and reduce those overly permissive roles that have been assigned.
So this is the SaaSpocalypse scenario right here. This is a large company with thousands of
customers, potentially tens of thousands of AWS accounts connected to them, and an attacker that's
able to break into the Okta AWS account that's trusted by all those companies,
will very easily be able to find a role, because they have list roles, by the way,
and then assume a role with as many privileges as they need, and then take over Okta's customers'
accounts. So they did get the external ID right, though, but the privileges, not so much.
If you are an Okta customer, look at their updated documentation, update your environment.
If you are from Okta, and I'm wrong here, and you are working on this, and they might be just
waiting for legal approval or going through some extra hoops internally to be able to notify
customers, etc., which I know nothing about, I can only assume they have very talented,
competent security and legal people at Okta. So, talk to them, if you are that person,
and you have not been involved in this, and get this moving, because there's a lot of potential
risk involved. And something that, I mean, Chris Ferris has essentially the same thing,
only for natural control, so a vendor that didn't know how to specify less permissive
roles essentially asked for, you know, let me manage all of your VPC, what could possibly go
wrong? What I have found really, really often, though, is that a lot of vendors ask for the read
only AWS managed access policy. And it seems harmless enough, and it's a start, because at
least they're not asking to be able to change things in your environment, so yay, I guess.
But the problem is that read only access is very misleading. If you actually look at the
privileges it's providing, it allows you to do things like read all S3 objects on every bucket
on your environment, and list all buckets, perform queries on databases like DynamoDB,
and really dangerous things that would give a vendor access to, like, company data.
And I've seen this be requested by people, by vendors like, you know, cloud cost optimization,
or CSPM providers. They have no business getting access to your data. They only need to see,
like, your billing, or even how things are configured, right? But not have access to your
actual confidential data, right? So one quick replacement here would be to use the view only
access, which is a much better policy in that regard. But, you know, apply list privilege as
much as possible, and not necessarily the existing policies are going to be best suited to
what you need, right? So in closing, what do I recommend? If you're a customer, you know,
you need to be aware of the third party access to your AWS accounts. You need to have governance
over it. You need a process to do a regular review, not only on initial onboarding, because those
things... privileges only grow, they never diminish. So you need to have regular reviews
of what actual privileges are being given to third parties. Ensure there's still business need. We,
on our consulting practice, have seen many times the companies that had no longer a contract with
our clients, but the access they were given was never revoked. So people need to go account by
account and revoke that. So you need to be looking for that, and you can automate that to look for
trust policies and so on and so forth. There's an amazing tool written by Kinnard McQuaid called
Cloud Splaining that will go through your accounts, and it will look for overprivilege and things that
can be abused to exfiltrate data or to do privilege escalation. It's automated. It's awesome. Use it
in Parliament by Scott Piper. It can also help you do some QA of policies to make sure required
things are not missing, like you're accidentally using something that's not working well. You're
an external ID, things like that, and monitor using CloudTrail. If you're a vendor, right,
you have the brunt of the work. You have the biggest responsibility here, right? And you
have the biggest liability, probably, because you have the sun's liability of all your customers.
Minimize as much as you can the privileges asked of customers. And I know this is a balancing act,
because if you ask for too few privileges, then when you do need more privileges,
you have to go back to customers and ask them to change things, which is bad.
But I would err in the side of caution. But then again, that's why I work in security.
Make sure you're using external IDs correctly. If you can't easily enforce that the external
IDs are unique, use a random UUID. It's such a big key space that statistically you probably
won't be in trouble. Ensure customers are correctly implementing the trust policy.
I would say standardize the role names as well and use CloudFormation. So the customers don't
have to manually set up anything. They just drop CloudFormation stack and everything is done
beforehand. There are plenty of examples online if you want to look it up. Or I can provide one
later if anyone requests. Limit the attack surface of that account that everyone trusts.
You need to run as few things as possible. You need to have as few humans as possible
that have access to that one account that every customer is trusting. You need to reduce that
attack surface as much as you can. Because if that one account gets broken into, then that's
a very bad day for you. The upside is if you're using cross-account access, as soon as you kick
the attacker out, they can no longer access your customers. And your customers don't need to do
anything themselves. But still, the value for an attacker of breaking into that one trusted
account is huge because of all that concentrated access. So you need to be extra careful.
Separate it from everything else that you can. And monitor extensively what's happening in that
account using CloudTrail. Monitoring is going to be a lot easier if you have less things running on it.