1
00:00:00,000 --> 00:00:00,920
Hi, everyone.

2
00:00:00,920 --> 00:00:01,820
I'm Alex Sierra.

3
00:00:01,820 --> 00:00:08,420
I'm going to be talking about possible SaaSpocalypse, the complexity and power of AWS cross-account access.

4
00:00:08,540 --> 00:00:11,780
Just a little very quick tidbit about myself.

5
00:00:12,320 --> 00:00:16,000
I am the founder of Tenshi Security, as was just described.

6
00:00:16,000 --> 00:00:24,200
I'm based out of Sao Paulo, Brazil, and I'm currently focusing exclusively on cloud security as my day job on the new company.

7
00:00:24,200 --> 00:00:37,860
This is where my interest in this topic came along because of engagements with customers that had a lot of SaaS providers that were representing an interesting level of risk to them.

8
00:00:38,060 --> 00:00:40,160
So let's get started.

9
00:00:40,460 --> 00:00:42,380
Why should you care?

10
00:00:42,380 --> 00:00:49,000
Why should you be interested in learning more about cross-account access in AWS?

11
00:00:49,000 --> 00:01:04,600
The thing is, as the ATT&CK framework mentions, abusing a relationship that already exists between a vendor and a customer, or what's also known as supply chain attacks, can be a very effective way for an attacker to gain a foothold into an organization.

12
00:01:04,600 --> 00:01:08,940
There are very well-known APT groups that have used that tactic in the past.

13
00:01:08,960 --> 00:01:15,180
There's very well-known incidents that were brought about by this sort of technique.

14
00:01:15,320 --> 00:01:28,080
And especially when you talk about the cloud, a lot of companies are already either all in the cloud, startups, new projects, even at large companies, and existing infrastructures being migrated to the cloud at a frantic pace.

15
00:01:28,440 --> 00:01:48,960
And it becomes a lot easier to grant access to your cloud accounts to all sorts of SaaS providers, CSPM solutions or cloud security partial management solutions, a single pane of glass, multi-cloud goodness for managing all the things, identity providers,

16
00:01:48,960 --> 00:02:13,820
backup solutions, IT management, all sorts of products are taking advantage of the fact that using the cloud service provider APIs to gain access to a customer's account allows you to do a lot more automation and allows you to scale a lot better those services that would have been a lot more complex to integrate with on-premises environments.

17
00:02:13,820 --> 00:02:22,840
And so the tendency is that those SaaS providers are going to be connected to a lot of customers, gaining access to a lot of accounts, and that's a huge concentrator of risk.

18
00:02:22,940 --> 00:02:36,340
If you have a single provider that has access to thousands of different AWS accounts around the world with a decent amount of privileges, that'll be a jackpot for any attacker.

19
00:02:36,340 --> 00:02:43,800
So that's a huge risk concentrator, which is why we need to get granting those accesses right as defenders.

20
00:02:44,380 --> 00:02:55,540
And so I'm going to start with a very quick, you know, a thousand feet high overview of AWS IAM, just for people that are not very familiar with the concepts.

21
00:02:55,840 --> 00:03:02,080
But AWS has a few very interesting concepts that we need to cover.

22
00:03:02,080 --> 00:03:06,260
First of all, on AWS, AWS accounts are self-contained.

23
00:03:06,260 --> 00:03:17,040
The identities exist within the account, the privileges that they have exist within the account, and the resources they access, the APIs, the actions they execute, all live within an account.

24
00:03:17,040 --> 00:03:24,780
It's not possible directly for one, say, IAM user to action something on another account, right?

25
00:03:24,780 --> 00:03:29,820
Everything that happens typically happens inside a single account.

26
00:03:29,820 --> 00:03:42,060
It's very different from, for example, Azure, where you have an Active Directory user and it has access to several subscriptions, which kind of would be the equivalence of AWS accounts.

27
00:03:42,060 --> 00:03:47,780
It's the same user, they're just being given privileges on different subscriptions.

28
00:03:47,780 --> 00:03:50,140
That's not what happens on AWS, right?

29
00:03:50,140 --> 00:04:05,320
So there are some resources on AWS that you can share across accounts explicitly, most infamously S3 buckets, which people, you know, since they can share it with the world, tend to do when they shouldn't.

30
00:04:05,480 --> 00:04:22,600
But other things like EBS snapshots, AMIs, and if you say AMIs, I hate you, but AMIs are other kinds of resources that you can share either publicly with the whole world or with specific AWS accounts.

31
00:04:22,600 --> 00:04:39,960
But more generally, if you are doing some sort of work, if you are a SaaS provider and need to have some sort of access into a customer AWS account, you need to have some sort of identity inside that account to be able to be given privileges and access those actions and resources inside the customer account.

32
00:04:40,580 --> 00:04:53,300
And so the way that this has been done typically is either you create something like an IAM user, which is one sort of identity that has long lasting credentials.

33
00:04:53,300 --> 00:04:56,480
So if you have, you can have a password to access a console.

34
00:04:56,480 --> 00:05:00,240
So for managed services provided, it's typically what you probably get.

35
00:05:00,360 --> 00:05:08,600
Or if you're doing automated access, which is more typical in the SaaS providers, you would have an access key ID and a secret access key, right?

36
00:05:08,940 --> 00:05:21,140
But AWS also provides another way of doing this, which is using the STS or security token service, where you create a role, which is a sort of a user that can't access the console, say.

37
00:05:21,780 --> 00:05:30,900
And the way this works is you assign privileges to a role, you say who can assume that role or become that role.

38
00:05:31,320 --> 00:05:35,260
And then the security token service will provide temporary credentials.

39
00:05:35,260 --> 00:05:36,720
And then there will be an extra thing.

40
00:05:36,720 --> 00:05:50,480
So it will be the access key ID and the secret access key and the session token that you can use to call APIs, AWS APIs, limited by those privileges, specific period of time.

41
00:05:50,480 --> 00:05:52,300
So there are temporary credentials.

42
00:05:53,700 --> 00:06:01,720
And so what you need to do to be able to do this is you have to configure the trust policy on the role, which says who can assume that role.

43
00:06:01,720 --> 00:06:04,700
You assign it the proper permissions on the permission policy.

44
00:06:04,920 --> 00:06:09,180
When you are the administrator and you are creating the role, it can set a maximum session duration.

45
00:06:09,280 --> 00:06:10,780
But that's not enough.

46
00:06:10,780 --> 00:06:20,080
The identity that's going to assume that role also needs to have privileges to call the STS assume role action, which is very aptly named.

47
00:06:21,220 --> 00:06:32,140
And so this is what option one of how a SaaS provider can access AWS accounts for customers looks like.

48
00:06:32,140 --> 00:06:39,400
And this looks like what you would do on Azure as well, and what has been done over time for a variety of solutions.

49
00:06:39,440 --> 00:06:57,140
Essentially, you create an IAM user, you assign it the necessary privileges with identity policies, you create long lived access keys, and then the customer somehow uses a trusted channel, don't you love those, to send those credentials to the MSP.

50
00:06:57,140 --> 00:07:03,000
It can be just going on a web portal to do the self onboarding, right?

51
00:07:03,000 --> 00:07:18,960
And then what happens is inside the vendor, that's where the trick is, any machine, any lambda, any container, or any person that needs to perform actions on that customer's account, will need to be able to retrieve those credentials.

52
00:07:18,960 --> 00:07:22,200
So those credentials need to be stored long term.

53
00:07:22,620 --> 00:07:35,140
If they are encrypted, they need to use reversible encryption, because every time that vendor needs to call API's on behalf of that customer to access their AWS account, those credentials need to be there.

54
00:07:35,340 --> 00:07:38,660
And that's kind of the problem, right?

55
00:07:38,840 --> 00:07:40,720
Credentials are secret, right?

56
00:07:40,720 --> 00:07:48,220
If they aren't leaked, that's basically everything that an attacker needs to be able to access that customer's account.

57
00:07:48,580 --> 00:07:53,140
So what happens is there's a lot of burden associated with maintaining secrets.

58
00:07:54,000 --> 00:07:56,740
You need to rotate them regularly, right?

59
00:07:56,740 --> 00:07:58,800
The customer will probably mandate that.

60
00:07:58,800 --> 00:08:07,100
So that's a lot of work for the customer, that's a lot of work for the vendor, the SaaS vendor, especially if a lot of custom accounts are involved, right?

61
00:08:07,640 --> 00:08:09,920
They're very easy to lose, right?

62
00:08:09,920 --> 00:08:12,300
I mean, all sorts of ways.

63
00:08:12,300 --> 00:08:19,540
And since every time they need to be used, they will be accessed, that's an opportunity to lose them, to leak them, right?

64
00:08:19,540 --> 00:08:23,040
So someone can break into the worker machines.

65
00:08:23,040 --> 00:08:26,220
If there are human beings involved, their machines can be hacked.

66
00:08:27,780 --> 00:08:30,800
Backups can be accidentally published somewhere.

67
00:08:30,800 --> 00:08:34,240
All sorts of things can happen where those secrets leak.

68
00:08:34,440 --> 00:08:55,340
And if that happens, if the secrets that a SaaS vendor uses to access 1000 AWS accounts from customers are lost, the only way to get back to a secure state is to ask all of those customers to rotate those credentials, to invalidate the old credentials,

69
00:08:55,340 --> 00:08:58,080
generate new ones and reconfigure them on the MSP.

70
00:08:58,080 --> 00:09:04,220
So that generates a lot of work, a lot of downtime, and it has to be done in a hurry to minimize the exposure window, right?

71
00:09:04,220 --> 00:09:26,340
And so if one of your customers didn't get the email, is on vacation, whatever, that environment will continue to be exposed, and there's nothing that the SaaS vendor can do unless plead with their customers to rotate the secrets and be exposed to the liability of any losses the customers incur because of their breach,

72
00:09:26,340 --> 00:09:29,400
because of their leaking of those secrets.

73
00:09:29,640 --> 00:09:33,060
So yeah, that's not ideal, right?

74
00:09:34,040 --> 00:09:36,500
Option B looks more interesting.

75
00:09:36,500 --> 00:09:39,660
That's where we use roles to do cross-account access.

76
00:09:39,660 --> 00:09:49,980
And the way this works is you create a role on the customer environment, and the trust policy says the vendor AWS account can assume that role.

77
00:09:49,980 --> 00:09:55,120
And it's important to keep in mind this is being assigned to the entire account.

78
00:09:55,120 --> 00:10:03,800
It's up to the vendor to control who inside their account gets the STS assume role privileges to assume the role.

79
00:10:03,800 --> 00:10:07,960
As far as the customer is concerned, they're trusting the entire SaaS vendor account.

80
00:10:07,960 --> 00:10:10,140
They have to trust an account ID, right?

81
00:10:10,140 --> 00:10:22,820
So they create a role, they set up a trust policy saying, I trust this vendor's account, and then they assign the privileges to that role using the permissions policy that the vendor needs to perform their work.

82
00:10:22,860 --> 00:10:32,420
And now on the vendor side, what they do is they need to control essentially who gets to call STS assume role on the customer's role, right?

83
00:10:32,420 --> 00:10:33,500
That's it.

84
00:10:33,500 --> 00:10:48,260
Whenever a machine, an instance, a container, Lambda, or a person needs to work on that customer's account, they need to know what the unique identifier of that role is, or the ARN in Amazon parlance, the ARN.

85
00:10:49,560 --> 00:11:01,980
And so they assume that role, they get temporary credentials, they don't store it anywhere, those are ephemeral credentials, they just keep it in memory, use it for the few API calls they need to do, and then they throw them away.

86
00:11:01,980 --> 00:11:05,340
And they repeat that process each time, right?

87
00:11:05,340 --> 00:11:12,000
And so a lot of risk is already being minimized here, because those credentials are temporary.

88
00:11:12,000 --> 00:11:17,540
Even if they do leak, they will last a short amount of time.

89
00:11:17,540 --> 00:11:28,420
And if you have thousands of accounts, but you're only accessing a few at a time, even if an attacker compromises all of your worker instances, they probably can read secrets in memory, say.

90
00:11:28,720 --> 00:11:37,080
If they can't also have the ability to call assume role, they will only get secrets for a few customers, and they will not last long.

91
00:11:37,080 --> 00:11:39,440
So there's a lot of mitigation already taking place.

92
00:11:39,440 --> 00:11:44,720
But the most important one is, this is the only state that the vendor needs to keep.

93
00:11:44,720 --> 00:11:51,040
They only need to know what ARN they need to assume, the ARN of the role that they need to assume.

94
00:11:51,040 --> 00:11:52,680
This is not a secret.

95
00:11:52,680 --> 00:12:10,200
Even if this leaks, if the database containing all the customer ARNs that they can connect to is leaked, there's nothing an attacker can do with it, because they still need to break into the SaaS vendor account and call STS assume role from that account.

96
00:12:10,200 --> 00:12:12,400
Because this is what's happening.

97
00:12:12,500 --> 00:12:18,700
AWS, because any identities on the customer account are being authenticated against AWS.

98
00:12:18,700 --> 00:12:22,960
Any identities inside the vendor account are being authenticated against AWS.

99
00:12:22,960 --> 00:12:39,820
And then AWS acts as a middleman to say, if a request comes to assume that role, AWS is assuring that this is coming from a principle, from an identity that really belongs to the SaaS vendor account and has been properly authenticated.

100
00:12:39,820 --> 00:12:41,820
You don't need another secret.

101
00:12:41,820 --> 00:12:46,320
AWS is kind of making the trust transitive in that sense.

102
00:12:46,320 --> 00:12:55,120
And so an attacker that just has the state the vendor keeps here, still can't do anything and just hack into the vendor.

103
00:12:55,180 --> 00:12:56,600
But it's not perfect.

104
00:12:56,600 --> 00:12:57,760
This is still naive.

105
00:12:57,760 --> 00:13:00,740
as I described, because there's a problem.

106
00:13:00,740 --> 00:13:03,580
We have what's called the confused deputy problem.

107
00:13:03,680 --> 00:13:05,060
So here's what happens.

108
00:13:05,680 --> 00:13:10,380
Imagine you go into G Suite today.

109
00:13:10,380 --> 00:13:16,680
And you say, look, I own, I don't know, Capital One.com, right, or Google.com.

110
00:13:16,700 --> 00:13:23,000
And I want to create a G Suite domain, you know, and create email addresses and send emails on behalf of that domain.

111
00:13:23,500 --> 00:13:29,220
What they will tell you is, first of all, you need to prove to me that you own that domain.

112
00:13:29,220 --> 00:13:37,360
Otherwise, you're just someone that's trying to do something evil here and impersonate people from that place.

113
00:13:37,360 --> 00:13:41,100
You need to prove first, somehow that you own that DNS domain.

114
00:13:41,100 --> 00:13:54,040
Same if you're issuing certificates on Let's Encrypt, they'll ask you to post like a public TXT record saying, proving that you are the admin of that domain.

115
00:13:54,120 --> 00:14:01,300
Otherwise, people could just issue certificates on behalf of domains they don't own and send up phishing sites and things like that.

116
00:14:01,540 --> 00:14:03,500
This is the same problem that's going to happen here.

117
00:14:03,500 --> 00:14:14,540
Someone could log into one of the SaaS tools and find out a target's account ID and pretend to own that account and say, look, I own this account ID.

118
00:14:14,540 --> 00:14:16,220
Can you please do a vulnerability scan?

119
00:14:16,220 --> 00:14:18,420
Or can you do a backup of the data there?

120
00:14:19,220 --> 00:14:30,640
And then if that account was set up previously by the customer, by the actual owner to work with that same SaaS vendor, the account does trust the SaaS vendor account.

121
00:14:30,920 --> 00:14:32,260
So it will work.

122
00:14:32,260 --> 00:14:41,960
The SaaS vendor will, let's say, scan, if it's a CSPM solution, it's going to scan that account twice, once on behalf of the actual owner and another time on behalf of the attacker.

123
00:14:41,960 --> 00:14:50,940
And so the attacker will have access to whatever information that SaaS vendor or functionality that SaaS vendor provides to an account they don't own.

124
00:14:50,940 --> 00:14:53,500
So that's a serious problem.

125
00:14:53,620 --> 00:15:10,620
And so what AWS did to solve this is to create something similar to what Let's Encrypt does or G Suite does, which is we're going to issue a challenge that proves that you are an admin of that target account.

126
00:15:10,760 --> 00:15:19,380
And actually, that you are an admin of that role that you're asking me to assume, to let you assume.

127
00:15:19,660 --> 00:15:23,000
And so they created what's called an external ID.

128
00:15:23,000 --> 00:15:27,500
This is a non-secret value, which is simply a challenge.

129
00:15:27,500 --> 00:15:38,240
The same way that the value that you're adding to a TXT record, to prove to Let's Encrypt or G Suite, that you are the admin of that DNS domain.

130
00:15:38,240 --> 00:16:01,820
This is simply a nonce, a token that the SaaS vendor will choose and say to the customer or to the prospective customer saying, if you really own this account, you're going to configure the trust policy on the role to only allow me in if this exact nonce is used,

131
00:16:01,820 --> 00:16:04,460
this exact external ID is used.

132
00:16:04,500 --> 00:16:08,080
And so on the left here, you see what that trust policy would look like.

133
00:16:08,120 --> 00:16:29,060
So you would have the STS assume role allow, you would have on the customer side, you would configure that on the role, you would put the ID of the SaaS vendor AWS account, and then you would add a condition, which is something that's standard on IAM policy syntax,

134
00:16:29,060 --> 00:16:35,880
saying that STS external ID needs to match exactly the value that was created by the SaaS vendor.

135
00:16:36,560 --> 00:16:49,420
And so what happens on the other side is that the vendor, when they do call assume role, besides telling AWS which role they want to assume, they will also pass an optional external ID.

136
00:16:49,700 --> 00:16:58,640
And so if the two things match, then the activity will be successful, the role, the temporary credential will be returned, and everything will be fine.

137
00:16:58,940 --> 00:16:59,300
Right?

138
00:16:59,300 --> 00:17:02,380
And so in a nutshell, this is what this looks like.

139
00:17:02,380 --> 00:17:13,680
The customer creates the role as before, but they add this extra condition to make sure that they're using an external ID that the SaaS vendor has created, right?

140
00:17:13,680 --> 00:17:15,760
And they again, assign the correct privileges.

141
00:17:15,840 --> 00:17:19,860
The SaaS vendor now has to do more work, though.

142
00:17:19,860 --> 00:17:33,720
What they have to do is to make sure that they are choosing the challenge, and that the challenge is unique, in the sense that they don't repeat or it's very unlikely that they will repeat that challenge, right?

143
00:17:33,720 --> 00:17:46,560
To make sure that anyone that can change the trust policy on the role they're assuming can be assumed to be an admin of that account and of that role, right?

144
00:17:47,340 --> 00:17:49,880
Once that happens, everything is the same.

145
00:17:49,880 --> 00:18:03,440
The thing that changed is now there's an additional piece of state as well that the vendors need to keep around, which is not only they have to know what the customer role arm is, but they also need to know what that challenge value is.

146
00:18:03,440 --> 00:18:07,800
But let's keep in mind, neither of those are secrets.

147
00:18:07,800 --> 00:18:28,680
Again, even if an attacker got a hold of all of that information, if a database dump was leaked, they still don't have any access into customer accounts unless they also were able to break into somehow the SaaS vendor account and can issue STS assume role calls from there,

148
00:18:28,680 --> 00:18:31,600
which is the key part, right?

149
00:18:31,600 --> 00:18:37,740
Where they need to be authenticated into the SaaS vendor account, right?

150
00:18:37,740 --> 00:18:43,080
So this is vastly superior to the scenario we had with the shared secret.

151
00:18:43,280 --> 00:18:44,100
Right?

152
00:18:45,040 --> 00:18:49,500
So a few caveats here that are important to keep in mind, right?

153
00:18:49,500 --> 00:19:10,220
The first one is you can't just check if the operation as a vendor, you need to confirm that in order to confirm that that account belongs to the person that's claiming that account, you can't just do an assume role passing the external ID and seeing if it succeeds.

154
00:19:10,220 --> 00:19:11,560
Because here's the problem.

155
00:19:12,190 --> 00:19:28,340
If the customer incorrectly sets the trust policy to not have that condition, it just says, I trust that account ID, but they don't add the condition block that requires the external ID, that operation will succeed.

156
00:19:28,340 --> 00:19:29,220
Here's the thing.

157
00:19:29,220 --> 00:19:38,060
The way that condition keys work on IAM policies is that they're like environment variables, right?

158
00:19:38,060 --> 00:19:41,780
You've added a value that can be used or not.

159
00:19:41,780 --> 00:19:44,880
But if it's not used, it's not going to have an effect, right?

160
00:19:44,880 --> 00:19:56,280
So basically, when the vendor calls STS assume role and adds the external ID on their end, unless the trust policy is checking for it, it won't have an impact.

161
00:19:56,280 --> 00:20:04,380
AWS is not treating this as a special value, that if it's not checked on the policy, that the evaluation is going to fail.

162
00:20:04,380 --> 00:20:12,380
So essentially, you need to check twice as a vendor, you need to check that you can do the assume role with the external ID.

163
00:20:12,380 --> 00:20:19,020
But you also need to check that you cannot that it fails if you do not provide the external ID.

164
00:20:19,020 --> 00:20:27,000
That's the only way that you can be sure that the customer really wrote the trust policy correctly with the condition there.

165
00:20:27,060 --> 00:20:29,700
So you need to check those two things.

166
00:20:29,700 --> 00:20:31,240
And that's very important.

167
00:20:31,720 --> 00:20:40,300
The other thing that's really important is what privileges the SaaS vendor will give to the worker.

168
00:20:40,300 --> 00:20:55,560
I'm using this general term to describe the instances, the lambdas, the containers, or the human beings that will be able to use those credentials to assume the roles and do API calls on the customer accounts.

169
00:20:55,620 --> 00:20:57,840
There are no conditions involved, I'm sorry.

170
00:20:58,280 --> 00:21:14,080
And so, since you do not know in advance what all of the account IDs of all the customers are, you might be tempted to do the first version here on the top that has Drake pretty sad.

171
00:21:14,180 --> 00:21:20,640
And so asterisks are bad, especially if they're by themselves, they get really angry when they're alone.

172
00:21:20,640 --> 00:21:25,900
And so having a resource asterisk there is a horrible idea, please do not do that.

173
00:21:25,900 --> 00:21:43,180
Basically, what this is doing is allowing that worker to have unrestricted STS assume role to any role on any account that has a trust policy, trust in your account, but also any role inside the SaaS vendor account as well.

174
00:21:43,180 --> 00:22:05,790
So if you have a role lying around that allows for administrative privileges, and that will be the case if your account is part of an AWS organization, then whoever gets those privileges, or if an attacker breaks into your worker, if that human being is a malicious insider of their laptop,

175
00:22:05,790 --> 00:22:13,430
their credentials are stolen or something like that, then an attacker can easily use this to escalate privileges and take over the vendor account.

176
00:22:13,510 --> 00:22:15,850
So that's a really bad idea.

177
00:22:15,870 --> 00:22:24,530
You want to do something that looks like the second version there, where you have standardized the role names on the customers.

178
00:22:24,630 --> 00:22:26,830
So that already mitigates a lot of risk.

179
00:22:26,830 --> 00:22:38,450
Because even if you just had the allow statement there, it would only allow a role with that name on the SaaS vendor account to be executed.

180
00:22:38,450 --> 00:22:43,170
But I would also add that deny as well, that then covers that possibility.

181
00:22:43,170 --> 00:22:59,430
So that second version is allowing you to do STS assume roles, only two roles, in that case called SaaS cross account role, you can choose whatever name you want, but on any account other than your own account.

182
00:23:00,270 --> 00:23:09,330
And so that would prevent you from overprivileging your workers and allowing an attacker or a malicious insider to do a privilege escalation.

183
00:23:09,330 --> 00:23:12,810
So that's really important to keep in mind.

184
00:23:14,790 --> 00:23:33,690
It would have been pretty nice if you were able to use SCPs to do something similar, to create an SCP that says the only identity within the SaaS vendor account that can do assume roles for other accounts is this lambda or this container task or whatever.

185
00:23:33,930 --> 00:23:36,470
But I haven't been able to do it.

186
00:23:36,470 --> 00:23:47,110
And the reason is this code here on the left would have worked a treat, I think, and probably a lot of people that know a lot more than I do will correct me soon on this.

187
00:23:47,250 --> 00:23:48,610
But here's the problem.

188
00:23:48,610 --> 00:23:57,730
For some weird reason that I have no idea what they are, SCPs do not support the not resource syntax.

189
00:23:57,730 --> 00:24:00,090
They only support resource.

190
00:24:00,170 --> 00:24:04,390
They support not action, but they do not support not resource.

191
00:24:04,390 --> 00:24:09,130
So I can't say I'll deny for anything that's not on my account.

192
00:24:09,690 --> 00:24:13,170
That can't be done on an SCP.

193
00:24:13,170 --> 00:24:14,750
So why?

194
00:24:14,750 --> 00:24:15,930
I mean, why?

195
00:24:16,070 --> 00:24:33,630
In any case, that means that you need to protect, as you should anyway, your root account privileges, for example, or anyone with the administrator privileges, because they will still be able to assume roles on all your customers as well if they have those privileges on the SaaS vendor account.

196
00:24:35,690 --> 00:24:37,770
CloudTrail is always your friend.

197
00:24:37,770 --> 00:24:45,690
You can see those STS assume role activities happening on the customer account as well.

198
00:24:45,690 --> 00:24:47,350
This will be logged there.

199
00:24:47,390 --> 00:24:54,730
So if you are a vendor or if you are a customer, you need to monitor those activities and make sure that you're not seeing something weird.

200
00:24:55,030 --> 00:25:17,330
So for example, any attempts that fail, that are trying to use a different external ID, or once you know what the identity, what the temporary credentials were assigned to a vendor are, if you look for what API calls they're doing with those temporary credentials later on,

201
00:25:17,330 --> 00:25:21,750
you can try to find patterns, especially for things that are blocked.

202
00:25:21,930 --> 00:25:32,470
If you have a vendor that asks you for the privileges they need, and they have automated tasks running, it shouldn't, I think, have a lot of blocked activities.

203
00:25:32,470 --> 00:25:36,970
They shouldn't be trying to do things that they didn't ask you the privileges to do.

204
00:25:37,070 --> 00:25:45,070
So there are lots of things that you can very easily monitor that should be low volume, high impact events that you can monitor using CloudTrail.

205
00:25:45,130 --> 00:25:55,390
So not only the STS assume role operations themselves, and you can see that the external ID is not a secret because it's explicitly included on the request parameters there.

206
00:25:55,390 --> 00:25:58,250
AWS does not consider that to be secret.

207
00:25:59,190 --> 00:26:17,710
And so monitoring the STS assume role operations themselves, maybe you find out about cross account access that you didn't know existed, because you see that on the log on your scene, or you keep track of the identities assigned to the vendor, and then you see what other activities they performed.

208
00:26:17,730 --> 00:26:31,990
And especially you pay attention to things that might indicate enumeration, like reconnaissance, and denied activities, which again, for automated workloads, probably shouldn't happen that often, right?

209
00:26:31,990 --> 00:26:34,670
And probably worth your time investigating.

210
00:26:35,630 --> 00:26:45,510
And so I want to highlight the research from Kasten Broughton, I hope I'm pronouncing that right, which is amazing.

211
00:26:45,690 --> 00:26:50,930
He looked at this, and he said, how many vendors are doing this right?

212
00:26:50,930 --> 00:26:55,310
And so he tested about 200 vendors, which is awesome.

213
00:26:55,310 --> 00:26:56,710
This is a lot of vendors.

214
00:26:57,130 --> 00:27:19,290
And to my surprise, he found that even though all of the advantages we just described to using the assume role method, about 50% of the vendors he tested chose to work exclusively with IAM user credentials, long-lived credentials, they chose to continue having to deal in secrets with all of the disadvantages that we mentioned,

215
00:27:19,290 --> 00:27:24,510
which is to me like a wasted opportunity to do better, to be quite frank.

216
00:27:24,630 --> 00:27:41,450
And of those 50% or 100 user vendors that did use the assume role way of getting access to customer accounts, 98% of them, so just two vendors, did this right.

217
00:27:41,450 --> 00:27:44,350
So 98% of them got this wrong.

218
00:27:45,670 --> 00:27:52,510
Actually did that second check to see if the assume role failed, if they didn't provide the external ID.

219
00:27:52,510 --> 00:28:04,290
So only two of them, two out of 100, actually checked if the customer configured the policy correctly, which is kind of depressing if you ask me.

220
00:28:04,610 --> 00:28:16,190
And so the thing here is that they would be amenable to a customer configuring things incorrectly, and then essentially the confused deputy problem is still there.

221
00:28:16,190 --> 00:28:32,650
37% of them don't understand what external IDs are, and they explicitly, by design, allow customers to pick their own external IDs on the UI when they're onboarding their integration into the SaaS vendor.

222
00:28:32,650 --> 00:28:39,530
And so this defeats the whole purpose, the whole purpose of having the external ID in the first place.

223
00:28:39,530 --> 00:28:46,550
It's as if Let's Encrypt said, so configure any TXT record you want to prove you own this domain.

224
00:28:46,550 --> 00:28:48,490
That's not how it works.

225
00:28:48,490 --> 00:29:01,110
Another 15% unwittingly allowed it to happen this way because of failures, there was a way to bypass the UI, go straight to the API and do this, or some other vulnerability allowed Kasten to do it.

226
00:29:01,110 --> 00:29:19,130
So over half of the vendors, 52% of the vendors that used assumed roles had a way for the customer, or the attacker in this case, to just pick whatever external ID they wanted and essentially force the confused deputy problems to happen.

227
00:29:19,350 --> 00:29:20,610
So that's a bit depressing.

228
00:29:20,610 --> 00:29:21,850
That's amazing research.

229
00:29:21,850 --> 00:29:26,950
Check out his talk on Forward Cloud SAC where he delves into this into more detail.

230
00:29:28,770 --> 00:29:30,410
And privileges.

231
00:29:30,410 --> 00:29:35,210
Whenever you're giving people privileges, you need to be very wary of what those privileges are.

232
00:29:35,810 --> 00:29:41,510
You need to make sure you are using this privilege, otherwise you're going to be in a world of pain, right?

233
00:29:41,750 --> 00:29:45,690
And so cloud is pretty new, right?

234
00:29:45,990 --> 00:29:51,030
And IAM for AWS is really complicated.

235
00:29:51,850 --> 00:30:06,410
I'm pretty sure if someone, you know, works the mathematical proofs, it's going to be Turing complete at this point, which makes me particularly mad they didn't do, you know, not resource on SCPs because, you know, anyway.

236
00:30:06,650 --> 00:30:17,650
But you should, of course, minimize the privileges that you are assigning to this vendor-customer relationship.

237
00:30:17,890 --> 00:30:21,130
And it's an interest of both parties involved, right?

238
00:30:21,130 --> 00:30:26,270
As a customer, of course, you want to minimize the amount of privileges anyone has from outside into your environment.

239
00:30:26,270 --> 00:30:29,510
But as a vendor, you don't want that liability either, right?

240
00:30:29,510 --> 00:30:50,730
If you accidentally ask for more privileges than you need and you can be potentially blamed for illegal access to, I don't know, credit card data, PII, trade secrets, all sorts of things, that's a lot of problems that can cause you as far as legal liability,

241
00:30:50,730 --> 00:30:53,690
fines, and all sorts of fun things.

242
00:30:53,890 --> 00:31:00,470
So it's in the interest of both parties to keep those privileges to a minimum, right?

243
00:31:00,470 --> 00:31:03,890
And so, of course, everyone does that, right?

244
00:31:06,070 --> 00:31:08,510
No, not really, no.

245
00:31:08,630 --> 00:31:19,630
And so Ben Reeser, and again, I hope I'm saying this right, published this tweet recently, I think last week, and it was very timely for my talk.

246
00:31:19,630 --> 00:31:21,070
Thank you very much, Ben.

247
00:31:21,070 --> 00:31:27,010
Essentially, he's a user of Okta, which is a publicly traded company.

248
00:31:27,550 --> 00:31:30,710
It's a SaaS identity provider, right?

249
00:31:30,730 --> 00:31:36,750
And a lot of large corporations use Okta for identity management.

250
00:31:36,870 --> 00:31:48,230
And they were asking for very, very overbroad privileges there, like they were asking that they were able to assume any role inside of your account.

251
00:31:48,410 --> 00:31:50,590
You can see how that's problematic, right?

252
00:31:50,590 --> 00:31:55,810
I mean, an IDP would need to have pretty high privileges, right?

253
00:31:55,810 --> 00:32:00,030
Anyway, but this is just too much, right?

254
00:32:00,030 --> 00:32:13,150
They can very easily be abused to gain admin level on any AWS account pretty easily, especially again, if you are using AWS organizations, you should write an SSL.

255
00:32:13,150 --> 00:32:15,970
And so it's a pretty serious problem.

256
00:32:15,970 --> 00:32:25,310
And Ben, I have not been involved in the disclosure process, or I haven't talked directly to Okta about this.

257
00:32:25,310 --> 00:32:34,590
What I hear from Ben, and I asked for his permission before sharing this and his opinion on this, is that Okta has been very slow to respond.

258
00:32:34,950 --> 00:32:38,610
He's been six months trying to convince them that this is serious.

259
00:32:38,610 --> 00:32:41,510
So far, they have updated their documentation.

260
00:32:41,750 --> 00:32:53,990
But as far as Ben knows, or from what he told me, they have not notified existing customers to change and reduce those overly permissive roles that have been assigned.

261
00:32:53,990 --> 00:32:57,010
So this is the SaaSpocalypse scenario right here.

262
00:32:57,010 --> 00:33:17,230
This is a large company with thousands of customers, potentially tens of thousands of AWS accounts connected to them, and an attacker that's able to break into the Okta AWS account that's trusted by all those companies, will very easily be able to find a role,

263
00:33:17,230 --> 00:33:25,870
because they have list roles, by the way, and then assume a role with as many privileges as they need, and then take over Okta's customers' accounts.

264
00:33:26,630 --> 00:33:31,390
So they did get the external ID right, though, but the privileges, not so much.

265
00:33:31,930 --> 00:33:37,650
If you are an Okta customer, look at their updated documentation, update your environment.

266
00:33:37,990 --> 00:33:54,870
If you are from Okta, and I'm wrong here, and you are working on this, and they might be just waiting for legal approval or going through some extra hoops internally to be able to notify customers, etc., which I know nothing about, I can only assume they have very talented,

267
00:33:54,870 --> 00:33:59,010
competent security and legal people at Okta.

268
00:33:59,990 --> 00:34:08,750
So, talk to them, if you are that person, and you have not been involved in this, and get this moving, because there's a lot of potential risk involved.

269
00:34:11,230 --> 00:34:27,910
And something that, I mean, Chris Ferris has essentially the same thing, only for natural control, so a vendor that didn't know how to specify less permissive roles essentially asked for, you know, let me manage all of your VPC, what could possibly go wrong?

270
00:34:28,090 --> 00:34:38,910
What I have found really, really often, though, is that a lot of vendors ask for the read only AWS managed access policy.

271
00:34:39,090 --> 00:34:47,950
And it seems harmless enough, and it's a start, because at least they're not asking to be able to change things in your environment, so yay, I guess.

272
00:34:47,950 --> 00:34:52,390
But the problem is that read only access is very misleading.

273
00:34:52,390 --> 00:35:09,730
If you actually look at the privileges it's providing, it allows you to do things like read all S3 objects on every bucket on your environment, and list all buckets, perform queries on databases like DynamoDB, and really dangerous things that would give a vendor access to,

274
00:35:09,730 --> 00:35:11,370
like, company data.

275
00:35:11,470 --> 00:35:19,140
And I've seen this be requested by people, by vendors like, you know, cloud cost optimization, or CSPM providers.

276
00:35:19,140 --> 00:35:21,340
They have no business getting access to your data.

277
00:35:21,340 --> 00:35:25,880
They only need to see, like, your billing, or even how things are configured, right?

278
00:35:25,880 --> 00:35:30,400
But not have access to your actual confidential data, right?

279
00:35:30,400 --> 00:35:38,160
So one quick replacement here would be to use the view only access, which is a much better policy in that regard.

280
00:35:38,720 --> 00:35:50,640
But, you know, apply list privilege as much as possible, and not necessarily the existing policies are going to be best suited to what you need, right?

281
00:35:50,640 --> 00:35:55,800
So in closing, what do I recommend?

282
00:35:55,800 --> 00:36:01,220
If you're a customer, you know, you need to be aware of the third party access to your AWS accounts.

283
00:36:01,220 --> 00:36:02,800
You need to have governance over it.

284
00:36:02,800 --> 00:36:10,620
You need a process to do a regular review, not only on initial onboarding, because those things... privileges only grow, they never diminish.

285
00:36:10,620 --> 00:36:15,740
So you need to have regular reviews of what actual privileges are being given to third parties.

286
00:36:15,740 --> 00:36:17,480
Ensure there's still business need.

287
00:36:17,480 --> 00:36:27,700
We, on our consulting practice, have seen many times the companies that had no longer a contract with our clients, but the access they were given was never revoked.

288
00:36:28,060 --> 00:36:31,000
So people need to go account by account and revoke that.

289
00:36:31,000 --> 00:36:36,540
So you need to be looking for that, and you can automate that to look for trust policies and so on and so forth.

290
00:36:36,720 --> 00:36:51,140
There's an amazing tool written by Kinnard McQuaid called Cloud Splaining that will go through your accounts, and it will look for overprivilege and things that can be abused to exfiltrate data or to do privilege escalation.

291
00:36:51,140 --> 00:36:52,020
It's automated.

292
00:36:52,020 --> 00:36:52,640
It's awesome.

293
00:36:52,640 --> 00:36:55,000
Use it in Parliament by Scott Piper.

294
00:36:55,000 --> 00:37:03,620
It can also help you do some QA of policies to make sure required things are not missing, like you're accidentally using something that's not working well.

295
00:37:03,620 --> 00:37:07,520
You're an external ID, things like that, and monitor using CloudTrail.

296
00:37:07,860 --> 00:37:12,500
If you're a vendor, right, you have the brunt of the work.

297
00:37:12,500 --> 00:37:14,520
You have the biggest responsibility here, right?

298
00:37:14,520 --> 00:37:19,440
And you have the biggest liability, probably, because you have the sun's liability of all your customers.

299
00:37:20,100 --> 00:37:22,780
Minimize as much as you can the privileges asked of customers.

300
00:37:22,780 --> 00:37:31,180
And I know this is a balancing act, because if you ask for too few privileges, then when you do need more privileges, you have to go back to customers and ask them to change things, which is bad.

301
00:37:31,180 --> 00:37:34,660
But I would err in the side of caution.

302
00:37:34,660 --> 00:37:36,380
But then again, that's why I work in security.

303
00:37:37,060 --> 00:37:39,840
Make sure you're using external IDs correctly.

304
00:37:39,840 --> 00:37:46,640
If you can't easily enforce that the external IDs are unique, use a random UUID.

305
00:37:46,640 --> 00:37:51,620
It's such a big key space that statistically you probably won't be in trouble.

306
00:37:51,920 --> 00:37:55,940
Ensure customers are correctly implementing the trust policy.

307
00:37:55,940 --> 00:38:01,480
I would say standardize the role names as well and use CloudFormation.

308
00:38:01,480 --> 00:38:03,860
So the customers don't have to manually set up anything.

309
00:38:03,860 --> 00:38:08,480
They just drop CloudFormation stack and everything is done beforehand.

310
00:38:08,480 --> 00:38:12,660
There are plenty of examples online if you want to look it up.

311
00:38:12,660 --> 00:38:15,780
Or I can provide one later if anyone requests.

312
00:38:16,520 --> 00:38:19,540
Limit the attack surface of that account that everyone trusts.

313
00:38:19,540 --> 00:38:21,760
You need to run as few things as possible.

314
00:38:21,760 --> 00:38:28,760
You need to have as few humans as possible that have access to that one account that every customer is trusting.

315
00:38:28,760 --> 00:38:32,180
You need to reduce that attack surface as much as you can.

316
00:38:32,280 --> 00:38:39,860
Because if that one account gets broken into, then that's a very bad day for you.

317
00:38:39,860 --> 00:38:46,160
The upside is if you're using cross-account access, as soon as you kick the attacker out, they can no longer access your customers.

318
00:38:46,160 --> 00:38:48,580
And your customers don't need to do anything themselves.

319
00:38:48,580 --> 00:38:57,980
But still, the value for an attacker of breaking into that one trusted account is huge because of all that concentrated access.

320
00:38:57,980 --> 00:38:59,800
So you need to be extra careful.

321
00:38:59,920 --> 00:39:02,720
Separate it from everything else that you can.

322
00:39:02,900 --> 00:39:07,040
And monitor extensively what's happening in that account using CloudTrail.

323
00:39:07,040 --> 00:39:11,180
Monitoring is going to be a lot easier if you have less things running on it.