I'm Nimrod. I'm a solution architect at BridgeCrew.
And over the next 35 minutes or so, I'd love to walk you through
how you can utilize open source to do least privilege
using infrastructure as code. Sorry.
So, a bit about me. I'm a solution architect at BridgeCrew.
BridgeCrew is a cloud security company that helps identify
misconfigurations in both real cloud environments
and infrastructure as code frameworks, such as Terraform, CloudFormation, etc.
And we're doing all of this using grid automation.
As part of my role in BridgeCrew, I'm involved with a few
open source projects around AWS and Terraform,
and security, which I either co-maintain
or contribute to continuously.
And my newest open source tool is called AirIAM,
which I hope at the end of this talk you'll also be a fan of.
Let's dive in.
So, just a quick overview.
AWS AirIAM is a service by AWS that manages
all the privileges in your AWS account,
in every AWS account. That means it includes
all the privileges both for things that you need,
your employees, your servers,
but also for things that run in the background,
like AWS services and access across services.
So, it starts out really cool, and it allows that for everything.
But the problem is that it is hard because
it includes a lot of specifics and details.
You create a user for every employee, and maybe
for integrations, and for your CICD pipeline,
and your servers, and it just becomes way too hectic.
So, let's try to break it down and see
why it is that way.
When I look at AIAM, I think of it
more like a road system. So, we have our users
and our servers, and we have the resources,
our data, our buckets, our databases,
and all of those are destinations.
And IAM, or privileges that are described in IAM,
are like the roads between those entities.
So, usually when you start out, you don't need a road planner.
You have maybe two or three employees,
and you set up maybe a server. That server
serves your app. It can be behind a load balancer.
It's pretty simple stuff. You won't need more than maybe
two or three roles and top four or five users.
But we don't use the cloud for this
simple infrastructure. We use the cloud because it allows us
to grow and scale and use more
and more services. And sooner rather than later,
you'll find yourself stuck in this kind of situation.
You'll be adding more services, RDS,
ECS, CloudFormation, CloudFront,
more and more services, and all of those
require different roles and different users.
Some of them are maybe even third-party tools.
And you also have different access patterns.
You can start out by everybody using their own
IAM users and then switch out
to maybe using SSO and identity providers,
which mean you have roles and not set of credentials.
And all of that leaves layer
over layer of unused and unnecessary privileges,
which can be used by bad actors.
Now, as a security person,
which is kind of what I do, when I
see IAM, I have only one question I really want
to answer. And that is, who has access to what?
It should be pretty simple,
but I will now show you why it isn't that simple,
specifically in AWS IAM, but that happens also in
GCP, Azure, and other cloud providers.
So let's start out with a simple question.
Who has admin access, which is full
access, to my AWS account?
When I asked this question in the beginning, naturally
I went to the administrator access, which is a managed AWS
policy, and I tried to understand who has access
to that policy. Who has that policy attached?
So you can see here, this is an example account.
We have two roles, which have admin
access to the account, and also a group. Now,
the group in itself is not any user
or role, but we would like to know who is part of that group.
So I clicked on the group, and now I see the group
has 11 users, all of them have admin access to the
account. The problem with IAM
in AWS is that this isn't a complete list
of all the admins in the account.
And the reason is simple. IAM allows you to create
your own policy documents, which are simple JSONs,
in which you describe what you allow and deny for each
entity. It could be a role, it could be a user,
it could be a group. So this is an example of an
inline policy, which means it's a policy for one specific
role, which allows all actions
on all resources. Well, that's what I would call an
admin. And this isn't listed anywhere. It won't
show up anywhere. You'll have to go entity by
entity, policy document by policy document
to find these entities.
So this example shows us two problems
we have with IAM. One is that it is easily bypassed.
You can just create your own policy, it won't be
listed anywhere, no one will know, and you can give any
permission you'd like. The second one is that
IAM supports complex relationships. Let's say you have a user.
You can attach a managed policy to it, or you can
put an inline policy. But as we've just seen, you can also
do that using a group, right? You can have the
user be a part of a group, and that group will have the policies
attached or put on. But that's not
all that IAM supports. IAM supports assuming
role as well. So that means a user can assume
a role and get the policies that are set on that
role, or assume through that role another role.
And that's a long chain of privilege excavation
that you won't be able to see using
simple console and going through the entities
in IAM. But that's not all.
Let's take a look at another example and see two
other things we can find in IAM. So
this is a list of the 11 admins from the admin group
we've seen before. If we look at them, we'll see
that 10 of them have access keys which are older than 90
days. So best practice is to rotate
your keys every 90 days since these are static credentials
that if discovered, they can be used to abuse the account.
We can also see that
four of these users have passwords and no
MFA set. Now you might notice that most
of them do not have password access. That's because we've
moved over to using SSO. And these
four users unfortunately did not do what was
required of them, or they've been handled since then.
But they do pose
brute force risk to the account just by having passwords
with no MFA. And if that's not enough, we also
see that two users have not been used for
more than half a year, more than 200 days. There's
really no reason to keep live access keys
that have such a high level of access to the
account without them being used at all.
So this example shows us two things.
One is that it's difficult to enforce patterns over time.
You have to manually go through the user list
and find the, I would say, the
people who are out of line
or the resources that are misconfigured.
But it's also hard to trace and verify usage. It could be that
a person was using this account a lot for a year
and has since stopped, but
we won't know it unless we check. And we have to check it
per user, per role, per entity in IAM.
So
we have all of these problems which cause
layer and layer of garbage and stuff that's left behind
and waiting to be now used.
And we were wondering how we can do it.
So what I developed
is called Air IAM. It's completely open source. You can
check out the link. I'll share it later too.
And what it does is distribute IAM through
Terraform. It creates Terraform code from your
existing IAM. Let's walk through and see
what it does.
So before we created the open source, obviously, we had to do
a few manual iterations to see
what works and what doesn't. And we came
to this flow of conclusions
and actions. So at first
we would like to collect all the IAM entities from the
account and their usage using AWS
APIs, maybe CloudTrail data and so
forth. And then for each entity
cross-reference it with its usage and decide
whether the entity is in use or not. Whether the entity is
in use or not is a question that depends on
the threshold for usage. I would
say the default best practice is 90 days.
I've seen it go down to 30 or up to 365
days. So it really depends on your preference.
Once we've decided what's not used, it's easy. We can mark
it for division. And we will need to go through the
permissions that are in use.
So we will have to go through the entities
and look at their permissions, look at what
they've actually used using Access Analyzer or
CloudTrail and create, I would say,
a minimized version of the policies that they have
based only on stuff that they use, meaning we won't
be able to impair any workloads,
not dev, not production, not anything. If it is
in use, we're leaving it as is. The third
one is to create a recommendation. That
usually included a very
large workbook with a few sheets in it
that describes what we want to do for each entity with
a few different, I would say, categories.
And after we've done that, we'll take that to the relevant
account stakeholder or
the person who actually uses the account and knows
the types of usage in the account.
For approval, that might
seem like a pretty long process. You actually
won't understand how long it is. I've added
some time estimations to the different steps. I would
say the most difficult part is to create
a recommendation and then get the approval from the relevant
stakeholders. I've seen it take weeks
and it's just a lot of misunderstanding
and meetings and getting
formal approval. And then I found out that
even implementing the changes after I have the action items
and I have the approval, it takes me a while. An
hour is for a medium-sized account.
It could take a lot longer for bigger accounts.
So after iterating through this a few
times and making sure that we understand what we need,
we decided, like we do in many other
fields here at Bridgecrew, that we want to automate it.
When we wanted to build
the automation, we decided that we have
four main, I would say, goals we
wanted to reach through the automation tool.
So the first one is that we want to identify unused
entities and easily remove them. When I say
unused entities, I also mean unused policy attachments.
So that means policies that are not in use at all by the
role or user or group that has that
policy attached. I want to also remove that.
We want to be able to detect drifts and enable rollback.
Let's say I made some modification and it caused
downtime. For some reason, something went wrong.
I can always roll back and make sure everything works.
I want to be able to handle scales
of thousands of entities. Again, when I say entities,
also the relationships between the users and the policies
and the roles and the groups. So it very easily
reaches the 1,000, 2,000 threshold on big accounts.
And then the fourth one,
which is very important, is to be able to take
the plan to get approval from the relevant stakeholders.
But I want something that I can share and people will
easily understand and either say that's fine or not.
So the tool itself,
ARIAM, is pretty easy. All it does is
collect data from AWS APIs, both
IAM and Access Advisor, cross-references it
and either outputs it as CLI
output for your convenience or
Terraform code, which we'll see in a bit how
that, I would say, leverages
the power to the maximum.
So let's start talking about the tool.
ARIAM contains three commands
built in. The first one is called Find Unused.
What it does is collect all of the data from
the APIs and decides whether the entity is
used or not. The default setting is
90 days. Let's do a short demo and see how it works.
Great. So I've already installed
ARIAM on this computer, although it is
very easy to install. And I can just run
ARIAM Find Unused.
So it now gets all the IAM
configurations for the account. It will then collect the usage
for all the users and the roles in the account. And
right after that, it will output all
the findings. Awesome. So you can see
it's generating the reports, editing it,
and now it will start outputting the unused
entities that it found in the account. So as you can see, we
have here four users who have not been used.
It says last used 365 days ago.
That's the maximum that AWS promises that
there was no usage in. We can
also see there's one active access key that isn't being used.
You'll notice that it's for a user that's not
listed under the unused users. So
ARIAM does not give you duplicate results. If
the user is unused, it won't say that its access keys are unused,
although he might have access keys that are unused.
Sorry.
We also have one user, that's my user, which
has password access to the console with MFA,
but hasn't used it for over 100 days.
If it doesn't have MFA, it will say it here.
So you'll be able to understand who is, I would say,
the bigger problem. We also have
four roles which are unused. Those are roles
that haven't been used for, I'd say, over
100 days. Yes. We have two groups which
are redundant. One of them has no members, which means
it's just laying there and it's a security hazard
because, let's say, it does have any policies attached.
If you attach a user to it, it will automatically gain
all those permissions. So even
it's just leaving behind privileges that maybe an attacker can
utilize. And then group without policies.
Groups can have members but have no policies
attached. And since a group is not an entity
that can be used in AWS, just
groups' privileges, there's no point in having those
groups. And since we want to make
AWS IAM as manageable as possible,
it also recommends deleting those kinds of groups.
We have a redundant policy that's a policy that's not
attached to any user, group, or role. It's just
clutter that's being left behind in the account.
And we also have two policy attachments.
For policies that are unused by the relevant
principle. So this user, S3User1,
is not using any privileges given to him by Amazon
API Gateway Administrator. But he does have that privilege
given to him, which can cause him to make
actions which he shouldn't, such as creating or even deleting
existing API gateways.
Great.
So I hope we've got all of that clear.
And this helps us actually create some
of the permissions baseline, since it allows us to remove
attachments of policies that are unused but given.
And create more of a list privilege model for
our IAM. The second command is called
recommend groups. What it does is try to look
through the users that you have in the
account, group them according to their
relevant privileges, and actually
create recommendations for three
types of groups currently. One is the admins group, which are
active admins. The second one is power users,
which are for users who have write access to the account
but not admin access. And the third one
is the read-only group. That's relevant for
all the users who have read access to the account but don't have any
write access. This command
helps us finalize the permissions baseline or
the changes we want to... the things that are in use
and create the recommendation, what we want the account to look
like in the end. So let's
see that as well.
I'll hit ARIM, recommend
groups.
Awesome.
So in this account, we have two active users.
One is my user, which is an admin and requires admin
access. And also this S3 user one,
which has write access to S3.
We don't have any read users in this account, so it doesn't
offer me the read-only. There's no point in creating empty groups
as we've talked before, but if there were any, it would show them
here.
So we have the recommendation, it's just not actionable yet.
We have everything we want to do, but we can't show it
to anyone and we can't just hit play and make that work.
That's where the third command comes in.
It's called ARIM Terraform. And what it does is create
Terraform code and state file. I'll show you
how it works and we'll talk a bit about Terraform
so you understand all the power that this command brings with this.
So,
I'll type in ARIM Terraform
and what it does is it will initialize Terraform,
it will create Terraform files for me, and then it will import
the existing Terraform entities into the local
state file. Let's see how that looks.
So I'll change to the results there
and you'll see that it created just now.
This is in my time zone. It created
the .terraform directory, which includes all the
files that are necessary by Terraform to run.
This is a temporary file, so you can ignore it.
It creates .tf files that define all the
different entities and their relationships using Terraform
HCL code. It also creates a state file
and the backup, which again, you don't need to
care about because it is only temporary
and for system usage. Let's take a look
at groups.tf. So,
what this shows you is the
four groups that we have in the account and the
policies that they have attached, if any. So, for example,
we have the admins group and the admins group has
administrator access attached to it. We have a group
without members and that group has two policies attached,
the RDS full access and the user change password.
We also have a group without policies, which
has no policies attached, and the S3 group,
which has the Amazon S3 full access.
Just so we can see it for real,
I've opened the
console for it and this is the groups in the account.
You'll see these are exactly the four groups we have and
you can see that the group without policies actually
does not have policies and the group without members has
these two permissions.
So, the same goes for
roles. Let's see the roles file, for example.
So, for every role, we'll have
the role itself. We'll also have the assume role
policy document and if it has any
policy attachments, such as this one, it will be listed
as well. You'll notice that the syntax is
not using any hard-coded strings.
Basically, ARN creates the entities and connects
them so you can actually see who has access to
what. Awesome.
So, that's all the code. Let's take a look at the state file.
So, the way Terraform works
is it creates both the code and the state file
and the state file describes what connects
the entities that were defined in code with the actual
ARNs in the cloud environment. So, for example,
we have here the AWS IAM group admins
as we've seen it in the code and that's connected
to this instance with this ARN.
So, if I copy it and I go to admins,
that's exactly the ARN of the group. So, that means
this code describes
this group and when I
will run Terraform apply or Terraform destroy or Terraform
plan, it will go and check what's configured in the
code and what's configured in the state file and
fix the differences. So, if it needs to delete
or create or modify anything that's been
described in the code and is not as is in the
console, in the actual account, it will be fixed.
Awesome.
So,
once I've created this, I can actually create the
Terraform code, if you'd like, without the import stage
which takes a while.
So, this is what it does
and if I type ls-la,
you'll see that everything has updated
except, well, it just finished now.
So, if I run it again, you'll see that everything will change
except the state file. See that? Awesome.
So, the state file did not change, but all the tf files
got updated.
So, if I go into the results
and type Terraform plan,
you'll see that all the differences that it wants
to do is unrelated to actual
entity modifications. This Terraform code
describes the actual entities almost
one-to-one. You see it has a force destroy flag
added to the users. That's so when you select
to destroy a user, it will actually go and
destroy, including the access keys,
the password access, everything. So,
I'll run Terraform apply just to make sure we don't have
any differences. You'll see it takes
a few seconds. And then
we have the Terraform output form
and the actual account exactly in line.
Hold on. I don't want to break it. Yes.
Awesome. So, now
let's say I want to delete all the unused entities that I've just
found. Instead of going through and deleting
one-by-one the policies and the groups and the users and the
roles, I can just type here
minus minus without
unused. Great. So,
now it created the Terraform files. But if we'll take a look, for
example, at groups.tf, you'll see
that we only have two groups now, which are the two groups that
both have users and have policies attached.
Pretty simple.
Now, when I run Terraform plan again,
you'll see that it offers to
delete everything for me. And it will
be much higher than five or ten
entities. It actually plans to destroy 40
through users, roles,
policies, and all the relationships between
them. And it will take just a few seconds and
the Terraform will apply. That's all you need.
Pretty slick.
Awesome.
So, we've created a recommendation that's
very actionable. I can take it and
apply it. But I don't have any way to show it
and get approval for it. But another very
powerful plus side of using Terraform is that it's
code. And for code, we have
version control systems. I like GitHub, so these are
print screens from GitHub. But basically, what you
can do is commit the
Terraform code that is outputted by default with
everything to a master branch, check out a new
branch, run it without unused, for example,
and open a PR. And that PR talks
in a very simple language that developers can understand,
even less technical guys can understand that
if there's a group here that's called group without
members and has a minus in front of it, it will
probably get deleted. So you can use that
to actually have all the conversation around
getting approval and making
everything, I would say,
approved or agreed upon. And then,
again, just a simple Terraform apply will make all
those changes for you.
So that completes our entire flow. Although
we have three commands to help
crunch through all the data and understand the phases,
in general, you can just run directly RIM
Terraform, it will do all the work for you. It will find all the
unused entities, it will import them into Terraform,
and you'll be able to actually
run it again without unused or with unused,
run Terraform apply, everything will be in your hands.
Now, you've run it once, that's great. You have your
IAM set up clean and sparkling and beautiful.
But the problem with the cloud is that
well, it's both a plus and a minus, I would say,
is that people can
manually change and make modifications.
Now, using RIM on a
weekly basis, monthly basis, on a regular basis,
you'll be able to scan your IAM and identify
drifts in existing configurations. That means if there were any
modifications or deletions, you'll be able to find
those things in code and
actually automatically update your
Terraform state and your Terraform code.
You'll also be able to identify changes in usage status.
So we've seen the problem where we had admins who have been unused
for a long time. If you scan on a regular basis,
it will by itself find entities that have been unused
for 90 days or if you modify your threshold
by your threshold and mark them for deletion
and recommend you delete them. It will also help you identify
new configurations. So let's say you have a
new team or you have a new service or you even started a new Lambda
and you created a role for it, that role will
automatically be integrated into your Terraform.
It will be imported into your state file and you can add it
to your repository where you will still have it
managed and you'll be able to see it.
And nothing will go without
knowledge. And then the last
thing that ARAM and Terraform enable
to you is to use static analysis tools to identify
bad practices. So I would like
to take this to the next slide actually
which shows, I would say, two
really great tools that will help you
use infrastructure as code
and static analysis for your permissions
in your IAM. So the first one is called
Chekhov. It's also pretty easy to install and I already
have it set up here.
So Chekhov, what it does is identify bad practices
in Terraform. So if I run... let's see...
the last thing I did was without the unused
right? So I want to do it one second with
everything. So right now I created Terraform for
the existing IAM setup. And if I go to
the results tab, I can just run Chekhov
minus the results. This will make
Chekhov scan all the Terraform files that are in the results
directory. And we'll see in a few seconds
that it has quite a lot of findings.
So if I scroll up, yes, quite a lot indeed.
So it found, it ran 56
checks and 23 of them failed. This
includes misconfigured admins and
bad assumer role policies and
policies that are attached directly to users as this
would require justification. Otherwise, it is really
recommended to use groups. Let's say
I ran it without the unused, just for
checking what that would mean. By deleting all the unused
entities and running Chekhov again, I will see
how much cleaner the account is. And the
account is a lot cleaner. So we only have two failed checks
and both of them can be either justified or
fixed. And all in all, we only
have 11 checks that were scanned.
That means a lot of the overhead was cleaned simply
by deleting unused stuff.
Awesome.
Another really cool tool is PolicySentry.
PolicySentry allows you to build IAM policies
from scratch using YAML syntax
and easy, simple CRUD syntax.
So you have create, read and
update and delete syntax for different resources and
different services. And that creates an IAM
policy that's probably better than what
your average developer would create.
If you're talking about scanning your existing
configurations, I also have a few tools up my
sleeve. So the first one, I would say the category
is finer and more granular
disk privilege for IAM.
We have CloudTracker by Geolabs and Ardvark
plus RepoKid by Netflix. They both
use CloudTrail data, so you need to have CloudTrail read
access to analyze the policies and
mark specific privileges and break
down all the stars and the jokers in your
policy to understand what you're giving that isn't really
in use. I would say that CloudTrail
visibility or coverage is not 100%.
Not all actions are by default recorded
in CloudTrail. And so using
CloudTrail comes with a...
maybe you should take it
suspiciously. Another really
cool tool that I'd like to show is called CloudSplaining.
CloudSplaining is another tool by Salesforce that identifies
possible risks in your IAM configurations.
So it has four different classes, privilege escalation,
data exfiltration, other stuff.
Let me show you real quick.
So I run CloudSplaining. It's a very simple
setup again. So what it does
is it scans all of the policies that are in the
account and it outputs a very detailed and rich
report about all of the stuff that you can find
in your account. You can actually go by the policies
or by the principles, which is what I prefer, and just
take a look, for example,
who has access to infrastructure
modification. So say the misconfigured admin
user has... well, since he's a misconfigured
admin user, he has all the bad stuff that he can.
If we'll take a look at the misconfigured IAM user,
you'll see that
he has this very bad policy,
which we've talked about. It allows everything on everything.
So it shows that he can
do a lot of infrastructure modifications,
do privilege escalation, do resource exposure,
and data exfiltration. You can actually go
through this by risk and start
diving deeper into that.
And then the final
tool I'd like to show you is called PrincipleMapper
or PMapper. It's by NCC Group. It helps
identify possible privilege escalations in the account.
So if you remember, we had the graph where a user
can assume a role and that role can assume another role
and each role has its own privilege, I would say
policy context of policies.
So this tool creates a graph. You see it here.
An example, it's a very partial
graph. It has these two roles
and this user, for example, can assume this role.
And so the user, which is called
EC2 Manager, can assume the role and gain all the privileges
that are attached to EC2 role for SSM.
And this allows you to map all the different roles
and maybe if you identify a role that is super powerful,
check who can assume it and make sure
that that's in check.
Yep, that's it for the slideshow.
All right.
I guess to start off with
one question for me, I'm assuming that
if you wanted to make a change, a specific change
instead of the whole IAM recommendation made by
AIR IAM, you would just go ahead and edit the
Terraform file, right? And so are there any
specific config files within AIR IAM
that you can use to specify what to detect
and what to ignore or anything like that?
So Terraform itself allows you to
maybe, I would say, create or modify or destroy
anything you'd like within the code and the state file.
So AIR IAM gives you all of the changes you'd like.
You can run Terraform apply as I did in the slideshow,
but you can also specify a specific target,
be it a user, a group, or a policy, and Terraform will
modify only that part of the findings.
Got it. And then another one,
do you plan on bringing any of these
features to GCP or
Azure, something along those lines?
Definitely. So that's something that we've heard before.
We are planning to do this. It requires
a complete different set of, I would say,
you need to use the other
SDKs and it will require more knowledge because
GCP IAM is not exactly like AWS IAM.
It has a lot more of privileges
given by the resources and not the principles,
but we are definitely planning to go in that direction.
Got it. Another question
from Artemis in chat. How well do these tools
work in a multi-account environment?
That's a great question. So
I would start out by saying that IAM permissions are
limited to the account. You can give cross-account
permissions, but that would be from
one account you give the other account permissions.
So the context of every, I would
say, analysis is a specific AWS account.
So you can use ARAM using all the default
AWS credentials tools. You can use the profile and specify
the minus minus profile flag. You can use
AWS environment variables to scan different accounts,
but every ARAM scan is limited to the context of
one account since that is the context of IAM.