So hi, I'm Justin Pagliarani. I'm a staff security engineer at Flatiron. What that means is I'm the
team lead for the cloud security team. Prior to Flatiron, I worked at Bishop Fox as a pen tester
and various quasi-governmental organizations before that. So I've been around for a little
while. Yeah, I guess that's kind of me. Suheer? Sure. So I'm a senior security engineer at Prime
Health. I've been here like three years. And then before that, I was a software engineer and did
software engineering for like three years. And that's pretty much about me. Yep. So for anybody
who doesn't know about Flatiron, we basically use technology to try and fight cancer. We're not
going to be talking about any of that today. We're going to be talking about cloud security
misconfiguration, detection, and remediation. So kind of going into that, we can move to the
next slide. Sorry, we were playing the slide dance. We got some...
Scott Piper from Summit Route has helped us out quite a bit with this, as well as Reza,
who is our summer intern. He's a PhD student, and he did some really great work for us as well,
helping us build this out and put it to good use in our environment. So kind of on from there,
we'll get into the agenda. We're going to kind of give you the background of the problem we were
facing, kind of our first iteration of a solution, kind of where we're at now, the things that we've
solved for, a quick demo, and a Q&A. So kind of our general background here is we kind of wanted
to know how our critical resources are configured. We wanted to be able to detect if things become
configured badly, and if they become misconfigured, can we fix it in a reasonable amount of time
automatically without having to be on call at night, because nobody wants to do that.
So kind of generally why we want to do this is because cloud configuration problems are
pretty easy to accidentally do in the new kind of DevOps, well, newish DevOps paradigm,
where we've now decided that the software engineers or sysadmins, mistakes happen,
and we end up with lots of things like, you know, S3 buckets being exposed, you know,
people losing IAM tokens through IMDS, all kinds of weird things, basically through like
silly misconfigurations that if you kind of like thought through, you know, you would catch pretty
easily, like having an S3 bucket be available publicly when it has sensitive information
is like logically a very simple mistake. But because of the complexities of how we are
building and utilizing cloud resources, it's something that kind of happens a lot. So at
Flatiron, we handle millions and millions of patient records, you know, raw PII from doctors.
So, you know, an S3 bucket with that information becoming publicly available would one, be a
disaster for us from a business standpoint, but also be a disaster for our patients. And when
you're dealing with cancer patients, the last damn thing they need is to be dealing with identity
theft, or even the thoughts of it. You know, we don't we don't want to do that. Stress is
bad for health outcomes. So we're trying to avoid that. So kind of our first iteration on going
about fixing this was to use lambdas, you know, just do serverless bits of code to check over
our resources every 10 minutes or so. We had a bunch of lambdas checking for a lot of different
things. It generally would do one lambda per resource or one lambda per misconfiguration
that would detect an issue and then fix it automatically, just kind of like your standard
polling model. And when we first started doing this, we had many fewer resources. We were a much
smaller company. It worked great. Things were fantastic. And it was great. So kind of the
general architecture of how this worked was we had separate lambdas for each remediation. We had
cloud watch events or we would be watching the cloud watch events from our primary account,
which we call our security account. And then the lambda would assume roles into each of our other
accounts to kind of look at resources that way. So we kind of had like a centralized, isolated
deputy that would then kind of move on to the different accounts and monitor all the resources
and then fix them if things were broken. And it worked. You know, we had great things. So,
we would be getting JIRA tickets that would tell us like, oh, somebody stood up this S3
bucket and it was public accidentally, but we fixed it right away. So no harm, no foul.
A fidelity of about 10 minutes. So that was not so bad. You know, it worked. It was great. Why
did we move on? Well, the general problem was we were using tons of API calls, lots and lots. And
as the environment grew, it got to be worse. That got to be very costly. It was causing production
issues and everyone was kind of mad at us. It was not a great thing. So, you know, we ended up with
lots of rate limiting, which was, I mean, not the worst thing in the world, but it wasn't great. And
it became more and more of a problem as time went on. So, you know, we kind of wanted to avoid this,
you know, every 10 minutes things were starting to get screwed up. So kind of our general
post-mortem was we had these cron, like basically cron jobs running to check these things. It worked
very well. But, you know, polling is just kind of crappy in general. So we wanted to move away
from that and move into something better because just the number of API calls in a cloud environment
was getting costly from a number of different standpoints. And then also just the general
fidelity of having a 10 minute delay in response time was not necessarily great. You know, if we
could act a little bit more, I guess a quicker response would be better, you know, when you're
dealing with as much PII as we do. So kind of enter the remediation framework V2. So Saheer, take it
away. All right. So all these problems which we saw before, like multiple API calls and also having
exponential backoffs, which were like impacting other stuff, right? And the issue also was like
we had like cron based polling, which will depend on like how is your cron frequency. Like we had
like 10 minutes, but for some reasons, for some stuff we had like 20 minutes also. It was not real
time or it was not that easy to scale. So that all brought us to like work on the V2.0 of that,
like how we can like aggregate all of these independent lambdas into one unified code base.
So instead of like multiple lambdas, we have one single stuff where we have like multiple modules
for like let's say S3, lambda, RDS, or AMIs, or EC2s. So we try to like unify everything and try
to add like event driven models. So because we know like what events would be responsible for
making things, like for example, like when we talk about S3 buckets, you would know that put bucket
ACL or put bucket policy would be one of those calls, like which would make the bucket public,
right? By making either slapping an ACL or slapping a bucket policy, which basically exposes
your S3 bucket. Similarly, we have the same API calls for other resources, like let's say
EC2 AMIs or snapshots. So we wanted to capture on that and make a framework which can tap onto
these events which are happening and then maybe trigger our framework. So if we move ahead,
we'll see the architecture, like how exactly it is set up. So the V2.0 is basically,
you can use like multiple accounts with it, multiple regions. So let's say if you have like
10 accounts, you can basically configure your members in 10 accounts and also the multiple
regions. So how we do here is we have our CloudTrail event rules, which basically capture
on the events which happen in, let's say, account A. And those events could be like, hey, like EC2
modify snapshot attribute. And what happens here is that sends the events on event bus
to your master account where your remediator function lives. So event bus basically will
send that to the master account. And then on the master account, you have same pattern rules which
would capture on the EC2 modify snapshot attribute. And then that is captured by the
event translator, which is like Lambda. So it will capture the raw event and then massage it
and send that to SQS queue. And then the SQS queue is piped to the remediator Lambda, basically.
And the Lambda, remediator Lambda will basically look at the resource, try to enumerate it,
what does it look like? Does the snapshot look like it has been shared publicly? Does the snapshot
look like it has been shared with a random AWS account, which probably we don't own?
Based on that decision, the remediator Lambda will make the decision to remediate it or not
remediate it. So there are event translator and event forwarder here and remediator function,
which makes it event-driven. Moving on, like we mentioned the limitations of V1,
which was based on the pooling model and we were using Cron for triggering our previous Lambdas
every 10 minutes or 20 minutes. So we thought we would maybe get rid of the pooling-based stuff,
but not really. So we still have Cron, we still have the pooling-based approach
in our V2 where we pool for certain issues, let's say, for example, missing MFA or old access keys
of inactive users. But some of these events are really hard to capture on your Cloudflare API logs
or the event rules, like let's say old access keys, how do you capture on the API calls?
You have to enumerate the user, find when it was created, when it was used and stuff.
So we have the pooling stuff, which basically runs on a Cron, say nightly or whatnot. And then
what it does is it will go to each account, assume a role and do the enumeration and then send
everything, every finding to an SQS queue, which is sent to a remediator function, same remediator
function, which will again enumerate the resource and make the decision if it should be remediated
or not. All right, so now who needs API calls? So what we see here is we saw a big drop in number
of API calls which our remediator function was making. The one spike you see is the polar function,
which was running nightly for describing the resources. And also if we move ahead and see
how many request limit exceeded errors we got, it was almost like zero. Maybe it won't translate
exactly the same for you, but what we saw was we experienced zero pushbacks from AWS.
Okay, so what are we watching exactly? So these are the list of things which we are watching
currently. So if you look at the EC2, we have a check for publicly accessible AMI or
an AMI shared with an unknown account, or EBS snapshots publicly exposed, or same applies if
it is shared with an unknown account. Missing tags, IMDS v2. And then one extra thing which we have
here is a check for publicly exposed EC2 instances. And what we have done is they categorize
that off like a dev account. So you have like an enrollment variable where you can specify what
are your dev accounts. And let's say if you want more stringent controls on that account, you can
make that rule that hey, no EC2 should be exposed in dev account. And similarly, we have like
multiple other remediation for IAM users, like look for inactive console access, look for inactive
access keys, missing MFA for RDS if RDS is publicly exposed, if the snapshot is exposed,
or if the storage is not encrypted. Similarly, it applies to Redshift, S3, and IAM role if it's
or Lambda. I think, yeah, this list is like pretty much like self-explanatory.
All right, so moving on to the demo, which we have, let me... Yeah, and to jump in real quick,
adding remediations to this is also, you know, pretty easy. It's not like this is just hard
code. It is like a monolith. It's very modular. So writing this is really as easy as writing a
pretty simple Lambda that checks whatever it is that you're wanting to check. So a lot of these
are sort of kind of designed around the requirements of our environment, but we've
built this in such a way that you can add things to your environment. So for whatever reason you
want all of your assets to be publicly available in your DMZ account or something, like you can
enforce that kind of thing too. It's really not too complicated to write one of these modules,
and it's basically plug and play. Yeah, exactly. Yeah, so like just to mention, so our main
intention was like make it modular so you can add more controls. I can actually show that, like if I
go to GitHub, AWS. All right, so if you go to resources, just a small look here on the remediator
and auditors, so you basically have like multiple checks for each module. So what
you can really maybe add for things that you want to watch for, let's say you want to watch
for API gateways or you want to watch for ECS and whatnot. So you can basically add a module here.
All right, so jumping onto the demo, let's make some things like publicly exposed. Let's make an
EMI exposed. So we have a test EMI here, and what we are going to do is modify image permissions
and make it publicly exposed. Save it. Let's do the same thing for snapshots meanwhile.
So I have this, modify permissions, make it public, save it, go back to the EMI,
refresh. Most likely it might be done by now, maybe not, we'll see. So it was made public,
now it has been converted to private. It's almost real time, like probably took like
three, four seconds, I think. And go back to snapshots. If we look in the permissions,
it's private. So we can do the same exercise for S3 also. Let's go to S3
and just choose any of the buckets. Let's slap public access, list object,
and I can see because something is blocking my view. All right, so ACL public,
and we can do similar stuff for lambdas, wait for IAM role also. IAM, let's go to role,
I might have a test role here. What we'll do is edit trust relationship and just allow everyone
with the principal star to assume this role. Update the trust relationship. Yeah, sure.
It says overly permissive policies. Let's go back and we'll see if S3 has been fixed.
Okay, so the ACL was removed and it's back again private. Let's see IAM, what has happened to this
thing. Okay, so it basically changed the trust relationship and added deny on principal star.
So basically, no one can assume this role. So yeah, so basically, what you can do is that you
can have like multiple services and it basically will respond to it almost in real time. Let's say
you're making RDS or modifying an RDS instance and making it publicly exposed. It will capture
the event and try to see if the star configuration right now is bad. As in, is it publicly exposed?
Is it not encrypted? And make the decision. Some things which are challenging is RDS and Redshift
because when you spin up an RDS, what happens is it takes a shitload of time, like 20 minutes,
30 minutes. So when you're using an event-driven approach, the lambdas will trigger immediately,
but when they try to enumerate the RDS instance, it might not be in the available state.
And that what happens is like your lambdas will see and that it's not available and they'll skip
it. But once your RDS is available and it is public facing and you don't and then you can't
catch it unless your polar runs like by a cron. So what we have done is like we have added some
like repeat invocation, like let's check for the RDS. If it is not available, maybe repeat
the same message to SQS, send it again like after like five minutes so it will keep on checking
every five minutes. Is it available now? Is it available now? If it is available, check it and
then make the decision like if it is public, make it private. If it is not encrypted, stop the
instance. All right, so let's jump on to the Q&A. Do we have any questions?
Oh yeah, a couple of questions for me. So in this case, those little modules that you showed
is when it comes to detecting something as well as responding to it. The response part,
is that also built into each of the modules? Is that how? So yeah, so basically what happens
I'll show you. If I go to the code, so if I go to resources, event translator,
main.py. So what happens like for each of these API calls, which are saying like create queue,
set SQS, queue attributes. So you have like for DB, we have like some delays like 120
seconds. Similarly for these, like they are almost real-time, right? Whenever this event is seen,
forward the event almost real-time to the SQS queue, and then it will trigger the
remediator function. But some of the other calls we have, like we have added some delays.
Yeah, yes, yes. And then kind of the beauty of some of this is that because it's all relatively
short Python code, if you wanted to integrate this with like, I don't know, if you have like
some kind of compliance tool that has detection or remediation built in and has like an API,
you can pretty easily leverage those like REST APIs or whatever to do the detection and the
remediation for you. And this just kind of ties all the automation together because most of those
tools that we've experimented with don't really have that. So that's something we're kind of
working on building out as well as leveraging some of the other like detection and response
tools so that this kind of just glues it all together and makes it event-driven. So you have
lots of flexibility to do stuff like that. And the remediator modules are relatively short. It's
basically just examining the event and then doing, you know, making API calls to do what it is you
want to do. So yeah, the remediations are built in and very easily customizable. Yeah, that basically
almost answered my second question, which was, you know, the polar function, which detects something,
that the CloudWatch events doesn't cover, that could easily be substituted or, you know, something
could be added on to it, which is an open source tool that does those checks or any additional
stuff that you want to do. Yeah, definitely. I mean, if you look at the PolarCore right now,
it's doing similar stuff, but doing like a sanity check every night, let's say,
it's doing the same RDS check, describe RDS, describe Snapshot, describe AMI, and also
including the other checks, like IAM user, if they are not using the access keys and stuff. So
we have clubbed everything, but let's say if you don't want to use something, some of the stuff,
you can basically remove it. So it will, like, do sanity check, like, once a night, but you can
modify the cron schedule. Let's say you want to do your, like, once an hour or once every four
hours, you can do that. It's very customizable. And also the whole deployment is all, like,
data form and matchstrips. So it's very easy to deploy and easy to destroy.
Got it. We have a couple of questions here from the chat, from John. In the what are we watching
slide under EC2, why are you watching IMDS v2 to ensure v1 is not used?
Yeah, that's exactly it. So, you know, we have a large environment with lots of apps,
and because it's medical apps, there's lots of, like, legacy components,
and there's lots of, you know, complexity. So, yeah, the IMDS v2 stuff is mostly monitoring
what is still using IMDS v1. And then so here, correct me if I'm wrong, it also flags when
things are using IMDS v2 to, like, say it's compliant, but it doesn't, like,
it's not automatically killing things. I don't believe it's written.
If you're in a state where you can, like, you are, like, okay, I'm in that state where my EC2
are supposed to be on IMDS v2, you can toggle the plug and stop and add the check for stop
instances if they are not, like, on v1 or on v2. And then second question from Scott.
Is the AWS remediation framework intended to complement or supplement
the use of AWS config with its own remediation processes?
So what we have, like, thought of, like, was to add config also as one of the events.
Also, like, right now we have polling and we have event-driven. So we thought of, like,
moving to config and, like, enumerate it, but we haven't, like, done the full enumeration part,
like how config can, like, help us in achieving the similar state which we are achieving by
event-driven plus polling. I'm sure, like, config has the custom rules and whatnot,
which you can, like, add and do the remediation based on the custom lambdas.
But, yeah, something which for to-do we want to, like, do, maybe add, like, a config integration,
maybe if you want to have three options of, like, leveraging the same framework. Because
even with custom model tool, you'll have to have, like, your lambdas to make the remediation.
And so...
Oh, go ahead.
Oh, no, go ahead.
I just wanted to ask, kind of going off of that,
any other sneak peek or future stuff you have in the plan for this tool?
Yeah. So, I mean, Sahir and I have both kind of mentioned it. One of the things that we're
wanting to look at is more integration into other tools, as well as things like AWS config,
to kind of leverage this stuff. We are a relatively small team. I think we have, well,
we have three people, basically, kind of managing our pretty large cloud environment. So, you know,
when it comes to leveraging existing tooling, our general philosophy is kind of use it all
and automate the triage to clean stuff up. So, really, what we're trying to get into is,
can we leverage existing checks and remediations from things like config,
security monkey is dead, but, like, tools that are similar to security monkey and things like that,
and leverage those checks and remediations instead of writing our own,
using kind of like the rest APIs and all the fun, nice features that people are putting out there.
So, you know, if we see a tool that makes this kind of stuff easy, the future state that we see
for the remediation framework is kind of gobbling it up and consuming it to make this stuff more
effective, giving us greater confidence that the detective capabilities are actually correct to,
like, lower false positive rate, or just to make development quicker.
Got it. And another question for me, you mentioned that your team over there is quite small.
I know that in order to get to a place where you guys are at with this tool to remediate right away,
or even with a certain delay, get those remediations going in the cloud environment,
you need to have sort of a in-level kind of in-depth knowledge of your cloud environment,
as well as buy-in from other teams. Any kind of insights as to how you got there,
fun stories, or anything like that? I think me and Sahir can both give some insight into this. So
Sahir has been at Flatiron much longer than me, and he is a very beloved resource among the entire
company, where I am kind of known more as the jackass who comes in and tells everybody no all
the time. But I like doing that. So one of the things that we found that really made these things
very successful was communicating to our engineers, because, you know, they're engineers,
they like to see cool shit. And like showing them that this is cool, it means that they can
experiment more as they're developing and building without worry. Oh, we lost Justin.
Justin, if you could try again for the past five seconds.
Yeah, so that developers could experiment a little more without fear of exposing, you know,
all of our patients' data. They actually really enjoyed that idea. And then we also found that
when we showed them that we had the capabilities to do these things, they suddenly had all kinds
of ideas as to how they could make use of it as part of their deployment, development, and testing
processes. So being able to tear down test instances after, you know, 15 minutes after
their tests completed, and things like that. So a lot of the way that we got buy-in from this stuff
was showing them how not only did it make the company and our patients safer, it made their day
easier and gave them the ability with, you know, a lot without having to build a whole new solution,
they could like automate their testing processes and have it all be event driven,
which was stuff that they wanted. So, you know, just being willing to be transparent and communicate
very closely with them and be helpful and show them how this could be like a boon to their
processes was really, really integral in getting people excited and on board with this, where we
could just start flipping switches and turning remediations on, because suddenly, you know,
they didn't have to worry about being the person who exposed an S3 bucket full of, you know,
20 million patients or whatever. So here probably has another view on this.
No, I think this new cover pretty much everything. Yeah, so it's more on, like,
how are we going to give more flexibility to developers? Like, are we confident, like,
making a segregated account where we can, like, let them do whatever they want and still have
some sort of guardrails to watch for, some sort of guardrails to, like, detect things, along with
SCPs and other preventative controls? Like, what if still things happen? Like, can we have, like,
a framework which watches for, let's say, S3 bucket or AMIs and whatnot and fixes it automatically?
So yeah, so ensuring that level of maturity is what we're aiming for. And I'm pretty sure, like,
software engineering kind of is on board on that. And then any kind of quick insight into
what the cost savings were like from v1 to v2? Oh, so we haven't, like, compared,
like, specifically the cost saving. We did, like, actually save a lot from the API calls
perspective. Like, if I, if you look back at the slides which we had from the v1, we were making
tons of API call, like, for the most API calls spent was on snapshots, describing snapshots.
We got to describe a snapshot and then you're described for each snapshot attribute. So it was,
like, a very, very iterative. So we had, like, so many API calls which were impacting other stuff
with, like, the whole situation where Ansible and stuff, because we were almost, like, getting
pushed back on the EC2 resources, always hitting the limits. So v2 basically gave us some room,
like, that we don't have to, like, describe a poll all the time. We can do it when certain
events which we care about happens. So that was some of the more, like, API saving, the number
of API call savings, but not specifically cost. That we haven't, like, compared yet.
Yeah, the cost savings for me was our SREs quit calling, saying that API, you know, we were making
too many API calls and people were getting rate limited, which is worth, like, a zillion dollars.
Because, like, if there's... I haven't made this clear, the one thing that I don't want to be doing
is being on call and getting calls at night pisses me off to no end, especially when it's an angry SRE,
because the SREs do not give a shit what I have to say. They are mad that things aren't working.
So I... that stopped and that is worth just... it's invaluable. That's really all I care about is a
quick calling. I think we can all agree there. A couple more questions from the chat. Have you
tested or compared this with AWS GuardDuty? So we do have... we do use, like, GuardDuty,
but then GuardDuty, like, it doesn't capture everything what we want. Let's say, sure, it
might have, like, for S3 checks and everything. But for, like, specific stuff, like, which we're
looking for, like, IAM roles and whatnot, I'm not sure if it is being covered in GuardDuty, but we
do complement that with, like, using GuardDuty, like, standalone. Yeah, to kind of... to add to that, you
know, like we said before, I don't necessarily believe in replacing free things. If there's
things that do things well, we will use them both as best we can and then try and, you know, write
some code to glue it together. So, you know, we have a lot of... when you ask these questions about
these things, very often the answer is we're using both of them as well as this.
Got it. And another one, for some cases, why not create a preventative SCP?
Sure, that's... yeah, this one, go ahead. Yeah, absolutely do that. You should do that. This is
another layer helping us, you know, validate that the SCP hasn't changed or that I didn't fat finger
the SCP or, honestly, I'm not that smart. I might have written a bad SCP and I kind of... I like
having the many eyes approach and, like, the many, you know, the defense in depth approach of, you
know, maybe I wrote a bad SCP and this is something we can use to detect... we can use to detect,
you know, something going public again. So we have both. We're using SCPs. You should be using
SCPs. They're a great preventative control. I also would say that having preventative controls
does not mean you should avoid having detective and responsive controls as well. You know,
you need multiple layers.