start	end	text
0	7340	So hi, I'm Justin Pagliarani. I'm a staff security engineer at Flatiron. What that means is I'm the
7340	13100	team lead for the cloud security team. Prior to Flatiron, I worked at Bishop Fox as a pen tester
13760	19020	and various quasi-governmental organizations before that. So I've been around for a little
19020	26120	while. Yeah, I guess that's kind of me. Suheer? Sure. So I'm a senior security engineer at Prime
26120	31100	Health. I've been here like three years. And then before that, I was a software engineer and did
31100	36220	software engineering for like three years. And that's pretty much about me. Yep. So for anybody
36220	42260	who doesn't know about Flatiron, we basically use technology to try and fight cancer. We're not
42260	45680	going to be talking about any of that today. We're going to be talking about cloud security
47000	52160	misconfiguration, detection, and remediation. So kind of going into that, we can move to the
52160	55140	next slide. Sorry, we were playing the slide dance. We got some...
57920	61900	Scott Piper from Summit Route has helped us out quite a bit with this, as well as Reza,
61900	67060	who is our summer intern. He's a PhD student, and he did some really great work for us as well,
67060	74000	helping us build this out and put it to good use in our environment. So kind of on from there,
74000	77900	we'll get into the agenda. We're going to kind of give you the background of the problem we were
77900	84740	facing, kind of our first iteration of a solution, kind of where we're at now, the things that we've
84740	92980	solved for, a quick demo, and a Q&A. So kind of our general background here is we kind of wanted
92980	98840	to know how our critical resources are configured. We wanted to be able to detect if things become
98840	105640	configured badly, and if they become misconfigured, can we fix it in a reasonable amount of time
105640	111320	automatically without having to be on call at night, because nobody wants to do that.
111400	121320	So kind of generally why we want to do this is because cloud configuration problems are
122120	128400	pretty easy to accidentally do in the new kind of DevOps, well, newish DevOps paradigm,
128400	134120	where we've now decided that the software engineers or sysadmins, mistakes happen,
134120	140060	and we end up with lots of things like, you know, S3 buckets being exposed, you know,
140060	146860	people losing IAM tokens through IMDS, all kinds of weird things, basically through like
146860	151760	silly misconfigurations that if you kind of like thought through, you know, you would catch pretty
151760	155540	easily, like having an S3 bucket be available publicly when it has sensitive information
155540	161680	is like logically a very simple mistake. But because of the complexities of how we are
161680	168320	building and utilizing cloud resources, it's something that kind of happens a lot. So at
168320	173980	Flatiron, we handle millions and millions of patient records, you know, raw PII from doctors.
174040	181080	So, you know, an S3 bucket with that information becoming publicly available would one, be a
181080	186180	disaster for us from a business standpoint, but also be a disaster for our patients. And when
186180	190500	you're dealing with cancer patients, the last damn thing they need is to be dealing with identity
190500	195220	theft, or even the thoughts of it. You know, we don't we don't want to do that. Stress is
195220	205280	bad for health outcomes. So we're trying to avoid that. So kind of our first iteration on going
205280	211980	about fixing this was to use lambdas, you know, just do serverless bits of code to check over
211980	217060	our resources every 10 minutes or so. We had a bunch of lambdas checking for a lot of different
217060	222740	things. It generally would do one lambda per resource or one lambda per misconfiguration
222740	227520	that would detect an issue and then fix it automatically, just kind of like your standard
227520	232140	polling model. And when we first started doing this, we had many fewer resources. We were a much
232140	238640	smaller company. It worked great. Things were fantastic. And it was great. So kind of the
238640	244420	general architecture of how this worked was we had separate lambdas for each remediation. We had
244420	250020	cloud watch events or we would be watching the cloud watch events from our primary account,
250020	255020	which we call our security account. And then the lambda would assume roles into each of our other
255020	258780	accounts to kind of look at resources that way. So we kind of had like a centralized, isolated
260100	264080	deputy that would then kind of move on to the different accounts and monitor all the resources
264080	270300	and then fix them if things were broken. And it worked. You know, we had great things. So,
270300	275600	we would be getting JIRA tickets that would tell us like, oh, somebody stood up this S3
275600	280500	bucket and it was public accidentally, but we fixed it right away. So no harm, no foul.
284560	290780	A fidelity of about 10 minutes. So that was not so bad. You know, it worked. It was great. Why
290780	296500	did we move on? Well, the general problem was we were using tons of API calls, lots and lots. And
296500	301300	as the environment grew, it got to be worse. That got to be very costly. It was causing production
301300	308860	issues and everyone was kind of mad at us. It was not a great thing. So, you know, we ended up with
308860	316560	lots of rate limiting, which was, I mean, not the worst thing in the world, but it wasn't great. And
316560	321000	it became more and more of a problem as time went on. So, you know, we kind of wanted to avoid this,
321000	327460	you know, every 10 minutes things were starting to get screwed up. So kind of our general
327460	333540	post-mortem was we had these cron, like basically cron jobs running to check these things. It worked
333540	340080	very well. But, you know, polling is just kind of crappy in general. So we wanted to move away
340080	345160	from that and move into something better because just the number of API calls in a cloud environment
345160	349520	was getting costly from a number of different standpoints. And then also just the general
349520	357440	fidelity of having a 10 minute delay in response time was not necessarily great. You know, if we
357440	363780	could act a little bit more, I guess a quicker response would be better, you know, when you're
363780	372860	dealing with as much PII as we do. So kind of enter the remediation framework V2. So Saheer, take it
372860	380060	away. All right. So all these problems which we saw before, like multiple API calls and also having
380060	385280	exponential backoffs, which were like impacting other stuff, right? And the issue also was like
385280	391340	we had like cron based polling, which will depend on like how is your cron frequency. Like we had
391340	396340	like 10 minutes, but for some reasons, for some stuff we had like 20 minutes also. It was not real
396340	402880	time or it was not that easy to scale. So that all brought us to like work on the V2.0 of that,
402880	408560	like how we can like aggregate all of these independent lambdas into one unified code base.
408560	413060	So instead of like multiple lambdas, we have one single stuff where we have like multiple modules
413060	420260	for like let's say S3, lambda, RDS, or AMIs, or EC2s. So we try to like unify everything and try
420260	426020	to add like event driven models. So because we know like what events would be responsible for
426020	430640	making things, like for example, like when we talk about S3 buckets, you would know that put bucket
430640	436340	ACL or put bucket policy would be one of those calls, like which would make the bucket public,
436340	441600	right? By making either slapping an ACL or slapping a bucket policy, which basically exposes
441600	447020	your S3 bucket. Similarly, we have the same API calls for other resources, like let's say
447020	453100	EC2 AMIs or snapshots. So we wanted to capture on that and make a framework which can tap onto
453100	458940	these events which are happening and then maybe trigger our framework. So if we move ahead,
458940	465720	we'll see the architecture, like how exactly it is set up. So the V2.0 is basically,
465720	473000	you can use like multiple accounts with it, multiple regions. So let's say if you have like
473000	477740	10 accounts, you can basically configure your members in 10 accounts and also the multiple
477740	484940	regions. So how we do here is we have our CloudTrail event rules, which basically capture
484940	491420	on the events which happen in, let's say, account A. And those events could be like, hey, like EC2
491420	499640	modify snapshot attribute. And what happens here is that sends the events on event bus
499640	504780	to your master account where your remediator function lives. So event bus basically will
504780	510160	send that to the master account. And then on the master account, you have same pattern rules which
510160	516200	would capture on the EC2 modify snapshot attribute. And then that is captured by the
516200	522820	event translator, which is like Lambda. So it will capture the raw event and then massage it
522820	528680	and send that to SQS queue. And then the SQS queue is piped to the remediator Lambda, basically.
528800	533440	And the Lambda, remediator Lambda will basically look at the resource, try to enumerate it,
533440	538020	what does it look like? Does the snapshot look like it has been shared publicly? Does the snapshot
538020	542580	look like it has been shared with a random AWS account, which probably we don't own?
542580	547720	Based on that decision, the remediator Lambda will make the decision to remediate it or not
547720	554480	remediate it. So there are event translator and event forwarder here and remediator function,
554480	563140	which makes it event-driven. Moving on, like we mentioned the limitations of V1,
563140	569140	which was based on the pooling model and we were using Cron for triggering our previous Lambdas
569140	574960	every 10 minutes or 20 minutes. So we thought we would maybe get rid of the pooling-based stuff,
574960	579620	but not really. So we still have Cron, we still have the pooling-based approach
580160	587500	in our V2 where we pool for certain issues, let's say, for example, missing MFA or old access keys
587500	593580	of inactive users. But some of these events are really hard to capture on your Cloudflare API logs
593580	599500	or the event rules, like let's say old access keys, how do you capture on the API calls?
599500	605180	You have to enumerate the user, find when it was created, when it was used and stuff.
605180	610620	So we have the pooling stuff, which basically runs on a Cron, say nightly or whatnot. And then
610620	617320	what it does is it will go to each account, assume a role and do the enumeration and then send
617320	622820	everything, every finding to an SQS queue, which is sent to a remediator function, same remediator
622820	627300	function, which will again enumerate the resource and make the decision if it should be remediated
627300	637360	or not. All right, so now who needs API calls? So what we see here is we saw a big drop in number
637360	642660	of API calls which our remediator function was making. The one spike you see is the polar function,
642660	648280	which was running nightly for describing the resources. And also if we move ahead and see
648280	653560	how many request limit exceeded errors we got, it was almost like zero. Maybe it won't translate
653560	660760	exactly the same for you, but what we saw was we experienced zero pushbacks from AWS.
665010	669790	Okay, so what are we watching exactly? So these are the list of things which we are watching
669790	676050	currently. So if you look at the EC2, we have a check for publicly accessible AMI or
676050	682690	an AMI shared with an unknown account, or EBS snapshots publicly exposed, or same applies if
682690	687790	it is shared with an unknown account. Missing tags, IMDS v2. And then one extra thing which we have
687790	693770	here is a check for publicly exposed EC2 instances. And what we have done is they categorize
694370	700070	that off like a dev account. So you have like an enrollment variable where you can specify what
700070	704970	are your dev accounts. And let's say if you want more stringent controls on that account, you can
704970	710670	make that rule that hey, no EC2 should be exposed in dev account. And similarly, we have like
710670	716730	multiple other remediation for IAM users, like look for inactive console access, look for inactive
716730	724370	access keys, missing MFA for RDS if RDS is publicly exposed, if the snapshot is exposed,
724370	730770	or if the storage is not encrypted. Similarly, it applies to Redshift, S3, and IAM role if it's
730770	737210	or Lambda. I think, yeah, this list is like pretty much like self-explanatory.
737990	744830	All right, so moving on to the demo, which we have, let me... Yeah, and to jump in real quick,
744830	751390	adding remediations to this is also, you know, pretty easy. It's not like this is just hard
751390	757370	code. It is like a monolith. It's very modular. So writing this is really as easy as writing a
757370	762170	pretty simple Lambda that checks whatever it is that you're wanting to check. So a lot of these
762170	768050	are sort of kind of designed around the requirements of our environment, but we've
768050	772470	built this in such a way that you can add things to your environment. So for whatever reason you
772470	777810	want all of your assets to be publicly available in your DMZ account or something, like you can
777810	782690	enforce that kind of thing too. It's really not too complicated to write one of these modules,
782690	787490	and it's basically plug and play. Yeah, exactly. Yeah, so like just to mention, so our main
787490	793290	intention was like make it modular so you can add more controls. I can actually show that, like if I
793950	810600	go to GitHub, AWS. All right, so if you go to resources, just a small look here on the remediator
812460	816120	and auditors, so you basically have like multiple checks for each module. So what
816120	819600	you can really maybe add for things that you want to watch for, let's say you want to watch
819600	825260	for API gateways or you want to watch for ECS and whatnot. So you can basically add a module here.
826720	831320	All right, so jumping onto the demo, let's make some things like publicly exposed. Let's make an
831320	839860	EMI exposed. So we have a test EMI here, and what we are going to do is modify image permissions
839860	845840	and make it publicly exposed. Save it. Let's do the same thing for snapshots meanwhile.
848670	858150	So I have this, modify permissions, make it public, save it, go back to the EMI,
858150	864710	refresh. Most likely it might be done by now, maybe not, we'll see. So it was made public,
864710	871010	now it has been converted to private. It's almost real time, like probably took like
871010	877230	three, four seconds, I think. And go back to snapshots. If we look in the permissions,
877230	883910	it's private. So we can do the same exercise for S3 also. Let's go to S3
885650	894070	and just choose any of the buckets. Let's slap public access, list object,
894070	899610	and I can see because something is blocking my view. All right, so ACL public,
900510	909030	and we can do similar stuff for lambdas, wait for IAM role also. IAM, let's go to role,
909030	915750	I might have a test role here. What we'll do is edit trust relationship and just allow everyone
915750	921990	with the principal star to assume this role. Update the trust relationship. Yeah, sure.
922630	927250	It says overly permissive policies. Let's go back and we'll see if S3 has been fixed.
929070	936050	Okay, so the ACL was removed and it's back again private. Let's see IAM, what has happened to this
936050	944540	thing. Okay, so it basically changed the trust relationship and added deny on principal star.
944540	950700	So basically, no one can assume this role. So yeah, so basically, what you can do is that you
950700	956580	can have like multiple services and it basically will respond to it almost in real time. Let's say
956580	962400	you're making RDS or modifying an RDS instance and making it publicly exposed. It will capture
962400	969000	the event and try to see if the star configuration right now is bad. As in, is it publicly exposed?
969000	973980	Is it not encrypted? And make the decision. Some things which are challenging is RDS and Redshift
973980	979400	because when you spin up an RDS, what happens is it takes a shitload of time, like 20 minutes,
979400	985140	30 minutes. So when you're using an event-driven approach, the lambdas will trigger immediately,
985140	990100	but when they try to enumerate the RDS instance, it might not be in the available state.
990560	995660	And that what happens is like your lambdas will see and that it's not available and they'll skip
995660	1000800	it. But once your RDS is available and it is public facing and you don't and then you can't
1000800	1006420	catch it unless your polar runs like by a cron. So what we have done is like we have added some
1006420	1011760	like repeat invocation, like let's check for the RDS. If it is not available, maybe repeat
1011760	1016620	the same message to SQS, send it again like after like five minutes so it will keep on checking
1016620	1020780	every five minutes. Is it available now? Is it available now? If it is available, check it and
1020780	1024960	then make the decision like if it is public, make it private. If it is not encrypted, stop the
1024960	1037780	instance. All right, so let's jump on to the Q&A. Do we have any questions?
1039320	1046840	Oh yeah, a couple of questions for me. So in this case, those little modules that you showed
1046840	1053320	is when it comes to detecting something as well as responding to it. The response part,
1053320	1059800	is that also built into each of the modules? Is that how? So yeah, so basically what happens
1059800	1070160	I'll show you. If I go to the code, so if I go to resources, event translator,
1071080	1076420	main.py. So what happens like for each of these API calls, which are saying like create queue,
1076420	1082560	set SQS, queue attributes. So you have like for DB, we have like some delays like 120
1083200	1088180	seconds. Similarly for these, like they are almost real-time, right? Whenever this event is seen,
1088180	1094420	forward the event almost real-time to the SQS queue, and then it will trigger the
1094420	1099020	remediator function. But some of the other calls we have, like we have added some delays.
1101860	1107980	Yeah, yes, yes. And then kind of the beauty of some of this is that because it's all relatively
1107980	1114760	short Python code, if you wanted to integrate this with like, I don't know, if you have like
1114760	1120380	some kind of compliance tool that has detection or remediation built in and has like an API,
1120380	1126080	you can pretty easily leverage those like REST APIs or whatever to do the detection and the
1126080	1130700	remediation for you. And this just kind of ties all the automation together because most of those
1130700	1134920	tools that we've experimented with don't really have that. So that's something we're kind of
1134920	1140720	working on building out as well as leveraging some of the other like detection and response
1140720	1145280	tools so that this kind of just glues it all together and makes it event-driven. So you have
1145280	1151140	lots of flexibility to do stuff like that. And the remediator modules are relatively short. It's
1151140	1155460	basically just examining the event and then doing, you know, making API calls to do what it is you
1155460	1161020	want to do. So yeah, the remediations are built in and very easily customizable. Yeah, that basically
1161020	1167640	almost answered my second question, which was, you know, the polar function, which detects something,
1167640	1173860	that the CloudWatch events doesn't cover, that could easily be substituted or, you know, something
1173860	1179260	could be added on to it, which is an open source tool that does those checks or any additional
1179260	1183960	stuff that you want to do. Yeah, definitely. I mean, if you look at the PolarCore right now,
1183960	1189980	it's doing similar stuff, but doing like a sanity check every night, let's say,
1189980	1194260	it's doing the same RDS check, describe RDS, describe Snapshot, describe AMI, and also
1194260	1199000	including the other checks, like IAM user, if they are not using the access keys and stuff. So
1199000	1203260	we have clubbed everything, but let's say if you don't want to use something, some of the stuff,
1203260	1207680	you can basically remove it. So it will, like, do sanity check, like, once a night, but you can
1207680	1213780	modify the cron schedule. Let's say you want to do your, like, once an hour or once every four
1213780	1218540	hours, you can do that. It's very customizable. And also the whole deployment is all, like,
1218540	1222680	data form and matchstrips. So it's very easy to deploy and easy to destroy.
1223920	1232880	Got it. We have a couple of questions here from the chat, from John. In the what are we watching
1232880	1240720	slide under EC2, why are you watching IMDS v2 to ensure v1 is not used?
1241220	1246940	Yeah, that's exactly it. So, you know, we have a large environment with lots of apps,
1246940	1251460	and because it's medical apps, there's lots of, like, legacy components,
1251460	1258140	and there's lots of, you know, complexity. So, yeah, the IMDS v2 stuff is mostly monitoring
1258140	1264600	what is still using IMDS v1. And then so here, correct me if I'm wrong, it also flags when
1264600	1268460	things are using IMDS v2 to, like, say it's compliant, but it doesn't, like,
1268460	1272820	it's not automatically killing things. I don't believe it's written.
1272820	1279240	If you're in a state where you can, like, you are, like, okay, I'm in that state where my EC2
1279240	1284840	are supposed to be on IMDS v2, you can toggle the plug and stop and add the check for stop
1284840	1294020	instances if they are not, like, on v1 or on v2. And then second question from Scott.
1294360	1298240	Is the AWS remediation framework intended to complement or supplement
1298240	1302220	the use of AWS config with its own remediation processes?
1302780	1309380	So what we have, like, thought of, like, was to add config also as one of the events.
1309380	1313920	Also, like, right now we have polling and we have event-driven. So we thought of, like,
1313920	1317920	moving to config and, like, enumerate it, but we haven't, like, done the full enumeration part,
1317920	1321560	like how config can, like, help us in achieving the similar state which we are achieving by
1321560	1325640	event-driven plus polling. I'm sure, like, config has the custom rules and whatnot,
1325640	1329260	which you can, like, add and do the remediation based on the custom lambdas.
1329260	1333640	But, yeah, something which for to-do we want to, like, do, maybe add, like, a config integration,
1333640	1337300	maybe if you want to have three options of, like, leveraging the same framework. Because
1337920	1342240	even with custom model tool, you'll have to have, like, your lambdas to make the remediation.
1344680	1345400	And so...
1345400	1346260	Oh, go ahead.
1346600	1347540	Oh, no, go ahead.
1347620	1349900	I just wanted to ask, kind of going off of that,
1349900	1355400	any other sneak peek or future stuff you have in the plan for this tool?
1355400	1360400	Yeah. So, I mean, Sahir and I have both kind of mentioned it. One of the things that we're
1360400	1365320	wanting to look at is more integration into other tools, as well as things like AWS config,
1365320	1371480	to kind of leverage this stuff. We are a relatively small team. I think we have, well,
1371480	1379200	we have three people, basically, kind of managing our pretty large cloud environment. So, you know,
1379200	1387000	when it comes to leveraging existing tooling, our general philosophy is kind of use it all
1388020	1394220	and automate the triage to clean stuff up. So, really, what we're trying to get into is,
1394220	1399320	can we leverage existing checks and remediations from things like config,
1399320	1403560	security monkey is dead, but, like, tools that are similar to security monkey and things like that,
1403560	1407300	and leverage those checks and remediations instead of writing our own,
1407300	1412800	using kind of like the rest APIs and all the fun, nice features that people are putting out there.
1412980	1418780	So, you know, if we see a tool that makes this kind of stuff easy, the future state that we see
1418780	1424540	for the remediation framework is kind of gobbling it up and consuming it to make this stuff more
1424540	1430740	effective, giving us greater confidence that the detective capabilities are actually correct to,
1430740	1435120	like, lower false positive rate, or just to make development quicker.
1438000	1446660	Got it. And another question for me, you mentioned that your team over there is quite small.
1446660	1453380	I know that in order to get to a place where you guys are at with this tool to remediate right away,
1453380	1458180	or even with a certain delay, get those remediations going in the cloud environment,
1458180	1464800	you need to have sort of a in-level kind of in-depth knowledge of your cloud environment,
1464800	1470660	as well as buy-in from other teams. Any kind of insights as to how you got there,
1470660	1477400	fun stories, or anything like that? I think me and Sahir can both give some insight into this. So
1477400	1483660	Sahir has been at Flatiron much longer than me, and he is a very beloved resource among the entire
1483660	1489900	company, where I am kind of known more as the jackass who comes in and tells everybody no all
1489900	1496440	the time. But I like doing that. So one of the things that we found that really made these things
1496440	1500800	very successful was communicating to our engineers, because, you know, they're engineers,
1500800	1505420	they like to see cool shit. And like showing them that this is cool, it means that they can
1505420	1518600	experiment more as they're developing and building without worry. Oh, we lost Justin.
1521300	1527380	Justin, if you could try again for the past five seconds.
1527440	1532820	Yeah, so that developers could experiment a little more without fear of exposing, you know,
1532820	1538500	all of our patients' data. They actually really enjoyed that idea. And then we also found that
1538500	1543140	when we showed them that we had the capabilities to do these things, they suddenly had all kinds
1543140	1547860	of ideas as to how they could make use of it as part of their deployment, development, and testing
1547860	1553820	processes. So being able to tear down test instances after, you know, 15 minutes after
1553820	1558640	their tests completed, and things like that. So a lot of the way that we got buy-in from this stuff
1558640	1563540	was showing them how not only did it make the company and our patients safer, it made their day
1563540	1570080	easier and gave them the ability with, you know, a lot without having to build a whole new solution,
1570080	1573940	they could like automate their testing processes and have it all be event driven,
1573940	1579220	which was stuff that they wanted. So, you know, just being willing to be transparent and communicate
1579220	1583800	very closely with them and be helpful and show them how this could be like a boon to their
1583800	1589600	processes was really, really integral in getting people excited and on board with this, where we
1589600	1594100	could just start flipping switches and turning remediations on, because suddenly, you know,
1594100	1598300	they didn't have to worry about being the person who exposed an S3 bucket full of, you know,
1598300	1604100	20 million patients or whatever. So here probably has another view on this.
1604560	1608640	No, I think this new cover pretty much everything. Yeah, so it's more on, like,
1608640	1613420	how are we going to give more flexibility to developers? Like, are we confident, like,
1613420	1618180	making a segregated account where we can, like, let them do whatever they want and still have
1618180	1623320	some sort of guardrails to watch for, some sort of guardrails to, like, detect things, along with
1623320	1628580	SCPs and other preventative controls? Like, what if still things happen? Like, can we have, like,
1628640	1633440	a framework which watches for, let's say, S3 bucket or AMIs and whatnot and fixes it automatically?
1633720	1637760	So yeah, so ensuring that level of maturity is what we're aiming for. And I'm pretty sure, like,
1637760	1644160	software engineering kind of is on board on that. And then any kind of quick insight into
1644940	1652940	what the cost savings were like from v1 to v2? Oh, so we haven't, like, compared,
1652940	1658620	like, specifically the cost saving. We did, like, actually save a lot from the API calls
1658620	1663800	perspective. Like, if I, if you look back at the slides which we had from the v1, we were making
1663800	1671260	tons of API call, like, for the most API calls spent was on snapshots, describing snapshots.
1671260	1675600	We got to describe a snapshot and then you're described for each snapshot attribute. So it was,
1675600	1681520	like, a very, very iterative. So we had, like, so many API calls which were impacting other stuff
1681520	1685480	with, like, the whole situation where Ansible and stuff, because we were almost, like, getting
1685480	1693520	pushed back on the EC2 resources, always hitting the limits. So v2 basically gave us some room,
1693520	1698240	like, that we don't have to, like, describe a poll all the time. We can do it when certain
1698240	1705400	events which we care about happens. So that was some of the more, like, API saving, the number
1705400	1710820	of API call savings, but not specifically cost. That we haven't, like, compared yet.
1711240	1717180	Yeah, the cost savings for me was our SREs quit calling, saying that API, you know, we were making
1717180	1721800	too many API calls and people were getting rate limited, which is worth, like, a zillion dollars.
1721800	1726220	Because, like, if there's... I haven't made this clear, the one thing that I don't want to be doing
1726220	1732300	is being on call and getting calls at night pisses me off to no end, especially when it's an angry SRE,
1732300	1736260	because the SREs do not give a shit what I have to say. They are mad that things aren't working.
1736260	1742440	So I... that stopped and that is worth just... it's invaluable. That's really all I care about is a
1742440	1748980	quick calling. I think we can all agree there. A couple more questions from the chat. Have you
1748980	1755180	tested or compared this with AWS GuardDuty? So we do have... we do use, like, GuardDuty,
1755180	1761040	but then GuardDuty, like, it doesn't capture everything what we want. Let's say, sure, it
1761040	1765040	might have, like, for S3 checks and everything. But for, like, specific stuff, like, which we're
1765040	1769640	looking for, like, IAM roles and whatnot, I'm not sure if it is being covered in GuardDuty, but we
1769640	1776880	do complement that with, like, using GuardDuty, like, standalone. Yeah, to kind of... to add to that, you
1776880	1782220	know, like we said before, I don't necessarily believe in replacing free things. If there's
1782220	1787420	things that do things well, we will use them both as best we can and then try and, you know, write
1787420	1792440	some code to glue it together. So, you know, we have a lot of... when you ask these questions about
1792440	1796600	these things, very often the answer is we're using both of them as well as this.
1798880	1805740	Got it. And another one, for some cases, why not create a preventative SCP?
1806100	1812680	Sure, that's... yeah, this one, go ahead. Yeah, absolutely do that. You should do that. This is
1812680	1819240	another layer helping us, you know, validate that the SCP hasn't changed or that I didn't fat finger
1819240	1827360	the SCP or, honestly, I'm not that smart. I might have written a bad SCP and I kind of... I like
1827360	1831040	having the many eyes approach and, like, the many, you know, the defense in depth approach of, you
1831040	1839140	know, maybe I wrote a bad SCP and this is something we can use to detect... we can use to detect,
1839140	1844020	you know, something going public again. So we have both. We're using SCPs. You should be using
1844020	1849380	SCPs. They're a great preventative control. I also would say that having preventative controls
1849380	1853320	does not mean you should avoid having detective and responsive controls as well. You know,
1853320	1854700	you need multiple layers.