WEBVTT 00:00.000 --> 00:02.660 So hi, I'm Justin Pagliarani. 00:02.660 --> 00:06.040 I'm a staff security engineer at Flatiron. 00:06.040 --> 00:10.200 What that means is I'm the team lead for the cloud security team. 00:10.420 --> 00:18.000 Prior to Flatiron, I worked at Bishop Fox as a pen tester and various quasi-governmental organizations before that. 00:18.060 --> 00:19.880 So I've been around for a little while. 00:21.340 --> 00:23.320 Yeah, I guess that's kind of me. 00:23.320 --> 00:23.720 Suheer? 00:23.720 --> 00:24.500 Sure. 00:24.500 --> 00:26.880 So I'm a senior security engineer at Prime Health. 00:27.040 --> 00:28.680 I've been here like three years. 00:28.680 --> 00:32.760 And then before that, I was a software engineer and did software engineering for like three years. 00:32.940 --> 00:34.600 And that's pretty much about me. 00:35.140 --> 00:35.480 Yep. 00:35.480 --> 00:41.640 So for anybody who doesn't know about Flatiron, we basically use technology to try and fight cancer. 00:41.860 --> 00:43.680 We're not going to be talking about any of that today. 00:43.680 --> 00:48.960 We're going to be talking about cloud security misconfiguration, detection, and remediation. 00:49.260 --> 00:52.720 So kind of going into that, we can move to the next slide. 00:52.720 --> 00:54.360 Sorry, we were playing the slide dance. 00:54.720 --> 00:55.140 We got some... 00:57.920 --> 01:03.220 Scott Piper from Summit Route has helped us out quite a bit with this, as well as Reza, who is our summer intern. 01:03.600 --> 01:11.280 He's a PhD student, and he did some really great work for us as well, helping us build this out and put it to good use in our environment. 01:11.840 --> 01:15.320 So kind of on from there, we'll get into the agenda. 01:15.680 --> 01:27.660 We're going to kind of give you the background of the problem we were facing, kind of our first iteration of a solution, kind of where we're at now, the things that we've solved for, a quick demo, and a Q&A. 01:28.920 --> 01:36.200 So kind of our general background here is we kind of wanted to know how our critical resources are configured. 01:36.320 --> 01:51.320 We wanted to be able to detect if things become configured badly, and if they become misconfigured, can we fix it in a reasonable amount of time automatically without having to be on call at night, because nobody wants to do that. 01:51.400 --> 02:14.120 So kind of generally why we want to do this is because cloud configuration problems are pretty easy to accidentally do in the new kind of DevOps, well, newish DevOps paradigm, where we've now decided that the software engineers or sysadmins, mistakes happen, 02:14.120 --> 02:32.080 and we end up with lots of things like, you know, S3 buckets being exposed, you know, people losing IAM tokens through IMDS, all kinds of weird things, basically through like silly misconfigurations that if you kind of like thought through, you know, you would catch pretty easily, 02:32.080 --> 02:38.560 like having an S3 bucket be available publicly when it has sensitive information is like logically a very simple mistake. 02:38.720 --> 02:47.800 But because of the complexities of how we are building and utilizing cloud resources, it's something that kind of happens a lot. 02:47.800 --> 02:53.980 So at Flatiron, we handle millions and millions of patient records, you know, raw PII from doctors. 02:54.040 --> 03:05.500 So, you know, an S3 bucket with that information becoming publicly available would one, be a disaster for us from a business standpoint, but also be a disaster for our patients. 03:05.500 --> 03:12.480 And when you're dealing with cancer patients, the last damn thing they need is to be dealing with identity theft, or even the thoughts of it. 03:12.740 --> 03:14.500 You know, we don't we don't want to do that. 03:14.880 --> 03:16.560 Stress is bad for health outcomes. 03:16.560 --> 03:18.340 So we're trying to avoid that. 03:20.600 --> 03:34.320 So kind of our first iteration on going about fixing this was to use lambdas, you know, just do serverless bits of code to check over our resources every 10 minutes or so. 03:34.380 --> 03:37.940 We had a bunch of lambdas checking for a lot of different things. 03:38.320 --> 03:48.720 It generally would do one lambda per resource or one lambda per misconfiguration that would detect an issue and then fix it automatically, just kind of like your standard polling model. 03:48.800 --> 03:51.720 And when we first started doing this, we had many fewer resources. 03:51.720 --> 03:53.280 We were a much smaller company. 03:53.300 --> 03:54.480 It worked great. 03:54.480 --> 03:55.820 Things were fantastic. 03:55.820 --> 03:56.860 And it was great. 03:56.860 --> 04:03.880 So kind of the general architecture of how this worked was we had separate lambdas for each remediation. 04:04.020 --> 04:11.840 We had cloud watch events or we would be watching the cloud watch events from our primary account, which we call our security account. 04:12.140 --> 04:17.100 And then the lambda would assume roles into each of our other accounts to kind of look at resources that way. 04:17.100 --> 04:26.140 So we kind of had like a centralized, isolated deputy that would then kind of move on to the different accounts and monitor all the resources and then fix them if things were broken. 04:28.320 --> 04:29.180 And it worked. 04:29.180 --> 04:30.180 You know, we had great things. 04:30.180 --> 04:38.920 So, we would be getting JIRA tickets that would tell us like, oh, somebody stood up this S3 bucket and it was public accidentally, but we fixed it right away. 04:38.920 --> 04:40.500 So no harm, no foul. 04:44.560 --> 04:46.220 A fidelity of about 10 minutes. 04:46.220 --> 04:47.980 So that was not so bad. 04:49.000 --> 04:50.100 You know, it worked. 04:50.100 --> 04:50.620 It was great. 04:50.620 --> 04:51.820 Why did we move on? 04:52.180 --> 04:56.420 Well, the general problem was we were using tons of API calls, lots and lots. 04:56.420 --> 04:58.860 And as the environment grew, it got to be worse. 04:58.980 --> 05:00.200 That got to be very costly. 05:00.200 --> 05:04.540 It was causing production issues and everyone was kind of mad at us. 05:04.760 --> 05:06.820 It was not a great thing. 05:07.140 --> 05:16.480 So, you know, we ended up with lots of rate limiting, which was, I mean, not the worst thing in the world, but it wasn't great. 05:16.480 --> 05:19.040 And it became more and more of a problem as time went on. 05:19.040 --> 05:25.040 So, you know, we kind of wanted to avoid this, you know, every 10 minutes things were starting to get screwed up. 05:26.020 --> 05:33.180 So kind of our general post-mortem was we had these cron, like basically cron jobs running to check these things. 05:33.180 --> 05:34.980 It worked very well. 05:34.980 --> 05:38.640 But, you know, polling is just kind of crappy in general. 05:38.640 --> 05:48.120 So we wanted to move away from that and move into something better because just the number of API calls in a cloud environment was getting costly from a number of different standpoints. 05:48.120 --> 05:56.520 And then also just the general fidelity of having a 10 minute delay in response time was not necessarily great. 05:56.580 --> 06:05.920 You know, if we could act a little bit more, I guess a quicker response would be better, you know, when you're dealing with as much PII as we do. 06:07.500 --> 06:11.960 So kind of enter the remediation framework V2. 06:12.020 --> 06:13.440 So Saheer, take it away. 06:13.640 --> 06:14.000 All right. 06:14.000 --> 06:24.020 So all these problems which we saw before, like multiple API calls and also having exponential backoffs, which were like impacting other stuff, right? 06:24.020 --> 06:30.940 And the issue also was like we had like cron based polling, which will depend on like how is your cron frequency. 06:30.940 --> 06:35.660 Like we had like 10 minutes, but for some reasons, for some stuff we had like 20 minutes also. 06:35.680 --> 06:38.700 It was not real time or it was not that easy to scale. 06:38.700 --> 06:48.560 So that all brought us to like work on the V2.0 of that, like how we can like aggregate all of these independent lambdas into one unified code base. 06:48.560 --> 06:57.720 So instead of like multiple lambdas, we have one single stuff where we have like multiple modules for like let's say S3, lambda, RDS, or AMIs, or EC2s. 06:57.720 --> 07:01.900 So we try to like unify everything and try to add like event driven models. 07:01.900 --> 07:16.340 So because we know like what events would be responsible for making things, like for example, like when we talk about S3 buckets, you would know that put bucket ACL or put bucket policy would be one of those calls, like which would make the bucket public, 07:16.340 --> 07:16.560 right? 07:16.560 --> 07:22.480 By making either slapping an ACL or slapping a bucket policy, which basically exposes your S3 bucket. 07:22.480 --> 07:28.840 Similarly, we have the same API calls for other resources, like let's say EC2 AMIs or snapshots. 07:28.840 --> 07:37.360 So we wanted to capture on that and make a framework which can tap onto these events which are happening and then maybe trigger our framework. 07:37.560 --> 07:42.160 So if we move ahead, we'll see the architecture, like how exactly it is set up. 07:42.400 --> 07:48.920 So the V2.0 is basically, you can use like multiple accounts with it, multiple regions. 07:49.680 --> 07:58.520 So let's say if you have like 10 accounts, you can basically configure your members in 10 accounts and also the multiple regions. 07:58.520 --> 08:08.260 So how we do here is we have our CloudTrail event rules, which basically capture on the events which happen in, let's say, account A. 08:08.480 --> 08:13.480 And those events could be like, hey, like EC2 modify snapshot attribute. 08:13.560 --> 08:23.380 And what happens here is that sends the events on event bus to your master account where your remediator function lives. 08:23.400 --> 08:26.480 So event bus basically will send that to the master account. 08:26.480 --> 08:34.160 And then on the master account, you have same pattern rules which would capture on the EC2 modify snapshot attribute. 08:34.160 --> 08:38.300 And then that is captured by the event translator, which is like Lambda. 08:38.400 --> 08:45.080 So it will capture the raw event and then massage it and send that to SQS queue. 08:45.160 --> 08:48.680 And then the SQS queue is piped to the remediator Lambda, basically. 08:48.800 --> 08:54.680 And the Lambda, remediator Lambda will basically look at the resource, try to enumerate it, what does it look like? 08:54.680 --> 08:57.340 Does the snapshot look like it has been shared publicly? 08:57.340 --> 09:02.580 Does the snapshot look like it has been shared with a random AWS account, which probably we don't own? 09:02.580 --> 09:08.980 Based on that decision, the remediator Lambda will make the decision to remediate it or not remediate it. 09:09.380 --> 09:16.960 So there are event translator and event forwarder here and remediator function, which makes it event-driven. 09:17.360 --> 09:31.360 Moving on, like we mentioned the limitations of V1, which was based on the pooling model and we were using Cron for triggering our previous Lambdas every 10 minutes or 20 minutes. 09:31.360 --> 09:36.620 So we thought we would maybe get rid of the pooling-based stuff, but not really. 09:36.620 --> 09:49.100 So we still have Cron, we still have the pooling-based approach in our V2 where we pool for certain issues, let's say, for example, missing MFA or old access keys of inactive users. 09:49.100 --> 09:59.500 But some of these events are really hard to capture on your Cloudflare API logs or the event rules, like let's say old access keys, how do you capture on the API calls? 09:59.500 --> 10:05.180 You have to enumerate the user, find when it was created, when it was used and stuff. 10:05.180 --> 10:09.940 So we have the pooling stuff, which basically runs on a Cron, say nightly or whatnot. 10:09.940 --> 10:28.220 And then what it does is it will go to each account, assume a role and do the enumeration and then send everything, every finding to an SQS queue, which is sent to a remediator function, same remediator function, which will again enumerate the resource and make the decision if it should be remediated or not. 10:30.220 --> 10:33.100 All right, so now who needs API calls? 10:33.100 --> 10:40.560 So what we see here is we saw a big drop in number of API calls which our remediator function was making. 10:40.560 --> 10:45.640 The one spike you see is the polar function, which was running nightly for describing the resources. 10:46.040 --> 10:52.700 And also if we move ahead and see how many request limit exceeded errors we got, it was almost like zero. 10:52.700 --> 11:00.760 Maybe it won't translate exactly the same for you, but what we saw was we experienced zero pushbacks from AWS. 11:05.010 --> 11:07.410 Okay, so what are we watching exactly? 11:07.410 --> 11:10.690 So these are the list of things which we are watching currently. 11:10.770 --> 11:24.210 So if you look at the EC2, we have a check for publicly accessible AMI or an AMI shared with an unknown account, or EBS snapshots publicly exposed, or same applies if it is shared with an unknown account. 11:24.210 --> 11:26.290 Missing tags, IMDS v2. 11:26.290 --> 11:31.830 And then one extra thing which we have here is a check for publicly exposed EC2 instances. 11:32.090 --> 11:36.010 And what we have done is they categorize that off like a dev account. 11:36.010 --> 11:41.130 So you have like an enrollment variable where you can specify what are your dev accounts. 11:41.130 --> 11:49.450 And let's say if you want more stringent controls on that account, you can make that rule that hey, no EC2 should be exposed in dev account. 11:49.670 --> 12:06.710 And similarly, we have like multiple other remediation for IAM users, like look for inactive console access, look for inactive access keys, missing MFA for RDS if RDS is publicly exposed, if the snapshot is exposed, or if the storage is not encrypted. 12:06.710 --> 12:12.030 Similarly, it applies to Redshift, S3, and IAM role if it's or Lambda. 12:13.610 --> 12:17.210 I think, yeah, this list is like pretty much like self-explanatory. 12:17.990 --> 12:28.710 All right, so moving on to the demo, which we have, let me... Yeah, and to jump in real quick, adding remediations to this is also, you know, pretty easy. 12:28.710 --> 12:31.690 It's not like this is just hard code. 12:31.690 --> 12:32.990 It is like a monolith. 12:33.110 --> 12:34.350 It's very modular. 12:34.350 --> 12:41.150 So writing this is really as easy as writing a pretty simple Lambda that checks whatever it is that you're wanting to check. 12:41.150 --> 12:51.350 So a lot of these are sort of kind of designed around the requirements of our environment, but we've built this in such a way that you can add things to your environment. 12:51.350 --> 12:59.390 So for whatever reason you want all of your assets to be publicly available in your DMZ account or something, like you can enforce that kind of thing too. 12:59.390 --> 13:04.550 It's really not too complicated to write one of these modules, and it's basically plug and play. 13:04.970 --> 13:05.510 Yeah, exactly. 13:05.510 --> 13:10.610 Yeah, so like just to mention, so our main intention was like make it modular so you can add more controls. 13:10.710 --> 13:19.110 I can actually show that, like if I go to GitHub, AWS. 13:24.540 --> 13:35.900 All right, so if you go to resources, just a small look here on the remediator and auditors, so you basically have like multiple checks for each module. 13:35.900 --> 13:43.420 So what you can really maybe add for things that you want to watch for, let's say you want to watch for API gateways or you want to watch for ECS and whatnot. 13:43.420 --> 13:45.260 So you can basically add a module here. 13:46.720 --> 13:50.940 All right, so jumping onto the demo, let's make some things like publicly exposed. 13:50.940 --> 13:52.600 Let's make an EMI exposed. 13:53.760 --> 14:01.860 So we have a test EMI here, and what we are going to do is modify image permissions and make it publicly exposed. 14:02.220 --> 14:03.180 Save it. 14:03.500 --> 14:05.840 Let's do the same thing for snapshots meanwhile. 14:08.670 --> 14:18.890 So I have this, modify permissions, make it public, save it, go back to the EMI, refresh. 14:18.890 --> 14:22.410 Most likely it might be done by now, maybe not, we'll see. 14:22.590 --> 14:27.530 So it was made public, now it has been converted to private. 14:27.690 --> 14:32.890 It's almost real time, like probably took like three, four seconds, I think. 14:32.930 --> 14:35.110 And go back to snapshots. 14:35.110 --> 14:38.490 If we look in the permissions, it's private. 14:39.110 --> 14:42.890 So we can do the same exercise for S3 also. 14:42.890 --> 14:48.810 Let's go to S3 and just choose any of the buckets. 14:49.330 --> 14:57.230 Let's slap public access, list object, and I can see because something is blocking my view. 14:57.230 --> 15:05.010 All right, so ACL public, and we can do similar stuff for lambdas, wait for IAM role also. 15:05.490 --> 15:10.930 IAM, let's go to role, I might have a test role here. 15:10.990 --> 15:18.430 What we'll do is edit trust relationship and just allow everyone with the principal star to assume this role. 15:18.910 --> 15:20.690 Update the trust relationship. 15:20.830 --> 15:21.990 Yeah, sure. 15:22.630 --> 15:24.630 It says overly permissive policies. 15:24.630 --> 15:27.250 Let's go back and we'll see if S3 has been fixed. 15:29.070 --> 15:33.650 Okay, so the ACL was removed and it's back again private. 15:34.050 --> 15:36.690 Let's see IAM, what has happened to this thing. 15:39.360 --> 15:44.540 Okay, so it basically changed the trust relationship and added deny on principal star. 15:44.540 --> 15:46.360 So basically, no one can assume this role. 15:48.060 --> 15:56.300 So yeah, so basically, what you can do is that you can have like multiple services and it basically will respond to it almost in real time. 15:56.300 --> 16:01.780 Let's say you're making RDS or modifying an RDS instance and making it publicly exposed. 16:01.780 --> 16:07.540 It will capture the event and try to see if the star configuration right now is bad. 16:07.540 --> 16:09.000 As in, is it publicly exposed? 16:09.000 --> 16:10.180 Is it not encrypted? 16:10.180 --> 16:11.100 And make the decision. 16:11.100 --> 16:20.300 Some things which are challenging is RDS and Redshift because when you spin up an RDS, what happens is it takes a shitload of time, like 20 minutes, 30 minutes. 16:20.300 --> 16:30.100 So when you're using an event-driven approach, the lambdas will trigger immediately, but when they try to enumerate the RDS instance, it might not be in the available state. 16:30.560 --> 16:36.020 And that what happens is like your lambdas will see and that it's not available and they'll skip it. 16:36.020 --> 16:43.960 But once your RDS is available and it is public facing and you don't and then you can't catch it unless your polar runs like by a cron. 16:43.960 --> 16:49.920 So what we have done is like we have added some like repeat invocation, like let's check for the RDS. 16:49.920 --> 16:57.360 If it is not available, maybe repeat the same message to SQS, send it again like after like five minutes so it will keep on checking every five minutes. 16:57.360 --> 16:58.140 Is it available now? 16:58.140 --> 16:59.080 Is it available now? 16:59.080 --> 17:03.140 If it is available, check it and then make the decision like if it is public, make it private. 17:03.140 --> 17:05.680 If it is not encrypted, stop the instance. 17:10.720 --> 17:15.880 All right, so let's jump on to the Q&A. 17:16.080 --> 17:17.780 Do we have any questions? 17:19.320 --> 17:22.600 Oh yeah, a couple of questions for me. 17:23.160 --> 17:31.480 So in this case, those little modules that you showed is when it comes to detecting something as well as responding to it. 17:31.480 --> 17:35.920 The response part, is that also built into each of the modules? 17:35.920 --> 17:36.880 Is that how? 17:36.880 --> 17:41.380 So yeah, so basically what happens I'll show you. 17:41.380 --> 17:52.220 If I go to the code, so if I go to resources, event translator, main.py. 17:52.220 --> 17:58.520 So what happens like for each of these API calls, which are saying like create queue, set SQS, queue attributes. 17:58.520 --> 18:04.240 So you have like for DB, we have like some delays like 120 seconds. 18:04.420 --> 18:06.520 Similarly for these, like they are almost real-time, right? 18:06.520 --> 18:15.760 Whenever this event is seen, forward the event almost real-time to the SQS queue, and then it will trigger the remediator function. 18:15.860 --> 18:19.020 But some of the other calls we have, like we have added some delays. 18:21.860 --> 18:23.880 Yeah, yes, yes. 18:23.880 --> 18:40.380 And then kind of the beauty of some of this is that because it's all relatively short Python code, if you wanted to integrate this with like, I don't know, if you have like some kind of compliance tool that has detection or remediation built in and has like an API, 18:40.380 --> 18:47.660 you can pretty easily leverage those like REST APIs or whatever to do the detection and the remediation for you. 18:47.660 --> 18:53.840 And this just kind of ties all the automation together because most of those tools that we've experimented with don't really have that. 18:53.880 --> 19:04.300 So that's something we're kind of working on building out as well as leveraging some of the other like detection and response tools so that this kind of just glues it all together and makes it event-driven. 19:04.600 --> 19:07.240 So you have lots of flexibility to do stuff like that. 19:07.240 --> 19:10.760 And the remediator modules are relatively short. 19:10.980 --> 19:15.940 It's basically just examining the event and then doing, you know, making API calls to do what it is you want to do. 19:15.940 --> 19:18.980 So yeah, the remediations are built in and very easily customizable. 19:19.640 --> 19:41.280 Yeah, that basically almost answered my second question, which was, you know, the polar function, which detects something, that the CloudWatch events doesn't cover, that could easily be substituted or, you know, something could be added on to it, which is an open source tool that does those checks or any additional stuff that you want to do. 19:41.280 --> 19:41.880 Yeah, definitely. 19:41.880 --> 19:56.700 I mean, if you look at the PolarCore right now, it's doing similar stuff, but doing like a sanity check every night, let's say, it's doing the same RDS check, describe RDS, describe Snapshot, describe AMI, and also including the other checks, like IAM user, 19:56.700 --> 19:58.880 if they are not using the access keys and stuff. 19:58.880 --> 20:04.680 So we have clubbed everything, but let's say if you don't want to use something, some of the stuff, you can basically remove it. 20:04.820 --> 20:09.140 So it will, like, do sanity check, like, once a night, but you can modify the cron schedule. 20:09.140 --> 20:14.660 Let's say you want to do your, like, once an hour or once every four hours, you can do that. 20:14.660 --> 20:15.700 It's very customizable. 20:16.140 --> 20:20.380 And also the whole deployment is all, like, data form and matchstrips. 20:20.380 --> 20:22.680 So it's very easy to deploy and easy to destroy. 20:23.920 --> 20:25.300 Got it. 20:25.520 --> 20:30.740 We have a couple of questions here from the chat, from John. 20:30.860 --> 20:40.720 In the what are we watching slide under EC2, why are you watching IMDS v2 to ensure v1 is not used? 20:41.220 --> 20:43.140 Yeah, that's exactly it. 20:43.140 --> 20:53.380 So, you know, we have a large environment with lots of apps, and because it's medical apps, there's lots of, like, legacy components, and there's lots of, you know, complexity. 20:53.380 --> 21:01.420 So, yeah, the IMDS v2 stuff is mostly monitoring what is still using IMDS v1. 21:01.420 --> 21:10.780 And then so here, correct me if I'm wrong, it also flags when things are using IMDS v2 to, like, say it's compliant, but it doesn't, like, it's not automatically killing things. 21:10.780 --> 21:12.820 I don't believe it's written. 21:12.820 --> 21:28.560 If you're in a state where you can, like, you are, like, okay, I'm in that state where my EC2 are supposed to be on IMDS v2, you can toggle the plug and stop and add the check for stop instances if they are not, like, on v1 or on v2. 21:31.360 --> 21:34.020 And then second question from Scott. 21:34.360 --> 21:42.220 Is the AWS remediation framework intended to complement or supplement the use of AWS config with its own remediation processes? 21:42.780 --> 21:49.380 So what we have, like, thought of, like, was to add config also as one of the events. 21:49.380 --> 21:52.720 Also, like, right now we have polling and we have event-driven. 21:52.820 --> 22:02.960 So we thought of, like, moving to config and, like, enumerate it, but we haven't, like, done the full enumeration part, like how config can, like, help us in achieving the similar state which we are achieving by event-driven plus polling. 22:03.140 --> 22:09.260 I'm sure, like, config has the custom rules and whatnot, which you can, like, add and do the remediation based on the custom lambdas. 22:09.260 --> 22:17.200 But, yeah, something which for to-do we want to, like, do, maybe add, like, a config integration, maybe if you want to have three options of, like, leveraging the same framework. 22:17.200 --> 22:22.240 Because even with custom model tool, you'll have to have, like, your lambdas to make the remediation. 22:24.680 --> 22:25.400 And so... 22:25.400 --> 22:26.260 Oh, go ahead. 22:26.600 --> 22:27.540 Oh, no, go ahead. 22:27.620 --> 22:35.400 I just wanted to ask, kind of going off of that, any other sneak peek or future stuff you have in the plan for this tool? 22:35.400 --> 22:36.220 Yeah. 22:36.220 --> 22:39.540 So, I mean, Sahir and I have both kind of mentioned it. 22:39.540 --> 22:48.420 One of the things that we're wanting to look at is more integration into other tools, as well as things like AWS config, to kind of leverage this stuff. 22:48.540 --> 22:50.760 We are a relatively small team. 22:50.760 --> 22:58.020 I think we have, well, we have three people, basically, kind of managing our pretty large cloud environment. 22:58.140 --> 23:11.860 So, you know, when it comes to leveraging existing tooling, our general philosophy is kind of use it all and automate the triage to clean stuff up. 23:11.860 --> 23:27.300 So, really, what we're trying to get into is, can we leverage existing checks and remediations from things like config, security monkey is dead, but, like, tools that are similar to security monkey and things like that, and leverage those checks and remediations instead of writing our own, 23:27.300 --> 23:32.800 using kind of like the rest APIs and all the fun, nice features that people are putting out there. 23:32.980 --> 23:50.740 So, you know, if we see a tool that makes this kind of stuff easy, the future state that we see for the remediation framework is kind of gobbling it up and consuming it to make this stuff more effective, giving us greater confidence that the detective capabilities are actually correct to, 23:50.740 --> 23:55.120 like, lower false positive rate, or just to make development quicker. 23:58.000 --> 23:59.360 Got it. 24:00.240 --> 24:06.660 And another question for me, you mentioned that your team over there is quite small. 24:06.660 --> 24:24.800 I know that in order to get to a place where you guys are at with this tool to remediate right away, or even with a certain delay, get those remediations going in the cloud environment, you need to have sort of a in-level kind of in-depth knowledge of your cloud environment, 24:24.800 --> 24:27.220 as well as buy-in from other teams. 24:27.240 --> 24:33.520 Any kind of insights as to how you got there, fun stories, or anything like that? 24:33.540 --> 24:37.360 I think me and Sahir can both give some insight into this. 24:37.360 --> 24:50.780 So Sahir has been at Flatiron much longer than me, and he is a very beloved resource among the entire company, where I am kind of known more as the jackass who comes in and tells everybody no all the time. 24:50.780 --> 24:52.440 But I like doing that. 24:52.580 --> 25:02.300 So one of the things that we found that really made these things very successful was communicating to our engineers, because, you know, they're engineers, they like to see cool shit. 25:02.300 --> 25:10.220 And like showing them that this is cool, it means that they can experiment more as they're developing and building without worry. 25:16.160 --> 25:18.600 Oh, we lost Justin. 25:21.300 --> 25:27.380 Justin, if you could try again for the past five seconds. 25:27.440 --> 25:34.320 Yeah, so that developers could experiment a little more without fear of exposing, you know, all of our patients' data. 25:34.360 --> 25:36.620 They actually really enjoyed that idea. 25:36.720 --> 25:48.720 And then we also found that when we showed them that we had the capabilities to do these things, they suddenly had all kinds of ideas as to how they could make use of it as part of their deployment, development, and testing processes. 25:49.020 --> 25:56.180 So being able to tear down test instances after, you know, 15 minutes after their tests completed, and things like that. 25:56.180 --> 26:10.080 So a lot of the way that we got buy-in from this stuff was showing them how not only did it make the company and our patients safer, it made their day easier and gave them the ability with, you know, a lot without having to build a whole new solution, 26:10.080 --> 26:16.160 they could like automate their testing processes and have it all be event driven, which was stuff that they wanted. 26:16.180 --> 26:32.440 So, you know, just being willing to be transparent and communicate very closely with them and be helpful and show them how this could be like a boon to their processes was really, really integral in getting people excited and on board with this, where we could just start flipping switches and turning remediations on, 26:32.440 --> 26:39.960 because suddenly, you know, they didn't have to worry about being the person who exposed an S3 bucket full of, you know, 20 million patients or whatever. 26:39.980 --> 26:44.100 So here probably has another view on this. 26:44.560 --> 26:46.940 No, I think this new cover pretty much everything. 26:47.140 --> 26:51.480 Yeah, so it's more on, like, how are we going to give more flexibility to developers? 26:51.860 --> 27:05.280 Like, are we confident, like, making a segregated account where we can, like, let them do whatever they want and still have some sort of guardrails to watch for, some sort of guardrails to, like, detect things, along with SCPs and other preventative controls? 27:05.280 --> 27:07.340 Like, what if still things happen? 27:07.340 --> 27:13.440 Like, can we have, like, a framework which watches for, let's say, S3 bucket or AMIs and whatnot and fixes it automatically? 27:13.720 --> 27:16.740 So yeah, so ensuring that level of maturity is what we're aiming for. 27:16.740 --> 27:20.780 And I'm pretty sure, like, software engineering kind of is on board on that. 27:21.500 --> 27:30.080 And then any kind of quick insight into what the cost savings were like from v1 to v2? 27:30.320 --> 27:35.500 Oh, so we haven't, like, compared, like, specifically the cost saving. 27:35.680 --> 27:39.200 We did, like, actually save a lot from the API calls perspective. 27:39.200 --> 27:51.260 Like, if I, if you look back at the slides which we had from the v1, we were making tons of API call, like, for the most API calls spent was on snapshots, describing snapshots. 27:51.260 --> 27:54.940 We got to describe a snapshot and then you're described for each snapshot attribute. 27:54.940 --> 27:57.260 So it was, like, a very, very iterative. 27:57.380 --> 28:10.020 So we had, like, so many API calls which were impacting other stuff with, like, the whole situation where Ansible and stuff, because we were almost, like, getting pushed back on the EC2 resources, always hitting the limits. 28:10.300 --> 28:16.940 So v2 basically gave us some room, like, that we don't have to, like, describe a poll all the time. 28:16.940 --> 28:20.360 We can do it when certain events which we care about happens. 28:20.420 --> 28:28.840 So that was some of the more, like, API saving, the number of API call savings, but not specifically cost. 28:28.980 --> 28:30.820 That we haven't, like, compared yet. 28:31.240 --> 28:41.800 Yeah, the cost savings for me was our SREs quit calling, saying that API, you know, we were making too many API calls and people were getting rate limited, which is worth, like, a zillion dollars. 28:41.800 --> 28:54.680 Because, like, if there's... I haven't made this clear, the one thing that I don't want to be doing is being on call and getting calls at night pisses me off to no end, especially when it's an angry SRE, because the SREs do not give a shit what I have to say. 28:54.680 --> 28:56.260 They are mad that things aren't working. 28:56.260 --> 29:00.760 So I... that stopped and that is worth just... it's invaluable. 29:00.780 --> 29:03.060 That's really all I care about is a quick calling. 29:04.180 --> 29:06.180 I think we can all agree there. 29:06.180 --> 29:08.140 A couple more questions from the chat. 29:08.140 --> 29:11.740 Have you tested or compared this with AWS GuardDuty? 29:12.160 --> 29:19.400 So we do have... we do use, like, GuardDuty, but then GuardDuty, like, it doesn't capture everything what we want. 29:19.400 --> 29:22.680 Let's say, sure, it might have, like, for S3 checks and everything. 29:22.740 --> 29:32.680 But for, like, specific stuff, like, which we're looking for, like, IAM roles and whatnot, I'm not sure if it is being covered in GuardDuty, but we do complement that with, like, using GuardDuty, like, standalone. 29:34.660 --> 29:41.840 Yeah, to kind of... to add to that, you know, like we said before, I don't necessarily believe in replacing free things. 29:41.840 --> 29:48.980 If there's things that do things well, we will use them both as best we can and then try and, you know, write some code to glue it together. 29:48.980 --> 29:56.600 So, you know, we have a lot of... when you ask these questions about these things, very often the answer is we're using both of them as well as this. 29:58.880 --> 29:59.720 Got it. 29:59.720 --> 30:05.740 And another one, for some cases, why not create a preventative SCP? 30:06.100 --> 30:08.760 Sure, that's... yeah, this one, go ahead. 30:08.980 --> 30:10.360 Yeah, absolutely do that. 30:10.360 --> 30:11.540 You should do that. 30:11.800 --> 30:22.340 This is another layer helping us, you know, validate that the SCP hasn't changed or that I didn't fat finger the SCP or, honestly, I'm not that smart. 30:22.340 --> 30:39.140 I might have written a bad SCP and I kind of... I like having the many eyes approach and, like, the many, you know, the defense in depth approach of, you know, maybe I wrote a bad SCP and this is something we can use to detect... we can use to detect, 30:39.140 --> 30:40.680 you know, something going public again. 30:40.680 --> 30:42.080 So we have both. 30:42.080 --> 30:43.340 We're using SCPs. 30:43.340 --> 30:44.900 You should be using SCPs. 30:44.940 --> 30:46.880 They're a great preventative control. 30:47.340 --> 30:52.960 I also would say that having preventative controls does not mean you should avoid having detective and responsive controls as well. 30:53.140 --> 30:54.700 You know, you need multiple layers.