1
00:00:00,000 --> 00:00:02,660
So hi, I'm Justin Pagliarani.

2
00:00:02,660 --> 00:00:06,040
I'm a staff security engineer at Flatiron.

3
00:00:06,040 --> 00:00:10,200
What that means is I'm the team lead for the cloud security team.

4
00:00:10,420 --> 00:00:18,000
Prior to Flatiron, I worked at Bishop Fox as a pen tester and various quasi-governmental organizations before that.

5
00:00:18,060 --> 00:00:19,880
So I've been around for a little while.

6
00:00:21,340 --> 00:00:23,320
Yeah, I guess that's kind of me.

7
00:00:23,320 --> 00:00:23,720
Suheer?

8
00:00:23,720 --> 00:00:24,500
Sure.

9
00:00:24,500 --> 00:00:26,880
So I'm a senior security engineer at Prime Health.

10
00:00:27,040 --> 00:00:28,680
I've been here like three years.

11
00:00:28,680 --> 00:00:32,760
And then before that, I was a software engineer and did software engineering for like three years.

12
00:00:32,940 --> 00:00:34,600
And that's pretty much about me.

13
00:00:35,140 --> 00:00:35,480
Yep.

14
00:00:35,480 --> 00:00:41,640
So for anybody who doesn't know about Flatiron, we basically use technology to try and fight cancer.

15
00:00:41,860 --> 00:00:43,680
We're not going to be talking about any of that today.

16
00:00:43,680 --> 00:00:48,960
We're going to be talking about cloud security misconfiguration, detection, and remediation.

17
00:00:49,260 --> 00:00:52,720
So kind of going into that, we can move to the next slide.

18
00:00:52,720 --> 00:00:54,360
Sorry, we were playing the slide dance.

19
00:00:54,720 --> 00:00:55,140
We got some...

20
00:00:57,920 --> 00:01:03,220
Scott Piper from Summit Route has helped us out quite a bit with this, as well as Reza, who is our summer intern.

21
00:01:03,600 --> 00:01:11,280
He's a PhD student, and he did some really great work for us as well, helping us build this out and put it to good use in our environment.

22
00:01:11,840 --> 00:01:15,320
So kind of on from there, we'll get into the agenda.

23
00:01:15,680 --> 00:01:27,660
We're going to kind of give you the background of the problem we were facing, kind of our first iteration of a solution, kind of where we're at now, the things that we've solved for, a quick demo, and a Q&A.

24
00:01:28,920 --> 00:01:36,200
So kind of our general background here is we kind of wanted to know how our critical resources are configured.

25
00:01:36,320 --> 00:01:51,320
We wanted to be able to detect if things become configured badly, and if they become misconfigured, can we fix it in a reasonable amount of time automatically without having to be on call at night, because nobody wants to do that.

26
00:01:51,400 --> 00:02:14,120
So kind of generally why we want to do this is because cloud configuration problems are pretty easy to accidentally do in the new kind of DevOps, well, newish DevOps paradigm, where we've now decided that the software engineers or sysadmins, mistakes happen,

27
00:02:14,120 --> 00:02:32,080
and we end up with lots of things like, you know, S3 buckets being exposed, you know, people losing IAM tokens through IMDS, all kinds of weird things, basically through like silly misconfigurations that if you kind of like thought through, you know, you would catch pretty easily,

28
00:02:32,080 --> 00:02:38,560
like having an S3 bucket be available publicly when it has sensitive information is like logically a very simple mistake.

29
00:02:38,720 --> 00:02:47,800
But because of the complexities of how we are building and utilizing cloud resources, it's something that kind of happens a lot.

30
00:02:47,800 --> 00:02:53,980
So at Flatiron, we handle millions and millions of patient records, you know, raw PII from doctors.

31
00:02:54,040 --> 00:03:05,500
So, you know, an S3 bucket with that information becoming publicly available would one, be a disaster for us from a business standpoint, but also be a disaster for our patients.

32
00:03:05,500 --> 00:03:12,480
And when you're dealing with cancer patients, the last damn thing they need is to be dealing with identity theft, or even the thoughts of it.

33
00:03:12,740 --> 00:03:14,500
You know, we don't we don't want to do that.

34
00:03:14,880 --> 00:03:16,560
Stress is bad for health outcomes.

35
00:03:16,560 --> 00:03:18,340
So we're trying to avoid that.

36
00:03:20,600 --> 00:03:34,320
So kind of our first iteration on going about fixing this was to use lambdas, you know, just do serverless bits of code to check over our resources every 10 minutes or so.

37
00:03:34,380 --> 00:03:37,940
We had a bunch of lambdas checking for a lot of different things.

38
00:03:38,320 --> 00:03:48,720
It generally would do one lambda per resource or one lambda per misconfiguration that would detect an issue and then fix it automatically, just kind of like your standard polling model.

39
00:03:48,800 --> 00:03:51,720
And when we first started doing this, we had many fewer resources.

40
00:03:51,720 --> 00:03:53,280
We were a much smaller company.

41
00:03:53,300 --> 00:03:54,480
It worked great.

42
00:03:54,480 --> 00:03:55,820
Things were fantastic.

43
00:03:55,820 --> 00:03:56,860
And it was great.

44
00:03:56,860 --> 00:04:03,880
So kind of the general architecture of how this worked was we had separate lambdas for each remediation.

45
00:04:04,020 --> 00:04:11,840
We had cloud watch events or we would be watching the cloud watch events from our primary account, which we call our security account.

46
00:04:12,140 --> 00:04:17,100
And then the lambda would assume roles into each of our other accounts to kind of look at resources that way.

47
00:04:17,100 --> 00:04:26,140
So we kind of had like a centralized, isolated deputy that would then kind of move on to the different accounts and monitor all the resources and then fix them if things were broken.

48
00:04:28,320 --> 00:04:29,180
And it worked.

49
00:04:29,180 --> 00:04:30,180
You know, we had great things.

50
00:04:30,180 --> 00:04:38,920
So, we would be getting JIRA tickets that would tell us like, oh, somebody stood up this S3 bucket and it was public accidentally, but we fixed it right away.

51
00:04:38,920 --> 00:04:40,500
So no harm, no foul.

52
00:04:44,560 --> 00:04:46,220
A fidelity of about 10 minutes.

53
00:04:46,220 --> 00:04:47,980
So that was not so bad.

54
00:04:49,000 --> 00:04:50,100
You know, it worked.

55
00:04:50,100 --> 00:04:50,620
It was great.

56
00:04:50,620 --> 00:04:51,820
Why did we move on?

57
00:04:52,180 --> 00:04:56,420
Well, the general problem was we were using tons of API calls, lots and lots.

58
00:04:56,420 --> 00:04:58,860
And as the environment grew, it got to be worse.

59
00:04:58,980 --> 00:05:00,200
That got to be very costly.

60
00:05:00,200 --> 00:05:04,540
It was causing production issues and everyone was kind of mad at us.

61
00:05:04,760 --> 00:05:06,820
It was not a great thing.

62
00:05:07,140 --> 00:05:16,480
So, you know, we ended up with lots of rate limiting, which was, I mean, not the worst thing in the world, but it wasn't great.

63
00:05:16,480 --> 00:05:19,040
And it became more and more of a problem as time went on.

64
00:05:19,040 --> 00:05:25,040
So, you know, we kind of wanted to avoid this, you know, every 10 minutes things were starting to get screwed up.

65
00:05:26,020 --> 00:05:33,180
So kind of our general post-mortem was we had these cron, like basically cron jobs running to check these things.

66
00:05:33,180 --> 00:05:34,980
It worked very well.

67
00:05:34,980 --> 00:05:38,640
But, you know, polling is just kind of crappy in general.

68
00:05:38,640 --> 00:05:48,120
So we wanted to move away from that and move into something better because just the number of API calls in a cloud environment was getting costly from a number of different standpoints.

69
00:05:48,120 --> 00:05:56,520
And then also just the general fidelity of having a 10 minute delay in response time was not necessarily great.

70
00:05:56,580 --> 00:06:05,920
You know, if we could act a little bit more, I guess a quicker response would be better, you know, when you're dealing with as much PII as we do.

71
00:06:07,500 --> 00:06:11,960
So kind of enter the remediation framework V2.

72
00:06:12,020 --> 00:06:13,440
So Saheer, take it away.

73
00:06:13,640 --> 00:06:14,000
All right.

74
00:06:14,000 --> 00:06:24,020
So all these problems which we saw before, like multiple API calls and also having exponential backoffs, which were like impacting other stuff, right?

75
00:06:24,020 --> 00:06:30,940
And the issue also was like we had like cron based polling, which will depend on like how is your cron frequency.

76
00:06:30,940 --> 00:06:35,660
Like we had like 10 minutes, but for some reasons, for some stuff we had like 20 minutes also.

77
00:06:35,680 --> 00:06:38,700
It was not real time or it was not that easy to scale.

78
00:06:38,700 --> 00:06:48,560
So that all brought us to like work on the V2.0 of that, like how we can like aggregate all of these independent lambdas into one unified code base.

79
00:06:48,560 --> 00:06:57,720
So instead of like multiple lambdas, we have one single stuff where we have like multiple modules for like let's say S3, lambda, RDS, or AMIs, or EC2s.

80
00:06:57,720 --> 00:07:01,900
So we try to like unify everything and try to add like event driven models.

81
00:07:01,900 --> 00:07:16,340
So because we know like what events would be responsible for making things, like for example, like when we talk about S3 buckets, you would know that put bucket ACL or put bucket policy would be one of those calls, like which would make the bucket public,

82
00:07:16,340 --> 00:07:16,560
right?

83
00:07:16,560 --> 00:07:22,480
By making either slapping an ACL or slapping a bucket policy, which basically exposes your S3 bucket.

84
00:07:22,480 --> 00:07:28,840
Similarly, we have the same API calls for other resources, like let's say EC2 AMIs or snapshots.

85
00:07:28,840 --> 00:07:37,360
So we wanted to capture on that and make a framework which can tap onto these events which are happening and then maybe trigger our framework.

86
00:07:37,560 --> 00:07:42,160
So if we move ahead, we'll see the architecture, like how exactly it is set up.

87
00:07:42,400 --> 00:07:48,920
So the V2.0 is basically, you can use like multiple accounts with it, multiple regions.

88
00:07:49,680 --> 00:07:58,520
So let's say if you have like 10 accounts, you can basically configure your members in 10 accounts and also the multiple regions.

89
00:07:58,520 --> 00:08:08,260
So how we do here is we have our CloudTrail event rules, which basically capture on the events which happen in, let's say, account A.

90
00:08:08,480 --> 00:08:13,480
And those events could be like, hey, like EC2 modify snapshot attribute.

91
00:08:13,560 --> 00:08:23,380
And what happens here is that sends the events on event bus to your master account where your remediator function lives.

92
00:08:23,400 --> 00:08:26,480
So event bus basically will send that to the master account.

93
00:08:26,480 --> 00:08:34,160
And then on the master account, you have same pattern rules which would capture on the EC2 modify snapshot attribute.

94
00:08:34,160 --> 00:08:38,300
And then that is captured by the event translator, which is like Lambda.

95
00:08:38,400 --> 00:08:45,080
So it will capture the raw event and then massage it and send that to SQS queue.

96
00:08:45,160 --> 00:08:48,680
And then the SQS queue is piped to the remediator Lambda, basically.

97
00:08:48,800 --> 00:08:54,680
And the Lambda, remediator Lambda will basically look at the resource, try to enumerate it, what does it look like?

98
00:08:54,680 --> 00:08:57,340
Does the snapshot look like it has been shared publicly?

99
00:08:57,340 --> 00:09:02,580
Does the snapshot look like it has been shared with a random AWS account, which probably we don't own?

100
00:09:02,580 --> 00:09:08,980
Based on that decision, the remediator Lambda will make the decision to remediate it or not remediate it.

101
00:09:09,380 --> 00:09:16,960
So there are event translator and event forwarder here and remediator function, which makes it event-driven.

102
00:09:17,360 --> 00:09:31,360
Moving on, like we mentioned the limitations of V1, which was based on the pooling model and we were using Cron for triggering our previous Lambdas every 10 minutes or 20 minutes.

103
00:09:31,360 --> 00:09:36,620
So we thought we would maybe get rid of the pooling-based stuff, but not really.

104
00:09:36,620 --> 00:09:49,100
So we still have Cron, we still have the pooling-based approach in our V2 where we pool for certain issues, let's say, for example, missing MFA or old access keys of inactive users.

105
00:09:49,100 --> 00:09:59,500
But some of these events are really hard to capture on your Cloudflare API logs or the event rules, like let's say old access keys, how do you capture on the API calls?

106
00:09:59,500 --> 00:10:05,180
You have to enumerate the user, find when it was created, when it was used and stuff.

107
00:10:05,180 --> 00:10:09,940
So we have the pooling stuff, which basically runs on a Cron, say nightly or whatnot.

108
00:10:09,940 --> 00:10:28,220
And then what it does is it will go to each account, assume a role and do the enumeration and then send everything, every finding to an SQS queue, which is sent to a remediator function, same remediator function, which will again enumerate the resource and make the decision if it should be remediated or not.

109
00:10:30,220 --> 00:10:33,100
All right, so now who needs API calls?

110
00:10:33,100 --> 00:10:40,560
So what we see here is we saw a big drop in number of API calls which our remediator function was making.

111
00:10:40,560 --> 00:10:45,640
The one spike you see is the polar function, which was running nightly for describing the resources.

112
00:10:46,040 --> 00:10:52,700
And also if we move ahead and see how many request limit exceeded errors we got, it was almost like zero.

113
00:10:52,700 --> 00:11:00,760
Maybe it won't translate exactly the same for you, but what we saw was we experienced zero pushbacks from AWS.

114
00:11:05,010 --> 00:11:07,410
Okay, so what are we watching exactly?

115
00:11:07,410 --> 00:11:10,690
So these are the list of things which we are watching currently.

116
00:11:10,770 --> 00:11:24,210
So if you look at the EC2, we have a check for publicly accessible AMI or an AMI shared with an unknown account, or EBS snapshots publicly exposed, or same applies if it is shared with an unknown account.

117
00:11:24,210 --> 00:11:26,290
Missing tags, IMDS v2.

118
00:11:26,290 --> 00:11:31,830
And then one extra thing which we have here is a check for publicly exposed EC2 instances.

119
00:11:32,090 --> 00:11:36,010
And what we have done is they categorize that off like a dev account.

120
00:11:36,010 --> 00:11:41,130
So you have like an enrollment variable where you can specify what are your dev accounts.

121
00:11:41,130 --> 00:11:49,450
And let's say if you want more stringent controls on that account, you can make that rule that hey, no EC2 should be exposed in dev account.

122
00:11:49,670 --> 00:12:06,710
And similarly, we have like multiple other remediation for IAM users, like look for inactive console access, look for inactive access keys, missing MFA for RDS if RDS is publicly exposed, if the snapshot is exposed, or if the storage is not encrypted.

123
00:12:06,710 --> 00:12:12,030
Similarly, it applies to Redshift, S3, and IAM role if it's or Lambda.

124
00:12:13,610 --> 00:12:17,210
I think, yeah, this list is like pretty much like self-explanatory.

125
00:12:17,990 --> 00:12:28,710
All right, so moving on to the demo, which we have, let me... Yeah, and to jump in real quick, adding remediations to this is also, you know, pretty easy.

126
00:12:28,710 --> 00:12:31,690
It's not like this is just hard code.

127
00:12:31,690 --> 00:12:32,990
It is like a monolith.

128
00:12:33,110 --> 00:12:34,350
It's very modular.

129
00:12:34,350 --> 00:12:41,150
So writing this is really as easy as writing a pretty simple Lambda that checks whatever it is that you're wanting to check.

130
00:12:41,150 --> 00:12:51,350
So a lot of these are sort of kind of designed around the requirements of our environment, but we've built this in such a way that you can add things to your environment.

131
00:12:51,350 --> 00:12:59,390
So for whatever reason you want all of your assets to be publicly available in your DMZ account or something, like you can enforce that kind of thing too.

132
00:12:59,390 --> 00:13:04,550
It's really not too complicated to write one of these modules, and it's basically plug and play.

133
00:13:04,970 --> 00:13:05,510
Yeah, exactly.

134
00:13:05,510 --> 00:13:10,610
Yeah, so like just to mention, so our main intention was like make it modular so you can add more controls.

135
00:13:10,710 --> 00:13:19,110
I can actually show that, like if I go to GitHub, AWS.

136
00:13:24,540 --> 00:13:35,900
All right, so if you go to resources, just a small look here on the remediator and auditors, so you basically have like multiple checks for each module.

137
00:13:35,900 --> 00:13:43,420
So what you can really maybe add for things that you want to watch for, let's say you want to watch for API gateways or you want to watch for ECS and whatnot.

138
00:13:43,420 --> 00:13:45,260
So you can basically add a module here.

139
00:13:46,720 --> 00:13:50,940
All right, so jumping onto the demo, let's make some things like publicly exposed.

140
00:13:50,940 --> 00:13:52,600
Let's make an EMI exposed.

141
00:13:53,760 --> 00:14:01,860
So we have a test EMI here, and what we are going to do is modify image permissions and make it publicly exposed.

142
00:14:02,220 --> 00:14:03,180
Save it.

143
00:14:03,500 --> 00:14:05,840
Let's do the same thing for snapshots meanwhile.

144
00:14:08,670 --> 00:14:18,890
So I have this, modify permissions, make it public, save it, go back to the EMI, refresh.

145
00:14:18,890 --> 00:14:22,410
Most likely it might be done by now, maybe not, we'll see.

146
00:14:22,590 --> 00:14:27,530
So it was made public, now it has been converted to private.

147
00:14:27,690 --> 00:14:32,890
It's almost real time, like probably took like three, four seconds, I think.

148
00:14:32,930 --> 00:14:35,110
And go back to snapshots.

149
00:14:35,110 --> 00:14:38,490
If we look in the permissions, it's private.

150
00:14:39,110 --> 00:14:42,890
So we can do the same exercise for S3 also.

151
00:14:42,890 --> 00:14:48,810
Let's go to S3 and just choose any of the buckets.

152
00:14:49,330 --> 00:14:57,230
Let's slap public access, list object, and I can see because something is blocking my view.

153
00:14:57,230 --> 00:15:05,010
All right, so ACL public, and we can do similar stuff for lambdas, wait for IAM role also.

154
00:15:05,490 --> 00:15:10,930
IAM, let's go to role, I might have a test role here.

155
00:15:10,990 --> 00:15:18,430
What we'll do is edit trust relationship and just allow everyone with the principal star to assume this role.

156
00:15:18,910 --> 00:15:20,690
Update the trust relationship.

157
00:15:20,830 --> 00:15:21,990
Yeah, sure.

158
00:15:22,630 --> 00:15:24,630
It says overly permissive policies.

159
00:15:24,630 --> 00:15:27,250
Let's go back and we'll see if S3 has been fixed.

160
00:15:29,070 --> 00:15:33,650
Okay, so the ACL was removed and it's back again private.

161
00:15:34,050 --> 00:15:36,690
Let's see IAM, what has happened to this thing.

162
00:15:39,360 --> 00:15:44,540
Okay, so it basically changed the trust relationship and added deny on principal star.

163
00:15:44,540 --> 00:15:46,360
So basically, no one can assume this role.

164
00:15:48,060 --> 00:15:56,300
So yeah, so basically, what you can do is that you can have like multiple services and it basically will respond to it almost in real time.

165
00:15:56,300 --> 00:16:01,780
Let's say you're making RDS or modifying an RDS instance and making it publicly exposed.

166
00:16:01,780 --> 00:16:07,540
It will capture the event and try to see if the star configuration right now is bad.

167
00:16:07,540 --> 00:16:09,000
As in, is it publicly exposed?

168
00:16:09,000 --> 00:16:10,180
Is it not encrypted?

169
00:16:10,180 --> 00:16:11,100
And make the decision.

170
00:16:11,100 --> 00:16:20,300
Some things which are challenging is RDS and Redshift because when you spin up an RDS, what happens is it takes a shitload of time, like 20 minutes, 30 minutes.

171
00:16:20,300 --> 00:16:30,100
So when you're using an event-driven approach, the lambdas will trigger immediately, but when they try to enumerate the RDS instance, it might not be in the available state.

172
00:16:30,560 --> 00:16:36,020
And that what happens is like your lambdas will see and that it's not available and they'll skip it.

173
00:16:36,020 --> 00:16:43,960
But once your RDS is available and it is public facing and you don't and then you can't catch it unless your polar runs like by a cron.

174
00:16:43,960 --> 00:16:49,920
So what we have done is like we have added some like repeat invocation, like let's check for the RDS.

175
00:16:49,920 --> 00:16:57,360
If it is not available, maybe repeat the same message to SQS, send it again like after like five minutes so it will keep on checking every five minutes.

176
00:16:57,360 --> 00:16:58,140
Is it available now?

177
00:16:58,140 --> 00:16:59,080
Is it available now?

178
00:16:59,080 --> 00:17:03,140
If it is available, check it and then make the decision like if it is public, make it private.

179
00:17:03,140 --> 00:17:05,680
If it is not encrypted, stop the instance.

180
00:17:10,720 --> 00:17:15,880
All right, so let's jump on to the Q&A.

181
00:17:16,080 --> 00:17:17,780
Do we have any questions?

182
00:17:19,320 --> 00:17:22,600
Oh yeah, a couple of questions for me.

183
00:17:23,160 --> 00:17:31,480
So in this case, those little modules that you showed is when it comes to detecting something as well as responding to it.

184
00:17:31,480 --> 00:17:35,920
The response part, is that also built into each of the modules?

185
00:17:35,920 --> 00:17:36,880
Is that how?

186
00:17:36,880 --> 00:17:41,380
So yeah, so basically what happens I'll show you.

187
00:17:41,380 --> 00:17:52,220
If I go to the code, so if I go to resources, event translator, main.py.

188
00:17:52,220 --> 00:17:58,520
So what happens like for each of these API calls, which are saying like create queue, set SQS, queue attributes.

189
00:17:58,520 --> 00:18:04,240
So you have like for DB, we have like some delays like 120 seconds.

190
00:18:04,420 --> 00:18:06,520
Similarly for these, like they are almost real-time, right?

191
00:18:06,520 --> 00:18:15,760
Whenever this event is seen, forward the event almost real-time to the SQS queue, and then it will trigger the remediator function.

192
00:18:15,860 --> 00:18:19,020
But some of the other calls we have, like we have added some delays.

193
00:18:21,860 --> 00:18:23,880
Yeah, yes, yes.

194
00:18:23,880 --> 00:18:40,380
And then kind of the beauty of some of this is that because it's all relatively short Python code, if you wanted to integrate this with like, I don't know, if you have like some kind of compliance tool that has detection or remediation built in and has like an API,

195
00:18:40,380 --> 00:18:47,660
you can pretty easily leverage those like REST APIs or whatever to do the detection and the remediation for you.

196
00:18:47,660 --> 00:18:53,840
And this just kind of ties all the automation together because most of those tools that we've experimented with don't really have that.

197
00:18:53,880 --> 00:19:04,300
So that's something we're kind of working on building out as well as leveraging some of the other like detection and response tools so that this kind of just glues it all together and makes it event-driven.

198
00:19:04,600 --> 00:19:07,240
So you have lots of flexibility to do stuff like that.

199
00:19:07,240 --> 00:19:10,760
And the remediator modules are relatively short.

200
00:19:10,980 --> 00:19:15,940
It's basically just examining the event and then doing, you know, making API calls to do what it is you want to do.

201
00:19:15,940 --> 00:19:18,980
So yeah, the remediations are built in and very easily customizable.

202
00:19:19,640 --> 00:19:41,280
Yeah, that basically almost answered my second question, which was, you know, the polar function, which detects something, that the CloudWatch events doesn't cover, that could easily be substituted or, you know, something could be added on to it, which is an open source tool that does those checks or any additional stuff that you want to do.

203
00:19:41,280 --> 00:19:41,880
Yeah, definitely.

204
00:19:41,880 --> 00:19:56,700
I mean, if you look at the PolarCore right now, it's doing similar stuff, but doing like a sanity check every night, let's say, it's doing the same RDS check, describe RDS, describe Snapshot, describe AMI, and also including the other checks, like IAM user,

205
00:19:56,700 --> 00:19:58,880
if they are not using the access keys and stuff.

206
00:19:58,880 --> 00:20:04,680
So we have clubbed everything, but let's say if you don't want to use something, some of the stuff, you can basically remove it.

207
00:20:04,820 --> 00:20:09,140
So it will, like, do sanity check, like, once a night, but you can modify the cron schedule.

208
00:20:09,140 --> 00:20:14,660
Let's say you want to do your, like, once an hour or once every four hours, you can do that.

209
00:20:14,660 --> 00:20:15,700
It's very customizable.

210
00:20:16,140 --> 00:20:20,380
And also the whole deployment is all, like, data form and matchstrips.

211
00:20:20,380 --> 00:20:22,680
So it's very easy to deploy and easy to destroy.

212
00:20:23,920 --> 00:20:25,300
Got it.

213
00:20:25,520 --> 00:20:30,740
We have a couple of questions here from the chat, from John.

214
00:20:30,860 --> 00:20:40,720
In the what are we watching slide under EC2, why are you watching IMDS v2 to ensure v1 is not used?

215
00:20:41,220 --> 00:20:43,140
Yeah, that's exactly it.

216
00:20:43,140 --> 00:20:53,380
So, you know, we have a large environment with lots of apps, and because it's medical apps, there's lots of, like, legacy components, and there's lots of, you know, complexity.

217
00:20:53,380 --> 00:21:01,420
So, yeah, the IMDS v2 stuff is mostly monitoring what is still using IMDS v1.

218
00:21:01,420 --> 00:21:10,780
And then so here, correct me if I'm wrong, it also flags when things are using IMDS v2 to, like, say it's compliant, but it doesn't, like, it's not automatically killing things.

219
00:21:10,780 --> 00:21:12,820
I don't believe it's written.

220
00:21:12,820 --> 00:21:28,560
If you're in a state where you can, like, you are, like, okay, I'm in that state where my EC2 are supposed to be on IMDS v2, you can toggle the plug and stop and add the check for stop instances if they are not, like, on v1 or on v2.

221
00:21:31,360 --> 00:21:34,020
And then second question from Scott.

222
00:21:34,360 --> 00:21:42,220
Is the AWS remediation framework intended to complement or supplement the use of AWS config with its own remediation processes?

223
00:21:42,780 --> 00:21:49,380
So what we have, like, thought of, like, was to add config also as one of the events.

224
00:21:49,380 --> 00:21:52,720
Also, like, right now we have polling and we have event-driven.

225
00:21:52,820 --> 00:22:02,960
So we thought of, like, moving to config and, like, enumerate it, but we haven't, like, done the full enumeration part, like how config can, like, help us in achieving the similar state which we are achieving by event-driven plus polling.

226
00:22:03,140 --> 00:22:09,260
I'm sure, like, config has the custom rules and whatnot, which you can, like, add and do the remediation based on the custom lambdas.

227
00:22:09,260 --> 00:22:17,200
But, yeah, something which for to-do we want to, like, do, maybe add, like, a config integration, maybe if you want to have three options of, like, leveraging the same framework.

228
00:22:17,200 --> 00:22:22,240
Because even with custom model tool, you'll have to have, like, your lambdas to make the remediation.

229
00:22:24,680 --> 00:22:25,400
And so...

230
00:22:25,400 --> 00:22:26,260
Oh, go ahead.

231
00:22:26,600 --> 00:22:27,540
Oh, no, go ahead.

232
00:22:27,620 --> 00:22:35,400
I just wanted to ask, kind of going off of that, any other sneak peek or future stuff you have in the plan for this tool?

233
00:22:35,400 --> 00:22:36,220
Yeah.

234
00:22:36,220 --> 00:22:39,540
So, I mean, Sahir and I have both kind of mentioned it.

235
00:22:39,540 --> 00:22:48,420
One of the things that we're wanting to look at is more integration into other tools, as well as things like AWS config, to kind of leverage this stuff.

236
00:22:48,540 --> 00:22:50,760
We are a relatively small team.

237
00:22:50,760 --> 00:22:58,020
I think we have, well, we have three people, basically, kind of managing our pretty large cloud environment.

238
00:22:58,140 --> 00:23:11,860
So, you know, when it comes to leveraging existing tooling, our general philosophy is kind of use it all and automate the triage to clean stuff up.

239
00:23:11,860 --> 00:23:27,300
So, really, what we're trying to get into is, can we leverage existing checks and remediations from things like config, security monkey is dead, but, like, tools that are similar to security monkey and things like that, and leverage those checks and remediations instead of writing our own,

240
00:23:27,300 --> 00:23:32,800
using kind of like the rest APIs and all the fun, nice features that people are putting out there.

241
00:23:32,980 --> 00:23:50,740
So, you know, if we see a tool that makes this kind of stuff easy, the future state that we see for the remediation framework is kind of gobbling it up and consuming it to make this stuff more effective, giving us greater confidence that the detective capabilities are actually correct to,

242
00:23:50,740 --> 00:23:55,120
like, lower false positive rate, or just to make development quicker.

243
00:23:58,000 --> 00:23:59,360
Got it.

244
00:24:00,240 --> 00:24:06,660
And another question for me, you mentioned that your team over there is quite small.

245
00:24:06,660 --> 00:24:24,800
I know that in order to get to a place where you guys are at with this tool to remediate right away, or even with a certain delay, get those remediations going in the cloud environment, you need to have sort of a in-level kind of in-depth knowledge of your cloud environment,

246
00:24:24,800 --> 00:24:27,220
as well as buy-in from other teams.

247
00:24:27,240 --> 00:24:33,520
Any kind of insights as to how you got there, fun stories, or anything like that?

248
00:24:33,540 --> 00:24:37,360
I think me and Sahir can both give some insight into this.

249
00:24:37,360 --> 00:24:50,780
So Sahir has been at Flatiron much longer than me, and he is a very beloved resource among the entire company, where I am kind of known more as the jackass who comes in and tells everybody no all the time.

250
00:24:50,780 --> 00:24:52,440
But I like doing that.

251
00:24:52,580 --> 00:25:02,300
So one of the things that we found that really made these things very successful was communicating to our engineers, because, you know, they're engineers, they like to see cool shit.

252
00:25:02,300 --> 00:25:10,220
And like showing them that this is cool, it means that they can experiment more as they're developing and building without worry.

253
00:25:16,160 --> 00:25:18,600
Oh, we lost Justin.

254
00:25:21,300 --> 00:25:27,380
Justin, if you could try again for the past five seconds.

255
00:25:27,440 --> 00:25:34,320
Yeah, so that developers could experiment a little more without fear of exposing, you know, all of our patients' data.

256
00:25:34,360 --> 00:25:36,620
They actually really enjoyed that idea.

257
00:25:36,720 --> 00:25:48,720
And then we also found that when we showed them that we had the capabilities to do these things, they suddenly had all kinds of ideas as to how they could make use of it as part of their deployment, development, and testing processes.

258
00:25:49,020 --> 00:25:56,180
So being able to tear down test instances after, you know, 15 minutes after their tests completed, and things like that.

259
00:25:56,180 --> 00:26:10,080
So a lot of the way that we got buy-in from this stuff was showing them how not only did it make the company and our patients safer, it made their day easier and gave them the ability with, you know, a lot without having to build a whole new solution,

260
00:26:10,080 --> 00:26:16,160
they could like automate their testing processes and have it all be event driven, which was stuff that they wanted.

261
00:26:16,180 --> 00:26:32,440
So, you know, just being willing to be transparent and communicate very closely with them and be helpful and show them how this could be like a boon to their processes was really, really integral in getting people excited and on board with this, where we could just start flipping switches and turning remediations on,

262
00:26:32,440 --> 00:26:39,960
because suddenly, you know, they didn't have to worry about being the person who exposed an S3 bucket full of, you know, 20 million patients or whatever.

263
00:26:39,980 --> 00:26:44,100
So here probably has another view on this.

264
00:26:44,560 --> 00:26:46,940
No, I think this new cover pretty much everything.

265
00:26:47,140 --> 00:26:51,480
Yeah, so it's more on, like, how are we going to give more flexibility to developers?

266
00:26:51,860 --> 00:27:05,280
Like, are we confident, like, making a segregated account where we can, like, let them do whatever they want and still have some sort of guardrails to watch for, some sort of guardrails to, like, detect things, along with SCPs and other preventative controls?

267
00:27:05,280 --> 00:27:07,340
Like, what if still things happen?

268
00:27:07,340 --> 00:27:13,440
Like, can we have, like, a framework which watches for, let's say, S3 bucket or AMIs and whatnot and fixes it automatically?

269
00:27:13,720 --> 00:27:16,740
So yeah, so ensuring that level of maturity is what we're aiming for.

270
00:27:16,740 --> 00:27:20,780
And I'm pretty sure, like, software engineering kind of is on board on that.

271
00:27:21,500 --> 00:27:30,080
And then any kind of quick insight into what the cost savings were like from v1 to v2?

272
00:27:30,320 --> 00:27:35,500
Oh, so we haven't, like, compared, like, specifically the cost saving.

273
00:27:35,680 --> 00:27:39,200
We did, like, actually save a lot from the API calls perspective.

274
00:27:39,200 --> 00:27:51,260
Like, if I, if you look back at the slides which we had from the v1, we were making tons of API call, like, for the most API calls spent was on snapshots, describing snapshots.

275
00:27:51,260 --> 00:27:54,940
We got to describe a snapshot and then you're described for each snapshot attribute.

276
00:27:54,940 --> 00:27:57,260
So it was, like, a very, very iterative.

277
00:27:57,380 --> 00:28:10,020
So we had, like, so many API calls which were impacting other stuff with, like, the whole situation where Ansible and stuff, because we were almost, like, getting pushed back on the EC2 resources, always hitting the limits.

278
00:28:10,300 --> 00:28:16,940
So v2 basically gave us some room, like, that we don't have to, like, describe a poll all the time.

279
00:28:16,940 --> 00:28:20,360
We can do it when certain events which we care about happens.

280
00:28:20,420 --> 00:28:28,840
So that was some of the more, like, API saving, the number of API call savings, but not specifically cost.

281
00:28:28,980 --> 00:28:30,820
That we haven't, like, compared yet.

282
00:28:31,240 --> 00:28:41,800
Yeah, the cost savings for me was our SREs quit calling, saying that API, you know, we were making too many API calls and people were getting rate limited, which is worth, like, a zillion dollars.

283
00:28:41,800 --> 00:28:54,680
Because, like, if there's... I haven't made this clear, the one thing that I don't want to be doing is being on call and getting calls at night pisses me off to no end, especially when it's an angry SRE, because the SREs do not give a shit what I have to say.

284
00:28:54,680 --> 00:28:56,260
They are mad that things aren't working.

285
00:28:56,260 --> 00:29:00,760
So I... that stopped and that is worth just... it's invaluable.

286
00:29:00,780 --> 00:29:03,060
That's really all I care about is a quick calling.

287
00:29:04,180 --> 00:29:06,180
I think we can all agree there.

288
00:29:06,180 --> 00:29:08,140
A couple more questions from the chat.

289
00:29:08,140 --> 00:29:11,740
Have you tested or compared this with AWS GuardDuty?

290
00:29:12,160 --> 00:29:19,400
So we do have... we do use, like, GuardDuty, but then GuardDuty, like, it doesn't capture everything what we want.

291
00:29:19,400 --> 00:29:22,680
Let's say, sure, it might have, like, for S3 checks and everything.

292
00:29:22,740 --> 00:29:32,680
But for, like, specific stuff, like, which we're looking for, like, IAM roles and whatnot, I'm not sure if it is being covered in GuardDuty, but we do complement that with, like, using GuardDuty, like, standalone.

293
00:29:34,660 --> 00:29:41,840
Yeah, to kind of... to add to that, you know, like we said before, I don't necessarily believe in replacing free things.

294
00:29:41,840 --> 00:29:48,980
If there's things that do things well, we will use them both as best we can and then try and, you know, write some code to glue it together.

295
00:29:48,980 --> 00:29:56,600
So, you know, we have a lot of... when you ask these questions about these things, very often the answer is we're using both of them as well as this.

296
00:29:58,880 --> 00:29:59,720
Got it.

297
00:29:59,720 --> 00:30:05,740
And another one, for some cases, why not create a preventative SCP?

298
00:30:06,100 --> 00:30:08,760
Sure, that's... yeah, this one, go ahead.

299
00:30:08,980 --> 00:30:10,360
Yeah, absolutely do that.

300
00:30:10,360 --> 00:30:11,540
You should do that.

301
00:30:11,800 --> 00:30:22,340
This is another layer helping us, you know, validate that the SCP hasn't changed or that I didn't fat finger the SCP or, honestly, I'm not that smart.

302
00:30:22,340 --> 00:30:39,140
I might have written a bad SCP and I kind of... I like having the many eyes approach and, like, the many, you know, the defense in depth approach of, you know, maybe I wrote a bad SCP and this is something we can use to detect... we can use to detect,

303
00:30:39,140 --> 00:30:40,680
you know, something going public again.

304
00:30:40,680 --> 00:30:42,080
So we have both.

305
00:30:42,080 --> 00:30:43,340
We're using SCPs.

306
00:30:43,340 --> 00:30:44,900
You should be using SCPs.

307
00:30:44,940 --> 00:30:46,880
They're a great preventative control.

308
00:30:47,340 --> 00:30:52,960
I also would say that having preventative controls does not mean you should avoid having detective and responsive controls as well.

309
00:30:53,140 --> 00:30:54,700
You know, you need multiple layers.