1 00:00:00,240 --> 00:00:03,359 so next up we've got Marco manini hello 2 00:00:03,359 --> 00:00:06,680 everyone Marco is the tech lead for the 3 00:00:06,680 --> 00:00:08,320 Marco is a tech lead for detection 4 00:00:08,320 --> 00:00:10,240 engineering and at the at the company 5 00:00:10,240 --> 00:00:11,960 Thought Machine which is a cloud native 6 00:00:11,960 --> 00:00:14,320 core Banking Company where he focuses on 7 00:00:14,320 --> 00:00:16,960 instant response security monitoring and 8 00:00:16,960 --> 00:00:19,279 datadriven solutions to several security 9 00:00:19,279 --> 00:00:21,680 problems today Marco is going to explain 10 00:00:21,680 --> 00:00:23,080 different logging patterns that you can 11 00:00:23,080 --> 00:00:26,880 find in public clouds like AWS gcp Azure 12 00:00:26,880 --> 00:00:28,320 and Lessons Learned From building and 13 00:00:28,320 --> 00:00:30,000 rebuilding security logging into 14 00:00:30,000 --> 00:00:32,000 different scales the title of his talk 15 00:00:32,000 --> 00:00:33,559 is security logging in the cloud 16 00:00:33,559 --> 00:00:35,559 trade-offs to consider and patterns to 17 00:00:35,559 --> 00:00:37,239 maximize the effectiveness of security 18 00:00:37,239 --> 00:00:41,199 data pipelines everyone please welcome 19 00:00:44,960 --> 00:00:47,800 Marco I'm so grateful you said the title 20 00:00:47,800 --> 00:00:49,360 because I think I W I was going to 21 00:00:49,360 --> 00:00:52,480 repeat it a lot of times so security l 22 00:00:52,480 --> 00:00:54,160 in the cloud for 23 00:00:54,160 --> 00:00:56,440 short this is a big introduction don't 24 00:00:56,440 --> 00:00:58,359 worry I'm not going to read everything 25 00:00:58,359 --> 00:01:01,039 but let's start with why this stock I 26 00:01:01,039 --> 00:01:03,120 think will be interesting for people in 27 00:01:03,120 --> 00:01:05,199 this room so a lot of the times and 28 00:01:05,199 --> 00:01:06,840 especially after you come from Defcon so 29 00:01:06,840 --> 00:01:09,880 you will go back to your uh offices your 30 00:01:09,880 --> 00:01:11,840 businesses and you will be okay guys we 31 00:01:11,840 --> 00:01:14,360 need to enable all the logs there's this 32 00:01:14,360 --> 00:01:17,320 attacks there is uh hackers that can do 33 00:01:17,320 --> 00:01:19,520 things that if they have that we have no 34 00:01:19,520 --> 00:01:22,119 logs we will not even be able to know if 35 00:01:22,119 --> 00:01:25,799 we have been hacked and especially after 36 00:01:25,799 --> 00:01:27,640 you start doing it you realize that the 37 00:01:27,640 --> 00:01:30,400 cloud contains a lot of locks a lot of 38 00:01:30,400 --> 00:01:32,000 different sources and a lot of different 39 00:01:32,000 --> 00:01:34,759 things you can uh introspect and the 40 00:01:34,759 --> 00:01:37,000 cloud really if you go to the 41 00:01:37,000 --> 00:01:39,280 documentation for all the cloud 42 00:01:39,280 --> 00:01:42,119 providers they will even have warnings 43 00:01:42,119 --> 00:01:43,600 not the warning of are you sure you want 44 00:01:43,600 --> 00:01:46,079 to do this it's you should do it it will 45 00:01:46,079 --> 00:01:48,040 cost you money we don't know how much 46 00:01:48,040 --> 00:01:50,079 but it will cost you in big letters so 47 00:01:50,079 --> 00:01:53,200 be aware there will be cost that's in 48 00:01:53,200 --> 00:01:55,840 their documentation and also in general 49 00:01:55,840 --> 00:01:57,799 doing data loging pipelines it's not an 50 00:01:57,799 --> 00:02:00,799 easy problem like when we we were doing 51 00:02:00,799 --> 00:02:03,360 things on Prem a lot of companies would 52 00:02:03,360 --> 00:02:05,680 just leave the logs for their Linux 53 00:02:05,680 --> 00:02:06,920 machines for example the windows 54 00:02:06,920 --> 00:02:08,758 machines in the machine so when there 55 00:02:08,758 --> 00:02:10,000 was an incident the forensic 56 00:02:10,000 --> 00:02:12,239 investigator will still grab the 57 00:02:12,239 --> 00:02:15,200 physical machine and investigate it in 58 00:02:15,200 --> 00:02:17,319 the cloud that's a bit more difficult 59 00:02:17,319 --> 00:02:19,040 but on the other hand the cloud 60 00:02:19,040 --> 00:02:21,120 providers have given us the tools to 61 00:02:21,120 --> 00:02:24,959 actually move to a 25 Century style data 62 00:02:24,959 --> 00:02:26,800 loging and 63 00:02:26,800 --> 00:02:30,680 analytics uh platforms so 64 00:02:30,680 --> 00:02:34,239 let's go for f who am I I was introduced 65 00:02:34,239 --> 00:02:37,280 very well but I am 66 00:02:37,280 --> 00:02:40,800 like if I didn't say I'm the kind of 67 00:02:40,800 --> 00:02:42,319 person that have told every company I 68 00:02:42,319 --> 00:02:44,120 have work in we should lock everything 69 00:02:44,120 --> 00:02:45,800 you may not get that idea from this 70 00:02:45,800 --> 00:02:49,000 stock because I say some blasphemy 71 00:02:49,000 --> 00:02:51,440 between like data people which is maybe 72 00:02:51,440 --> 00:02:53,440 we don't need this data but I'm the kind 73 00:02:53,440 --> 00:02:55,319 of person that will tell we need to Lo 74 00:02:55,319 --> 00:02:57,000 everything because otherwise we won't be 75 00:02:57,000 --> 00:02:59,560 able to find the threats and especially 76 00:02:59,560 --> 00:03:01,120 we will need it if we get hacked to 77 00:03:01,120 --> 00:03:04,560 foran investigations in the club so some 78 00:03:04,560 --> 00:03:06,560 DET of th machine apart from doing core 79 00:03:06,560 --> 00:03:08,720 Banking and payments I know exciting 80 00:03:08,720 --> 00:03:12,120 stuff uh we are Cloud native environment 81 00:03:12,120 --> 00:03:15,080 multic cloud and kubernetes Native which 82 00:03:15,080 --> 00:03:19,360 means that we cannot only rely on one 83 00:03:19,360 --> 00:03:21,560 cloud and their way of doing things and 84 00:03:21,560 --> 00:03:23,159 between implementing logging on every 85 00:03:23,159 --> 00:03:25,920 single Cloud especially L us then gcp 86 00:03:25,920 --> 00:03:28,519 and now I'm starting with a z I'm 87 00:03:28,519 --> 00:03:30,720 realizing that the tiny between the 88 00:03:30,720 --> 00:03:32,879 clouds make a huge difference in terms 89 00:03:32,879 --> 00:03:34,319 of cost and 90 00:03:34,319 --> 00:03:37,159 performance and then kubernetes 91 00:03:37,159 --> 00:03:41,319 kubernetes very fun and the logging it's 92 00:03:41,319 --> 00:03:43,200 even more 93 00:03:43,200 --> 00:03:45,560 complicated and the problem is funny 94 00:03:45,560 --> 00:03:47,760 enough kubernetes it's like a cloud 95 00:03:47,760 --> 00:03:50,280 inside the cloud like you will often 96 00:03:50,280 --> 00:03:52,400 have like the cloud at the end of day 97 00:03:52,400 --> 00:03:55,040 it's just an API for people to the stuff 98 00:03:55,040 --> 00:03:56,799 for people to manage the machines so 99 00:03:56,799 --> 00:03:59,200 people to manage everything kuet is the 100 00:03:59,200 --> 00:04:00,640 same thing 101 00:04:00,640 --> 00:04:03,599 uh so you have an API and they and then 102 00:04:03,599 --> 00:04:05,599 the kubernetes will manage the compute 103 00:04:05,599 --> 00:04:08,519 will manage the pods and everything so 104 00:04:08,519 --> 00:04:11,000 coet needs to be introspected as much as 105 00:04:11,000 --> 00:04:13,760 a club and then you go every single 106 00:04:13,760 --> 00:04:15,640 layer of abstraction and every single 107 00:04:15,640 --> 00:04:18,238 layer has their own W of logs and you 108 00:04:18,238 --> 00:04:19,798 pay for all of 109 00:04:19,798 --> 00:04:24,280 them so here I want to start with what 110 00:04:24,280 --> 00:04:26,199 the problem that we have which is 111 00:04:26,199 --> 00:04:27,759 engineering and SEC engineering is the 112 00:04:27,759 --> 00:04:30,520 art of constraints like if we had 113 00:04:30,520 --> 00:04:32,280 infinite time and infinite resources 114 00:04:32,280 --> 00:04:33,960 security wouldn't be a problem we'll 115 00:04:33,960 --> 00:04:35,600 patch everything we'll have all the logs 116 00:04:35,600 --> 00:04:38,280 and we will be able to have perfect 117 00:04:38,280 --> 00:04:40,600 security but in the real world that may 118 00:04:40,600 --> 00:04:42,919 not be as easy 119 00:04:42,919 --> 00:04:45,320 so uh because we are not protected we 120 00:04:45,320 --> 00:04:46,880 cannot protect everything we will need 121 00:04:46,880 --> 00:04:49,759 locks to understand what's happening and 122 00:04:49,759 --> 00:04:52,160 to investigate if something goes wrong 123 00:04:52,160 --> 00:04:54,080 it's a tiny loog like the presentation 124 00:04:54,080 --> 00:04:56,320 has a lot of graphs and diagrams so at 125 00:04:56,320 --> 00:04:58,240 least you get a nice image before we go 126 00:04:58,240 --> 00:05:01,080 into the boring part so I want also to 127 00:05:01,080 --> 00:05:03,120 preface that a loog can be thought as a 128 00:05:03,120 --> 00:05:04,840 representation of an event or an action 129 00:05:04,840 --> 00:05:06,840 performed in a system this seems very 130 00:05:06,840 --> 00:05:09,320 broad because I think of logs on not 131 00:05:09,320 --> 00:05:12,080 only like for example a easy to instance 132 00:05:12,080 --> 00:05:15,840 being spun but also things like uh an 133 00:05:15,840 --> 00:05:17,880 EMP a new employee has been added to 134 00:05:17,880 --> 00:05:20,240 payroll all of these data you need to 135 00:05:20,240 --> 00:05:21,840 correlate it you need to understand 136 00:05:21,840 --> 00:05:23,680 everything as a as a whole and you needs 137 00:05:23,680 --> 00:05:27,600 to be able to be match with each other 138 00:05:27,600 --> 00:05:30,000 like if an employee has not being able 139 00:05:30,000 --> 00:05:33,800 to payroll but this employee it's adding 140 00:05:33,800 --> 00:05:36,720 compute that's a strange and may the 141 00:05:36,720 --> 00:05:39,400 compute be a crypt Miner and that 142 00:05:39,400 --> 00:05:41,520 employee may be an account created by an 143 00:05:41,520 --> 00:05:44,479 admin like that's the kind of thing that 144 00:05:44,479 --> 00:05:49,280 we need to worry about so data that Dev 145 00:05:49,280 --> 00:05:50,639 SE 146 00:05:50,639 --> 00:05:53,160 Ops when you are building a a data 147 00:05:53,160 --> 00:05:55,680 application you need to think about the 148 00:05:55,680 --> 00:05:57,800 users in this case the users will be 149 00:05:57,800 --> 00:05:59,960 analysts and people do for 150 00:05:59,960 --> 00:06:02,080 investigations the problem is the 151 00:06:02,080 --> 00:06:04,000 constraints for security loging are very 152 00:06:04,000 --> 00:06:06,960 stringent like very few systems you have 153 00:06:06,960 --> 00:06:09,240 you need real time immediate search very 154 00:06:09,240 --> 00:06:11,520 long retention like years of retention 155 00:06:11,520 --> 00:06:13,440 highly contextualized data precise data 156 00:06:13,440 --> 00:06:15,400 so you cannot do sampling and it needs 157 00:06:15,400 --> 00:06:17,120 to be extremely reliable and tamper 158 00:06:17,120 --> 00:06:19,160 proof which means that you cannot take 159 00:06:19,160 --> 00:06:21,680 out of the shortcuts that other systems 160 00:06:21,680 --> 00:06:24,280 can and also it needs to be easy to use 161 00:06:24,280 --> 00:06:25,960 and visualize and aggregate things which 162 00:06:25,960 --> 00:06:28,880 is sometimes data people will forget 163 00:06:28,880 --> 00:06:30,360 that the dat and the department that we 164 00:06:30,360 --> 00:06:32,599 build hard for humans to 165 00:06:32,599 --> 00:06:36,120 use so all of this actually the cloud 166 00:06:36,120 --> 00:06:39,599 providers have build things to help you 167 00:06:39,599 --> 00:06:42,400 achieve these constraints the only thing 168 00:06:42,400 --> 00:06:45,360 it's price if you put the slider on 169 00:06:45,360 --> 00:06:47,800 price to the maximum don't worry you 170 00:06:47,800 --> 00:06:50,800 will get that but the bill may be like 171 00:06:50,800 --> 00:06:53,720 your finan te may not be happy with this 172 00:06:53,720 --> 00:06:56,879 and then on the other hand hacking like 173 00:06:56,879 --> 00:06:59,759 where the hacking lives where why why 174 00:06:59,759 --> 00:07:02,599 hacking is not going to go away in more 175 00:07:02,599 --> 00:07:05,400 Society it's because the constraints 176 00:07:05,400 --> 00:07:07,199 that we have 177 00:07:07,199 --> 00:07:11,280 build H gaps in these gaps hacking and 178 00:07:11,280 --> 00:07:14,400 red teaming and exploit 179 00:07:14,400 --> 00:07:17,560 research lives like if you cannot have 180 00:07:17,560 --> 00:07:19,199 if you cannot solve every problem 181 00:07:19,199 --> 00:07:21,560 because of the price of Hosting theing 182 00:07:21,560 --> 00:07:24,120 difficulty or even the price of licenses 183 00:07:24,120 --> 00:07:26,759 if there may be a vendor that solve or 184 00:07:26,759 --> 00:07:28,319 security problems but probably their 185 00:07:28,319 --> 00:07:30,440 license is going to be 186 00:07:30,440 --> 00:07:34,840 outrageous so logs are expensive and 187 00:07:34,840 --> 00:07:36,639 they can depending on the log it's more 188 00:07:36,639 --> 00:07:38,080 or less expensive but in the cloud are 189 00:07:38,080 --> 00:07:42,000 vital so this talk is how we can get to 190 00:07:42,000 --> 00:07:43,560 as much security as we can in terms of 191 00:07:43,560 --> 00:07:47,560 logging without price being a constraint 192 00:07:47,560 --> 00:07:49,639 or it will always a constraint but it 193 00:07:49,639 --> 00:07:52,199 shouldn't be something that I have seen 194 00:07:52,199 --> 00:07:54,639 some places and I have seen people tell 195 00:07:54,639 --> 00:07:57,520 me uh security people will go to a 196 00:07:57,520 --> 00:07:59,680 company and we will say we need to lock 197 00:07:59,680 --> 00:08:02,199 this thing everyone I think well not 198 00:08:02,199 --> 00:08:03,400 everyone but a lot of people can be 199 00:08:03,400 --> 00:08:06,000 traumatized by enable bpc flow logs for 200 00:08:06,000 --> 00:08:10,560 the first time like yeah exactly uh 201 00:08:10,560 --> 00:08:11,879 security people will be like we need 202 00:08:11,879 --> 00:08:15,759 this definitely enable it go get a nice 203 00:08:15,759 --> 00:08:17,680 weekend come back and then the finance 204 00:08:17,680 --> 00:08:21,759 team will be at their desk going um do 205 00:08:21,759 --> 00:08:24,680 we what do we spend 50k 206 00:08:24,680 --> 00:08:26,919 more and you were like this is for 207 00:08:26,919 --> 00:08:29,800 security so we need to find a way 208 00:08:29,800 --> 00:08:31,599 they will probably traumatize and try to 209 00:08:31,599 --> 00:08:33,000 stop you every time you do something 210 00:08:33,000 --> 00:08:36,080 again so let's talk about some case 211 00:08:36,080 --> 00:08:39,440 studies on like particular examples of 212 00:08:39,440 --> 00:08:41,200 how this will 213 00:08:41,200 --> 00:08:43,080 work 214 00:08:43,080 --> 00:08:46,560 so logs should scale with the amount of 215 00:08:46,560 --> 00:08:48,959 instances I am and then General 216 00:08:48,959 --> 00:08:52,560 resources that you have Okay so this 217 00:08:52,560 --> 00:08:54,519 should be like a nice 218 00:08:54,519 --> 00:08:58,839 linear uh graph but if you see here it 219 00:08:58,839 --> 00:09:00,760 doesn't tend to be like this the more 220 00:09:00,760 --> 00:09:03,920 you grow the more problems that can 221 00:09:03,920 --> 00:09:05,519 appear in your loging this for example 222 00:09:05,519 --> 00:09:08,880 would be a cloud tril loging which is I 223 00:09:08,880 --> 00:09:10,720 think the the one that you should really 224 00:09:10,720 --> 00:09:14,760 really enable like well like the they 225 00:09:14,760 --> 00:09:16,200 call it management 226 00:09:16,200 --> 00:09:18,800 API because otherwise you will not know 227 00:09:18,800 --> 00:09:20,880 what people are doing on the environment 228 00:09:20,880 --> 00:09:23,000 the thing is 229 00:09:23,000 --> 00:09:26,560 what F like this function of scaling 230 00:09:26,560 --> 00:09:30,720 it's managed by different parameters and 231 00:09:30,720 --> 00:09:33,000 in in this case you have human produce 232 00:09:33,000 --> 00:09:35,800 logs which I'm not that worried about 233 00:09:35,800 --> 00:09:38,360 them because humans don't tend to wake 234 00:09:38,360 --> 00:09:40,800 up one day and do 10 million actions in 235 00:09:40,800 --> 00:09:45,200 one second you find some humans like 236 00:09:45,200 --> 00:09:48,640 that that will be a problem and then you 237 00:09:48,640 --> 00:09:51,120 have application interaction this I'm 238 00:09:51,120 --> 00:09:52,959 I'm going to talk later about that but 239 00:09:52,959 --> 00:09:54,680 this can become very problematic very 240 00:09:54,680 --> 00:09:57,600 quickly and then Baseline operation this 241 00:09:57,600 --> 00:09:59,959 is something 242 00:09:59,959 --> 00:10:02,200 that when we think about the cloud we 243 00:10:02,200 --> 00:10:05,000 think about that but you you will see 244 00:10:05,000 --> 00:10:06,680 that can become quite problematic in 245 00:10:06,680 --> 00:10:08,440 certain 246 00:10:08,440 --> 00:10:10,480 circumstances 247 00:10:10,480 --> 00:10:14,360 so I already explained uh a bit of what 248 00:10:14,360 --> 00:10:15,920 they are but the application interaction 249 00:10:15,920 --> 00:10:18,079 it's the fre like you can find friction 250 00:10:18,079 --> 00:10:20,040 between apps like apps that require 251 00:10:20,040 --> 00:10:21,839 authentication from other apps apps that 252 00:10:21,839 --> 00:10:24,480 trigger other apps and in general things 253 00:10:24,480 --> 00:10:26,440 that manage other things and the things 254 00:10:26,440 --> 00:10:27,760 that reveal other 255 00:10:27,760 --> 00:10:30,120 things this 256 00:10:30,120 --> 00:10:32,079 in the errors between these interactions 257 00:10:32,079 --> 00:10:34,040 you can have explosion of logs and 258 00:10:34,040 --> 00:10:38,079 especially if you get apps that don't 259 00:10:38,079 --> 00:10:40,519 work well with each other you may have 260 00:10:40,519 --> 00:10:42,959 like you will see later like explosion 261 00:10:42,959 --> 00:10:44,880 like Lo 262 00:10:44,880 --> 00:10:46,920 explosions uh then you have Bas on 263 00:10:46,920 --> 00:10:48,760 operations these are things that can be 264 00:10:48,760 --> 00:10:50,680 too noisy just by configuration you can 265 00:10:50,680 --> 00:10:52,800 have a misconfiguration in which you get 266 00:10:52,800 --> 00:10:55,000 an application that checks the health 267 00:10:55,000 --> 00:10:56,760 and then someone goes I want the health 268 00:10:56,760 --> 00:10:58,760 to be known at real time so they have a 269 00:10:58,760 --> 00:11:02,360 refresh fresh time of 1 270 00:11:02,360 --> 00:11:05,200 second maybe knowing the CPU usage per 271 00:11:05,200 --> 00:11:08,839 second no not that useful 272 00:11:08,839 --> 00:11:11,959 but this can very easily because of 273 00:11:11,959 --> 00:11:14,720 configuration and assumptions of where 274 00:11:14,720 --> 00:11:18,040 the cost go become kind of 275 00:11:18,040 --> 00:11:19,920 unmanageable and think at the end of day 276 00:11:19,920 --> 00:11:21,880 we want to know everything all the time 277 00:11:21,880 --> 00:11:24,920 so how we can make this happen so this 278 00:11:24,920 --> 00:11:28,680 is also an example it's not like the 279 00:11:28,680 --> 00:11:31,120 problem with the cloud and in in general 280 00:11:31,120 --> 00:11:34,200 when you go to documentation of logging 281 00:11:34,200 --> 00:11:37,120 it's there is a very easy question which 282 00:11:37,120 --> 00:11:40,000 is how much something will cost the 283 00:11:40,000 --> 00:11:41,920 cloud provider doesn't doesn't say and 284 00:11:41,920 --> 00:11:43,440 it doesn't give you any tools to 285 00:11:43,440 --> 00:11:46,360 actually understand where how much would 286 00:11:46,360 --> 00:11:48,440 you pay for these logs I mean you know 287 00:11:48,440 --> 00:11:50,040 how much would you pay for gigabyte but 288 00:11:50,040 --> 00:11:51,240 you don't know how many logs you will 289 00:11:51,240 --> 00:11:53,120 you will have because the problem is 290 00:11:53,120 --> 00:11:55,160 when you start looking into this you may 291 00:11:55,160 --> 00:11:57,399 find that at different levels of scale 292 00:11:57,399 --> 00:11:59,639 your logs start being polluted with 293 00:11:59,639 --> 00:12:01,920 things that you don't want so you always 294 00:12:01,920 --> 00:12:04,240 want to be GRE in the high signal locks 295 00:12:04,240 --> 00:12:06,760 these are for example uh newest stuff 296 00:12:06,760 --> 00:12:10,320 being created containers uh that are 297 00:12:10,320 --> 00:12:13,320 suspicious H new administrator accounts 298 00:12:13,320 --> 00:12:16,560 being created is the kind of logs that 299 00:12:16,560 --> 00:12:18,639 if something bad happens you will look 300 00:12:18,639 --> 00:12:19,399 at 301 00:12:19,399 --> 00:12:24,000 them the blue ones are still useful but 302 00:12:24,000 --> 00:12:27,320 are things like Recon like how to find 303 00:12:27,320 --> 00:12:30,040 reconnaissance like uh that may be like 304 00:12:30,040 --> 00:12:32,079 a million logs may be just related to 305 00:12:32,079 --> 00:12:35,480 one scanner in these cases these logs 306 00:12:35,480 --> 00:12:37,880 don't provide that much value but they 307 00:12:37,880 --> 00:12:40,279 are still relevant and you want to keep 308 00:12:40,279 --> 00:12:41,880 them but you want to find ways of 309 00:12:41,880 --> 00:12:44,519 actually store them giving up one of the 310 00:12:44,519 --> 00:12:46,480 other constraints like for example 311 00:12:46,480 --> 00:12:49,320 immediate search or high retention like 312 00:12:49,320 --> 00:12:51,600 you can like those logs are the only 313 00:12:51,600 --> 00:12:53,920 ones I would say you should be safe 314 00:12:53,920 --> 00:12:55,959 sampling them and cloud trail in this in 315 00:12:55,959 --> 00:12:58,399 this particular case actually does this 316 00:12:58,399 --> 00:12:59,560 distinction 317 00:12:59,560 --> 00:13:01,279 and you can have something they call 318 00:13:01,279 --> 00:13:04,079 event filtering in which in general in 319 00:13:04,079 --> 00:13:06,079 in cloud trail you can set you can 320 00:13:06,079 --> 00:13:07,880 divide between like gr operations and 321 00:13:07,880 --> 00:13:10,560 then read operations and only collect 322 00:13:10,560 --> 00:13:14,040 read operations sorry well only right 323 00:13:14,040 --> 00:13:15,920 operations is will be the one that you 324 00:13:15,920 --> 00:13:19,199 want and in that case they like L us has 325 00:13:19,199 --> 00:13:21,240 identified that it's clearly you need to 326 00:13:21,240 --> 00:13:23,120 have the distinction between High signal 327 00:13:23,120 --> 00:13:24,880 and L signal and also I want to say 328 00:13:24,880 --> 00:13:26,240 something that security teams don't tend 329 00:13:26,240 --> 00:13:29,720 to look at which is errors 330 00:13:29,720 --> 00:13:32,720 like this can become out of hand very 331 00:13:32,720 --> 00:13:34,440 quickly and I have seen applications 332 00:13:34,440 --> 00:13:36,680 that are so efficient that if if they 333 00:13:36,680 --> 00:13:39,199 don't have the proper IM am they will 334 00:13:39,199 --> 00:13:40,680 tell you but they will tell you at a 335 00:13:40,680 --> 00:13:43,480 rate of a million logs per minute like 336 00:13:43,480 --> 00:13:46,639 great go fantastic 337 00:13:46,639 --> 00:13:49,480 but that may be a problem the thing is 338 00:13:49,480 --> 00:13:52,160 these errors often only the security 339 00:13:52,160 --> 00:13:54,360 team are looking at them because this 340 00:13:54,360 --> 00:13:57,120 would be erors of authorization and if 341 00:13:57,120 --> 00:13:58,440 people are not managing their 342 00:13:58,440 --> 00:14:01,440 application ations properly they these 343 00:14:01,440 --> 00:14:03,000 Heros will start polluting your 344 00:14:03,000 --> 00:14:04,480 environment and it only takes like a 345 00:14:04,480 --> 00:14:06,360 couple of applications to do this and 346 00:14:06,360 --> 00:14:09,199 start exploding on your logs and the 347 00:14:09,199 --> 00:14:11,440 thing is I would 348 00:14:11,440 --> 00:14:14,519 say fix the errors like tell the teams 349 00:14:14,519 --> 00:14:17,160 tell the te like security we tend to be 350 00:14:17,160 --> 00:14:19,440 silent thinking that our value is 351 00:14:19,440 --> 00:14:21,839 security but when we have a really good 352 00:14:21,839 --> 00:14:24,480 data stack and observability stack we 353 00:14:24,480 --> 00:14:25,839 are able 354 00:14:25,839 --> 00:14:29,720 to like help other parts of of the of 355 00:14:29,720 --> 00:14:33,519 engineering to be better and to make H 356 00:14:33,519 --> 00:14:36,279 systems more efficient and if they 357 00:14:36,279 --> 00:14:38,320 eliminate the errors we pay less locks 358 00:14:38,320 --> 00:14:41,079 and now when an attacker is trying to 359 00:14:41,079 --> 00:14:43,480 hack into our system and they get they 360 00:14:43,480 --> 00:14:45,639 don't set things properly they will 361 00:14:45,639 --> 00:14:48,120 trigger errors and now that error is a 362 00:14:48,120 --> 00:14:50,320 high signal lock so there is a lot of 363 00:14:50,320 --> 00:14:55,560 value in helping engineering Sol of 364 00:14:55,560 --> 00:14:59,639 Errors like for example uh unauthorized 365 00:14:59,639 --> 00:15:03,399 I I wouldn't say the tool but I'm sure 366 00:15:03,399 --> 00:15:08,000 people will find them so uh here I 367 00:15:08,000 --> 00:15:11,959 already talked about uh this it's 368 00:15:11,959 --> 00:15:14,240 example of errors and misconfigurations 369 00:15:14,240 --> 00:15:15,920 like setting refresh rates that are way 370 00:15:15,920 --> 00:15:19,880 too low services that logs are way way 371 00:15:19,880 --> 00:15:23,279 too verose like there may be and this is 372 00:15:23,279 --> 00:15:25,880 the cloud doesn't do this as badly but 373 00:15:25,880 --> 00:15:27,399 there will be applications that they 374 00:15:27,399 --> 00:15:28,680 will send the lock and they would just 375 00:15:28,680 --> 00:15:31,560 send a stack trace on the lock and if 376 00:15:31,560 --> 00:15:34,319 there is an constant error that gives 377 00:15:34,319 --> 00:15:36,720 you the whole stack Trace that can 378 00:15:36,720 --> 00:15:38,920 become completely unmanageable very 379 00:15:38,920 --> 00:15:41,600 quickly and by 380 00:15:41,600 --> 00:15:43,920 unmanageable the cloud will manage it 381 00:15:43,920 --> 00:15:46,839 you will just get the B 382 00:15:47,600 --> 00:15:50,079 later the good things about modern uh 383 00:15:50,079 --> 00:15:53,880 Cloud operations then this thing it's 384 00:15:53,880 --> 00:15:56,560 about like underlying errors I have 385 00:15:56,560 --> 00:15:59,639 encountered like some loog sources that 386 00:15:59,639 --> 00:16:02,199 90% or more of the logs were just an 387 00:16:02,199 --> 00:16:05,040 error from very efficient 388 00:16:05,040 --> 00:16:08,440 applications this uh makes my life a lot 389 00:16:08,440 --> 00:16:11,800 easier because less errors more signal 390 00:16:11,800 --> 00:16:14,000 great and less 391 00:16:14,000 --> 00:16:17,920 cost so these are also some examples 392 00:16:17,920 --> 00:16:21,480 that I have um how called shoot my foot 393 00:16:21,480 --> 00:16:25,680 off without realizing so Cloud errors in 394 00:16:25,680 --> 00:16:27,480 authentication third party Services 395 00:16:27,480 --> 00:16:29,440 infrastructure scope constant changes 396 00:16:29,440 --> 00:16:32,920 that's you really cannot filter this out 397 00:16:32,920 --> 00:16:35,399 you really need to be looking at them 398 00:16:35,399 --> 00:16:37,920 but if infrastructure has code is the is 399 00:16:37,920 --> 00:16:40,000 misconfigured that may pollute the 400 00:16:40,000 --> 00:16:41,920 system a lot because these are the kind 401 00:16:41,920 --> 00:16:44,160 of logs that you really need to care and 402 00:16:44,160 --> 00:16:47,160 put them in cash or real expensive 403 00:16:47,160 --> 00:16:50,240 storage then cage logs this obility 404 00:16:50,240 --> 00:16:52,759 tooling fantastic but you need to be 405 00:16:52,759 --> 00:16:55,639 careful of how how are those refresher 406 00:16:55,639 --> 00:16:57,480 rates 407 00:16:57,480 --> 00:16:59,000 how 408 00:16:59,000 --> 00:17:02,480 how much ination is and how they attack 409 00:17:02,480 --> 00:17:04,199 the API because a lot of the things 410 00:17:04,199 --> 00:17:06,799 would just go constantly against the API 411 00:17:06,799 --> 00:17:11,039 as fast as they can then on VM ISU and 412 00:17:11,039 --> 00:17:13,799 instance of some kind 413 00:17:13,799 --> 00:17:17,520 CIS this I I think I could write a whole 414 00:17:17,520 --> 00:17:20,000 book like you really need if you really 415 00:17:20,000 --> 00:17:21,480 want to understand what's happening on 416 00:17:21,480 --> 00:17:22,640 your environment 417 00:17:22,640 --> 00:17:24,959 CIS will tell you we'll tell you 418 00:17:24,959 --> 00:17:27,160 everything has happened real time there 419 00:17:27,160 --> 00:17:29,960 is very few ways of escaping them 420 00:17:29,960 --> 00:17:31,720 because that's in Linux that's how the 421 00:17:31,720 --> 00:17:35,559 Kel Works problem is that can explode 422 00:17:35,559 --> 00:17:38,240 extremely like just an application doing 423 00:17:38,240 --> 00:17:41,520 csols they can if they are extremely 424 00:17:41,520 --> 00:17:45,400 efficient this can even bring down 425 00:17:45,400 --> 00:17:49,799 performance on the like on on the 426 00:17:49,799 --> 00:17:52,840 service now with evf things are better 427 00:17:52,840 --> 00:17:55,440 but back in the day with aitd that could 428 00:17:55,440 --> 00:17:56,640 become a performance problem very 429 00:17:56,640 --> 00:17:58,520 quickly and a performance problem on a 430 00:17:58,520 --> 00:18:00,720 machine can become an expensive bill on 431 00:18:00,720 --> 00:18:03,400 a storage especially if you need to 432 00:18:03,400 --> 00:18:04,840 retain things for 433 00:18:04,840 --> 00:18:08,520 years excuse did you say audit yeah did 434 00:18:08,520 --> 00:18:11,000 you say audit D 435 00:18:11,000 --> 00:18:14,480 or uh now evf fantastic audit D I also 436 00:18:14,480 --> 00:18:15,640 love it it's 437 00:18:15,640 --> 00:18:18,360 great I don't I'm not sure like Finance 438 00:18:18,360 --> 00:18:20,840 people love it 439 00:18:20,840 --> 00:18:24,480 but we we can always win uh so BBC flow 440 00:18:24,480 --> 00:18:27,240 locks this I really love this example 441 00:18:27,240 --> 00:18:29,000 because this is one of those 442 00:18:29,000 --> 00:18:32,400 extremely useful logs if you just get 443 00:18:32,400 --> 00:18:33,919 the log that you want if there is an 444 00:18:33,919 --> 00:18:36,640 incident bpc flow Lo will often tell you 445 00:18:36,640 --> 00:18:39,679 what got out of the cluster how the AER 446 00:18:39,679 --> 00:18:42,679 move inside but the problem is this even 447 00:18:42,679 --> 00:18:45,520 in an incident this would be a very very 448 00:18:45,520 --> 00:18:49,960 tiny part of these locks and especially 449 00:18:49,960 --> 00:18:51,720 BP FL locks the cloud will just 450 00:18:51,720 --> 00:18:54,120 recommend it you enable it for the Bing 451 00:18:54,120 --> 00:18:56,760 which means useless for for forensics 452 00:18:56,760 --> 00:18:58,600 sample sample them which means means 453 00:18:58,600 --> 00:19:02,280 that if you just get a sample of 10% you 454 00:19:02,280 --> 00:19:04,840 have a 90% of not seeing the L that you 455 00:19:04,840 --> 00:19:07,760 want and aggregation which aggregation 456 00:19:07,760 --> 00:19:09,240 is one of the things that for security I 457 00:19:09,240 --> 00:19:11,600 think we will have to consider but the 458 00:19:11,600 --> 00:19:14,159 problem is it adds delay which is one of 459 00:19:14,159 --> 00:19:15,600 the constraints that we have with 460 00:19:15,600 --> 00:19:18,320 security logging because if we have a 15 461 00:19:18,320 --> 00:19:20,919 minutes delay on the logs arriving it's 462 00:19:20,919 --> 00:19:23,679 great for costs because for example 463 00:19:23,679 --> 00:19:25,640 instead of getting every single request 464 00:19:25,640 --> 00:19:27,600 you just aggregate a million locks into 465 00:19:27,600 --> 00:19:31,000 one nice lock problem is if an attacker 466 00:19:31,000 --> 00:19:33,080 is very quickly that aggregation would 467 00:19:33,080 --> 00:19:36,080 be oh that's a cool attack that happened 468 00:19:36,080 --> 00:19:38,000 50 minutes 469 00:19:38,000 --> 00:19:40,600 ago and the TG is doing other 470 00:19:40,600 --> 00:19:43,760 things so going to the second case of 471 00:19:43,760 --> 00:19:47,080 study so the first one was about locks 472 00:19:47,080 --> 00:19:49,880 signal the second one is about the 473 00:19:49,880 --> 00:19:51,840 pipelines because the pipelines actually 474 00:19:51,840 --> 00:19:55,039 matter like the order you do things and 475 00:19:55,039 --> 00:19:56,840 where do you put the data it's actually 476 00:19:56,840 --> 00:19:59,280 quite important to make make sure your 477 00:19:59,280 --> 00:20:03,480 system uh is performant at many 478 00:20:03,480 --> 00:20:08,080 levels so this a particular example this 479 00:20:08,080 --> 00:20:10,640 I went for eks a locks one of the things 480 00:20:10,640 --> 00:20:14,520 that keeps me up at night I guess and 481 00:20:14,520 --> 00:20:16,520 these locks can be 482 00:20:16,520 --> 00:20:20,480 extremely uh costly because it's as I 483 00:20:20,480 --> 00:20:22,440 said it's an API like it's a cloud 484 00:20:22,440 --> 00:20:24,440 inside the cloud and if you are managing 485 00:20:24,440 --> 00:20:29,159 eks logs let's say like here I ask 486 00:20:29,159 --> 00:20:31,400 and people who are running like a th000 487 00:20:31,400 --> 00:20:34,960 plus notes with eks they a lot of them 488 00:20:34,960 --> 00:20:37,919 are getting 1 tab per day of 489 00:20:37,919 --> 00:20:40,760 locks managing one tab should not be 490 00:20:40,760 --> 00:20:43,440 that difficult in in nowadays but if you 491 00:20:43,440 --> 00:20:46,120 really want these logs to be useful for 492 00:20:46,120 --> 00:20:48,000 security that's meeting the constraint 493 00:20:48,000 --> 00:20:50,200 that we have then we are encountering 494 00:20:50,200 --> 00:20:53,320 some serious price here I'm not going to 495 00:20:53,320 --> 00:20:55,120 go with like every single piece that I 496 00:20:55,120 --> 00:20:58,760 have another diagram that goes into how 497 00:20:58,760 --> 00:21:00,880 things are but thing it's I would say 498 00:21:00,880 --> 00:21:03,159 here there are like one of the three 499 00:21:03,159 --> 00:21:05,919 most common ways I'm seeing security 500 00:21:05,919 --> 00:21:09,120 logging in terms of pipelines which is G 501 00:21:09,120 --> 00:21:12,559 Duty only for example if someone goes 502 00:21:12,559 --> 00:21:14,559 data is too difficult by just want to 503 00:21:14,559 --> 00:21:17,679 know if I hacked or not so enable G Duty 504 00:21:17,679 --> 00:21:19,559 G Duty starts checking the logs and 505 00:21:19,559 --> 00:21:22,440 tells you when something is wrong this 506 00:21:22,440 --> 00:21:24,799 fantastic as an starting great and in 507 00:21:24,799 --> 00:21:27,120 terms of cost this will be the most 508 00:21:27,120 --> 00:21:29,640 efficient one but then you lose a lot of 509 00:21:29,640 --> 00:21:32,520 the good like a lot of the constraints 510 00:21:32,520 --> 00:21:34,679 that we have which is it will very hard 511 00:21:34,679 --> 00:21:37,360 to know like what are other systems 512 00:21:37,360 --> 00:21:40,400 doing that are similar to The thater 513 00:21:40,400 --> 00:21:42,679 because thater will not tell you the 514 00:21:42,679 --> 00:21:44,799 context it will tell you a bit of 515 00:21:44,799 --> 00:21:48,679 context but you you will you will have a 516 00:21:48,679 --> 00:21:50,919 hard time understanding really how your 517 00:21:50,919 --> 00:21:54,159 application works and but in general 518 00:21:54,159 --> 00:21:56,720 it's not a bad start then if you just 519 00:21:56,720 --> 00:21:59,400 are like got Duty it's too difficult 520 00:21:59,400 --> 00:22:01,200 it's too expensive but it's not too 521 00:22:01,200 --> 00:22:03,640 difficult let me roll it myself this is 522 00:22:03,640 --> 00:22:04,760 a lot more 523 00:22:04,760 --> 00:22:07,799 expensive uh this is using Kinesis which 524 00:22:07,799 --> 00:22:09,840 means that you are not even doing High 525 00:22:09,840 --> 00:22:12,039 retention and you still have to do all 526 00:22:12,039 --> 00:22:14,760 the alerting this is not bad because 527 00:22:14,760 --> 00:22:17,039 then you can do some interesting things 528 00:22:17,039 --> 00:22:20,679 in this space but giving away guard Duty 529 00:22:20,679 --> 00:22:22,600 for custom alerting you really need to 530 00:22:22,600 --> 00:22:24,799 know what you are doing because most 531 00:22:24,799 --> 00:22:26,159 probably you will get less value for 532 00:22:26,159 --> 00:22:26,960 more 533 00:22:26,960 --> 00:22:28,679 money 534 00:22:28,679 --> 00:22:29,919 uh unless you're very good then if 535 00:22:29,919 --> 00:22:33,559 you're very good it's fine but so one of 536 00:22:33,559 --> 00:22:36,960 the things that I think are the more 537 00:22:36,960 --> 00:22:40,039 what I would believe now is changing a 538 00:22:40,039 --> 00:22:44,000 bit but they used to recommend more of a 539 00:22:44,000 --> 00:22:46,039 storing all the data in a three quering 540 00:22:46,039 --> 00:22:48,440 with Athena so you got all the nice ways 541 00:22:48,440 --> 00:22:50,120 of understanding data doing data 542 00:22:50,120 --> 00:22:52,120 analytics and then then getting a duty 543 00:22:52,120 --> 00:22:56,960 for a l this will be more than 40K a 544 00:22:56,960 --> 00:22:58,600 month 545 00:22:58,600 --> 00:23:01,360 and you may be asking what are these 546 00:23:01,360 --> 00:23:06,000 logs what why one terab what what what 547 00:23:06,000 --> 00:23:08,159 is this data and the thing is data teams 548 00:23:08,159 --> 00:23:09,480 you will ask them please I need these 549 00:23:09,480 --> 00:23:12,000 logs and they will be like it's 40K but 550 00:23:12,000 --> 00:23:14,240 I guess for security we'll have to do it 551 00:23:14,240 --> 00:23:17,520 but they are not they are not the ones 552 00:23:17,520 --> 00:23:21,159 knowing if those logs have value and 553 00:23:21,159 --> 00:23:22,760 from that databyte we have the issue 554 00:23:22,760 --> 00:23:25,559 that most of it it's R operations most 555 00:23:25,559 --> 00:23:28,600 of it most of it will at best show you 556 00:23:28,600 --> 00:23:30,679 reconnaissance but at wor it just show 557 00:23:30,679 --> 00:23:32,080 you the tools that you know that are 558 00:23:32,080 --> 00:23:37,120 running are running and that very 559 00:23:37,120 --> 00:23:40,000 interesting uh locks Lo logs are not 560 00:23:40,000 --> 00:23:43,200 even capture like for example the base 561 00:23:43,200 --> 00:23:46,279 the normal cage logs will not tell you 562 00:23:46,279 --> 00:23:49,000 exact operations inside a PO they will 563 00:23:49,000 --> 00:23:51,159 tell you that a PO has been executed but 564 00:23:51,159 --> 00:23:52,440 not the commands that have been run on 565 00:23:52,440 --> 00:23:54,240 it but then you will have a million 566 00:23:54,240 --> 00:23:56,880 times a obser tool that have check if 567 00:23:56,880 --> 00:23:58,799 the system is alive 568 00:23:58,799 --> 00:24:01,799 and there's no way natively of filtering 569 00:24:01,799 --> 00:24:03,919 these 570 00:24:03,919 --> 00:24:08,159 locks so this nice diagram love 571 00:24:08,159 --> 00:24:11,720 diagrams uh in general here the issue in 572 00:24:11,720 --> 00:24:15,520 C it's cloudwatch like cloudwatch it's 573 00:24:15,520 --> 00:24:20,080 50p per gigabyte which means that even 574 00:24:20,080 --> 00:24:22,120 before you touch S3 storage you are 575 00:24:22,120 --> 00:24:24,520 going to pay a very high tax on the logs 576 00:24:24,520 --> 00:24:27,919 even if you use kinis or S3 that 577 00:24:27,919 --> 00:24:30,080 cloudwatch taxs will happen other 578 00:24:30,080 --> 00:24:33,080 services from don't require you to go 579 00:24:33,080 --> 00:24:35,000 through cloudwatch or require to go to 580 00:24:35,000 --> 00:24:37,480 cloudwatch in ways that you can filter 581 00:24:37,480 --> 00:24:41,399 them filter them before you pay Google 582 00:24:41,399 --> 00:24:43,279 on the other hand for example they have 583 00:24:43,279 --> 00:24:44,799 instead of cloudwatch something called 584 00:24:44,799 --> 00:24:46,640 Cloud loging and they do something 585 00:24:46,640 --> 00:24:49,399 called exclusion filters which before 586 00:24:49,399 --> 00:24:53,159 you pay to put into a sync uh you can 587 00:24:53,159 --> 00:24:56,480 make a a search that will F the logs so 588 00:24:56,480 --> 00:24:58,200 if you know your application and you 589 00:24:58,200 --> 00:25:00,159 know that that application cannot be 590 00:25:00,159 --> 00:25:01,919 malicious or even if it's malicious you 591 00:25:01,919 --> 00:25:04,840 will not be able to know by looking at 592 00:25:04,840 --> 00:25:06,200 particular operation if they have gone 593 00:25:06,200 --> 00:25:07,960 malicious or not you can filter it and 594 00:25:07,960 --> 00:25:11,520 then only capture what you really need 595 00:25:11,520 --> 00:25:13,240 and when you have what you really need 596 00:25:13,240 --> 00:25:14,200 you are 597 00:25:14,200 --> 00:25:16,360 able 598 00:25:16,360 --> 00:25:20,960 to get that data into match faster cach 599 00:25:20,960 --> 00:25:23,640 and then do a lot more interesting 600 00:25:23,640 --> 00:25:26,640 stuff for this particular case I have 601 00:25:26,640 --> 00:25:28,919 thought a lot about and this is the the 602 00:25:28,919 --> 00:25:31,120 kind of like if you if you're using 603 00:25:31,120 --> 00:25:34,200 non-managed case notes for for eks audit 604 00:25:34,200 --> 00:25:35,880 or G or 605 00:25:35,880 --> 00:25:39,039 eks uh non manage Notch very simple 606 00:25:39,039 --> 00:25:40,840 there's an audit policy you cannot click 607 00:25:40,840 --> 00:25:42,399 here on the presentation obviously but 608 00:25:42,399 --> 00:25:44,240 when I share the the presentation you'll 609 00:25:44,240 --> 00:25:46,360 be able to in which you can just filter 610 00:25:46,360 --> 00:25:48,440 the policy change the policy and tell 611 00:25:48,440 --> 00:25:52,120 exactly what you want to to collect like 612 00:25:52,120 --> 00:25:53,960 cage it's actually very good in this you 613 00:25:53,960 --> 00:25:56,120 can filter things and then you can get 614 00:25:56,120 --> 00:26:00,520 different levels of logging like go 615 00:26:00,520 --> 00:26:02,360 deeper on getting the information of 616 00:26:02,360 --> 00:26:05,399 what's happening in this case be aware 617 00:26:05,399 --> 00:26:08,279 that this may contain Secrets if you 618 00:26:08,279 --> 00:26:10,360 like all the policy if you do them wrong 619 00:26:10,360 --> 00:26:12,240 they can be quite dangerous and in 620 00:26:12,240 --> 00:26:14,640 general the the manage service will be 621 00:26:14,640 --> 00:26:18,080 very good in in filtering all this data 622 00:26:18,080 --> 00:26:19,480 that you don't really want in your in 623 00:26:19,480 --> 00:26:22,399 your sim then you get the manage Gates 624 00:26:22,399 --> 00:26:24,159 clusters which is the point that we are 625 00:26:24,159 --> 00:26:26,159 encountering here here you cannot change 626 00:26:26,159 --> 00:26:29,200 this audit policy you cannot touch the 627 00:26:29,200 --> 00:26:31,279 control plane and you have to leave with 628 00:26:31,279 --> 00:26:33,039 what the cloud provider gives you 629 00:26:33,039 --> 00:26:35,320 otherwise it wouldn't be managed the 630 00:26:35,320 --> 00:26:37,559 thing is one 631 00:26:37,559 --> 00:26:40,120 thing I'm not sure I would recommend but 632 00:26:40,120 --> 00:26:42,520 it's something that we have been doing 633 00:26:42,520 --> 00:26:45,159 it's using ad Mission controller ad 634 00:26:45,159 --> 00:26:48,279 Mission controllers in kubernetes they 635 00:26:48,279 --> 00:26:51,080 subscribe to what's happening on the 636 00:26:51,080 --> 00:26:52,720 cluster so they subscribe to the control 637 00:26:52,720 --> 00:26:54,679 plane and they get every operation that 638 00:26:54,679 --> 00:26:57,000 makes a change on the clust so and 639 00:26:57,000 --> 00:26:59,080 that's exactly what we want and with 640 00:26:59,080 --> 00:27:01,640 aniss controller you can actually just 641 00:27:01,640 --> 00:27:04,520 analyze the and do alerting in the 642 00:27:04,520 --> 00:27:07,240 cluster or ship just a tiny subset of 643 00:27:07,240 --> 00:27:09,960 things going from a terabyte of logs to 644 00:27:09,960 --> 00:27:12,399 a gigabyte of logs of actually things 645 00:27:12,399 --> 00:27:14,559 that change the cluster with if you're 646 00:27:14,559 --> 00:27:17,279 an attacker you I'm sure you are happy 647 00:27:17,279 --> 00:27:18,799 doing recon but what you really want is 648 00:27:18,799 --> 00:27:19,919 doing 649 00:27:19,919 --> 00:27:22,200 changes uh then what interesting thing 650 00:27:22,200 --> 00:27:24,279 in terms of pipelines you need to think 651 00:27:24,279 --> 00:27:27,600 about retention and storage in AA for 652 00:27:27,600 --> 00:27:30,399 example Athena goes against S3 but you 653 00:27:30,399 --> 00:27:33,840 don't have to be a standard storage high 654 00:27:33,840 --> 00:27:36,399 high speed you can move them to glacia 655 00:27:36,399 --> 00:27:38,640 you can do life cycle policies that 656 00:27:38,640 --> 00:27:40,240 after a certain amount of time you 657 00:27:40,240 --> 00:27:42,519 assume that the logs won't be for Real 658 00:27:42,519 --> 00:27:44,360 Time inent response but it will be for 659 00:27:44,360 --> 00:27:46,799 forensic investigations which would have 660 00:27:46,799 --> 00:27:48,360 different requirements in terms of the 661 00:27:48,360 --> 00:27:52,879 speed and that can save you a lot of 662 00:27:53,200 --> 00:27:58,760 money final the final case study it's so 663 00:27:58,760 --> 00:28:01,279 let's say this has been the journey so 664 00:28:01,279 --> 00:28:03,679 far the start I was 665 00:28:03,679 --> 00:28:07,840 like you came to Defcon realize you need 666 00:28:07,840 --> 00:28:10,720 to have full visibility so you go back 667 00:28:10,720 --> 00:28:13,200 and go we need all the locks but then 668 00:28:13,200 --> 00:28:16,080 you went and did all the logs and your 669 00:28:16,080 --> 00:28:18,120 your finance department was like I don't 670 00:28:18,120 --> 00:28:20,159 think we can afford all the logs what do 671 00:28:20,159 --> 00:28:22,320 we do and when they ask you to start 672 00:28:22,320 --> 00:28:24,960 turning off logs you're like wait let's 673 00:28:24,960 --> 00:28:27,519 just get the high signal logs and that's 674 00:28:27,519 --> 00:28:31,159 when you start getting h a lot a lot 675 00:28:31,159 --> 00:28:33,039 more value from the logs and you start 676 00:28:33,039 --> 00:28:34,039 to understand better how your 677 00:28:34,039 --> 00:28:38,200 organization works and then you move to 678 00:28:38,200 --> 00:28:40,200 more efficient loging pipelines like 679 00:28:40,200 --> 00:28:43,600 same logs same high value but more 680 00:28:43,600 --> 00:28:46,640 efficient ways of uh getting your data 681 00:28:46,640 --> 00:28:48,440 and this is giving 682 00:28:48,440 --> 00:28:52,240 up things that you may want like for 683 00:28:52,240 --> 00:28:53,519 example you can Aggregate and you can 684 00:28:53,519 --> 00:28:55,480 filter you can transform your logs like 685 00:28:55,480 --> 00:28:58,240 aggregation of security locks is very 686 00:28:58,240 --> 00:29:00,440 useful like you don't need to know for 687 00:29:00,440 --> 00:29:03,720 example that h a like a 688 00:29:03,720 --> 00:29:08,200 million uh logs may be just one person 689 00:29:08,200 --> 00:29:09,440 accessing a 690 00:29:09,440 --> 00:29:13,480 server uh obviously like when we go from 691 00:29:13,480 --> 00:29:16,600 Pickups to like Network logs that we 692 00:29:16,600 --> 00:29:18,360 have gone and we have done an 693 00:29:18,360 --> 00:29:22,640 aggregation we are able to introspect 694 00:29:22,640 --> 00:29:26,320 less but we are not losing that much for 695 00:29:26,320 --> 00:29:29,519 compressing and like removing a lot of 696 00:29:29,519 --> 00:29:31,360 the things that we don't need from that 697 00:29:31,360 --> 00:29:34,559 data and then at the end if things go 698 00:29:34,559 --> 00:29:37,440 really Grim people will be like what if 699 00:29:37,440 --> 00:29:40,120 we have no logs what what should we do 700 00:29:40,120 --> 00:29:43,640 what is the forbidden thing about like 701 00:29:43,640 --> 00:29:46,799 going no logs I'm sure the answer will 702 00:29:46,799 --> 00:29:49,519 not surprise you but let's analyze this 703 00:29:49,519 --> 00:29:53,159 for real so if you cannot have logs like 704 00:29:53,159 --> 00:29:55,880 Cloud logs then you have forensics API 705 00:29:55,880 --> 00:29:57,640 based reports and assets and met data 706 00:29:57,640 --> 00:30:01,760 and Asian based detections so on this 707 00:30:01,760 --> 00:30:04,000 forensics it tended to be the old way of 708 00:30:04,000 --> 00:30:06,519 doing things the the old way of doing 709 00:30:06,519 --> 00:30:08,960 things had a problem now moving to the 710 00:30:08,960 --> 00:30:11,080 cloud which is the cloud things are 711 00:30:11,080 --> 00:30:14,519 virtual things are ephemeral so as a 712 00:30:14,519 --> 00:30:17,320 forensic investigator you may go to an 713 00:30:17,320 --> 00:30:20,200 incident and realize that the data that 714 00:30:20,200 --> 00:30:21,799 you need is no longer that because the 715 00:30:21,799 --> 00:30:24,799 machines are no longer that 716 00:30:24,799 --> 00:30:28,000 uh but forensic forensic scale with 717 00:30:28,000 --> 00:30:29,760 incidents so if you have very little 718 00:30:29,760 --> 00:30:32,519 incidents then you will have you will 719 00:30:32,519 --> 00:30:34,159 not be paying that much but forensic 720 00:30:34,159 --> 00:30:37,360 data can become it will it will become 721 00:30:37,360 --> 00:30:38,840 expensive but it will become expensive 722 00:30:38,840 --> 00:30:40,399 because the analysation and the data 723 00:30:40,399 --> 00:30:42,480 acquisition is quite 724 00:30:42,480 --> 00:30:44,840 complicated then you go on API based 725 00:30:44,840 --> 00:30:46,720 reports and asset an data this is kind 726 00:30:46,720 --> 00:30:51,080 of a weird thing I I probably there is a 727 00:30:51,080 --> 00:30:53,399 b word for this but for example these 728 00:30:53,399 --> 00:30:56,519 would be tools like OS query or this 729 00:30:56,519 --> 00:30:58,279 will be tools like Cloud query like 730 00:30:58,279 --> 00:31:02,240 there will be things that scan your uh 731 00:31:02,240 --> 00:31:04,600 cloud and you will be able to detect 732 00:31:04,600 --> 00:31:07,880 drift on it so for example you may scan 733 00:31:07,880 --> 00:31:10,720 all your users and then you scan again 734 00:31:10,720 --> 00:31:12,600 all your users and you then see that 735 00:31:12,600 --> 00:31:15,720 there's a new admin and the new admin 736 00:31:15,720 --> 00:31:17,399 has way too many permissions and it's 737 00:31:17,399 --> 00:31:20,440 not related to any of your normal 738 00:31:20,440 --> 00:31:22,559 changes that can be very useful and in 739 00:31:22,559 --> 00:31:24,880 terms of costs it will be very cheap 740 00:31:24,880 --> 00:31:26,760 because you are storing a snapshat here 741 00:31:26,760 --> 00:31:29,320 a snapch here and then you just need to 742 00:31:29,320 --> 00:31:31,200 keep few of those snapshots and doing 743 00:31:31,200 --> 00:31:32,799 the correlation is not that compute 744 00:31:32,799 --> 00:31:36,200 intensive the problem is then again 745 00:31:36,200 --> 00:31:38,919 frequency if you do a frequency of one 746 00:31:38,919 --> 00:31:41,320 day between a snapshot a and a snapshot 747 00:31:41,320 --> 00:31:44,159 B an has a full day to use those arming 748 00:31:44,159 --> 00:31:46,279 credentials and it may be too late to do 749 00:31:46,279 --> 00:31:49,600 anything I think this approach it's 750 00:31:49,600 --> 00:31:54,399 quite good as a as a baseline And as a 751 00:31:54,399 --> 00:31:56,840 first approach to the problem like if 752 00:31:56,840 --> 00:31:58,559 you don't have no if you don't have any 753 00:31:58,559 --> 00:32:01,559 logs say going to unand visibility and 754 00:32:01,559 --> 00:32:03,440 everything being efficient will take a 755 00:32:03,440 --> 00:32:06,320 while these tools the tools that I think 756 00:32:06,320 --> 00:32:07,519 a lot of the open source Community are 757 00:32:07,519 --> 00:32:09,799 doing are very good to be able to get 758 00:32:09,799 --> 00:32:12,480 and knowing what is my environment doing 759 00:32:12,480 --> 00:32:14,679 what has my environment done between day 760 00:32:14,679 --> 00:32:17,960 a and day B then you have Asian based 761 00:32:17,960 --> 00:32:19,639 detections I think a lot of the vendors 762 00:32:19,639 --> 00:32:21,639 really go into this problem because you 763 00:32:21,639 --> 00:32:26,320 then move to compute and then they have 764 00:32:26,320 --> 00:32:29,600 the thing about this is if you know the 765 00:32:29,600 --> 00:32:31,120 attacker and you know the 766 00:32:31,120 --> 00:32:33,360 attacker very good you are able to make 767 00:32:33,360 --> 00:32:34,840 the intelligence and then put it in the 768 00:32:34,840 --> 00:32:36,919 agent which means that the agent will be 769 00:32:36,919 --> 00:32:39,720 very very adapt at finding the problem 770 00:32:39,720 --> 00:32:41,120 fixing the problem and just telling you 771 00:32:41,120 --> 00:32:43,919 that the thing has been fixed problem is 772 00:32:43,919 --> 00:32:46,360 this may have worked very well on on 773 00:32:46,360 --> 00:32:49,360 Prem or on endpoints and for fighting 774 00:32:49,360 --> 00:32:52,440 like commodity malware the issue is 775 00:32:52,440 --> 00:32:55,559 cloud problems don't touch compute that 776 00:32:55,559 --> 00:32:57,600 often and not only they don't touch 777 00:32:57,600 --> 00:33:00,720 compute that often it's the cloud native 778 00:33:00,720 --> 00:33:03,720 attackers they just go to the console 779 00:33:03,720 --> 00:33:06,080 and do operation that would be normal 780 00:33:06,080 --> 00:33:09,360 for for an engineer to do the problem is 781 00:33:09,360 --> 00:33:11,600 instead of deploying your new 782 00:33:11,600 --> 00:33:13,200 application they're just deploying a new 783 00:33:13,200 --> 00:33:14,080 crypto 784 00:33:14,080 --> 00:33:17,559 M so Asian based detections can be 785 00:33:17,559 --> 00:33:19,559 extremely useful but the problem is 786 00:33:19,559 --> 00:33:23,159 there are no the in the cloud by the 787 00:33:23,159 --> 00:33:28,600 sign they will struggle to replace 788 00:33:28,600 --> 00:33:32,519 logs and in a bit of animatic way 789 00:33:32,519 --> 00:33:34,000 because I'm running out of time I 790 00:33:34,000 --> 00:33:36,279 predicted that that's the end of the of 791 00:33:36,279 --> 00:33:38,000 the talk I have some conclusions and I 792 00:33:38,000 --> 00:33:40,240 have a meme so I like that it was only 793 00:33:40,240 --> 00:33:43,399 going to be diagrams and graphs but this 794 00:33:43,399 --> 00:33:47,120 lives R free my mind I have to 795 00:33:47,320 --> 00:33:49,360 say 796 00:33:49,360 --> 00:33:52,519 uh thank 797 00:33:55,200 --> 00:33:58,880 you I I have I think like two minutes 798 00:33:58,880 --> 00:33:59,679 for 799 00:33:59,679 --> 00:34:02,960 questions if anyone has a 800 00:34:02,960 --> 00:34:05,639 question are there any um open source 801 00:34:05,639 --> 00:34:08,239 Alternatives 802 00:34:08,239 --> 00:34:10,760 that kind of scale you're talking about 803 00:34:10,760 --> 00:34:11,679 you 804 00:34:11,679 --> 00:34:14,520 like that is a excellent question the 805 00:34:14,520 --> 00:34:18,239 issue is every every open source tool 806 00:34:18,239 --> 00:34:20,119 will actually already have to pay the 807 00:34:20,119 --> 00:34:22,800 cloudwatch tax so you are already paying 808 00:34:22,800 --> 00:34:25,199 the tax to the cloud provider anyway and 809 00:34:25,199 --> 00:34:27,879 then you like I really like a like 810 00:34:27,879 --> 00:34:28,800 I said I'm not going to do any 811 00:34:28,800 --> 00:34:30,560 endorsements but I really do like some 812 00:34:30,560 --> 00:34:32,320 open source alternative to look at the 813 00:34:32,320 --> 00:34:34,440 locks and process the locks in the cloud 814 00:34:34,440 --> 00:34:37,000 then move them to CMS and then analyze 815 00:34:37,000 --> 00:34:38,879 them because at the end of the day the 816 00:34:38,879 --> 00:34:40,599 correlation of the data is where most of 817 00:34:40,599 --> 00:34:43,239 the value is like the thing that 818 00:34:43,239 --> 00:34:46,000 security teams in companies can do that 819 00:34:46,000 --> 00:34:48,599 no vendor can that can do it's the the 820 00:34:48,599 --> 00:34:51,599 context you know your company and you 821 00:34:51,599 --> 00:34:53,918 spend time understanding your company 822 00:34:53,918 --> 00:34:55,879 that's the value that we 823 00:34:55,879 --> 00:34:59,480 bring and in the cloud I think we can 824 00:34:59,480 --> 00:35:04,320 move to a to the Utopia of getting to 825 00:35:04,320 --> 00:35:07,640 know what's good what should happen and 826 00:35:07,640 --> 00:35:09,880 everything outside of it be the thing 827 00:35:09,880 --> 00:35:12,040 that we investigate as malicious instead 828 00:35:12,040 --> 00:35:14,280 of just being like we know this is 829 00:35:14,280 --> 00:35:17,400 malicious because we we got a vendor 830 00:35:17,400 --> 00:35:19,280 telling us so we are going to only find 831 00:35:19,280 --> 00:35:23,280 evil we should find not 832 00:35:23,280 --> 00:35:26,160 good any do you have like a policy 833 00:35:26,160 --> 00:35:28,560 breakdown for like what type of policies 834 00:35:28,560 --> 00:35:31,200 you recommend to write or customize 835 00:35:31,200 --> 00:35:34,000 based off the laog type like let's say 836 00:35:34,000 --> 00:35:36,119 it's a kubernetes laws or let's say it's 837 00:35:36,119 --> 00:35:38,560 like a cloudwatch law these are the 838 00:35:38,560 --> 00:35:40,520 policies that you like recommend to 839 00:35:40,520 --> 00:35:42,320 apply to those so that you're actually 840 00:35:42,320 --> 00:35:43,240 getting 841 00:35:43,240 --> 00:35:46,480 useful yeah so the question is if we 842 00:35:46,480 --> 00:35:49,359 have policies to because I'm doing it 843 00:35:49,359 --> 00:35:51,880 for the for the for the future we have 844 00:35:51,880 --> 00:35:54,720 policies related to logs and related to 845 00:35:54,720 --> 00:35:57,319 the value of those logs uh well I 846 00:35:57,319 --> 00:36:01,000 haven't I haven't published them but we 847 00:36:01,000 --> 00:36:03,119 do have policies and we try to enforce 848 00:36:03,119 --> 00:36:05,040 them at the organizational level on 849 00:36:05,040 --> 00:36:06,800 every single Cloud to be like okay this 850 00:36:06,800 --> 00:36:09,160 is how you filter this is the value and 851 00:36:09,160 --> 00:36:11,240 we try to do documentation and we also 852 00:36:11,240 --> 00:36:14,079 do things like heal checks like we need 853 00:36:14,079 --> 00:36:16,119 like you need you need to help check the 854 00:36:16,119 --> 00:36:17,599 cloud provider will not help you in this 855 00:36:17,599 --> 00:36:20,119 you need to know that your your the logs 856 00:36:20,119 --> 00:36:22,920 that you want are coming and one of the 857 00:36:22,920 --> 00:36:24,520 interesting things is when you start 858 00:36:24,520 --> 00:36:27,440 doing the te engineering you realize 859 00:36:27,440 --> 00:36:31,119 that certain data sources always require 860 00:36:31,119 --> 00:36:34,319 certain things like if you go and go 861 00:36:34,319 --> 00:36:37,200 from your rules to your data you can 862 00:36:37,200 --> 00:36:40,079 build policies on the the logs 863 00:36:40,079 --> 00:36:42,319 supporting the rules that I care the 864 00:36:42,319 --> 00:36:44,040 rules like the rules that will call me 865 00:36:44,040 --> 00:36:47,119 at 3:00 a.m. this 100% important and 866 00:36:47,119 --> 00:36:51,079 they need to be stored in the best like 867 00:36:51,079 --> 00:36:52,920 CM that we have or the best layer of 868 00:36:52,920 --> 00:36:54,280 data that we 869 00:36:54,280 --> 00:36:57,560 have but yeah you need yeah you gring it 870 00:36:57,560 --> 00:37:00,960 gring this down is very important also 871 00:37:02,720 --> 00:37:06,960 [Applause] 872 00:37:06,960 --> 00:37:09,000 retention all right folks thank you for 873 00:37:09,000 --> 00:37:11,720 coming