1 00:00:00,000 --> 00:00:01,380 - Here's Aviv. 2 00:00:01,380 --> 00:00:02,213 Go on. 3 00:00:04,320 --> 00:00:05,153 - Thank you. 4 00:00:08,190 --> 00:00:11,010 So thank you everyone for coming to this session, 5 00:00:11,010 --> 00:00:13,980 I'm really excited to be here at DEFCON 30 6 00:00:13,980 --> 00:00:17,400 this time in person. And in this session, 7 00:00:17,400 --> 00:00:19,020 we're gonna talk about the journey, 8 00:00:19,020 --> 00:00:21,810 my journey from an isolated container 9 00:00:21,810 --> 00:00:24,720 to cluster admin in Service Fabric. 10 00:00:24,720 --> 00:00:27,840 So basically we're gonna talk about a vulnerability 11 00:00:27,840 --> 00:00:31,050 that I found in Service Fabric, 12 00:00:31,050 --> 00:00:34,860 which is kind of similar to Kubernetes in the sense that 13 00:00:34,860 --> 00:00:38,940 it is an orchestrator that runs everything 14 00:00:38,940 --> 00:00:42,390 on distributed systems and the vulnerability 15 00:00:42,390 --> 00:00:46,680 allows us to break out of the container in Service Fabric, 16 00:00:46,680 --> 00:00:48,660 specifically Docker container, 17 00:00:48,660 --> 00:00:53,193 and later to compromise the whole Service Fabric cluster. 18 00:00:54,540 --> 00:00:58,290 So, first and foremost, let's talk about me. 19 00:00:58,290 --> 00:00:59,910 My name is Aviv Sasson. 20 00:00:59,910 --> 00:01:03,600 I'm a proud dog owner, and over here on the right, 21 00:01:03,600 --> 00:01:04,470 you can see Lizzy. 22 00:01:04,470 --> 00:01:07,180 She was the assistant in the research 23 00:01:08,100 --> 00:01:11,460 and I'm a research team lead in Prisma Cloud 24 00:01:11,460 --> 00:01:12,450 at Palo Alto Networks. 25 00:01:12,450 --> 00:01:15,480 And basically my job is to look for 26 00:01:15,480 --> 00:01:18,810 zero days for vulnerabilities in the cloud, 27 00:01:18,810 --> 00:01:23,810 and it could be in CF project, in Kubernetes, Docker, runc, 28 00:01:24,150 --> 00:01:25,710 any part of the infrastructure, 29 00:01:25,710 --> 00:01:28,740 just cloud vendors and our job is to keep everyone safe 30 00:01:28,740 --> 00:01:32,913 by finding vulnerabilities and disclose them to the vendors. 31 00:01:37,380 --> 00:01:41,130 On the latest research, I chose to work on Service Fabric 32 00:01:41,130 --> 00:01:42,330 and the motivations. 33 00:01:42,330 --> 00:01:44,925 And the reason I chose Service Fabric 34 00:01:44,925 --> 00:01:47,490 the first one is that Service Fabric hosts 35 00:01:47,490 --> 00:01:52,490 more than 1 million applications and runs on millions 36 00:01:52,590 --> 00:01:56,460 of course, now this is according to Microsoft. 37 00:01:56,460 --> 00:02:00,480 So finding a vulnerability in such a software 38 00:02:00,480 --> 00:02:02,730 could have a really big impact. 39 00:02:02,730 --> 00:02:04,053 And other than that, 40 00:02:05,250 --> 00:02:09,150 there were almost no prior research to Service Fabric. 41 00:02:09,150 --> 00:02:13,920 So I found there was only one CVE to Service Fabric 42 00:02:13,920 --> 00:02:17,190 unlike Kubernetes that had more than 40 CVEs. 43 00:02:17,190 --> 00:02:18,900 So this is kind of interesting 44 00:02:18,900 --> 00:02:21,330 because if I will audit Service Fabric 45 00:02:21,330 --> 00:02:22,823 then perhaps I could find something 46 00:02:22,823 --> 00:02:26,160 that could have some impact. 47 00:02:26,160 --> 00:02:30,870 The third reason was that Microsoft released the source code 48 00:02:30,870 --> 00:02:35,870 of Service Fabric version 6.4 a few years ago 49 00:02:35,940 --> 00:02:37,710 and this is only one version, 50 00:02:37,710 --> 00:02:41,313 but by looking at the code of the oldest version, 51 00:02:42,180 --> 00:02:46,113 I could understand a lot about how Service Fabric works. 52 00:02:47,490 --> 00:02:50,850 So let's make a quick brief about Service Fabric 53 00:02:50,850 --> 00:02:51,683 and what it is. 54 00:02:51,683 --> 00:02:55,890 So, as I said earlier, it is very similar to Kubernetes 55 00:02:55,890 --> 00:03:00,360 and it is a platform for deploying and managing applications 56 00:03:00,360 --> 00:03:02,070 on distributed systems. 57 00:03:02,070 --> 00:03:05,520 So basically you can just take a bunch of machines, 58 00:03:05,520 --> 00:03:08,610 form them into a Service Fabric cluster, 59 00:03:08,610 --> 00:03:11,610 and then you could just deploy your applications 60 00:03:11,610 --> 00:03:15,063 in that cluster and manage them super easily. 61 00:03:15,990 --> 00:03:19,500 Now Service Fabric is developed by Microsoft 62 00:03:19,500 --> 00:03:23,470 and therefore is widely used by Microsoft 63 00:03:24,810 --> 00:03:29,430 and Microsoft discloses, a partial list of offerings 64 00:03:29,430 --> 00:03:31,530 and production and products 65 00:03:31,530 --> 00:03:34,200 that are powered by Service Fabric. 66 00:03:34,200 --> 00:03:36,990 So for example, we have our Azure Service Fabric 67 00:03:36,990 --> 00:03:41,610 Azure SQL, Cosmos DB, and many, many more, 68 00:03:41,610 --> 00:03:46,410 and some products like Cortana or Bing or Skype. 69 00:03:46,410 --> 00:03:48,690 So Service Fabric is everywhere. 70 00:03:48,690 --> 00:03:49,523 And for example, 71 00:03:49,523 --> 00:03:54,240 if you are as a customer deploying Azure SQL database, 72 00:03:54,240 --> 00:03:56,100 there's a good chance that you're running 73 00:03:56,100 --> 00:03:59,793 on a managed Service Fabric cluster on Azure. 74 00:04:02,460 --> 00:04:04,540 And just to demonstrate how common is it 75 00:04:05,520 --> 00:04:08,640 this screenshot was taken from a presentation 76 00:04:08,640 --> 00:04:12,180 of Microsoft in 2019. And as you can see, 77 00:04:12,180 --> 00:04:16,110 just one example is that Azure SQL database 78 00:04:16,110 --> 00:04:21,110 had over 1.6 million databases in 2019, 79 00:04:21,480 --> 00:04:26,480 all powered by Service Fabric. So I guess that in 2022, 80 00:04:27,330 --> 00:04:29,820 it could be even 3 million databases 81 00:04:29,820 --> 00:04:31,593 and that is a lot. 82 00:04:33,810 --> 00:04:36,300 Now in order to understand the vulnerability, 83 00:04:36,300 --> 00:04:39,510 we need to talk a little bit about how Service Fabric works 84 00:04:39,510 --> 00:04:42,390 and what is the architecture in the node. 85 00:04:42,390 --> 00:04:44,010 So, first of all, 86 00:04:44,010 --> 00:04:48,600 Service Fabric can run on both Linux and Windows. 87 00:04:48,600 --> 00:04:52,650 So you could have Windows Service Fabric cluster, 88 00:04:52,650 --> 00:04:54,660 or Linux Service Fabric cluster. 89 00:04:54,660 --> 00:04:55,650 And in this talk, 90 00:04:55,650 --> 00:04:59,103 we're gonna talk about Linux Service Fabric cluster. 91 00:05:00,240 --> 00:05:02,910 So on every node, on the cluster, 92 00:05:02,910 --> 00:05:05,310 you have some fabric components, 93 00:05:05,310 --> 00:05:08,640 which are actually is the core of Service Fabric. 94 00:05:08,640 --> 00:05:11,100 So you have the fabric getaway, 95 00:05:11,100 --> 00:05:14,460 you have file store service and on and on and on. 96 00:05:14,460 --> 00:05:17,190 And those processes are actually communicated 97 00:05:17,190 --> 00:05:20,343 with each other and with other nodes to form the cluster. 98 00:05:21,390 --> 00:05:22,223 Now, 99 00:05:22,223 --> 00:05:26,073 when you are deploying an application in Service Fabric. 100 00:05:26,970 --> 00:05:29,040 Just a second, I'll put it on mute. 101 00:05:29,040 --> 00:05:29,873 All right. 102 00:05:29,873 --> 00:05:34,110 So when you're deploying an application on Service Fabric, 103 00:05:34,110 --> 00:05:37,200 you can deploy it either as code or as a container. 104 00:05:37,200 --> 00:05:40,260 So basically there is a Docker daemon on each node 105 00:05:40,260 --> 00:05:45,180 and the applications will be deployed as containers. 106 00:05:45,180 --> 00:05:48,210 So if you're deploying nginx into your cluster 107 00:05:48,210 --> 00:05:49,713 it'll be an nginx container. 108 00:05:52,080 --> 00:05:56,700 So I started to look at the components on each node. 109 00:05:56,700 --> 00:05:59,790 And one component was really interesting 110 00:05:59,790 --> 00:06:03,660 and this one was the Data Collection Agent, 111 00:06:03,660 --> 00:06:07,590 and this agent does a bunch of things and among them, 112 00:06:07,590 --> 00:06:11,190 it collects logs from the host machine. 113 00:06:11,190 --> 00:06:15,300 So it runs as root on the host and collect the host logs. 114 00:06:15,300 --> 00:06:19,456 In addition, it is also collecting logs from containers 115 00:06:19,456 --> 00:06:21,153 on the same machine. 116 00:06:22,260 --> 00:06:27,260 And this is really, really interesting because this process 117 00:06:27,870 --> 00:06:31,020 kind of breaks the container installation 118 00:06:31,020 --> 00:06:33,240 in the way that on the one hand, 119 00:06:33,240 --> 00:06:35,100 it is not bound to the container. 120 00:06:35,100 --> 00:06:38,250 It does run as root on the host 121 00:06:38,250 --> 00:06:41,410 on every node, on every host on the cluster 122 00:06:43,590 --> 00:06:46,050 and it also interact. 123 00:06:46,050 --> 00:06:49,470 It handles files within containers. 124 00:06:49,470 --> 00:06:53,860 So I thought that perhaps 125 00:06:55,470 --> 00:07:00,330 I could interact with this process by modifying the logs 126 00:07:00,330 --> 00:07:04,500 inside the container, and that could maybe somehow 127 00:07:04,500 --> 00:07:07,290 translate into some interaction with DCA 128 00:07:07,290 --> 00:07:10,710 that will cause some kind of exploitation. 129 00:07:10,710 --> 00:07:15,330 So perhaps by doing so, we can manipulate DCA and exploit it 130 00:07:15,330 --> 00:07:17,583 and break out of the container. 131 00:07:21,240 --> 00:07:24,160 So I started to look at the code 132 00:07:25,230 --> 00:07:29,820 and I found a very interesting function called GetIndex. 133 00:07:29,820 --> 00:07:32,580 Now this function does three things. 134 00:07:32,580 --> 00:07:35,127 The first one is that 135 00:07:35,127 --> 00:07:37,590 and just for the sake of the explanation, 136 00:07:37,590 --> 00:07:40,200 we're gonna talk about paths 137 00:07:40,200 --> 00:07:43,050 and we're gonna refer to the relevant path as X. 138 00:07:43,050 --> 00:07:46,170 It could be any path, it would depend on the implementation. 139 00:07:46,170 --> 00:07:50,220 So let's say that GetIndex is being called for path X 140 00:07:50,220 --> 00:07:54,330 and then LoadFromFile will open X, 141 00:07:54,330 --> 00:07:57,840 read it and store it in the memory. 142 00:07:57,840 --> 00:08:02,280 Afterwards, GetIndex will verify 143 00:08:02,280 --> 00:08:05,610 some things inside that content of X. 144 00:08:05,610 --> 00:08:09,240 It will do some modifications, minor modifications, 145 00:08:09,240 --> 00:08:14,240 and will add some data into the content, in the memory. 146 00:08:16,080 --> 00:08:21,080 And after that, it'll override X with the modified data. 147 00:08:22,680 --> 00:08:27,680 So this scenario can result in a classical, Symlink race. 148 00:08:29,070 --> 00:08:34,070 In the sense that I could just take X, 149 00:08:34,170 --> 00:08:37,120 modify it to my malicious content 150 00:08:38,130 --> 00:08:41,400 and then after the verification and everything, 151 00:08:41,400 --> 00:08:44,850 I could just override X with a Symlink 152 00:08:44,850 --> 00:08:49,850 that points to any place that I want on the file system. 153 00:08:50,010 --> 00:08:52,290 And when SaveToFile will be called, 154 00:08:52,290 --> 00:08:56,400 it'll actually try to save the data, 155 00:08:56,400 --> 00:08:58,890 the malicious payload into X, 156 00:08:58,890 --> 00:09:02,010 which actually points to another location. 157 00:09:02,010 --> 00:09:05,590 And by that, I will be able to get a right primitive 158 00:09:06,750 --> 00:09:11,370 and right to any place that I want on the file system. 159 00:09:11,370 --> 00:09:16,370 And if I was able to do it from within the container, 160 00:09:17,370 --> 00:09:22,370 then I could actually write files into the host file system 161 00:09:22,590 --> 00:09:24,150 from within the container. 162 00:09:24,150 --> 00:09:29,150 So this right primitive could actually be translated later 163 00:09:29,430 --> 00:09:31,293 into a container escape. 164 00:09:33,150 --> 00:09:35,310 So let's talk about the requirements 165 00:09:35,310 --> 00:09:40,310 for exploiting this Symlink race and escape the container. 166 00:09:43,500 --> 00:09:46,552 So first we need to find a way to trigger, 167 00:09:46,552 --> 00:09:48,930 GetIndex from within the container 168 00:09:48,930 --> 00:09:53,853 'cause this function contains the vulnerable functionality. 169 00:09:55,020 --> 00:09:58,140 Second is that we need that GetIndex 170 00:09:58,140 --> 00:10:02,913 will be called on a path that we have permissions to modify, 171 00:10:03,780 --> 00:10:06,720 because if we cannot modify it and override it 172 00:10:06,720 --> 00:10:10,680 with a Symlink, then it doesn't morph anything. 173 00:10:10,680 --> 00:10:13,950 The third requirement is that 174 00:10:13,950 --> 00:10:17,940 we need to beat the race condition for the Symlink race. 175 00:10:17,940 --> 00:10:22,940 So between SaveToFile and between LoadFromFile 176 00:10:22,950 --> 00:10:24,330 and SaveToFile, 177 00:10:24,330 --> 00:10:28,050 we have a bunch of really three or four milliseconds. 178 00:10:28,050 --> 00:10:32,700 It's really hard to do the overriding of the file 179 00:10:32,700 --> 00:10:35,580 during that really short period. 180 00:10:35,580 --> 00:10:39,280 So let's see how can we fulfill those requirements 181 00:10:40,890 --> 00:10:43,770 but first I just wanna mention that until now 182 00:10:43,770 --> 00:10:46,830 we talked about just the theoretical part 183 00:10:46,830 --> 00:10:48,420 of the vulnerability 184 00:10:48,420 --> 00:10:52,710 and in order to test for real exploitation, 185 00:10:52,710 --> 00:10:55,620 I need to decide on a target and try to exploit it. 186 00:10:55,620 --> 00:10:59,250 So there are many products and services 187 00:10:59,250 --> 00:11:01,470 that are powered by Service Fabric 188 00:11:01,470 --> 00:11:04,740 and I chose to work on Azure Service Fabric, 189 00:11:04,740 --> 00:11:08,430 which is offering by Azure that allow you 190 00:11:08,430 --> 00:11:13,430 to deploy your own Service Fabric cluster and manage it. 191 00:11:14,610 --> 00:11:16,170 So Microsoft does not manage it. 192 00:11:16,170 --> 00:11:18,483 You are the manager of the cluster. 193 00:11:19,530 --> 00:11:21,600 And by managing the cluster, 194 00:11:21,600 --> 00:11:25,917 you can have all of the permissions to the containers, 195 00:11:25,917 --> 00:11:29,070 to the nodes and just to everything. 196 00:11:29,070 --> 00:11:33,386 So it'll be really easy to debug such environment 197 00:11:33,386 --> 00:11:34,800 for the exploitation. 198 00:11:34,800 --> 00:11:39,210 So I chose this one and for the operation system, 199 00:11:39,210 --> 00:11:42,600 I chose Ubuntu because I like Linux. 200 00:11:42,600 --> 00:11:46,470 And apparently this vulnerability cannot be exploited 201 00:11:46,470 --> 00:11:49,890 in Windows, which we will discuss later. 202 00:11:49,890 --> 00:11:51,753 So Ubuntu was a good choice. 203 00:11:53,040 --> 00:11:57,960 So first, let's talk about how can we trigger GetIndex 204 00:11:57,960 --> 00:11:59,850 the vulnerable function. 205 00:11:59,850 --> 00:12:01,800 So after diving into the code, 206 00:12:01,800 --> 00:12:05,220 I found out that DCA, the Data Collection Agent 207 00:12:05,220 --> 00:12:09,900 monitors the creation of marker files inside each container. 208 00:12:09,900 --> 00:12:14,430 So other components on the machine could create those files 209 00:12:14,430 --> 00:12:16,530 and just mark to the agent 210 00:12:16,530 --> 00:12:18,840 that it should execute some functionalities. 211 00:12:18,840 --> 00:12:20,460 This is just a way for communication 212 00:12:20,460 --> 00:12:22,713 between processes in Service Fabric. 213 00:12:23,640 --> 00:12:28,087 And one of those processes is process container, log.txt. 214 00:12:29,880 --> 00:12:31,740 So I tried to create this one 215 00:12:31,740 --> 00:12:34,590 and created many files and tried to see what happened. 216 00:12:34,590 --> 00:12:37,830 And apparently when I created it, 217 00:12:37,830 --> 00:12:42,030 DCA executed, GetIndex several times. 218 00:12:42,030 --> 00:12:44,400 So we got the first requirement 219 00:12:44,400 --> 00:12:47,130 and we were able to trigger GetIndex 220 00:12:47,130 --> 00:12:48,430 from within the container. 221 00:12:49,890 --> 00:12:54,210 Now we needed to make sure that there are files 222 00:12:54,210 --> 00:12:58,953 inside the containers that we can modify that GetIndex uses. 223 00:12:59,880 --> 00:13:02,280 So I used inotifywait, 224 00:13:02,280 --> 00:13:05,850 which is a CLI tool for interacting 225 00:13:05,850 --> 00:13:09,880 with the inotify kernel feature. 226 00:13:11,310 --> 00:13:13,590 And this feature actually is really cool. 227 00:13:13,590 --> 00:13:17,310 It allow you to monitor the access to some files 228 00:13:17,310 --> 00:13:22,310 and once the files that you configured is being accessed, 229 00:13:22,440 --> 00:13:26,190 then you'll get a notification immediately about it. 230 00:13:26,190 --> 00:13:30,840 So I set up the monitoring and created the file 231 00:13:30,840 --> 00:13:34,000 and apparently many paths inside the container 232 00:13:35,010 --> 00:13:36,060 were accessed. 233 00:13:36,060 --> 00:13:38,460 So getting to use them, and I could exploit them. 234 00:13:38,460 --> 00:13:40,650 And for the sake of this exploitation, 235 00:13:40,650 --> 00:13:44,493 I chose WorkSubFolderMap.det. 236 00:13:47,820 --> 00:13:51,630 The third part was to beat the race condition. 237 00:13:51,630 --> 00:13:53,490 Now, first we needed to know 238 00:13:53,490 --> 00:13:55,020 when the file was being accessed 239 00:13:55,020 --> 00:13:58,950 and it was easy because we had inotify for that, 240 00:13:58,950 --> 00:14:03,950 but there was a really, really short window of opportunity 241 00:14:04,230 --> 00:14:08,790 between LoadFromFile and SaveToFile. 242 00:14:08,790 --> 00:14:12,660 And when I tried to exploit it, I was not able to do so, 243 00:14:12,660 --> 00:14:16,680 but I figured that there might be some other things 244 00:14:16,680 --> 00:14:17,513 we can do. 245 00:14:17,513 --> 00:14:19,530 And actually, Lizzy thought about this idea. 246 00:14:19,530 --> 00:14:21,540 You can see her on the bike here. 247 00:14:21,540 --> 00:14:25,060 And I just took the content of 248 00:14:26,250 --> 00:14:28,980 WorkSubFolderMap.det, 249 00:14:28,980 --> 00:14:31,323 which is actually X that we talked about. 250 00:14:32,995 --> 00:14:35,760 And I just made it really, really, really long. 251 00:14:35,760 --> 00:14:40,497 And that created a situation where LoadToFile 252 00:14:41,790 --> 00:14:46,470 uses much more time for the reading and parsing. 253 00:14:46,470 --> 00:14:47,820 And during that time, 254 00:14:47,820 --> 00:14:51,510 I was able to override it with a Symlink. 255 00:14:51,510 --> 00:14:55,080 And when I did it, it was actually working every time 256 00:14:55,080 --> 00:14:57,963 and I beat the race condition systematically. 257 00:14:59,910 --> 00:15:02,407 So now we have the exploitation. 258 00:15:02,407 --> 00:15:04,470 It's great, it works. 259 00:15:04,470 --> 00:15:09,330 And I was able to override any path on the host machine. 260 00:15:09,330 --> 00:15:11,520 So now there are many techniques 261 00:15:11,520 --> 00:15:14,280 in order to gain remote code execution 262 00:15:14,280 --> 00:15:18,480 from a right primitive on the host. 263 00:15:18,480 --> 00:15:22,500 For example, I could just add malicious ssh keys. 264 00:15:22,500 --> 00:15:24,840 I could just add them to the file system, 265 00:15:24,840 --> 00:15:26,940 or I could just modify the file system 266 00:15:26,940 --> 00:15:30,180 to create a malicious user and use ssh 267 00:15:30,180 --> 00:15:32,760 and connect through using that user. 268 00:15:32,760 --> 00:15:34,410 Or if I just wanna go ballistics, 269 00:15:34,410 --> 00:15:37,440 I can override a benign file with a backdoor. 270 00:15:37,440 --> 00:15:38,670 So just for example, 271 00:15:38,670 --> 00:15:43,670 I could override bins and it is really presumptuous 272 00:15:44,940 --> 00:15:49,260 that in some point, kubernetes will be executed on the host 273 00:15:49,260 --> 00:15:54,260 and the exploit will work and my code will be executed. 274 00:15:55,980 --> 00:16:00,570 But all of those techniques are not applicable 275 00:16:00,570 --> 00:16:05,570 on this exploit because there are two very big limitations. 276 00:16:05,910 --> 00:16:10,320 The first one is that while overriding a file on the host, 277 00:16:10,320 --> 00:16:14,880 this file does not have any execution permissions. 278 00:16:14,880 --> 00:16:18,540 So I cannot count on someone else to execute it 279 00:16:18,540 --> 00:16:20,103 because it cannot be executed. 280 00:16:21,060 --> 00:16:25,920 The second limitation, which was more hard 281 00:16:25,920 --> 00:16:30,450 was that the file has to be in a specific format. 282 00:16:30,450 --> 00:16:32,310 And I said it before 283 00:16:32,310 --> 00:16:35,291 that between LoadFromFile and SaveToFile, 284 00:16:35,291 --> 00:16:39,603 there is a verification of the content in that file. 285 00:16:40,469 --> 00:16:43,860 And this content is gonna be written to the host. 286 00:16:43,860 --> 00:16:46,620 And according to the verification, 287 00:16:46,620 --> 00:16:51,620 this content has to be in this specific format. 288 00:16:52,230 --> 00:16:54,870 So it's gonna be version int's 289 00:16:54,870 --> 00:16:57,720 and the only thing that I could control 290 00:16:57,720 --> 00:17:00,810 are malicious strings and malicious int's. 291 00:17:00,810 --> 00:17:04,503 Now I could add as many rows as I want, 292 00:17:05,400 --> 00:17:08,760 but still this format is really weird. 293 00:17:08,760 --> 00:17:13,530 And even more, in order to beat the race condition, 294 00:17:13,530 --> 00:17:18,450 I had to first add a lot of data that looks like this 295 00:17:18,450 --> 00:17:19,620 into that file. 296 00:17:19,620 --> 00:17:23,850 So basically I had a weak right primitive to the host 297 00:17:23,850 --> 00:17:25,923 and the format should look like this. 298 00:17:27,540 --> 00:17:31,140 Now, mainly really banged our head against the wall 299 00:17:31,140 --> 00:17:33,690 on this one, trying to figure out how can we exploit it 300 00:17:33,690 --> 00:17:35,190 to gain remote code execution, 301 00:17:35,190 --> 00:17:38,190 because this does not look like the best script 302 00:17:38,190 --> 00:17:41,970 or PHP script or Python or binary code, 303 00:17:41,970 --> 00:17:43,383 or just anything else. 304 00:17:44,250 --> 00:17:48,213 It's just normal text, weird text format. 305 00:17:49,740 --> 00:17:50,910 But after a while, 306 00:17:50,910 --> 00:17:55,120 we found out that perhaps we could modify the file 307 00:17:56,100 --> 00:17:58,830 in order for it to look like 308 00:17:58,830 --> 00:18:02,070 a file that contains environment variables. 309 00:18:02,070 --> 00:18:05,880 And we tried to feed some processes with this file. 310 00:18:05,880 --> 00:18:10,880 And we actually found out that they're digesting this file 311 00:18:11,670 --> 00:18:14,100 as environment variables. 312 00:18:14,100 --> 00:18:19,100 So we potentially have a way to inject environment variables 313 00:18:19,650 --> 00:18:24,123 into processes on the host that runs this root. 314 00:18:26,250 --> 00:18:31,250 So for this exploitation, we chose etc environment. 315 00:18:31,440 --> 00:18:35,770 And this file is actually the default environment variable 316 00:18:36,780 --> 00:18:40,350 contains the different environment variables for new logins, 317 00:18:40,350 --> 00:18:41,940 but this was not really helpful 318 00:18:41,940 --> 00:18:43,680 because I didn't want to count 319 00:18:43,680 --> 00:18:47,790 on the fact that someone will log in to the machine 320 00:18:47,790 --> 00:18:51,450 in order for me to get a remote pod execution. 321 00:18:51,450 --> 00:18:52,503 And besides that, 322 00:18:53,610 --> 00:18:55,590 we are talking about production environment, 323 00:18:55,590 --> 00:18:58,110 Service Fabric is for production environments 324 00:18:58,110 --> 00:19:01,473 and I do not count on someone to log into that machine 325 00:19:01,473 --> 00:19:03,333 'cause it's kind of rare. 326 00:19:05,850 --> 00:19:09,480 So I tried to see what else can I do. 327 00:19:09,480 --> 00:19:12,450 And by investigating, I found that cron, 328 00:19:12,450 --> 00:19:17,450 which is the Linux task scheduler actually imports 329 00:19:19,320 --> 00:19:21,870 it is the environment and use it 330 00:19:21,870 --> 00:19:26,070 for the default environment variables on every new job 331 00:19:26,070 --> 00:19:27,960 that it executes. 332 00:19:27,960 --> 00:19:30,330 And this is really interesting. 333 00:19:30,330 --> 00:19:34,140 So I tried to see which jobs are actually running 334 00:19:34,140 --> 00:19:36,630 on Azure Service Fabric. 335 00:19:36,630 --> 00:19:39,030 And I found out there is one current job 336 00:19:39,030 --> 00:19:42,550 that runs every minute as root on the host 337 00:19:44,490 --> 00:19:49,490 and I could just use it and exploit and in my vulnerability 338 00:19:50,040 --> 00:19:53,730 and exploit it so that every minute my environment variables 339 00:19:53,730 --> 00:19:57,183 will be injected into this process. 340 00:19:58,110 --> 00:19:59,760 But it wasn't really enough for me 341 00:19:59,760 --> 00:20:01,950 because there might be other places 342 00:20:01,950 --> 00:20:05,430 except Azure Service Fabric that are vulnerable, 343 00:20:05,430 --> 00:20:09,090 who do not have this minutely cron job. 344 00:20:09,090 --> 00:20:13,260 So I started to look even deeper and I found out that 345 00:20:13,260 --> 00:20:16,530 even though you don't have any cron jobs 346 00:20:16,530 --> 00:20:18,213 in your cron scheduler, 347 00:20:19,230 --> 00:20:24,030 cron is actually run internal hourly jobs 348 00:20:24,030 --> 00:20:26,640 and one of them is running as root. 349 00:20:26,640 --> 00:20:29,460 So it doesn't matter if you have anything over there. 350 00:20:29,460 --> 00:20:33,420 The exploit could work on that internal job every hour 351 00:20:33,420 --> 00:20:38,313 and just inject environment variables into that. 352 00:20:40,920 --> 00:20:44,190 So now that we could inject environment variables 353 00:20:44,190 --> 00:20:46,020 into processes in the host, 354 00:20:46,020 --> 00:20:48,620 let's talk about how can we escalate and use it 355 00:20:48,620 --> 00:20:51,750 to gain remote code execution. 356 00:20:51,750 --> 00:20:55,413 So for this one, I use LD PRELOAD. 357 00:20:56,700 --> 00:21:01,177 Now this environment variables is used by the Linux linker. 358 00:21:04,290 --> 00:21:07,140 And basically when you are initializing a process, 359 00:21:07,140 --> 00:21:11,640 the linker will take a look at this environment variable. 360 00:21:11,640 --> 00:21:14,190 And if it points to any path 361 00:21:14,190 --> 00:21:16,260 and if it finds this environment variable, 362 00:21:16,260 --> 00:21:18,480 then the first thing that it will do 363 00:21:18,480 --> 00:21:21,630 is to load the shared object 364 00:21:21,630 --> 00:21:24,720 that is pointed by that variable. 365 00:21:24,720 --> 00:21:26,820 So that means that if I use it, 366 00:21:26,820 --> 00:21:29,120 if I upload the shared object to the container 367 00:21:30,180 --> 00:21:34,620 and set LD PRELOAD to point to the shared object, 368 00:21:34,620 --> 00:21:38,160 then I could actually inject shared objects 369 00:21:38,160 --> 00:21:42,543 into processes on the host, which is super awesome. 370 00:21:45,000 --> 00:21:47,280 So I compiled the shared object 371 00:21:47,280 --> 00:21:52,050 with a construction attribute and this shared object, 372 00:21:52,050 --> 00:21:56,550 the construction attribute just helped me to execute code 373 00:21:56,550 --> 00:21:59,400 when the shared object was being loaded 374 00:21:59,400 --> 00:22:03,720 because I didn't want to wait for any function, 375 00:22:03,720 --> 00:22:06,150 the shared object tool to get called 376 00:22:06,150 --> 00:22:08,130 so that my code will be executed. 377 00:22:08,130 --> 00:22:11,850 I just wanted, once the shared object was being loaded, 378 00:22:11,850 --> 00:22:15,420 then my code will be executed. So it really was helpful. 379 00:22:15,420 --> 00:22:18,040 And I uses reverse shell for this one 380 00:22:19,260 --> 00:22:22,860 and I just uploaded it with the container, 381 00:22:22,860 --> 00:22:25,110 did everything and it worked. 382 00:22:25,110 --> 00:22:30,110 I just completed the full chain of the container escape 383 00:22:30,180 --> 00:22:34,263 and got root privileges from the stance of a container. 384 00:22:37,363 --> 00:22:38,642 (audience applauds) 385 00:22:38,642 --> 00:22:39,475 Thank you. 386 00:22:43,430 --> 00:22:46,080 So I know I talked a lot, 387 00:22:46,080 --> 00:22:49,530 so I wanna just make a quick recap of all of the stages 388 00:22:49,530 --> 00:22:52,500 so you will understand the workflow of the vulnerability 389 00:22:52,500 --> 00:22:54,213 in a more high level. 390 00:22:55,140 --> 00:22:57,690 So first let's talk about the attack vector, 391 00:22:57,690 --> 00:23:01,680 where the attacker begin everything. 392 00:23:01,680 --> 00:23:04,980 So to begin the attack, 393 00:23:04,980 --> 00:23:09,570 the attacker needs to be in a container in Service Fabric 394 00:23:09,570 --> 00:23:10,590 and compromise it. 395 00:23:10,590 --> 00:23:14,430 Now this can be done in many ways. 396 00:23:14,430 --> 00:23:18,180 For example, if the container runs an old software, 397 00:23:18,180 --> 00:23:22,140 let's say old nginx server with a zero day or one day 398 00:23:22,140 --> 00:23:23,421 or a misconfiguration, 399 00:23:23,421 --> 00:23:25,721 an attacker could just super easily exploit it 400 00:23:26,790 --> 00:23:30,120 and get onto the container or compromise it. 401 00:23:30,120 --> 00:23:34,620 Now, after the attacker got on the container, 402 00:23:34,620 --> 00:23:37,500 we can start to exploit the vulnerability 403 00:23:37,500 --> 00:23:39,060 and by exploiting it, 404 00:23:39,060 --> 00:23:43,050 we could override etc environment on the host 405 00:23:43,050 --> 00:23:46,890 and afterwards new cron jobs on the host 406 00:23:46,890 --> 00:23:51,890 will actually import this file, this malicious file. 407 00:23:51,900 --> 00:23:54,030 It is the environment and use it. 408 00:23:54,030 --> 00:23:57,490 And in this file, you could find LD PRELOAD 409 00:23:58,440 --> 00:24:02,760 and LD PRELOAD will point to a malicious shared object 410 00:24:02,760 --> 00:24:06,270 that the attacker uploaded to the container before 411 00:24:06,270 --> 00:24:07,413 for the exploitation. 412 00:24:08,520 --> 00:24:13,520 Now, when this object will be loaded by the cron job 413 00:24:13,740 --> 00:24:18,740 it'll actually initiate reverse shell from root on the host 414 00:24:20,315 --> 00:24:23,130 to the attacker and the attacker could just do anything 415 00:24:23,130 --> 00:24:28,083 he wants on the host and finish the container escape. 416 00:24:31,020 --> 00:24:35,040 So that was the vulnerability. 417 00:24:35,040 --> 00:24:38,970 And up until now, we talked about the vulnerability. 418 00:24:38,970 --> 00:24:40,200 And from now on, 419 00:24:40,200 --> 00:24:45,000 I wanna talk about exploring the options 420 00:24:45,000 --> 00:24:47,280 of escalating further from that stance 421 00:24:47,280 --> 00:24:48,600 and just a quick disclaimer, 422 00:24:48,600 --> 00:24:51,870 Microsoft does not consider nodes as a security boundary 423 00:24:51,870 --> 00:24:53,103 in Service Fabric. 424 00:24:54,570 --> 00:24:58,320 So we have this great tool called Sftcl. 425 00:24:58,320 --> 00:25:02,160 It is a CLI tool for managing Service Fabric clusters. 426 00:25:02,160 --> 00:25:05,070 For those of you who help manage Kubernetes, 427 00:25:05,070 --> 00:25:06,483 it's kind of kubectl, 428 00:25:07,500 --> 00:25:12,500 and you can provide a private certificate to your Sftcl 429 00:25:12,690 --> 00:25:15,540 and use it in order to manage your cluster. 430 00:25:15,540 --> 00:25:18,840 And actually what happens under the hood 431 00:25:18,840 --> 00:25:23,070 is that Sfctl will use this certificate 432 00:25:23,070 --> 00:25:25,096 in order to authenticate 433 00:25:25,096 --> 00:25:30,096 to the managing endpoint of your Service Fabric cluster, 434 00:25:30,360 --> 00:25:34,080 and then it'll execute whatever you'll send it. 435 00:25:34,080 --> 00:25:36,780 So you can just deploy application, 436 00:25:36,780 --> 00:25:38,040 manage your applications, 437 00:25:38,040 --> 00:25:41,190 or just do anything that you want on the cluster 438 00:25:41,190 --> 00:25:44,253 using that matching endpoint and the certificate. 439 00:25:45,570 --> 00:25:49,020 So finding the certificate, 440 00:25:49,020 --> 00:25:51,180 gaining the certificate could help us 441 00:25:51,180 --> 00:25:54,130 to just compromise the whole cluster 442 00:25:55,140 --> 00:25:57,440 so I tried to see if I could find it anywhere. 443 00:25:59,550 --> 00:26:02,400 Now, after I exploited the vulnerability 444 00:26:02,400 --> 00:26:04,200 and got onto the host, 445 00:26:04,200 --> 00:26:09,150 I found the directory var/lib/waagent/ 446 00:26:09,150 --> 00:26:11,730 which was only accessible through roots on the host. 447 00:26:11,730 --> 00:26:14,550 And luckily because of the exploit, 448 00:26:14,550 --> 00:26:17,673 I had those privileges and could access it. 449 00:26:18,960 --> 00:26:20,850 And by looking in the directory, 450 00:26:20,850 --> 00:26:23,370 I find so many interesting things and among them, 451 00:26:23,370 --> 00:26:27,480 I actually found this desirable certificate 452 00:26:27,480 --> 00:26:30,393 that could allow us to compromise the whole cluster. 453 00:26:31,620 --> 00:26:33,600 So by using this certificate, 454 00:26:33,600 --> 00:26:36,000 I was able to do so many things. 455 00:26:36,000 --> 00:26:38,940 I could just manage the cluster in a few different ways. 456 00:26:38,940 --> 00:26:43,410 I could use Sfctl to gain full control over the cluster, 457 00:26:43,410 --> 00:26:45,810 just manage applications, deploy applications, 458 00:26:45,810 --> 00:26:48,453 delete applications, whatever I want. 459 00:26:49,470 --> 00:26:53,537 I could just send row request to the managing endpoints 460 00:26:56,463 --> 00:26:59,490 using the certificate, 461 00:26:59,490 --> 00:27:04,132 or if I just want to go ballistics, I could open my browser, 462 00:27:04,132 --> 00:27:09,132 insert the certificate and go into that endpoint. 463 00:27:10,110 --> 00:27:14,340 And if I do so, I'll get the Service Fabric Explorer, 464 00:27:14,340 --> 00:27:18,110 which is Gooey web based management endpoint. 465 00:27:20,820 --> 00:27:23,490 And over there, I could just manage everything super easily 466 00:27:23,490 --> 00:27:25,380 and whatever I want, 467 00:27:25,380 --> 00:27:28,623 over here, you can see how this Gooey looks like. 468 00:27:30,765 --> 00:27:35,765 So I was able to exploit this in Azure Service Fabric 469 00:27:36,840 --> 00:27:39,390 and a lot of companies and governments 470 00:27:39,390 --> 00:27:43,743 use Azure Service Fabric from all the areas. 471 00:27:45,690 --> 00:27:49,380 But other than that, other than Azure Service Fabric, 472 00:27:49,380 --> 00:27:53,700 there are other offerings and products that are based 473 00:27:53,700 --> 00:27:56,040 that are powered by Service Fabric 474 00:27:56,040 --> 00:27:58,710 and some of them are even multi-tenant. 475 00:27:58,710 --> 00:27:59,820 So for example, 476 00:27:59,820 --> 00:28:03,480 let's say that one morning, I wake up, 477 00:28:03,480 --> 00:28:08,480 I decided that I want to have a Azure SQL database in Azure 478 00:28:10,020 --> 00:28:13,050 for my production environment and I just deploy it 479 00:28:13,050 --> 00:28:16,260 and it'll actually deploy it in Service Fabric cluster. 480 00:28:16,260 --> 00:28:17,430 And on the other hand, 481 00:28:17,430 --> 00:28:21,780 an attacker can just wake up and will think to myself. 482 00:28:21,780 --> 00:28:23,820 Well, all right, so I wanna do some mess today. 483 00:28:23,820 --> 00:28:25,020 Let's see what I can do. 484 00:28:26,340 --> 00:28:31,290 So he could also open NSU database on Azure 485 00:28:32,640 --> 00:28:36,300 and by exploiting this vulnerability in that cluster, 486 00:28:36,300 --> 00:28:39,450 he could actually compromise, escalate 487 00:28:39,450 --> 00:28:40,830 to control the whole cluster 488 00:28:40,830 --> 00:28:44,820 and compromise all of the other tenants in that cluster. 489 00:28:44,820 --> 00:28:48,060 So I was really scared about this option 490 00:28:48,060 --> 00:28:52,623 and I tried to figure out if there's a POC for this one. 491 00:28:53,520 --> 00:28:58,520 So I chose three targets for the test and needless to say 492 00:29:00,270 --> 00:29:05,270 that I needed first initial access 493 00:29:05,340 --> 00:29:08,400 in order to execute my exploit. 494 00:29:08,400 --> 00:29:10,890 So for that, I used Azure Functions 495 00:29:10,890 --> 00:29:12,360 that allows you to run code 496 00:29:12,360 --> 00:29:14,280 and Azure Container Instances 497 00:29:14,280 --> 00:29:17,220 that also allow you to run code in Azure PostgreSQL. 498 00:29:17,220 --> 00:29:20,460 So a few months ago, we were able to 499 00:29:20,460 --> 00:29:25,460 just deploy all Azure PostgreSQL server with some one day, 500 00:29:27,270 --> 00:29:31,080 then you could just gain initial access on that container. 501 00:29:31,080 --> 00:29:32,880 And I tried to exploit those, 502 00:29:32,880 --> 00:29:35,250 the vulnerability on those platforms, 503 00:29:35,250 --> 00:29:39,120 but I discovered that they were actually secured 504 00:29:39,120 --> 00:29:41,550 and I couldn't exploit it. 505 00:29:41,550 --> 00:29:43,020 So I started to think 506 00:29:43,020 --> 00:29:46,800 and figured why couldn't I exploit it over there 507 00:29:46,800 --> 00:29:50,220 and I found out there are two requirements 508 00:29:50,220 --> 00:29:53,550 for the exploitation to work. 509 00:29:53,550 --> 00:29:58,550 The first one is that your container Service Fabric, 510 00:29:58,650 --> 00:30:02,580 your application need to have runtime access enabled. 511 00:30:02,580 --> 00:30:05,610 Now this option just allows you to read some data 512 00:30:05,610 --> 00:30:08,103 about the runtime from within the container. 513 00:30:09,180 --> 00:30:12,030 And it is configured by default 514 00:30:12,030 --> 00:30:14,370 on applications in Service Fabric. 515 00:30:14,370 --> 00:30:16,560 So it's basically being used, 516 00:30:16,560 --> 00:30:21,560 but Microsoft did configure their services. 517 00:30:21,600 --> 00:30:24,480 The one that I tested to have runtime access disabled 518 00:30:24,480 --> 00:30:27,840 so I couldn't exploit the vulnerability. 519 00:30:27,840 --> 00:30:28,890 Other than that, 520 00:30:28,890 --> 00:30:30,990 the vulnerability only works on Linux clusters. 521 00:30:30,990 --> 00:30:34,470 So that means that it needs to be a Linux host 522 00:30:34,470 --> 00:30:35,820 and a Linux container. 523 00:30:35,820 --> 00:30:37,410 So just for example, LCOW, 524 00:30:37,410 --> 00:30:39,990 which is Linux containers on Windows, 525 00:30:39,990 --> 00:30:42,360 which is common on Azure 526 00:30:42,360 --> 00:30:44,970 is not relevant to this vulnerability. 527 00:30:44,970 --> 00:30:46,983 So it's only Linux on Linux. 528 00:30:49,050 --> 00:30:51,840 So the real impact in here that I found 529 00:30:51,840 --> 00:30:55,950 was that there is a full container escape 530 00:30:55,950 --> 00:30:59,583 in Linux Service Fabric clusters by default. 531 00:31:00,510 --> 00:31:05,510 After that, I could have a full cluster compromise, 532 00:31:05,550 --> 00:31:09,960 and I actually was able to test this 533 00:31:09,960 --> 00:31:12,690 on other Service Fabrics successfully 534 00:31:12,690 --> 00:31:17,163 and beat the race condition every single time. 535 00:31:19,440 --> 00:31:22,260 So let's talk about the disclosure process. 536 00:31:22,260 --> 00:31:26,760 So we reported this issue to Microsoft 537 00:31:26,760 --> 00:31:29,970 through the Azure bag bounty program, 538 00:31:29,970 --> 00:31:34,410 of course, including a full operational exploit 539 00:31:34,410 --> 00:31:37,830 and for explanation, MSRC acknowledged the issue 540 00:31:37,830 --> 00:31:41,283 and classified it as remote code execution. 541 00:31:42,120 --> 00:31:47,120 And they also awarded us with $30,000 for the bug bounty. 542 00:31:47,400 --> 00:31:50,130 So I could buy a lot of great food for Lizzy 543 00:31:50,130 --> 00:31:51,420 with that money. 544 00:31:51,420 --> 00:31:53,970 And they actually reserved a CVE for us 545 00:31:53,970 --> 00:31:56,400 with a great number, which was so close to release 546 00:31:56,400 --> 00:31:58,743 but I guess next time. 547 00:32:02,580 --> 00:32:03,690 So after collaborating, 548 00:32:03,690 --> 00:32:06,420 they're talking about how we can fix this issue. 549 00:32:06,420 --> 00:32:11,420 Microsoft updated, release a fix with the update 550 00:32:11,730 --> 00:32:16,500 on June 14, 2022. 551 00:32:16,500 --> 00:32:18,930 Now, if you're using Azure Service Fabric 552 00:32:18,930 --> 00:32:21,720 with automatic updates, you don't have any reason to worry 553 00:32:21,720 --> 00:32:25,410 because Microsoft updated your software automatically. 554 00:32:25,410 --> 00:32:27,990 And if you use any of the products that I mentioned 555 00:32:27,990 --> 00:32:30,900 or the Azure offerings that I mentioned, 556 00:32:30,900 --> 00:32:33,900 or just any random stuff, and you are scared, 557 00:32:33,900 --> 00:32:35,310 then you should know that MSRC 558 00:32:35,310 --> 00:32:40,310 contacted all of the internal partners in Microsoft 559 00:32:40,650 --> 00:32:43,944 so they would update their production environments. 560 00:32:43,944 --> 00:32:48,323 So this vulnerability will be fixed, 561 00:32:49,440 --> 00:32:51,690 whether it works or doesn't work 562 00:32:51,690 --> 00:32:53,880 on the specific production environments. 563 00:32:53,880 --> 00:32:58,053 So that's how everything went out. 564 00:32:59,490 --> 00:33:03,330 And there's two takeaways 565 00:33:03,330 --> 00:33:07,350 that I want you to take from my story. 566 00:33:07,350 --> 00:33:11,010 And the first one is that we all have the cloud. 567 00:33:11,010 --> 00:33:14,100 We all have containers, but containers, 568 00:33:14,100 --> 00:33:18,990 isolation is a weak security boundary 569 00:33:18,990 --> 00:33:23,610 because there were so many vulnerabilities in runc 570 00:33:23,610 --> 00:33:28,610 and in Kubernetes, in Docker and just everything 571 00:33:29,100 --> 00:33:32,310 in cloud vendors, for example, we had AzureScape a year ago, 572 00:33:32,310 --> 00:33:34,200 which is a vulnerability that allowed you 573 00:33:34,200 --> 00:33:38,193 to the container installation on Azure Container Instances. 574 00:33:39,120 --> 00:33:41,220 And you know what, even if your code is great 575 00:33:41,220 --> 00:33:43,710 and up to date, and you're really happy, 576 00:33:43,710 --> 00:33:47,790 there could always be a cron vulnerability in your machine 577 00:33:47,790 --> 00:33:52,230 that will allow attacker to gain a full container escape. 578 00:33:52,230 --> 00:33:56,430 So let's take this into consideration, that it is weak 579 00:33:56,430 --> 00:33:59,340 and assume that it'll probably be broken 580 00:33:59,340 --> 00:34:00,663 in some point of time. 581 00:34:01,710 --> 00:34:04,290 And in order to mitigate this issue, 582 00:34:04,290 --> 00:34:06,090 the best approach on this one 583 00:34:06,090 --> 00:34:10,410 is to use multiple layers of security, 584 00:34:10,410 --> 00:34:12,690 which is the castle approach. 585 00:34:12,690 --> 00:34:13,860 So for example, 586 00:34:13,860 --> 00:34:18,120 we have this castle and if an intruder want to get in, 587 00:34:18,120 --> 00:34:22,410 it'll have to go through a river, a bridge, a wall, a door, 588 00:34:22,410 --> 00:34:24,810 a guard, Lizzy, another guard, 589 00:34:24,810 --> 00:34:26,730 another door and just everything. 590 00:34:26,730 --> 00:34:28,950 And even if Lizzy is falling asleep 591 00:34:28,950 --> 00:34:32,070 and the intruder just bypassed her, 592 00:34:32,070 --> 00:34:34,463 then there are also many other layers 593 00:34:34,463 --> 00:34:39,243 that will block the exploitation. 594 00:34:40,230 --> 00:34:41,790 So I guess that's the best way 595 00:34:41,790 --> 00:34:43,320 to do so if you're interested, 596 00:34:43,320 --> 00:34:46,140 I just gave a talk two months ago in open source summit 597 00:34:46,140 --> 00:34:49,950 about container security layers and how not to break them, 598 00:34:49,950 --> 00:34:52,410 how to configure them right. 599 00:34:52,410 --> 00:34:53,280 And other than that, 600 00:34:53,280 --> 00:34:55,440 you could just use many other layers 601 00:34:55,440 --> 00:35:00,393 in order to improve your security and avoid takeovers. 602 00:35:01,470 --> 00:35:04,593 So that was all for today. 603 00:35:05,820 --> 00:35:07,420 So I wanna thank you for coming. 604 00:35:08,910 --> 00:35:10,016 Thank you. 605 00:35:10,016 --> 00:35:13,016 (audience applauds) 606 00:35:16,590 --> 00:35:18,360 If you want, you can find me on this email 607 00:35:18,360 --> 00:35:20,430 and if you want to read some more details, 608 00:35:20,430 --> 00:35:23,460 you have our unit 42 blog post at Palo Alto Networks, 609 00:35:23,460 --> 00:35:27,990 or you could read the MSRC blog post by Microsoft. 610 00:35:27,990 --> 00:35:31,770 If you have any questions, we can do it right now. 611 00:35:31,770 --> 00:35:32,603 Thank you. 612 00:35:33,967 --> 00:35:36,967 (audience applauds)