1 00:00:00,542 --> 00:00:02,010 - We're about to get started. 2 00:00:02,010 --> 00:00:05,190 I'd like to welcome Justin and Samuel. 3 00:00:05,190 --> 00:00:06,300 Welcome to DEFCON. 4 00:00:06,300 --> 00:00:08,280 First time speakers. 5 00:00:08,280 --> 00:00:11,280 They'll be given a speech on Crossing the KASM, 6 00:00:11,280 --> 00:00:14,340 about an exploit that they performed. 7 00:00:14,340 --> 00:00:18,480 And as a reminder, this is a public service announcement. 8 00:00:18,480 --> 00:00:22,050 Watch your schedules online and use Hacker Tracker 9 00:00:22,050 --> 00:00:23,820 to be able to see the latest information 10 00:00:23,820 --> 00:00:25,893 and what's going on. 11 00:00:25,893 --> 00:00:28,050 There's a lot that goes on at DEFCON. 12 00:00:28,050 --> 00:00:31,950 So once again, please give a warm welcome to our speakers. 13 00:00:31,950 --> 00:00:32,783 Welcome guys. 14 00:00:32,783 --> 00:00:36,093 (audience applauding) 15 00:00:36,093 --> 00:00:36,926 - Hello. 16 00:00:36,926 --> 00:00:37,759 (audience clapping) 17 00:00:37,759 --> 00:00:38,592 Hello. 18 00:00:38,592 --> 00:00:39,930 So, I'm Sam. 19 00:00:39,930 --> 00:00:41,159 So, this is Crossing the KASM, 20 00:00:41,159 --> 00:00:42,750 a webapp pentesting story. 21 00:00:42,750 --> 00:00:45,000 So, this talk will get a little bit technical, 22 00:00:45,000 --> 00:00:47,133 but it's worth it in the end, I promise. 23 00:00:48,000 --> 00:00:49,620 So, who am I? 24 00:00:49,620 --> 00:00:51,510 I am Sam, AKA erbbysam. 25 00:00:51,510 --> 00:00:54,060 I'm a hacker and I also have two DEFCON black badges 26 00:00:54,060 --> 00:00:55,290 from a few years back. 27 00:00:55,290 --> 00:00:56,460 And most importantly, 28 00:00:56,460 --> 00:00:58,470 I got a puppy about six months ago. (laughs) 29 00:00:58,470 --> 00:01:00,652 And so there'll be more puppy pictures later, I promise. 30 00:01:00,652 --> 00:01:03,510 (audience talking) 31 00:01:03,510 --> 00:01:06,930 - Nice, and I'm Justin Gardner, AKA Rhynorater. 32 00:01:06,930 --> 00:01:08,730 I'm a full-time bug bounty hunter, 33 00:01:08,730 --> 00:01:11,970 and here are some additional facts about me. 34 00:01:11,970 --> 00:01:14,218 You can find me on Twitter at my handle Rhynorater, 35 00:01:14,218 --> 00:01:16,320 and my DMs are open if you guys have any questions 36 00:01:16,320 --> 00:01:19,773 about the presentation or have any similar interests. 37 00:01:20,970 --> 00:01:24,270 So, when we're dealing with a product called KASM, 38 00:01:24,270 --> 00:01:26,910 we had to do something punny, right. 39 00:01:26,910 --> 00:01:29,850 So, I figured I'd make this little presentation 40 00:01:29,850 --> 00:01:30,870 for you guys here, 41 00:01:30,870 --> 00:01:33,300 representing what it's like to be a bug bounty hunter 42 00:01:33,300 --> 00:01:35,460 on a pretty hardened target. 43 00:01:35,460 --> 00:01:40,170 So, here's Sam and I, in the middle of the desert. 44 00:01:40,170 --> 00:01:42,755 And on the other side of this KASM, 45 00:01:42,755 --> 00:01:47,070 we can see a mirage of RCE in the distance, 46 00:01:47,070 --> 00:01:49,653 but we have to go for it, we have no choice. 47 00:01:50,670 --> 00:01:55,670 And unfortunately, this mirage is separated from us 48 00:01:55,680 --> 00:01:58,350 by a technology called KASM 49 00:01:58,350 --> 00:02:01,740 and represented here by the deep chasm. 50 00:02:01,740 --> 00:02:04,770 This is a VDI software enterprise 51 00:02:04,770 --> 00:02:09,300 for essentially managing what applications 52 00:02:09,300 --> 00:02:12,660 your user can run and in what context they do that. 53 00:02:12,660 --> 00:02:15,180 So, we threw a bunch of exploits at it. (laughs) 54 00:02:15,180 --> 00:02:17,070 Nothing was working, nothing was sticking. 55 00:02:17,070 --> 00:02:21,750 But, in the end, we did find the bridge over this chasm. 56 00:02:21,750 --> 00:02:24,063 And this is that bridge. 57 00:02:25,410 --> 00:02:26,880 And we'll explain that a little bit more, 58 00:02:26,880 --> 00:02:29,280 but all you need to know for now is we just hooked this up 59 00:02:29,280 --> 00:02:31,200 and hopped right on over the KASM 60 00:02:31,200 --> 00:02:34,530 and we made our way to the RCE. 61 00:02:34,530 --> 00:02:35,970 So, I'm gonna give it back to Sam 62 00:02:35,970 --> 00:02:37,440 to give you a little bit more context 63 00:02:37,440 --> 00:02:39,360 on what the heck we're talking about right now 64 00:02:39,360 --> 00:02:41,190 and hopefully it'll make a little bit more sense 65 00:02:41,190 --> 00:02:42,150 in a moment. 66 00:02:42,150 --> 00:02:44,130 - Cool, thanks. 67 00:02:44,130 --> 00:02:45,930 So, for some important background, 68 00:02:45,930 --> 00:02:47,280 before we get into hacking, 69 00:02:48,210 --> 00:02:51,480 so KASM Workspaces is streaming containerized apps 70 00:02:51,480 --> 00:02:52,500 and desktops to end-users. 71 00:02:52,500 --> 00:02:55,770 So, it's really intended for the enterprise world 72 00:02:55,770 --> 00:02:58,410 where you want to administer your end-users 73 00:02:58,410 --> 00:03:02,310 running uBuntu or Chrome or Windows 74 00:03:02,310 --> 00:03:03,960 inside of Docker containers. 75 00:03:03,960 --> 00:03:08,310 And it will deploy them for you and then allow end-users 76 00:03:08,310 --> 00:03:10,953 to connect to them and then you can monitor them, 77 00:03:12,180 --> 00:03:14,340 do various things, enterprisey things with them. 78 00:03:14,340 --> 00:03:16,230 And it's built on top of Docker and VNC. 79 00:03:16,230 --> 00:03:17,400 And so it'd give you an idea 80 00:03:17,400 --> 00:03:21,060 of just how Dockerized this world is. 81 00:03:21,060 --> 00:03:22,110 It's turtles all the way down. 82 00:03:22,110 --> 00:03:23,970 It's Docker containers all the way down. 83 00:03:23,970 --> 00:03:26,070 And I apologize for putting so much information on a slide, 84 00:03:26,070 --> 00:03:28,140 but I'll give a visualization in a minute 85 00:03:28,140 --> 00:03:29,670 of what all this looks like. 86 00:03:29,670 --> 00:03:30,930 And so at the bottom of this list, 87 00:03:30,930 --> 00:03:33,270 you can actually see the uBuntu desktop. 88 00:03:33,270 --> 00:03:35,610 The uBuntu Docker container that I deployed. 89 00:03:35,610 --> 00:03:36,690 And at the top of this list, 90 00:03:36,690 --> 00:03:40,050 you can actually see the NGINX container that I deployed... 91 00:03:40,050 --> 00:03:43,893 Sorry, the NGINX KASM proxy. 92 00:03:45,150 --> 00:03:47,610 And so the KASM proxy is actually listening remotely. 93 00:03:47,610 --> 00:03:48,990 And then every other Docker container 94 00:03:48,990 --> 00:03:51,300 is listening on local ports. 95 00:03:51,300 --> 00:03:53,880 And if you're not familiar with Docker, 96 00:03:53,880 --> 00:03:56,040 you can kind of think of it as a virtual machine, 97 00:03:56,040 --> 00:03:57,750 but instead of running on bare metal, 98 00:03:57,750 --> 00:03:59,130 it runs on top of the operating system. 99 00:03:59,130 --> 00:04:01,350 So, it's really just kind of one layer up. 100 00:04:01,350 --> 00:04:03,750 And then it has a bunch of access 101 00:04:03,750 --> 00:04:05,490 to underlying operating system, 102 00:04:05,490 --> 00:04:07,343 depending on what permissions you give it. 103 00:04:08,250 --> 00:04:10,710 And so diving into a little more detail 104 00:04:10,710 --> 00:04:12,270 into that KASM proxy, 105 00:04:12,270 --> 00:04:14,430 when you actually spin up a new container, 106 00:04:14,430 --> 00:04:17,310 a new NGINX configuration 107 00:04:17,310 --> 00:04:20,160 actually gets pushed to the KASM proxy. 108 00:04:20,160 --> 00:04:22,500 And so this container is our D folder 109 00:04:22,500 --> 00:04:24,450 will be updated with a new file. 110 00:04:24,450 --> 00:04:25,860 And so when you push a new container, 111 00:04:25,860 --> 00:04:27,900 this folder gets updated 112 00:04:27,900 --> 00:04:30,120 and then NGINX actually gets restarted. 113 00:04:30,120 --> 00:04:31,860 And then the end-user can connect 114 00:04:31,860 --> 00:04:33,710 to the newly formed Docker container. 115 00:04:36,210 --> 00:04:39,400 And throughout this talk, we'll try and share at the bottom 116 00:04:39,400 --> 00:04:42,120 the commands we ran as well as references 117 00:04:42,120 --> 00:04:45,420 if you wanna repeat this or learn more about this. 118 00:04:45,420 --> 00:04:47,853 And so at a high level, 119 00:04:48,750 --> 00:04:51,030 if you deploy KASM workspaces onto a single server, 120 00:04:51,030 --> 00:04:52,950 this is what it'll actually look like. 121 00:04:52,950 --> 00:04:54,840 Your end-users will connect through, as well as you, 122 00:04:54,840 --> 00:04:57,060 will connect through KASM proxy, 123 00:04:57,060 --> 00:04:59,670 and they'll be able to connect to their end-user containers. 124 00:04:59,670 --> 00:05:01,140 So, every blue square here 125 00:05:01,140 --> 00:05:03,720 actually represents a Docker container. 126 00:05:03,720 --> 00:05:04,830 And in the bottom row 127 00:05:04,830 --> 00:05:08,580 are the kind of command and control containers 128 00:05:08,580 --> 00:05:13,020 that are used to manage the rest of the system. 129 00:05:13,020 --> 00:05:15,150 And they're all built on top of Python. 130 00:05:15,150 --> 00:05:17,523 It's either gonna be Tornado or CherryPy. 131 00:05:19,050 --> 00:05:21,390 And so these are the purpose of all the containers, 132 00:05:21,390 --> 00:05:23,070 the most important one being KASM proxy 133 00:05:23,070 --> 00:05:24,570 with route cellular traffic 134 00:05:24,570 --> 00:05:25,680 and then KASM API, 135 00:05:25,680 --> 00:05:28,710 which is actually the API you're interacting with 136 00:05:28,710 --> 00:05:31,020 when you're using the application. 137 00:05:31,020 --> 00:05:32,370 KASM agent is also important 138 00:05:32,370 --> 00:05:34,530 because that actually has the ability to, 139 00:05:34,530 --> 00:05:36,450 and permission to deploy new containers. 140 00:05:36,450 --> 00:05:37,997 So it's a privileged Dock container 141 00:05:37,997 --> 00:05:39,347 in the Docker (indistinct). 142 00:05:40,980 --> 00:05:42,030 With multiple servers, 143 00:05:42,030 --> 00:05:45,420 you actually end up with a series of secondary, 144 00:05:45,420 --> 00:05:46,740 or when you wanna deploy this 145 00:05:46,740 --> 00:05:48,300 into a real enterprise environment, 146 00:05:48,300 --> 00:05:49,470 when you have multiple users, 147 00:05:49,470 --> 00:05:51,570 you wanna deploy it across multiple machines. 148 00:05:51,570 --> 00:05:53,730 And so, in order to accomplish this, 149 00:05:53,730 --> 00:05:56,009 it actually will, on the primary server, 150 00:05:56,009 --> 00:05:57,813 it'll keep KASM API. 151 00:05:57,813 --> 00:06:00,240 And then KASM proxy will proxy through 152 00:06:00,240 --> 00:06:01,860 to the secondary servers 153 00:06:01,860 --> 00:06:04,380 to run KASM agent, 154 00:06:04,380 --> 00:06:06,390 which will then deploy your containers 155 00:06:06,390 --> 00:06:08,040 or your end-user containers. 156 00:06:08,040 --> 00:06:11,043 And this allows you to scale infinitely. 157 00:06:12,960 --> 00:06:16,560 And so, very high level hand wavy way, 158 00:06:16,560 --> 00:06:19,530 this is roughly what credentials look like in the system. 159 00:06:19,530 --> 00:06:23,640 Your end-user session tokens will get past the KASM API 160 00:06:23,640 --> 00:06:24,473 as a cookie. 161 00:06:24,473 --> 00:06:27,270 It'll also get passed through to their containers. 162 00:06:27,270 --> 00:06:29,880 And then KASM API will interact with the rest 163 00:06:29,880 --> 00:06:31,500 of the system using secret keys, 164 00:06:31,500 --> 00:06:35,103 which are all internal to the KASM system. 165 00:06:37,470 --> 00:06:40,680 So, at this point, we were given user credentials 166 00:06:40,680 --> 00:06:41,670 to a bug bounty program, 167 00:06:41,670 --> 00:06:42,990 and we were asked to look at this 168 00:06:42,990 --> 00:06:45,600 without source code access. 169 00:06:45,600 --> 00:06:47,790 We didn't find any vulnerabilities. 170 00:06:47,790 --> 00:06:50,160 This was actually really secure software. 171 00:06:50,160 --> 00:06:51,840 And this talk wouldn't be here today 172 00:06:51,840 --> 00:06:53,130 if this was insecure software, 173 00:06:53,130 --> 00:06:55,500 it would be a very boring talk. 174 00:06:55,500 --> 00:06:56,880 So, at this point, 175 00:06:56,880 --> 00:07:01,880 we given up looking at this without source code, 176 00:07:01,950 --> 00:07:04,163 and we wanted to start testing this with source code. 177 00:07:05,310 --> 00:07:06,240 And so before we did that, 178 00:07:06,240 --> 00:07:09,240 we actually stopped at their issue tracker 179 00:07:09,240 --> 00:07:10,073 and just made sure 180 00:07:10,073 --> 00:07:12,360 that nobody else reported any security issues first. 181 00:07:12,360 --> 00:07:13,890 Not really finding anything, 182 00:07:13,890 --> 00:07:14,853 we moved on. 183 00:07:17,040 --> 00:07:18,300 So, we went to their website 184 00:07:18,300 --> 00:07:19,680 to see if we could download their software 185 00:07:19,680 --> 00:07:21,030 and we came across this page. 186 00:07:21,030 --> 00:07:23,010 And this page is the most exciting thing you can see 187 00:07:23,010 --> 00:07:24,360 as a bug bounty hunter. 188 00:07:24,360 --> 00:07:26,010 I'll add some arrows to show why. 189 00:07:26,010 --> 00:07:28,380 I'll add a few more arrows. 190 00:07:28,380 --> 00:07:30,900 This button means that you don't have to call sales, 191 00:07:30,900 --> 00:07:32,730 You don't have to put your credit card down 192 00:07:32,730 --> 00:07:34,560 for some hopes of finding some bug in the future. 193 00:07:34,560 --> 00:07:36,870 You can just download software and see how it works. 194 00:07:36,870 --> 00:07:38,370 And as you can see in the community addition, 195 00:07:38,370 --> 00:07:39,750 it has all features in enterprise, 196 00:07:39,750 --> 00:07:41,550 which is exactly what you wanna see. 197 00:07:42,690 --> 00:07:44,730 So, at this point, 198 00:07:44,730 --> 00:07:46,710 we went through and installed this onto a server. 199 00:07:46,710 --> 00:07:49,560 And when we looked at the installation process, 200 00:07:49,560 --> 00:07:51,870 we saw it was actually downloading from Docker hub 201 00:07:51,870 --> 00:07:54,960 and installing the Dock containers that way. 202 00:07:54,960 --> 00:07:55,950 We then played the game 203 00:07:55,950 --> 00:07:57,960 of "find the API that you're calling." 204 00:07:57,960 --> 00:07:58,890 And, in this case, (laughs) 205 00:07:58,890 --> 00:08:01,530 we picked one called "API get user attributes." 206 00:08:01,530 --> 00:08:03,630 And we looked through all of the servers that were running, 207 00:08:03,630 --> 00:08:06,180 and we were able to find it in the KASM API binary. 208 00:08:07,260 --> 00:08:10,773 So, we copied the binary out of that Docker container. 209 00:08:12,870 --> 00:08:15,570 This is the proper way to reverse it. 210 00:08:15,570 --> 00:08:17,550 The actual way that we went about it 211 00:08:17,550 --> 00:08:19,380 was running strings on the binary 212 00:08:19,380 --> 00:08:22,080 and then Googling a whole bunch of stuff. 213 00:08:22,080 --> 00:08:24,960 But this is the actual way that you're, (laughs) 214 00:08:24,960 --> 00:08:26,970 this is the actual way that you're supposed to identify 215 00:08:26,970 --> 00:08:29,490 that it's running as a Python binary. 216 00:08:29,490 --> 00:08:31,200 You can then identify the version, 217 00:08:31,200 --> 00:08:33,540 drop into a Python environment, 218 00:08:33,540 --> 00:08:35,490 extract the Python files, 219 00:08:35,490 --> 00:08:36,481 compile Python files, 220 00:08:36,481 --> 00:08:39,270 and then decompile them using uncompile6. 221 00:08:39,270 --> 00:08:40,230 The steps are listed here, 222 00:08:40,230 --> 00:08:42,180 in case you wanna repeat this yourself. 223 00:08:43,170 --> 00:08:46,620 We then dove into client underscore API dot P-Y 224 00:08:46,620 --> 00:08:49,530 and we were able to find our get user attributes function. 225 00:08:49,530 --> 00:08:51,546 And this actually gave, well, 226 00:08:51,546 --> 00:08:55,230 this concluded our game of "find the endpoint." 227 00:08:55,230 --> 00:08:57,450 And now we can actually do the real security work 228 00:08:57,450 --> 00:08:58,980 of looking at all the endpoints 229 00:08:58,980 --> 00:09:00,430 for security vulnerabilities. 230 00:09:02,010 --> 00:09:04,380 And so at this point, I'd actually given up almost entirely. 231 00:09:04,380 --> 00:09:05,467 And then Justin came along and said, 232 00:09:05,467 --> 00:09:09,510 "Hey, there's when you create a new KASM, a workspace, 233 00:09:09,510 --> 00:09:12,360 there's a host header that gets passed through 234 00:09:12,360 --> 00:09:13,830 as one of the inputs." 235 00:09:13,830 --> 00:09:17,100 And so we were a little bit curious to see how that worked. 236 00:09:17,100 --> 00:09:19,080 So, like a good bug bounty hunter, 237 00:09:19,080 --> 00:09:23,310 we spun a Burp Collaborator and we got callbacks. 238 00:09:23,310 --> 00:09:25,740 And so we immediately thought this was a crit 239 00:09:25,740 --> 00:09:27,300 and game over. 240 00:09:27,300 --> 00:09:29,310 We found the coolest vulnerability ever. 241 00:09:29,310 --> 00:09:31,140 And, unfortunately, 242 00:09:31,140 --> 00:09:33,810 we were only getting our own session token back. 243 00:09:33,810 --> 00:09:36,150 So, it's not really a credit. 244 00:09:36,150 --> 00:09:39,720 It's more like a really limited blind SSRF 245 00:09:39,720 --> 00:09:41,070 from a security standpoint. 246 00:09:41,070 --> 00:09:43,740 It's not super interesting. 247 00:09:43,740 --> 00:09:45,780 And so we obviously wanted 248 00:09:45,780 --> 00:09:47,940 to maximize payout as bug bounty hunters. 249 00:09:47,940 --> 00:09:49,440 So, we decided to dive deeper. 250 00:09:50,610 --> 00:09:51,933 So, now's hacky time. 251 00:09:54,120 --> 00:09:54,953 - Alrighty. 252 00:09:54,953 --> 00:09:57,120 So, let's look a little bit more into this. 253 00:09:57,120 --> 00:09:58,860 We've seen so far how we were able 254 00:09:58,860 --> 00:10:00,480 to reverse engineer the software, 255 00:10:00,480 --> 00:10:02,520 get access to the code, 256 00:10:02,520 --> 00:10:04,650 and now we're gonna start finding some vulnerabilities 257 00:10:04,650 --> 00:10:07,560 and figuring out why we're getting that callback. 258 00:10:07,560 --> 00:10:09,410 So, as we mentioned to the beginning, 259 00:10:11,989 --> 00:10:14,850 (laughs) the configuration for NGINX for this software 260 00:10:14,850 --> 00:10:16,620 is a little bit strange. 261 00:10:16,620 --> 00:10:18,180 Whenever a container is spun up, 262 00:10:18,180 --> 00:10:20,010 there is a new configuration file 263 00:10:20,010 --> 00:10:24,240 injected into this containers dot D directory. 264 00:10:24,240 --> 00:10:27,420 And the callbacks that we were seeing from this 265 00:10:27,420 --> 00:10:32,420 was other parts of the system hitting these internal routes 266 00:10:32,520 --> 00:10:36,570 that they created to communicate with our new API. 267 00:10:36,570 --> 00:10:38,610 We can see that here in the proxy pass definition 268 00:10:38,610 --> 00:10:41,613 for both of the locations defined here. 269 00:10:42,450 --> 00:10:45,300 So, we had something interesting here. 270 00:10:45,300 --> 00:10:46,860 After playing around with it a little bit more, 271 00:10:46,860 --> 00:10:50,100 we realized that it would actually leak the session token 272 00:10:50,100 --> 00:10:53,670 of anybody who visited the page 273 00:10:53,670 --> 00:10:58,170 where the thumbnail of this container was displayed. 274 00:10:58,170 --> 00:11:01,500 So, if an admin came and they viewed that thumbnail, 275 00:11:01,500 --> 00:11:02,700 then their session would get leaked. 276 00:11:02,700 --> 00:11:05,220 So, that made us feel a little bit better, right? 277 00:11:05,220 --> 00:11:07,590 'Cause we have a vulnerability 278 00:11:07,590 --> 00:11:09,870 that we've definitely got secured. 279 00:11:09,870 --> 00:11:11,670 But, like Sam said, 280 00:11:11,670 --> 00:11:16,050 and my lovely desert graphic in the beginning showed, 281 00:11:16,050 --> 00:11:19,290 we saw RC in the distance and we wanted to follow it. 282 00:11:19,290 --> 00:11:21,030 So, we continued. 283 00:11:21,030 --> 00:11:26,030 So, looking back at this configuration right here, 284 00:11:26,370 --> 00:11:28,410 we thought about what other things we could do. 285 00:11:28,410 --> 00:11:32,010 And, of course, when you're injecting into a proxy pass, 286 00:11:32,010 --> 00:11:35,280 most people would think of SSRF. 287 00:11:35,280 --> 00:11:36,450 We already had blind SSF. 288 00:11:36,450 --> 00:11:38,790 Let's see if we can get it to be full read SSRF. 289 00:11:38,790 --> 00:11:40,290 So, the way that proxy pass works 290 00:11:40,290 --> 00:11:43,980 is when we passed this host header 291 00:11:43,980 --> 00:11:47,073 that we were able to inject from the command, 292 00:11:48,309 --> 00:11:50,100 the HTTP request we did before, 293 00:11:50,100 --> 00:11:51,540 we just changed the host header, 294 00:11:51,540 --> 00:11:54,300 that got injected right into the proxy pass here. 295 00:11:54,300 --> 00:11:55,740 And if we could hit these routes, 296 00:11:55,740 --> 00:11:57,664 then we could hit any arbitrary host 297 00:11:57,664 --> 00:12:00,960 from the perspective of the server, right? 298 00:12:00,960 --> 00:12:03,630 And in today's modern cloud environments, 299 00:12:03,630 --> 00:12:08,630 this normally results in at least something a little fun, 300 00:12:08,670 --> 00:12:11,550 if not full RC or takeover of the (indistinct) account 301 00:12:11,550 --> 00:12:13,410 that this is hosted on. 302 00:12:13,410 --> 00:12:16,290 So, we thought this would be a good target to go for, 303 00:12:16,290 --> 00:12:18,510 but we ran into some issues here. 304 00:12:18,510 --> 00:12:21,120 Yeah, so the goal would be full read SSRF. 305 00:12:21,120 --> 00:12:22,350 We ran into some issues here, 306 00:12:22,350 --> 00:12:24,600 hitting these NGINX configurations. 307 00:12:24,600 --> 00:12:26,400 The first one, as you can probably see, 308 00:12:26,400 --> 00:12:31,320 is the first entry here has the internal directive. 309 00:12:31,320 --> 00:12:34,440 The internal directive and NGINX only allows this route 310 00:12:34,440 --> 00:12:37,590 to be routed by internal redirects 311 00:12:37,590 --> 00:12:39,330 or other internal requests. 312 00:12:39,330 --> 00:12:40,163 So, there was no way 313 00:12:40,163 --> 00:12:41,973 that we could possibly hit that endpoint. 314 00:12:43,290 --> 00:12:45,840 The second one, we also thought looked great, 315 00:12:45,840 --> 00:12:50,280 but that GUID is not leaked to the user in any way, 316 00:12:50,280 --> 00:12:51,113 shape or form. 317 00:12:51,113 --> 00:12:52,770 It just gets pulled straight out of the database 318 00:12:52,770 --> 00:12:55,260 and used by other pieces of the API. 319 00:12:55,260 --> 00:12:58,020 And we couldn't leak it to ourselves 320 00:12:58,020 --> 00:13:02,880 because it's being overwritten by mod rewrite there. 321 00:13:02,880 --> 00:13:05,733 So, that resulted in a big sad. 322 00:13:06,600 --> 00:13:08,520 So, we had to think a little bit more 323 00:13:08,520 --> 00:13:10,290 about what we were gonna do. 324 00:13:10,290 --> 00:13:13,320 So next, we noticed that there are some variables 325 00:13:13,320 --> 00:13:16,230 that we can access within NGINX configuration file. 326 00:13:16,230 --> 00:13:18,120 The remote address is one of them 327 00:13:18,120 --> 00:13:19,601 and, actually, 328 00:13:19,601 --> 00:13:22,860 NGINX provides a really nice (laughs) alphabetized list 329 00:13:22,860 --> 00:13:25,200 of variables that we could go through. 330 00:13:25,200 --> 00:13:26,130 So, we went through, 331 00:13:26,130 --> 00:13:27,690 I don't know if you can see it in the screenshot there, 332 00:13:27,690 --> 00:13:31,380 but all 222 of those variables 333 00:13:31,380 --> 00:13:34,380 and there's nothing interesting in there, at all, 334 00:13:34,380 --> 00:13:35,610 not even one. 335 00:13:35,610 --> 00:13:38,190 So, we went through all of that 336 00:13:38,190 --> 00:13:41,043 and couldn't find anything fun to do with that. 337 00:13:42,270 --> 00:13:43,297 But, then we started thinking, 338 00:13:43,297 --> 00:13:46,980 "Okay, variables. What about environmental variables? 339 00:13:46,980 --> 00:13:48,720 Could we leak environmental variables 340 00:13:48,720 --> 00:13:52,140 from the host machine?" 341 00:13:52,140 --> 00:13:54,960 So we started, like any good hackers, 342 00:13:54,960 --> 00:13:58,230 we Googled how to get (laughs) access 343 00:13:58,230 --> 00:14:01,380 to environmental variables in NGINX configuration, 344 00:14:01,380 --> 00:14:06,380 and this lovely Stack Overflow article popped up. 345 00:14:06,420 --> 00:14:10,770 And it mentioned running arbitrary Lua and Pearl 346 00:14:10,770 --> 00:14:13,080 in the context of a NGINX configuration file. 347 00:14:13,080 --> 00:14:16,230 So, don't mind if we do. 348 00:14:16,230 --> 00:14:19,590 So, we decided to quickly pivot and go down that path 349 00:14:19,590 --> 00:14:21,090 and 350 00:14:21,090 --> 00:14:23,370 yeah no, it's not gonna be that easy for us, 351 00:14:23,370 --> 00:14:24,664 unfortunately. 352 00:14:24,664 --> 00:14:27,660 There is no Pearl or... 353 00:14:27,660 --> 00:14:29,940 I mean, there is no Lua or JS modules loaded. 354 00:14:29,940 --> 00:14:32,190 And even though they did tease us a little bit 355 00:14:32,190 --> 00:14:37,190 with defining a Pearl's module path in the NGINX command, 356 00:14:37,200 --> 00:14:39,660 that path did not exist. 357 00:14:39,660 --> 00:14:41,580 So, there was no Pearl modules being loaded 358 00:14:41,580 --> 00:14:44,340 and we couldn't use the Pearl modules. 359 00:14:44,340 --> 00:14:47,220 So, we're back to the drawing board 360 00:14:47,220 --> 00:14:48,397 and we're trying to come up and we're like, 361 00:14:48,397 --> 00:14:49,230 "Okay, well, 362 00:14:49,230 --> 00:14:52,110 maybe we can modify this NGINX configuration file 363 00:14:52,110 --> 00:14:53,757 to do something funky." 364 00:14:55,320 --> 00:14:56,179 So we start, 365 00:14:56,179 --> 00:14:58,140 this is the vulnerable request, 366 00:14:58,140 --> 00:15:02,910 we start injecting valid, or what we thought was valid, 367 00:15:02,910 --> 00:15:06,600 NGINX syntax into the host header and, big surprise, 368 00:15:06,600 --> 00:15:10,920 NGINX fussed at us and said the front-end proxy, 369 00:15:10,920 --> 00:15:12,570 from Sam's explanation earlier, 370 00:15:12,570 --> 00:15:14,970 was not accepting that host header 371 00:15:14,970 --> 00:15:17,703 because it had spaces and slashes in it. 372 00:15:18,600 --> 00:15:21,390 So, this is getting more and more, 373 00:15:21,390 --> 00:15:22,830 this is getting more and more difficult, right? 374 00:15:22,830 --> 00:15:23,670 So, what's the sitch here? 375 00:15:23,670 --> 00:15:24,503 Just to summarize. 376 00:15:24,503 --> 00:15:27,180 We have a blind SSRF, 377 00:15:27,180 --> 00:15:30,300 via NGINX configuration file injection. 378 00:15:30,300 --> 00:15:33,180 We've got callbacks that contain the victim's cookies, 379 00:15:33,180 --> 00:15:36,330 but we kind of wanted to avoid any social engineering, 380 00:15:36,330 --> 00:15:39,390 'cause this is bug bounty and that's how it is. 381 00:15:39,390 --> 00:15:43,230 And we can't get full read SSRF 382 00:15:43,230 --> 00:15:44,460 because of the internal directive 383 00:15:44,460 --> 00:15:46,140 and we couldn't find the GUID. 384 00:15:46,140 --> 00:15:47,790 There were no interesting variables to leak, 385 00:15:47,790 --> 00:15:50,010 no access to code modules 386 00:15:50,010 --> 00:15:53,910 and that we couldn't use any spaces or slashes 387 00:15:53,910 --> 00:15:55,110 in the host header. 388 00:15:55,110 --> 00:15:58,710 So, this was looking really rough. 389 00:15:58,710 --> 00:16:01,230 We were feeling down at this point. (laughs) 390 00:16:01,230 --> 00:16:03,723 And then Sam has this lovely idea. 391 00:16:04,860 --> 00:16:05,693 I'm gonna read. 392 00:16:05,693 --> 00:16:07,620 "I'm going to try and call the API directly 393 00:16:07,620 --> 00:16:09,720 from within the container and see if Python is okay 394 00:16:09,720 --> 00:16:11,220 with the spaces in the host header," 395 00:16:11,220 --> 00:16:12,480 which was a lovely idea. 396 00:16:12,480 --> 00:16:14,640 So, essentially what he's doing here, 397 00:16:14,640 --> 00:16:17,340 is he's using those end-user containers 398 00:16:17,340 --> 00:16:18,630 that we're able to spin up 399 00:16:18,630 --> 00:16:20,250 and he's trying to communicate directly 400 00:16:20,250 --> 00:16:22,890 with the KASM API from within the containers. 401 00:16:22,890 --> 00:16:26,910 So, he spun up a nice uBuntu box here and boom, 402 00:16:26,910 --> 00:16:28,350 we could reach the internal API 403 00:16:28,350 --> 00:16:30,270 without NGINX getting in our way 404 00:16:30,270 --> 00:16:31,290 and, low and behold, 405 00:16:31,290 --> 00:16:34,290 CherryPy is a little bit less finicky 406 00:16:34,290 --> 00:16:35,340 about their host headers 407 00:16:35,340 --> 00:16:37,767 and will accept pretty much whatever you throw at it 408 00:16:37,767 --> 00:16:38,820 and a little extra, 409 00:16:38,820 --> 00:16:40,830 which we'll find out later. 410 00:16:40,830 --> 00:16:41,663 So, that was great. 411 00:16:41,663 --> 00:16:43,080 We thought we had a way 412 00:16:43,080 --> 00:16:45,780 to inject into NGINX configurations 413 00:16:45,780 --> 00:16:47,730 and we proceeded to write up an exploit 414 00:16:47,730 --> 00:16:49,500 and take it to prod. 415 00:16:49,500 --> 00:16:50,580 And 416 00:16:50,580 --> 00:16:51,660 Nope. 417 00:16:51,660 --> 00:16:55,710 The prod was running the multiple server configuration 418 00:16:55,710 --> 00:16:58,290 and we cannot access the KASM API. 419 00:16:58,290 --> 00:16:59,123 So, at this point, 420 00:16:59,123 --> 00:17:01,320 Sam and I are about to lose our minds 421 00:17:01,320 --> 00:17:02,430 and we've been going back and forth 422 00:17:02,430 --> 00:17:04,620 on this vulnerability for a while 423 00:17:04,620 --> 00:17:06,120 and we were stuck. 424 00:17:06,120 --> 00:17:07,530 We were really stuck. 425 00:17:07,530 --> 00:17:09,210 So, we tried a bunch of things here. 426 00:17:09,210 --> 00:17:11,790 We tried HTTP request smuggling, 427 00:17:11,790 --> 00:17:13,020 We (laughs) dove in... 428 00:17:13,020 --> 00:17:15,120 I don't know why we were so obsessed 429 00:17:15,120 --> 00:17:16,080 with this vulnerability, 430 00:17:16,080 --> 00:17:17,670 but (laughs) we just really were. 431 00:17:17,670 --> 00:17:19,770 And we started reading the NGINX source. 432 00:17:19,770 --> 00:17:21,371 When you start reading NGINX source code, 433 00:17:21,371 --> 00:17:23,490 you know you're in trouble. 434 00:17:23,490 --> 00:17:26,250 So, we started (laughs) reading the NGINX source code 435 00:17:26,250 --> 00:17:28,980 and we couldn't find anything in the host header, 436 00:17:28,980 --> 00:17:31,890 parsing logic or anything like that there. 437 00:17:31,890 --> 00:17:34,870 But then, we repeated this process for CherryPy 438 00:17:36,810 --> 00:17:39,240 and we found something really interesting, 439 00:17:39,240 --> 00:17:44,240 regarding ISO 88 59 dash one headers, 440 00:17:44,790 --> 00:17:46,950 which I had never heard of before. 441 00:17:46,950 --> 00:17:48,540 So, upon finding this, 442 00:17:48,540 --> 00:17:51,780 I sent a frantic message to Sam (laughs) 443 00:17:51,780 --> 00:17:54,307 and I think you can see above, me saying, 444 00:17:54,307 --> 00:17:55,350 "Man, this bug," 445 00:17:55,350 --> 00:17:56,287 and then 15 minutes later, 446 00:17:56,287 --> 00:17:57,480 "Dude, I've got it." 447 00:17:57,480 --> 00:18:01,631 So, you can kind of feel the bug bounty hunter drain 448 00:18:01,631 --> 00:18:05,610 right there and the lows and the highs, right? 449 00:18:05,610 --> 00:18:09,753 And so let me explain a little bit about ISO 88 59 dash one. 450 00:18:11,100 --> 00:18:13,830 If that looks familiar to you for some reason, 451 00:18:13,830 --> 00:18:17,760 that's because it's used often in SMTP headers. 452 00:18:17,760 --> 00:18:20,790 So, if you've ever clicked see original in your emails, 453 00:18:20,790 --> 00:18:23,910 inside of Gmail, 454 00:18:23,910 --> 00:18:26,070 then you be looking at something like this. 455 00:18:26,070 --> 00:18:29,490 And one of the things that neither of us knew, 456 00:18:29,490 --> 00:18:31,800 and that I think is not very widely known, 457 00:18:31,800 --> 00:18:33,990 it's that headers are actually supposed, 458 00:18:33,990 --> 00:18:35,760 according to the original RFC, 459 00:18:35,760 --> 00:18:37,530 are supposed to be able to be parsed 460 00:18:37,530 --> 00:18:39,210 via 88 59 dash one, 461 00:18:39,210 --> 00:18:41,493 ISO 88 59 dash one encoding. 462 00:18:42,840 --> 00:18:45,450 So, that was exactly what we needed. 463 00:18:45,450 --> 00:18:47,100 And the CherryPy code, 464 00:18:47,100 --> 00:18:48,930 you can see up at the top-right, 465 00:18:48,930 --> 00:18:51,210 says, "Yeah, if there's an equal sign 466 00:18:51,210 --> 00:18:53,250 and then a question mark in the beginning, 467 00:18:53,250 --> 00:18:55,200 let's go ahead and try to decode this value 468 00:18:55,200 --> 00:18:59,640 as ISO 88 59 dash one." 469 00:18:59,640 --> 00:19:00,810 So, the format of this 470 00:19:00,810 --> 00:19:03,210 is actually very similar to URL encoding, 471 00:19:03,210 --> 00:19:07,860 which is super convenient for encoding web vulnerability 472 00:19:07,860 --> 00:19:09,690 or web payloads. 473 00:19:09,690 --> 00:19:12,030 And we can do something similar here. 474 00:19:12,030 --> 00:19:14,820 We have to define this little header in the beginning, 475 00:19:14,820 --> 00:19:19,530 the 88 59 dash one question mark Q question mark part, 476 00:19:19,530 --> 00:19:21,360 and then, essentially, we can base 60 477 00:19:21,360 --> 00:19:24,000 or URL encode our payload, 478 00:19:24,000 --> 00:19:27,450 and then just replace the percentage signs with equal signs 479 00:19:27,450 --> 00:19:29,610 and you get a fairly close 480 00:19:29,610 --> 00:19:34,590 to valid ISO 88 59 dash one format. 481 00:19:34,590 --> 00:19:37,500 So, I think this is one of the takeaways from this talk 482 00:19:37,500 --> 00:19:39,870 is that if you're having trouble smuggling headers 483 00:19:39,870 --> 00:19:42,000 through some sort of intermediary proxy, 484 00:19:42,000 --> 00:19:46,290 whether it be a WAF or a proxy in our scenario, 485 00:19:46,290 --> 00:19:47,820 this could be some way to bypass it 486 00:19:47,820 --> 00:19:50,550 if any of your web servers are, in the stack, 487 00:19:50,550 --> 00:19:52,680 are accepting in decoding 488 00:19:52,680 --> 00:19:55,083 or normalizing this ISO 80 59 value. 489 00:19:56,250 --> 00:19:57,180 So, you can see right here, 490 00:19:57,180 --> 00:20:00,390 just the character-to-character translation 491 00:20:00,390 --> 00:20:04,530 of what it looks like for that to be decoded. 492 00:20:04,530 --> 00:20:06,840 And the value that you see in red 493 00:20:06,840 --> 00:20:09,330 is what is gonna be passed to CherryPy in the end, 494 00:20:09,330 --> 00:20:10,630 when the header is parsed. 495 00:20:11,790 --> 00:20:12,623 And over to Sam. 496 00:20:12,623 --> 00:20:13,645 - Cool. 497 00:20:13,645 --> 00:20:16,360 So, at this point, 498 00:20:16,360 --> 00:20:19,200 we have a way to get through it, the KASM proxy, 499 00:20:19,200 --> 00:20:21,450 and we have a way to inject host headers. 500 00:20:21,450 --> 00:20:23,340 So, we're almost there, 501 00:20:23,340 --> 00:20:27,180 but we first have to get through NGINX configuration. 502 00:20:27,180 --> 00:20:28,290 And so here, 503 00:20:28,290 --> 00:20:30,300 what we're actually injecting into is we're not... 504 00:20:30,300 --> 00:20:32,280 We're injecting into multiple places with the same string. 505 00:20:32,280 --> 00:20:35,160 So, we're injecting into two proxy pass directives 506 00:20:35,160 --> 00:20:37,140 and then four add header directives 507 00:20:37,140 --> 00:20:41,190 where the string is wrapped in single quote characters. 508 00:20:41,190 --> 00:20:44,884 And so this was something that we had to bypass. 509 00:20:44,884 --> 00:20:47,700 And our end goal here really was to add our own location 510 00:20:47,700 --> 00:20:50,250 so that we could add arbitrary NGINX configuration. 511 00:20:51,180 --> 00:20:54,240 And so we tried adding to a new location point, 512 00:20:54,240 --> 00:20:56,040 to the top two injection points, 513 00:20:56,040 --> 00:20:59,520 and we didn't use any single quote characters 514 00:20:59,520 --> 00:21:01,380 so that it would be okay 515 00:21:01,380 --> 00:21:03,900 for the remaining four injection spots. 516 00:21:03,900 --> 00:21:05,100 Unfortunately, this doesn't work 517 00:21:05,100 --> 00:21:08,220 because when NGINX sees a duplicate location, 518 00:21:08,220 --> 00:21:09,810 it rejects the configuration. 519 00:21:09,810 --> 00:21:11,580 And so that didn't work. 520 00:21:11,580 --> 00:21:14,943 We got this error, duplicate location slash new. 521 00:21:15,840 --> 00:21:19,020 So, we then tried something a little bit more creative, 522 00:21:19,020 --> 00:21:23,040 where we actually used some quote magic, 523 00:21:23,040 --> 00:21:24,000 as how I'll describe it, 524 00:21:24,000 --> 00:21:27,660 to inject into the top two positions using quotes 525 00:21:27,660 --> 00:21:30,060 and then a comment to break out successfully. 526 00:21:30,060 --> 00:21:32,460 And then, on the last four injection points, 527 00:21:32,460 --> 00:21:33,840 because the single quotes would take priority 528 00:21:33,840 --> 00:21:37,650 over the double quotes, we won't escape from that string, 529 00:21:37,650 --> 00:21:39,060 this should have worked 530 00:21:39,060 --> 00:21:40,760 except for there's one evil dollar sign 531 00:21:40,760 --> 00:21:41,670 in a regular expression, 532 00:21:41,670 --> 00:21:44,010 right in the middle of where were trying to inject into 533 00:21:44,010 --> 00:21:46,950 and NGINX rejected this because a single dollar sign 534 00:21:46,950 --> 00:21:49,373 is an invalid variable name in NGINX. 535 00:21:50,700 --> 00:21:52,200 There, as an aside, 536 00:21:52,200 --> 00:21:54,420 there is actually a solution to this 537 00:21:54,420 --> 00:21:58,470 where you can put this geo dollar default dollar sign 538 00:21:58,470 --> 00:21:59,910 into your NGINX configuration, 539 00:21:59,910 --> 00:22:01,500 but it has to be done at a higher level 540 00:22:01,500 --> 00:22:02,333 than we had access to. 541 00:22:02,333 --> 00:22:06,810 We only had access to the location block. 542 00:22:06,810 --> 00:22:08,280 And then we could also break out of that 543 00:22:08,280 --> 00:22:09,210 into the server block. 544 00:22:09,210 --> 00:22:10,830 This had to be placed in the HTTP block, 545 00:22:10,830 --> 00:22:12,270 which we didn't have access to. 546 00:22:12,270 --> 00:22:14,490 So, this didn't work. 547 00:22:14,490 --> 00:22:15,420 So, at this point, 548 00:22:15,420 --> 00:22:19,020 you've reached the most boring part of our presentation. 549 00:22:19,020 --> 00:22:20,250 I promise you, it gets better from here. 550 00:22:20,250 --> 00:22:21,918 So, now is the time to wake up if you're sleeping. 551 00:22:21,918 --> 00:22:23,520 (person laughing) 552 00:22:23,520 --> 00:22:25,530 So, this is really exciting. 553 00:22:25,530 --> 00:22:29,340 So, it turns out you can't repeat static locations 554 00:22:29,340 --> 00:22:30,660 in NGINX configuration, 555 00:22:30,660 --> 00:22:32,820 but you cannot repeat regular expression. 556 00:22:32,820 --> 00:22:33,750 And this is due to the way 557 00:22:33,750 --> 00:22:35,940 that NGINX parses the configuration file. 558 00:22:35,940 --> 00:22:40,940 It will first compile a list of prefix paths, 559 00:22:40,950 --> 00:22:42,450 quickly find a match, 560 00:22:42,450 --> 00:22:44,490 but then if it falls back to regular expression, 561 00:22:44,490 --> 00:22:45,900 it will actually go through them one at a time 562 00:22:45,900 --> 00:22:47,910 and, first match it gets, 563 00:22:47,910 --> 00:22:49,140 it will send it through. 564 00:22:49,140 --> 00:22:49,973 And this allows you 565 00:22:49,973 --> 00:22:52,020 to actually send through multiple identical, 566 00:22:52,020 --> 00:22:53,280 regular expression, location paths. 567 00:22:53,280 --> 00:22:55,152 And so this was a huge bypass. 568 00:22:55,152 --> 00:22:55,985 So, as I was gonna say, 569 00:22:55,985 --> 00:22:57,910 so you can repeat regular expression and just respond. 570 00:22:57,910 --> 00:22:59,040 That's big. 571 00:22:59,040 --> 00:23:01,620 It was big. (laughs) 572 00:23:01,620 --> 00:23:04,710 So, at this point, we built gadgets. 573 00:23:04,710 --> 00:23:06,810 We weren't quite sure exactly how we were gonna exploit 574 00:23:06,810 --> 00:23:07,643 the system yet. 575 00:23:07,643 --> 00:23:09,870 So, we built a full red SSRF, finally, 576 00:23:09,870 --> 00:23:11,790 we built a local file inclusion 577 00:23:11,790 --> 00:23:13,620 where we could actually pull files off the system. 578 00:23:13,620 --> 00:23:14,940 As an aside, 579 00:23:14,940 --> 00:23:18,270 they were very secure and they built the NGINX proxy 580 00:23:18,270 --> 00:23:19,590 as a read-only file system, 581 00:23:19,590 --> 00:23:20,970 so we couldn't write any files to it, 582 00:23:20,970 --> 00:23:22,890 which also made this more difficult 583 00:23:22,890 --> 00:23:23,850 than it should have been. 584 00:23:23,850 --> 00:23:28,170 And we built a cross-site scripting payload as well. 585 00:23:28,170 --> 00:23:30,540 And so we spent a lot of time looking 586 00:23:30,540 --> 00:23:32,250 for other interesting tags that we could use 587 00:23:32,250 --> 00:23:33,900 to either achieve remote code execution 588 00:23:33,900 --> 00:23:36,900 or somehow cause interesting things to happen here. 589 00:23:36,900 --> 00:23:38,490 And there really weren't very many, 590 00:23:38,490 --> 00:23:39,540 but there's one other one, 591 00:23:39,540 --> 00:23:41,190 and this is a little bit obscure. 592 00:23:41,190 --> 00:23:44,070 So client, body and file only. 593 00:23:44,070 --> 00:23:45,660 And so what this will do is, 594 00:23:45,660 --> 00:23:48,540 when NGINX receives a post payload, 595 00:23:48,540 --> 00:23:52,110 NGINX will save the post payload to a file on disk. 596 00:23:52,110 --> 00:23:53,700 And the expectation is you then have 597 00:23:53,700 --> 00:23:57,352 some sidecar Python or Pearl process or Lua 598 00:23:57,352 --> 00:23:59,970 that would then pick it up and do something 599 00:23:59,970 --> 00:24:02,640 with that post payload and then delete the file. 600 00:24:02,640 --> 00:24:04,080 So, it's more of a functionality 601 00:24:04,080 --> 00:24:08,520 than it is like a security feature tag directive, sorry. 602 00:24:08,520 --> 00:24:10,113 And so why does this matter? 603 00:24:11,040 --> 00:24:13,560 So, we can actually combine this with our... 604 00:24:13,560 --> 00:24:15,840 So, if we could find a secret key that we could save, 605 00:24:15,840 --> 00:24:18,270 we could combine this with our LFI payload 606 00:24:18,270 --> 00:24:21,660 to steal some secret key that's somewhere in a post payload. 607 00:24:21,660 --> 00:24:24,000 So, that's exactly what we did. 608 00:24:24,000 --> 00:24:25,620 So, if you remember this, 609 00:24:25,620 --> 00:24:27,690 where we had these secondary servers, 610 00:24:27,690 --> 00:24:32,160 where we had connections going to KASM agent, 611 00:24:32,160 --> 00:24:34,230 one of the requests that went through to KASM agent 612 00:24:34,230 --> 00:24:36,510 was a health check, and this went through about 30 seconds. 613 00:24:36,510 --> 00:24:38,610 And in that health check post body, 614 00:24:38,610 --> 00:24:41,310 there was a secret key that could be used to communicate 615 00:24:41,310 --> 00:24:42,603 with that KASM agent. 616 00:24:43,710 --> 00:24:46,830 So we asked, "What if we get access to that secret key?" 617 00:24:46,830 --> 00:24:49,710 And so we tried to get it. (laughs) 618 00:24:49,710 --> 00:24:50,790 So, putting everything together, 619 00:24:50,790 --> 00:24:52,290 ignoring what's on the right side for right now, 620 00:24:52,290 --> 00:24:53,460 I'll get to that in a second. 621 00:24:53,460 --> 00:24:56,580 So, we carefully intercepted a single KASM major request 622 00:24:56,580 --> 00:24:59,730 using that client body and file only directive. 623 00:24:59,730 --> 00:25:02,970 We then forwarded along so that the system wouldn't think 624 00:25:02,970 --> 00:25:05,520 that the secondary server was unhealthy. 625 00:25:05,520 --> 00:25:07,920 If it thought the secondary server was unhealthy, 626 00:25:07,920 --> 00:25:12,810 it would take it offline and break the whole attack chain. 627 00:25:12,810 --> 00:25:17,040 We'll then use our LFI payload to steal that secret. 628 00:25:17,040 --> 00:25:20,490 So, breaking this out a little bit better, 629 00:25:20,490 --> 00:25:22,470 what we're actually doing here, 630 00:25:22,470 --> 00:25:23,850 we're putting this into a host header 631 00:25:23,850 --> 00:25:26,370 using our ISO 88 59 dash one encoding. 632 00:25:26,370 --> 00:25:27,570 So the first, in the yellow, 633 00:25:27,570 --> 00:25:31,380 we're using that to keep off working for this workspace. 634 00:25:31,380 --> 00:25:34,260 We're then, for the top two injection points, 635 00:25:34,260 --> 00:25:35,520 we're using... 636 00:25:35,520 --> 00:25:37,380 Oh sorry, for the bottom four injection points, sorry. 637 00:25:37,380 --> 00:25:39,360 We're putting this into a comment 638 00:25:39,360 --> 00:25:41,550 so that everything is commented off correctly, 639 00:25:41,550 --> 00:25:42,510 that's in blue. 640 00:25:42,510 --> 00:25:44,880 And white is really the magic here 641 00:25:44,880 --> 00:25:48,480 where we're intercepting the health check 642 00:25:48,480 --> 00:25:51,420 and then we're stealing the secret that contains in there. 643 00:25:51,420 --> 00:25:56,070 Then we're using a regular expression location 644 00:25:56,070 --> 00:25:59,130 to create an LFI backdoor 645 00:25:59,130 --> 00:26:00,720 that using a dot secret end. 646 00:26:00,720 --> 00:26:02,460 So, any path that ends in dot secret 647 00:26:02,460 --> 00:26:05,280 will be interpreted by this regular expression. 648 00:26:05,280 --> 00:26:07,860 And we're going use that to steal the file, 649 00:26:07,860 --> 00:26:09,450 and then we're gonna put an unused location 650 00:26:09,450 --> 00:26:11,520 at the end of all this to close it off nicely. 651 00:26:11,520 --> 00:26:14,760 So, we're also hit by a 255 character limit, 652 00:26:14,760 --> 00:26:16,590 which is why this is a little bit code golfed. 653 00:26:16,590 --> 00:26:18,690 So I do apologize, that's not super clear. 654 00:26:19,830 --> 00:26:21,502 So, running this, 655 00:26:21,502 --> 00:26:24,090 we, as you can see here, 656 00:26:24,090 --> 00:26:25,920 you end up with a series of files 657 00:26:25,920 --> 00:26:28,770 and we get our host token in each one of these files, 658 00:26:28,770 --> 00:26:30,630 which is super exciting. 659 00:26:30,630 --> 00:26:33,990 So, what can you do with KASM agent at this point? 660 00:26:33,990 --> 00:26:35,730 So, with KASM agent, 661 00:26:35,730 --> 00:26:37,680 it actually has a well documented API 662 00:26:37,680 --> 00:26:40,200 where you can either spin up new containers 663 00:26:40,200 --> 00:26:43,170 or you can make privileged calls 664 00:26:43,170 --> 00:26:45,540 to other containers to run commands. 665 00:26:45,540 --> 00:26:46,680 So, in this case, 666 00:26:46,680 --> 00:26:48,930 what we actually wanted to do was 667 00:26:48,930 --> 00:26:52,140 we wanted to have KASM agent make a Docker API call 668 00:26:52,140 --> 00:26:53,940 to run a command on itself. 669 00:26:53,940 --> 00:26:56,370 And so, because it was a privileged container 670 00:26:56,370 --> 00:26:59,370 that would allow us to access the internal operating system. 671 00:26:59,370 --> 00:27:01,500 So, using our host token, 672 00:27:01,500 --> 00:27:02,670 we sent that request to the... 673 00:27:02,670 --> 00:27:04,110 We sent the request on the right side here 674 00:27:04,110 --> 00:27:05,910 to the KASM agent to run a command. 675 00:27:05,910 --> 00:27:07,170 This command 676 00:27:07,170 --> 00:27:10,620 mounted the underlying operating system as route. 677 00:27:10,620 --> 00:27:12,720 And we simply touched a file to demonstrate 678 00:27:12,720 --> 00:27:14,190 that we had access. 679 00:27:14,190 --> 00:27:15,930 And as you can see here, 680 00:27:15,930 --> 00:27:19,110 we were able to achieve file right as route, 681 00:27:19,110 --> 00:27:20,370 which would've led to RCE 682 00:27:20,370 --> 00:27:22,520 if we had actually exploited this for real. 683 00:27:23,550 --> 00:27:24,390 At this point, 684 00:27:24,390 --> 00:27:25,511 we were asked to stop. 685 00:27:25,511 --> 00:27:28,200 The response we got back from the bug bounty program 686 00:27:28,200 --> 00:27:29,520 was, "Well that's worrying." 687 00:27:29,520 --> 00:27:31,173 And we got our RCE. 688 00:27:33,270 --> 00:27:34,634 So, we were very excited. 689 00:27:34,634 --> 00:27:36,199 (person laughing) 690 00:27:36,199 --> 00:27:37,595 (audience clapping) 691 00:27:37,595 --> 00:27:38,428 Thank you. 692 00:27:38,428 --> 00:27:41,428 (audience clapping) 693 00:27:45,420 --> 00:27:48,780 So, no good slide deck is complete 694 00:27:48,780 --> 00:27:50,340 without a vendor response slide. 695 00:27:50,340 --> 00:27:53,460 So, NGINX doesn't consider this to be a vulnerability, 696 00:27:53,460 --> 00:27:55,320 but they might introduce some checks in the future 697 00:27:55,320 --> 00:27:57,840 to prevent certain characters that are obviously invalid 698 00:27:57,840 --> 00:27:59,550 for host headers from going through. 699 00:27:59,550 --> 00:28:01,860 CherryPy actually address this for the host header. 700 00:28:01,860 --> 00:28:05,730 So, other headers are still vulnerable, however. 701 00:28:05,730 --> 00:28:08,790 KASM Workspaces has a configuration fix for this. 702 00:28:08,790 --> 00:28:12,149 And if you're running this, we would recommend you apply it. 703 00:28:12,149 --> 00:28:14,070 Additionally, in the latest version, 704 00:28:14,070 --> 00:28:16,230 it's been fixed as well. 705 00:28:16,230 --> 00:28:17,280 And in terms of a timeline here, 706 00:28:17,280 --> 00:28:18,420 this is just to give you an idea 707 00:28:18,420 --> 00:28:20,403 of roughly what this looks like. 708 00:28:21,300 --> 00:28:22,740 We submitted this on three dash one. 709 00:28:22,740 --> 00:28:26,220 On three six, we reported the additional ways 710 00:28:26,220 --> 00:28:27,270 that we could exploit the system. 711 00:28:27,270 --> 00:28:28,920 It then took us another 22 days 712 00:28:28,920 --> 00:28:31,203 to demonstrate the Docker container breakout. 713 00:28:33,270 --> 00:28:34,110 So, some takeaways. 714 00:28:34,110 --> 00:28:35,730 So, from the blue team side, 715 00:28:35,730 --> 00:28:37,980 you have to ask yourself how your organization is handling, 716 00:28:37,980 --> 00:28:40,890 running a server that has multiple domain names in general. 717 00:28:40,890 --> 00:28:42,450 You'll have a QA and prod. 718 00:28:42,450 --> 00:28:44,580 How do you handle the host header there? 719 00:28:44,580 --> 00:28:47,520 Are you trusting the end-users to supply that? 720 00:28:47,520 --> 00:28:48,780 Because that could be a little bit dangerous 721 00:28:48,780 --> 00:28:50,610 if they are doing something with it. 722 00:28:50,610 --> 00:28:51,780 You also have to be a little bit careful 723 00:28:51,780 --> 00:28:53,490 with local Docker networking. 724 00:28:53,490 --> 00:28:55,110 I mean, that's true in any networking environment, 725 00:28:55,110 --> 00:28:56,954 but that's especially true with Docker 726 00:28:56,954 --> 00:28:59,643 and mixed low and high privilege environments. 727 00:29:00,900 --> 00:29:03,150 And then on the red team side, 728 00:29:03,150 --> 00:29:05,100 we know you're spraying payloads everywhere. 729 00:29:05,100 --> 00:29:09,300 So, why not just add ISO 8-8-5-9 dash one to that list. 730 00:29:09,300 --> 00:29:11,220 And then, similarly, build gadgets. 731 00:29:11,220 --> 00:29:13,740 And then constantly reevaluate what you have 732 00:29:13,740 --> 00:29:16,110 and what you can do. 733 00:29:16,110 --> 00:29:17,449 What we presented here. 734 00:29:17,449 --> 00:29:20,520 We tried to follow a logical plot. 735 00:29:20,520 --> 00:29:22,020 We jumped around quite a bit 736 00:29:22,020 --> 00:29:24,150 when we're (laughs) actually exploiting this 737 00:29:24,150 --> 00:29:25,470 and then, similarly, collaborate. 738 00:29:25,470 --> 00:29:26,640 I think there were multiple times 739 00:29:26,640 --> 00:29:30,480 where we both wanted to throw in the towel 740 00:29:30,480 --> 00:29:31,680 and completely give up on this. 741 00:29:31,680 --> 00:29:33,210 And then one of us found something 742 00:29:33,210 --> 00:29:34,440 that just pushed us that much further, 743 00:29:34,440 --> 00:29:36,990 which is really why we worked on this for a month, (laughs) 744 00:29:36,990 --> 00:29:41,100 I think, which is overkill. (laughs) 745 00:29:41,100 --> 00:29:42,180 So, that's it. 746 00:29:42,180 --> 00:29:43,013 Questions? 747 00:29:46,350 --> 00:29:47,343 Going once. 748 00:29:51,867 --> 00:29:54,870 (microphone thudding) 749 00:29:54,870 --> 00:29:55,703 - [Audience Member] Hey. 750 00:29:55,703 --> 00:29:57,150 So, you found the CherryPy, 751 00:29:57,150 --> 00:29:59,640 you're able to use that different encoding. 752 00:29:59,640 --> 00:30:01,390 So, was that also affecting NGINX 753 00:30:02,610 --> 00:30:05,376 for the host encoding when you did the- 754 00:30:05,376 --> 00:30:06,630 - Could you say it one more time? 755 00:30:06,630 --> 00:30:07,463 We can't- - Yep. 756 00:30:07,463 --> 00:30:08,296 Oh, sorry. 757 00:30:08,296 --> 00:30:09,300 Test, can you hear me better? 758 00:30:09,300 --> 00:30:10,133 Okay. 759 00:30:10,133 --> 00:30:11,580 So, when you did the actual exploiters 760 00:30:11,580 --> 00:30:13,080 on the CherryPy server, 761 00:30:13,080 --> 00:30:16,560 did the actual encoding trick bypass on NGINX as well? 762 00:30:16,560 --> 00:30:17,430 - [Man] Can you hear him at all? 763 00:30:17,430 --> 00:30:18,263 - [Audience Member] Oh, sorry. 764 00:30:18,263 --> 00:30:21,095 (people chatting) 765 00:30:22,350 --> 00:30:23,640 - [Audience Member] Oh, yeah. 766 00:30:23,640 --> 00:30:26,790 So, when you're testing the actual proof of concept, 767 00:30:26,790 --> 00:30:28,673 you were on the CherryPy, correct? 768 00:30:28,673 --> 00:30:29,938 - [Man] Yeah, I think the- 769 00:30:29,938 --> 00:30:32,771 (people chatting) 770 00:30:36,210 --> 00:30:37,043 - [Audience Member] Yeah. 771 00:30:37,043 --> 00:30:39,120 So, when you actually exploit it on the CherryPy server, 772 00:30:39,120 --> 00:30:41,940 but the actual exploit was done on like... 773 00:30:41,940 --> 00:30:44,183 You able to do the encoding trick on NGINX as well? 774 00:30:45,686 --> 00:30:47,343 - [Man] Okay. Sorry, the speakers are- 775 00:30:47,343 --> 00:30:48,176 - Yeah, I know. 776 00:30:48,176 --> 00:30:50,640 Sorry about having a hard time hearing. 777 00:30:50,640 --> 00:30:53,094 Yeah, so the question was, I think, 778 00:30:53,094 --> 00:30:55,410 when we actually did the exploit, 779 00:30:55,410 --> 00:30:57,720 were we able to do it through NGINX as well? 780 00:30:57,720 --> 00:30:58,553 And I think the... 781 00:30:58,553 --> 00:30:59,610 Yeah, so we were. 782 00:30:59,610 --> 00:31:01,350 And so the cool thing 783 00:31:01,350 --> 00:31:06,240 about the ISO 88 59 dash one encoding 784 00:31:06,240 --> 00:31:09,480 is that all of those characters are valid characters 785 00:31:09,480 --> 00:31:11,520 to be in a host header on NGINX. 786 00:31:11,520 --> 00:31:14,144 So, just what ended up happening was all of those characters 787 00:31:14,144 --> 00:31:17,400 that were causing the problem in the NGINX configuration 788 00:31:17,400 --> 00:31:18,600 just were encoded 789 00:31:18,600 --> 00:31:21,090 and then flew right through the NGINX part. 790 00:31:21,090 --> 00:31:24,960 And then were decoded by CherryPy on the back-end. 791 00:31:24,960 --> 00:31:28,440 And that's what allowed us to bypass that little mismatch 792 00:31:28,440 --> 00:31:29,833 between the parsing of the headers 793 00:31:29,833 --> 00:31:32,520 on the reverse proxy there. 794 00:31:32,520 --> 00:31:33,741 So, yeah- 795 00:31:33,741 --> 00:31:35,211 - Can I actually say something about that as well? 796 00:31:35,211 --> 00:31:36,044 - Yeah, go for it. 797 00:31:36,044 --> 00:31:37,830 - So, within NGINX itself, 798 00:31:37,830 --> 00:31:41,190 the configuration parsing logic is heavily dependent 799 00:31:41,190 --> 00:31:42,990 on certain control characters, 800 00:31:42,990 --> 00:31:47,220 which were blocked by trying to traverse through NGINX. 801 00:31:47,220 --> 00:31:48,574 So, that's why we had to do this. 802 00:31:48,574 --> 00:31:49,407 Yeah. 803 00:31:50,610 --> 00:31:51,723 - Any other questions? 804 00:31:54,330 --> 00:31:57,150 All right. Well, you can find us on Twitter afterwards. 805 00:31:57,150 --> 00:31:58,290 If you have any other questions, 806 00:31:58,290 --> 00:31:59,460 our handles are down at the bottom 807 00:31:59,460 --> 00:32:00,458 and thank you guys so much for listening. 808 00:32:00,458 --> 00:32:03,291 (people clapping) 809 00:32:04,890 --> 00:32:06,728 (Justin laughing)