1 00:00:00,870 --> 00:00:01,743 - All right. 2 00:00:02,640 --> 00:00:05,370 Talk's gonna be a little about a container escape. 3 00:00:05,370 --> 00:00:06,420 Those of you that are 4 00:00:08,310 --> 00:00:09,960 unaware. 5 00:00:09,960 --> 00:00:10,793 Enjoy. 6 00:00:13,800 --> 00:00:14,750 - What's up DEFCON? 7 00:00:15,899 --> 00:00:17,949 (audience cheering) - Mic's are hot now. 8 00:00:20,910 --> 00:00:22,823 - After that, I think we're gonna do great. 9 00:00:24,810 --> 00:00:26,400 - Okay. It's good. - Awesome. 10 00:00:26,400 --> 00:00:28,170 - All right. 11 00:00:28,170 --> 00:00:29,003 I'm Dagan. 12 00:00:29,940 --> 00:00:31,830 - Hi, I'm Will Kline. 13 00:00:31,830 --> 00:00:34,290 So, we're all here, as you know, 14 00:00:34,290 --> 00:00:37,500 we're mostly like SRE DevOpsy type people, 15 00:00:37,500 --> 00:00:39,859 setting up clusters, we're not pen testers, 16 00:00:39,859 --> 00:00:43,110 but when we're going through setting up these clusters, 17 00:00:43,110 --> 00:00:45,240 we notice some things and kind of repeating patterns 18 00:00:45,240 --> 00:00:48,210 of security issues that aren't being talked about enough, 19 00:00:48,210 --> 00:00:51,090 and that's when we decide to put this talk together for. 20 00:00:51,090 --> 00:00:53,010 You don't have to be a Kubernetes developer 21 00:00:53,010 --> 00:00:54,930 to understand what we're talking about here, 22 00:00:54,930 --> 00:00:56,220 but you just gotta think like one, 23 00:00:56,220 --> 00:00:58,020 and there's no 0-Days today here. 24 00:00:58,020 --> 00:01:00,810 This has all been disclosed properly, 25 00:01:00,810 --> 00:01:02,340 and we're gonna have three different demos 26 00:01:02,340 --> 00:01:04,140 that we're gonna run through, different types of exploits 27 00:01:04,140 --> 00:01:06,090 that we wanna tell everyone about. 28 00:01:06,090 --> 00:01:08,310 - With that being said with our demos, 29 00:01:08,310 --> 00:01:10,620 at the end, I have a GitHub URL. 30 00:01:10,620 --> 00:01:14,460 The demo environment is set up in a local laptop 31 00:01:14,460 --> 00:01:17,310 using a make, and that file's available. 32 00:01:17,310 --> 00:01:21,450 So, anybody that wants to can go and just clone that repo. 33 00:01:21,450 --> 00:01:22,680 You'll be able to step through each 34 00:01:22,680 --> 00:01:25,110 of the demos we do today, yourself on your laptop, 35 00:01:25,110 --> 00:01:27,310 and there's also a step by step walkthrough. 36 00:01:30,000 --> 00:01:31,503 All right, let's get started. 37 00:01:32,490 --> 00:01:34,699 WTF is a Kubernetes. 38 00:01:34,699 --> 00:01:36,510 I like to think of Kubernetes 39 00:01:36,510 --> 00:01:39,240 as the operating system for the cloud. 40 00:01:39,240 --> 00:01:42,870 It takes anywhere from one to several thousand nodes 41 00:01:42,870 --> 00:01:43,890 or individual servers, 42 00:01:43,890 --> 00:01:47,460 and turns it into a single orchestrated system. 43 00:01:47,460 --> 00:01:49,860 That system can span an entire data center. 44 00:01:49,860 --> 00:01:53,340 It can span multiple data centers and multiple clouds. 45 00:01:53,340 --> 00:01:56,160 It's the ultimate shift-left platform. 46 00:01:56,160 --> 00:01:57,780 As a Kubernetes developer, 47 00:01:57,780 --> 00:02:00,750 not only are you able to take advantage of containerization, 48 00:02:00,750 --> 00:02:02,160 to bring your file system 49 00:02:02,160 --> 00:02:04,710 and all of your dependencies with your application, 50 00:02:04,710 --> 00:02:07,590 you're also able to declare the exact way 51 00:02:07,590 --> 00:02:09,570 that your application should be instantiated. 52 00:02:09,570 --> 00:02:12,420 You can say, I wanna run this command with these AGS. 53 00:02:12,420 --> 00:02:14,460 You can include your configuration. 54 00:02:14,460 --> 00:02:17,400 You can say, I want five instances of it, 55 00:02:17,400 --> 00:02:19,290 and I don't want any of those instances running 56 00:02:19,290 --> 00:02:21,210 in the same availability zone. 57 00:02:21,210 --> 00:02:23,640 You can go further and declare how much CPU time 58 00:02:23,640 --> 00:02:25,830 should be allotted to each of your applications. 59 00:02:25,830 --> 00:02:28,530 How much memory is reserved for your applications? 60 00:02:28,530 --> 00:02:30,270 You can say, this is what I want you to do 61 00:02:30,270 --> 00:02:31,980 to monitor the health of the application 62 00:02:31,980 --> 00:02:34,080 and how to respond when it becomes unhealthy, 63 00:02:34,080 --> 00:02:36,060 even how to scale the application. 64 00:02:36,060 --> 00:02:40,383 Kubernetes is an extraordinarily powerful system. 65 00:02:41,940 --> 00:02:42,960 - Yo, and the other thing 66 00:02:42,960 --> 00:02:44,370 that's really nice about Kubernetes is 67 00:02:44,370 --> 00:02:48,120 that you basically have decades of Google's experience, 68 00:02:48,120 --> 00:02:49,800 and other large cloud providers experience 69 00:02:49,800 --> 00:02:52,320 and setting up these large scale systems, 70 00:02:52,320 --> 00:02:54,720 and they've distilled all that down in a series of APIs 71 00:02:54,720 --> 00:02:56,880 to help you at any scale you want, 72 00:02:56,880 --> 00:02:59,520 whether it's like a raspberry pie up to the biggest Google, 73 00:02:59,520 --> 00:03:01,320 deploy these applications and manage them. 74 00:03:01,320 --> 00:03:02,940 You get a lot of lessons learned out of that, 75 00:03:02,940 --> 00:03:06,393 so it's really powerful system for managing your systems. 76 00:03:07,875 --> 00:03:10,350 So, today, we're gonna talk about Pwning a Cluster. 77 00:03:10,350 --> 00:03:11,940 So, I want to just quick little thing about 78 00:03:11,940 --> 00:03:13,530 what we consider this Pwning a Cluster is? 79 00:03:13,530 --> 00:03:18,060 So, basically, with all of that power to schedule workloads, 80 00:03:18,060 --> 00:03:20,539 spin up new instances, manage the networking. 81 00:03:20,539 --> 00:03:22,410 What does Pwning mean here? 82 00:03:22,410 --> 00:03:24,900 So, we're talking about running your own workloads. 83 00:03:24,900 --> 00:03:27,150 This could be as simple as Bitcoin minors. 84 00:03:27,150 --> 00:03:28,800 This could be something that you're using as more 85 00:03:28,800 --> 00:03:32,100 of an implant to pivot to other containers in the cluster, 86 00:03:32,100 --> 00:03:33,090 stuff like that. 87 00:03:33,090 --> 00:03:34,710 We're modifying existing workloads. 88 00:03:34,710 --> 00:03:36,720 So, that could be, if I'm running some application, 89 00:03:36,720 --> 00:03:39,570 that's managing throughput from user data. 90 00:03:39,570 --> 00:03:43,110 If I can manage to modify the configuration of that pod 91 00:03:43,110 --> 00:03:46,410 in a way that makes it vulnerable to further exploitation, 92 00:03:46,410 --> 00:03:48,180 that's called pwning, right? 93 00:03:48,180 --> 00:03:49,950 We're gonna talk about hijacking traffic. 94 00:03:49,950 --> 00:03:52,050 One of the tools that we talk about a little bit is Istio, 95 00:03:52,050 --> 00:03:54,720 which is the most robust and interesting man 96 00:03:54,720 --> 00:03:56,640 in the middle proxy you've ever seen in your life. 97 00:03:56,640 --> 00:03:57,960 It's incredibly powerful. 98 00:03:57,960 --> 00:04:00,240 And if you can take control of that, you can man and middle, 99 00:04:00,240 --> 00:04:03,787 literally anything in the cluster, it's really powerful. 100 00:04:03,787 --> 00:04:05,430 And then finally, the fourth one here is breaking out 101 00:04:05,430 --> 00:04:08,280 so that you are executing code on the host as root, 102 00:04:08,280 --> 00:04:10,200 and at that point you control everything 103 00:04:10,200 --> 00:04:13,200 and that's the pop calc up on the Windows-box, 104 00:04:13,200 --> 00:04:15,390 and prove you owned it. Right? 105 00:04:15,390 --> 00:04:18,210 We're gonna talk about three different attack angles today. 106 00:04:18,210 --> 00:04:20,790 And realistically, this is not a real cluster 107 00:04:20,790 --> 00:04:22,260 that we're gonna throw all this through. 108 00:04:22,260 --> 00:04:23,550 These vulnerabilities probably could 109 00:04:23,550 --> 00:04:24,660 have existed at the same time, 110 00:04:24,660 --> 00:04:26,340 but it's not like we ran into a cluster 111 00:04:26,340 --> 00:04:28,080 that had all of these at the same time. 112 00:04:28,080 --> 00:04:29,793 And again, No 0-Days today. 113 00:04:32,430 --> 00:04:34,237 All right, so we're gonna start with phase one here, 114 00:04:34,237 --> 00:04:35,070 "No More Secrets." 115 00:04:35,070 --> 00:04:36,840 So, I mentioned earlier, 116 00:04:36,840 --> 00:04:38,880 Istio, is giant man in the middle proxy. 117 00:04:38,880 --> 00:04:41,820 So, basically, the idea is that we're gonna inject something 118 00:04:41,820 --> 00:04:43,500 into every time we launch a container, 119 00:04:43,500 --> 00:04:44,790 and we're gonna hook onto it 120 00:04:44,790 --> 00:04:48,810 to take all those traffic through IP tables, 121 00:04:48,810 --> 00:04:53,220 we're gonna use "MTLS" Mutual TLS authentication, 122 00:04:53,220 --> 00:04:56,133 so that when you, as a developer writing a service, 123 00:04:57,141 --> 00:04:58,800 you don't have to figure out, how to make MTLS work? 124 00:04:58,800 --> 00:05:02,340 You just run your pod in a system with Istio, 125 00:05:02,340 --> 00:05:03,690 and you get the benefit of all that, 126 00:05:03,690 --> 00:05:05,610 and that's where Kubernetes becomes really powerful, 127 00:05:05,610 --> 00:05:08,190 because we can take all of these lessons learned 128 00:05:08,190 --> 00:05:10,500 and hard cryptographic problems and boil it down, 129 00:05:10,500 --> 00:05:12,090 so the developers don't have to think about it. 130 00:05:12,090 --> 00:05:14,910 It's kinda like just bolt on end-to-end encryption, 131 00:05:14,910 --> 00:05:16,890 and it works really well, most of the time. 132 00:05:16,890 --> 00:05:19,770 One of the other great things about it is this Kiali tool, 133 00:05:19,770 --> 00:05:20,730 which is visualization. 134 00:05:20,730 --> 00:05:22,170 So, now that you're intercepting all the traffic 135 00:05:22,170 --> 00:05:23,460 and watching it through the network, 136 00:05:23,460 --> 00:05:24,747 you can really figure out what you're doing 137 00:05:24,747 --> 00:05:26,297 and how your network's working? 138 00:05:27,930 --> 00:05:32,460 Quick shout out to JWTs, pronounced "JOT", JSON Web Token. 139 00:05:32,460 --> 00:05:34,410 Obviously, we're not talking that much about it, 140 00:05:34,410 --> 00:05:36,060 but I just wanted to bring it up as a concept 141 00:05:36,060 --> 00:05:38,040 of when you see this, 142 00:05:38,040 --> 00:05:40,190 it's a very common authentication protocol, 143 00:05:41,040 --> 00:05:42,990 how these little cookies are made? 144 00:05:42,990 --> 00:05:44,850 Jot.io is really great website for 145 00:05:44,850 --> 00:05:46,800 if you ever get one of those and you have to interpret it, 146 00:05:46,800 --> 00:05:50,250 and the important part here is that it gets HMAC at the end. 147 00:05:50,250 --> 00:05:52,500 So, as everyone knows, cryptography is perfect 148 00:05:52,500 --> 00:05:54,120 and very, very easy to implement. 149 00:05:54,120 --> 00:05:56,940 So we're gonna talk a little bit about something we noticed. 150 00:05:56,940 --> 00:05:59,663 - You guys ready to hear about a vulnerability? 151 00:05:59,663 --> 00:06:00,600 (Will laughing) 152 00:06:00,600 --> 00:06:02,250 - Yeah. 153 00:06:02,250 --> 00:06:05,970 So, one day Dagan's setting up Kiali on one of his clusters, 154 00:06:05,970 --> 00:06:07,770 and we're bringing it up, bringing it down. 155 00:06:07,770 --> 00:06:09,690 He's tweaking all the parameters in the Helm chart 156 00:06:09,690 --> 00:06:11,370 to figure out how to deploy it right? 157 00:06:11,370 --> 00:06:13,740 And after doing that, the first time it stands out. 158 00:06:13,740 --> 00:06:16,860 The first time you get that login page, hot-damn, got it. 159 00:06:16,860 --> 00:06:19,410 And then the next time you don't get the login page, 160 00:06:19,410 --> 00:06:21,960 and eventually he notices that he's taking these pods 161 00:06:21,960 --> 00:06:25,380 up and down and the login page doesn't refresh. 162 00:06:25,380 --> 00:06:26,940 He doesn't have to log in each time. 163 00:06:26,940 --> 00:06:29,820 So, in Kubernetes, pods are generally ephemeral, 164 00:06:29,820 --> 00:06:33,060 which means that they're not storing data between sessions, 165 00:06:33,060 --> 00:06:36,030 and there's ways to store persistent data in Kubernetes. 166 00:06:36,030 --> 00:06:37,200 So, we started digging into it like, 167 00:06:37,200 --> 00:06:38,700 okay, if it's keeping my session alive, 168 00:06:38,700 --> 00:06:40,560 where is it storing that information, 169 00:06:40,560 --> 00:06:42,560 that cryptography to do the HMAC siting. 170 00:06:44,100 --> 00:06:47,992 - So, I dug in, and there in Kubernetes, 171 00:06:47,992 --> 00:06:50,520 there's a lot of ways that you can persist data as well. 172 00:06:50,520 --> 00:06:52,620 We're saying in this particular case, 173 00:06:52,620 --> 00:06:55,860 what I would've expected to see is a Kubernetes secret, 174 00:06:55,860 --> 00:06:58,080 where install Kiali for the first time, 175 00:06:58,080 --> 00:06:59,880 it looks for the secret, it doesn't exist, 176 00:06:59,880 --> 00:07:00,843 so it creates it. 177 00:07:00,843 --> 00:07:03,420 From then on, it would see that same secret 178 00:07:03,420 --> 00:07:06,300 and use that as its signing key for the jot. 179 00:07:06,300 --> 00:07:07,497 But there was no secret, 180 00:07:07,497 --> 00:07:09,510 and I can delete the entire name space, 181 00:07:09,510 --> 00:07:13,140 anything associated with it, and I was still logged in. 182 00:07:13,140 --> 00:07:15,660 The only piece of persistence was the cookie 183 00:07:15,660 --> 00:07:17,673 that was sitting in my browser, 184 00:07:18,600 --> 00:07:22,080 which meant there was two possibilities, equally terrifying. 185 00:07:22,080 --> 00:07:24,450 The first is there's no authentication 186 00:07:24,450 --> 00:07:25,470 of the cookie whatsoever. 187 00:07:25,470 --> 00:07:27,390 You just simply say, Hey, I'm admin, 188 00:07:27,390 --> 00:07:29,790 and Kiali's like, I'll take your word for it. 189 00:07:29,790 --> 00:07:32,070 The other is a hard-coded secret 190 00:07:32,070 --> 00:07:33,990 that's being used to sign the key. 191 00:07:33,990 --> 00:07:36,090 So, I dug through the code. 192 00:07:36,090 --> 00:07:40,893 You can see bottom right corner, Kiali, hello! 193 00:07:43,110 --> 00:07:45,413 See how we doing on time? I got a couple minutes. 194 00:07:46,511 --> 00:07:49,710 So anyway, I went over to Jot-IO, 195 00:07:49,710 --> 00:07:54,090 and I threw that in and I was able to just recreate 196 00:07:54,090 --> 00:07:56,940 or fabricate a perfectly valid jot, 197 00:07:56,940 --> 00:07:59,733 that would allow me to access Kiali, as an admin. 198 00:08:00,930 --> 00:08:03,930 The reason that that happened is because 199 00:08:03,930 --> 00:08:05,820 they had an insecure default. 200 00:08:05,820 --> 00:08:07,920 It could be configured in and overwritten 201 00:08:07,920 --> 00:08:09,330 in the Helm chart to specify, 202 00:08:09,330 --> 00:08:12,187 but nowhere did anybody take the time to say, 203 00:08:12,187 --> 00:08:13,890 "When you're installing this application, 204 00:08:13,890 --> 00:08:15,480 you should definitely do this." 205 00:08:15,480 --> 00:08:17,643 So, nowhere, did it actually happen. 206 00:08:18,480 --> 00:08:20,133 So, let's take a look. 207 00:08:24,990 --> 00:08:26,190 So, here's the Kiali login page. 208 00:08:26,190 --> 00:08:27,090 I'll go ahead and refresh it, 209 00:08:27,090 --> 00:08:30,060 just to show I'm not logged in, nothing up my sleeves. 210 00:08:30,060 --> 00:08:33,993 I'm gonna come over and take to Jot-IO and fabricate a jot. 211 00:08:35,580 --> 00:08:37,380 The sample on the left isn't signed, 212 00:08:37,380 --> 00:08:38,400 'cause there's no secret in here. 213 00:08:38,400 --> 00:08:40,263 So, I'll put in Kiali as a secret. 214 00:08:42,780 --> 00:08:44,310 I now have a signed jot. 215 00:08:44,310 --> 00:08:46,310 Just gonna copy that onto the clipboard. 216 00:08:48,870 --> 00:08:52,380 Using developer tools on my browser, I'll add a cookie. 217 00:08:52,380 --> 00:08:54,750 It's just called Kiali-Token, 218 00:08:54,750 --> 00:08:56,550 and the contents of the cookie are just the jot, 219 00:08:56,550 --> 00:08:57,900 which is totally valid, 220 00:08:57,900 --> 00:08:59,760 assuming it wasn't already signed 221 00:08:59,760 --> 00:09:02,283 by an completely hard coded secret. 222 00:09:03,150 --> 00:09:04,600 Now, when I refresh the page. 223 00:09:07,301 --> 00:09:12,301 (Dagan laughing) (audience applauding) 224 00:09:13,980 --> 00:09:16,140 Fortunately from the attacker, 225 00:09:16,140 --> 00:09:18,150 Kiali, is extraordinarily powerful. 226 00:09:18,150 --> 00:09:20,070 There's a lot that I can do here. 227 00:09:20,070 --> 00:09:21,630 It's a powerful service mesh that allows me 228 00:09:21,630 --> 00:09:24,810 to completely rewrite how traffic gets managed. 229 00:09:24,810 --> 00:09:27,420 If I wanna send 10% of your login traffic 230 00:09:27,420 --> 00:09:29,910 to my server instead, I can do that. 231 00:09:29,910 --> 00:09:32,280 If I want to add entirely new ingress points 232 00:09:32,280 --> 00:09:35,070 to things that shouldn't be secure or shouldn't be exposed, 233 00:09:35,070 --> 00:09:36,360 I can do that. 234 00:09:36,360 --> 00:09:38,763 I can pretty much do anything that I want to do. 235 00:09:39,810 --> 00:09:42,240 - So, Dagan, we were sitting at the bar that day, 236 00:09:42,240 --> 00:09:44,160 and you were telling me the story about this. 237 00:09:44,160 --> 00:09:45,691 One of the things you mentioned was that 238 00:09:45,691 --> 00:09:46,767 when you kept reloading the page the next day, 239 00:09:46,767 --> 00:09:48,780 and the next day, it still stayed up. 240 00:09:48,780 --> 00:09:50,130 How did you make that work? 241 00:09:50,130 --> 00:09:52,860 - Well, my talk on security ramifications of time travel 242 00:09:52,860 --> 00:09:55,803 was rejected, but I'll explain. 243 00:09:57,720 --> 00:10:02,720 Here's the code that Kiali uses to authenticate a cookie. 244 00:10:02,820 --> 00:10:05,220 Sorry, a jot. 245 00:10:05,220 --> 00:10:07,800 The line 67 there, it's the second one from the top. 246 00:10:07,800 --> 00:10:11,310 It just says to the library, Hey, can you parse this token? 247 00:10:11,310 --> 00:10:14,340 And if it's valid, give back the claims. 248 00:10:14,340 --> 00:10:16,830 But the developers didn't understand is 249 00:10:16,830 --> 00:10:20,760 that particular library, it didn't care what time it was? 250 00:10:20,760 --> 00:10:24,510 It just said, yes, this is a validly signed jot. 251 00:10:24,510 --> 00:10:26,850 What's not on there is any attempt to say, 252 00:10:26,850 --> 00:10:31,850 well, the expiration date is that in the past, wasn't there. 253 00:10:32,340 --> 00:10:35,090 So, you could do a replay attack with this, no problem. 254 00:10:38,010 --> 00:10:40,200 - So, if we imagine our fictitious cluster here, 255 00:10:40,200 --> 00:10:42,960 we're able to get to this Kiali service. 256 00:10:42,960 --> 00:10:45,450 We're able to forge credentials, and with that, 257 00:10:45,450 --> 00:10:47,490 we can pretty much read anything in the cluster, 258 00:10:47,490 --> 00:10:49,110 and we can write to Istio, 259 00:10:49,110 --> 00:10:52,480 which is the networking throbbing, 260 00:10:52,480 --> 00:10:55,560 the nervous system of this cluster, 261 00:10:55,560 --> 00:10:57,330 and we can reroute that wherever we want. 262 00:10:57,330 --> 00:11:00,150 The default for installing Kiali, is not read only. 263 00:11:00,150 --> 00:11:02,220 So, at this point we can man in the middle proxy, 264 00:11:02,220 --> 00:11:04,860 literally anything we want, we can blow up the system, 265 00:11:04,860 --> 00:11:07,800 we can shut it down, we can steal credentials. 266 00:11:07,800 --> 00:11:09,990 So, at this point, it's face to say, 267 00:11:09,990 --> 00:11:11,840 we've pretty much pwned that cluster. 268 00:11:13,127 --> 00:11:14,940 It's not much left to do there. 269 00:11:14,940 --> 00:11:15,773 - So, that's all right. 270 00:11:15,773 --> 00:11:17,310 Thanks for coming to our DEFCON talk. 271 00:11:17,310 --> 00:11:18,600 - But, Dagan! - What? 272 00:11:18,600 --> 00:11:19,920 - We promised them three demos. 273 00:11:19,920 --> 00:11:22,710 So, what if I said earlier, 274 00:11:22,710 --> 00:11:24,810 the default wasn't that it was read only, 275 00:11:24,810 --> 00:11:26,070 let's say it was read only. 276 00:11:26,070 --> 00:11:27,540 What do we do with read only now? 277 00:11:27,540 --> 00:11:28,373 - Okay. 278 00:11:28,373 --> 00:11:30,510 That is admittedly a little bit of a challenge, 279 00:11:30,510 --> 00:11:34,323 but fortunately this talk doesn't end there. 280 00:11:35,190 --> 00:11:36,360 So, let's go back and we'll say, 281 00:11:36,360 --> 00:11:37,800 we're gonna make this harder on ourselves, 282 00:11:37,800 --> 00:11:40,950 and let's say that somebody responsible named Will Kline, 283 00:11:40,950 --> 00:11:41,967 set up this instance of Kiali, 284 00:11:41,967 --> 00:11:43,773 and so it is in view only mode. 285 00:11:44,790 --> 00:11:45,640 What could we do? 286 00:11:48,510 --> 00:11:50,040 Enter my friend, Fleet. 287 00:11:50,040 --> 00:11:53,310 So, is anybody out there familiar with Fleet with Rancher? 288 00:11:53,310 --> 00:11:56,035 Okay. So you guys know. - Two victims. 289 00:11:56,035 --> 00:11:56,868 (both guys laughing) 290 00:11:56,868 --> 00:11:58,320 - Do you work for Rancher? - Ooh. 291 00:11:58,320 --> 00:12:01,633 - Listen up. (both guys laughing) 292 00:12:02,615 --> 00:12:03,960 So, no, this was responsibly disclosed 293 00:12:03,960 --> 00:12:06,630 and it has been resolved, but Fleet is GitOps. 294 00:12:06,630 --> 00:12:10,500 So, for those not familiar with GitOps, it's essentially, 295 00:12:10,500 --> 00:12:14,610 it's a really popular and great way, genuinely a good way, 296 00:12:14,610 --> 00:12:16,950 of administering cloud infrastructure. 297 00:12:16,950 --> 00:12:18,900 Where you put your configuration, 298 00:12:18,900 --> 00:12:21,270 you put whatever the manifest that you need to deploy 299 00:12:21,270 --> 00:12:24,570 from into your cluster, into a Git repository, 300 00:12:24,570 --> 00:12:26,700 and then you can have multiple clusters pointed 301 00:12:26,700 --> 00:12:28,110 at the same Git repository, 302 00:12:28,110 --> 00:12:31,170 and it's an easy way to build that scale. 303 00:12:31,170 --> 00:12:33,990 Also it's Git, so it's actually very auditable, 304 00:12:33,990 --> 00:12:35,070 and you can go back and say, 305 00:12:35,070 --> 00:12:37,870 Hey, Will, you deployed this thing, why did you do that? 306 00:12:39,600 --> 00:12:43,380 There's other instances of GitOps tools for Kubernetes. 307 00:12:43,380 --> 00:12:44,880 Flux is a popular one. 308 00:12:44,880 --> 00:12:46,713 Argo CD is a little bit more mature. 309 00:12:47,670 --> 00:12:49,410 But anyway, that's Fleet. 310 00:12:49,410 --> 00:12:51,570 - Well, and the other thing that's great about GitOps tools, 311 00:12:51,570 --> 00:12:54,180 like Fleet is, you can basically, 312 00:12:54,180 --> 00:12:55,440 everyone's had this situation, 313 00:12:55,440 --> 00:12:56,580 if you're any sort of cloud admin, 314 00:12:56,580 --> 00:12:58,530 where you set something up, it's running good, 315 00:12:58,530 --> 00:13:00,097 and then somebody comes along, says, 316 00:13:00,097 --> 00:13:02,100 "Oh, needed to expand that disc a little bit," 317 00:13:02,100 --> 00:13:05,070 and "Oh, needed to redirect that and change labels here." 318 00:13:05,070 --> 00:13:06,510 And as time progresses, 319 00:13:06,510 --> 00:13:08,010 you just get farther and farther away 320 00:13:08,010 --> 00:13:10,800 from whatever you thought the truth of your cluster was? 321 00:13:10,800 --> 00:13:12,960 GitOps tools will constantly be thinking that 322 00:13:12,960 --> 00:13:14,017 and looking at it and saying like, 323 00:13:14,017 --> 00:13:15,307 "Nope, back to what Git says," 324 00:13:15,307 --> 00:13:16,830 "Nope, back to what Git says." 325 00:13:16,830 --> 00:13:18,990 So, you always can quickly understand 326 00:13:18,990 --> 00:13:21,660 what's running in your cluster and how it's being deployed? 327 00:13:21,660 --> 00:13:23,640 And that's just generally a really good thing. 328 00:13:23,640 --> 00:13:25,140 Makes life easy. 329 00:13:25,140 --> 00:13:25,973 - Makes sense. 330 00:13:28,890 --> 00:13:30,900 Let's talk about Logging. 331 00:13:30,900 --> 00:13:33,360 After been professionally developing for about 15 years, 332 00:13:33,360 --> 00:13:35,340 and one of the things that I've come to learn, 333 00:13:35,340 --> 00:13:39,240 is that developers either whisper or they shout, 334 00:13:39,240 --> 00:13:40,490 when it comes to logging. 335 00:13:41,340 --> 00:13:42,780 Unfortunately in this case, 336 00:13:42,780 --> 00:13:45,580 Fleet was one that was more of a shouter. 337 00:13:45,580 --> 00:13:48,630 And as I was digging through the logs one day, 338 00:13:48,630 --> 00:13:51,273 in fact, it's even a little bit worse than that. 339 00:13:52,920 --> 00:13:55,260 When you're running Fleet with multiple clusters 340 00:13:55,260 --> 00:13:56,520 and that's not what we're doing in our demo, 341 00:13:56,520 --> 00:13:58,159 but just so you know how bad this was. 342 00:13:58,159 --> 00:14:01,620 If there's an error message, it gets displayed in the UI 343 00:14:01,620 --> 00:14:03,810 within a product called Rancher dashboard. 344 00:14:03,810 --> 00:14:06,030 And so it's very, very helpful. 345 00:14:06,030 --> 00:14:07,800 And I was working with Fleet, 346 00:14:07,800 --> 00:14:10,320 and I tried to display something and it didn't work. 347 00:14:10,320 --> 00:14:12,870 So, there was an error message that popped up that said, 348 00:14:12,870 --> 00:14:16,482 I tried to get this GitRepo, and I wasn't able to. 349 00:14:16,482 --> 00:14:20,100 The problem was Fleet uses a library 350 00:14:20,100 --> 00:14:22,050 called "GO-GETTER" from HashiCorp, 351 00:14:22,050 --> 00:14:25,620 and the way that go-getter works is you provide a URL 352 00:14:25,620 --> 00:14:27,570 that tells it everything it needs to know 353 00:14:27,570 --> 00:14:30,570 to go and get whatever file you're asking for. 354 00:14:30,570 --> 00:14:32,190 It can work with, Git repositories. 355 00:14:32,190 --> 00:14:34,348 It's a very valuable tool. 356 00:14:34,348 --> 00:14:37,800 But in order to get the Git repository, 357 00:14:37,800 --> 00:14:40,560 you need to give credentials. 358 00:14:40,560 --> 00:14:45,300 And so those credentials need to be embedded in the URL, 359 00:14:45,300 --> 00:14:48,270 and when the error comes back and says, 360 00:14:48,270 --> 00:14:51,720 I couldn't get this entire URL, 361 00:14:51,720 --> 00:14:54,180 and the credentials are in it, 362 00:14:54,180 --> 00:14:56,460 the error message that gets logged 363 00:14:56,460 --> 00:14:59,553 includes the credentials to the get repository, 364 00:15:00,480 --> 00:15:02,100 which we just said, 365 00:15:02,100 --> 00:15:04,680 you can use to deploy anything you want 366 00:15:04,680 --> 00:15:06,810 to all of your clusters. 367 00:15:06,810 --> 00:15:09,303 So, what could we do with something like that? 368 00:15:14,100 --> 00:15:16,740 So, as previously mentioned, Kiali's super-powerful. 369 00:15:16,740 --> 00:15:18,060 One of the things that lets you do 370 00:15:18,060 --> 00:15:20,880 is look at logs of individual workloads. 371 00:15:20,880 --> 00:15:23,970 So, I'm gonna take a look at the fleet-agent logs, 372 00:15:23,970 --> 00:15:24,900 and oh my goodness, 373 00:15:24,900 --> 00:15:28,110 is that a base 64 encoded SSH private key? 374 00:15:28,110 --> 00:15:29,130 Fortunately it's encoded, 375 00:15:29,130 --> 00:15:31,140 so nobody knows how to get access to this now, 376 00:15:31,140 --> 00:15:32,163 it's perfectly safe. 377 00:15:34,770 --> 00:15:36,360 This is my actual GitHub private key, 378 00:15:36,360 --> 00:15:37,950 by the way. - Yeah, of course. 379 00:15:37,950 --> 00:15:39,990 You're running low on entropy, so you only have one. 380 00:15:39,990 --> 00:15:41,550 - Let's see, a basic 64 decode. 381 00:15:41,550 --> 00:15:43,830 Oh my gosh, it's an actual private key. 382 00:15:43,830 --> 00:15:45,930 So, what else could I do with this? 383 00:15:45,930 --> 00:15:47,700 Well, one of the things that it did 384 00:15:47,700 --> 00:15:49,200 that the log message told me also 385 00:15:49,200 --> 00:15:51,540 is the actual URL to the GitRepo, 386 00:15:51,540 --> 00:15:54,060 that it was trying to read. 387 00:15:54,060 --> 00:15:56,460 Could I perhaps clone that and see what's there? 388 00:16:04,320 --> 00:16:05,163 Yep, I can. 389 00:16:07,590 --> 00:16:09,600 And it turns out it's a Helm chart, 390 00:16:09,600 --> 00:16:11,800 which I can do some interesting things with. 391 00:16:13,290 --> 00:16:16,680 It turns out the reason that there was an error is, 392 00:16:16,680 --> 00:16:19,320 if you look at the branches listed, it's only Maine. 393 00:16:19,320 --> 00:16:21,090 Maine, is the only branch that exists, 394 00:16:21,090 --> 00:16:23,370 but this is looking for the developed branch. 395 00:16:23,370 --> 00:16:26,280 So, what would happen if I created a developed branch, 396 00:16:26,280 --> 00:16:29,583 put my payload into it and pushed it back up to the GitRepo? 397 00:16:31,440 --> 00:16:32,490 Well, let's find out. 398 00:16:35,370 --> 00:16:36,520 I'll just create a jot, 399 00:16:37,440 --> 00:16:40,500 and I'm gonna put my malicious payload into that jot. 400 00:16:40,500 --> 00:16:42,650 This one's set to just pop a reverse shell. 401 00:16:48,863 --> 00:16:51,863 (audience laughing) 402 00:16:56,160 --> 00:16:58,010 Apparently I have rights permissions. 403 00:16:59,640 --> 00:17:01,320 And now it's a waiting game. 404 00:17:01,320 --> 00:17:04,233 GitOps does its thing and I have my reverse shell. 405 00:17:05,546 --> 00:17:08,713 (audience applauding) 406 00:17:11,805 --> 00:17:12,638 - Keep going. 407 00:17:12,638 --> 00:17:14,970 - So, to keep this sort of going, 408 00:17:14,970 --> 00:17:16,830 we're gonna add some more contravenes. 409 00:17:16,830 --> 00:17:19,770 Again, Fleet has global permissions. 410 00:17:19,770 --> 00:17:22,350 I could install anything that I wanted at this point. 411 00:17:22,350 --> 00:17:26,940 I truly own this cluster, but what if again, 412 00:17:26,940 --> 00:17:28,440 somebody named Will Kline set this up 413 00:17:28,440 --> 00:17:31,110 and actually constrained the default cluster role 414 00:17:31,110 --> 00:17:32,160 that Fleet runs under, 415 00:17:32,160 --> 00:17:33,710 so it can only do some things. 416 00:17:33,710 --> 00:17:36,420 It can deploy a pod, but it can't deploy a cluster role 417 00:17:36,420 --> 00:17:39,170 and cluster role bindings, which would be pretty smart. 418 00:17:41,610 --> 00:17:43,710 - So, basically, we're looking at this cluster now. 419 00:17:43,710 --> 00:17:45,450 If we think about it, we've gone in and we've taken 420 00:17:45,450 --> 00:17:47,790 a service that's generally kind of an available thing. 421 00:17:47,790 --> 00:17:49,320 We've gotten past that. 422 00:17:49,320 --> 00:17:51,420 We kinda locked it down a little bit retroactively there, 423 00:17:51,420 --> 00:17:53,640 but we can see the logs now of all the pods. 424 00:17:53,640 --> 00:17:54,690 We're looking at Fleet. 425 00:17:54,690 --> 00:17:56,070 We manage to get access to the GitOps. 426 00:17:56,070 --> 00:17:58,200 Now, one of the things you're thinking in your head is like, 427 00:17:58,200 --> 00:18:00,120 who in the world would possibly just use 428 00:18:00,120 --> 00:18:03,330 their actual SSH credentials while setting up their GitOps? 429 00:18:03,330 --> 00:18:05,490 Well, I find this all the time in the real world, 430 00:18:05,490 --> 00:18:08,093 it happens pretty often, especially with GitLab tokens, 431 00:18:08,093 --> 00:18:10,470 where the developer will go in 432 00:18:10,470 --> 00:18:13,710 and generate a personal token with rewrite access, 433 00:18:13,710 --> 00:18:14,543 instead of going in 434 00:18:14,543 --> 00:18:16,980 and creating a deploy token with only read access. 435 00:18:16,980 --> 00:18:18,000 If we only had read access, 436 00:18:18,000 --> 00:18:19,620 this would be a lot harder to do, 437 00:18:19,620 --> 00:18:22,020 but there's a lot that you can do with that information. 438 00:18:22,020 --> 00:18:24,840 Once we have write access to the GitOps repo, 439 00:18:24,840 --> 00:18:26,730 now we can push and we can do whatever we want, 440 00:18:26,730 --> 00:18:28,123 just like we are the cluster admin. 441 00:18:28,123 --> 00:18:30,180 It's really convenient for cluster admins, 442 00:18:30,180 --> 00:18:31,980 and it's also really convenient for anybody else 443 00:18:31,980 --> 00:18:34,380 who wants to run things on the cluster. 444 00:18:34,380 --> 00:18:36,660 So, really at this point, it's kind of pwned, 445 00:18:36,660 --> 00:18:38,730 because we could literally run anything we want 446 00:18:38,730 --> 00:18:41,160 because the get off tool is just Bob sledding, 447 00:18:41,160 --> 00:18:43,170 whatever we put in to get into your cluster, 448 00:18:43,170 --> 00:18:46,170 and that's how it work, and it's working as planned. 449 00:18:46,170 --> 00:18:50,674 - So, what went wrong is the developers at Rancher, 450 00:18:50,674 --> 00:18:53,850 were using a library and they used it properly, 451 00:18:53,850 --> 00:18:55,470 but they didn't fully understand 452 00:18:55,470 --> 00:18:57,783 what contents would be in that error message? 453 00:18:57,783 --> 00:19:00,540 So, they got an error and they logged the error, 454 00:19:00,540 --> 00:19:02,310 which is normally a reasonable thing to do. 455 00:19:02,310 --> 00:19:06,030 But, sometimes that can turn into an information disclosure. 456 00:19:06,030 --> 00:19:07,706 It's kind of understandable to be fair. 457 00:19:07,706 --> 00:19:10,290 When I traced it, the error was passed up 458 00:19:10,290 --> 00:19:12,330 through nine different call stacks. 459 00:19:12,330 --> 00:19:13,320 So, it was just like, 460 00:19:13,320 --> 00:19:14,850 here's an error, here's an error, here's an error. 461 00:19:14,850 --> 00:19:16,380 And when you get to the very top one, 462 00:19:16,380 --> 00:19:18,080 it was like, I don't know, log it. 463 00:19:20,730 --> 00:19:22,380 That way we know what went wrong? 464 00:19:24,540 --> 00:19:27,420 - All right, and so we're gonna talk about phase three here. 465 00:19:27,420 --> 00:19:30,150 So, let's pretend that we had in this fictitious cluster, 466 00:19:30,150 --> 00:19:31,650 we have a read only Kiali. 467 00:19:31,650 --> 00:19:33,450 We have a super locked down our back 468 00:19:33,450 --> 00:19:35,221 around our GitOps tool, 469 00:19:35,221 --> 00:19:37,871 but let's look around and see what we can find there? 470 00:19:39,420 --> 00:19:42,240 So, in this cluster, we're working on together. 471 00:19:42,240 --> 00:19:44,070 We were using this tool called Longhorn. 472 00:19:44,070 --> 00:19:47,280 Now, when I said earlier, that Kubernetes is this whole API, 473 00:19:47,280 --> 00:19:50,040 really the API describes how to do 474 00:19:50,040 --> 00:19:51,495 all the things you wanna do in your cluster. 475 00:19:51,495 --> 00:19:54,914 I wanna launch a service or I wanna have an end point. 476 00:19:54,914 --> 00:19:58,410 But there's actually the API specific implementations 477 00:19:58,410 --> 00:19:59,910 vary cluster to cluster. 478 00:19:59,910 --> 00:20:02,910 So, there's a thing called the container storage interface. 479 00:20:02,910 --> 00:20:05,400 It represents how is persistent data 480 00:20:05,400 --> 00:20:07,215 going to be stored on the cluster? 481 00:20:07,215 --> 00:20:09,330 Normally the pods are completely ephemeral, 482 00:20:09,330 --> 00:20:10,573 so you lose it, you lose it. 483 00:20:10,573 --> 00:20:14,370 Longhorn is a microservice based storage system 484 00:20:14,370 --> 00:20:15,420 that runs on Kubernetes. 485 00:20:15,420 --> 00:20:17,640 So at this point you have Kubernetes storage, 486 00:20:17,640 --> 00:20:18,930 feeding Kubernetes storage. 487 00:20:18,930 --> 00:20:20,882 So, it's a pretty neat project in that sense. 488 00:20:20,882 --> 00:20:23,693 It's made by Rancher and there's basically, 489 00:20:23,693 --> 00:20:25,140 there's a couple different pods running, 490 00:20:25,140 --> 00:20:27,270 but there are two ones that we wanna talk about today. 491 00:20:27,270 --> 00:20:29,910 One of them is an instant manager, which means that, 492 00:20:29,910 --> 00:20:33,180 which is running on every single node as a Damon set. 493 00:20:33,180 --> 00:20:36,480 And the Damon set says when a new auto scale node pops up, 494 00:20:36,480 --> 00:20:39,780 I want to launch my pod on that one and do some things. 495 00:20:39,780 --> 00:20:42,480 So, it's running every everywhere in the cluster, 496 00:20:42,480 --> 00:20:43,313 this Damon set. 497 00:20:43,313 --> 00:20:45,380 And then there's also the Longhorn manager 498 00:20:45,380 --> 00:20:48,780 in which kind of wrangles, all of those instance managers 499 00:20:48,780 --> 00:20:49,950 running all the different instances. 500 00:20:49,950 --> 00:20:52,530 So, these have a communication between them 501 00:20:52,530 --> 00:20:55,080 and they manage the provisioning of storage 502 00:20:55,080 --> 00:20:56,580 and all the different actions. 503 00:20:57,600 --> 00:21:01,200 So, as part of our hardening getting things ready, 504 00:21:01,200 --> 00:21:03,960 I pull up Longhorn and I'm looking at some 505 00:21:03,960 --> 00:21:06,270 of the pod security policies, PSPS, 506 00:21:06,270 --> 00:21:08,910 those are being deprecated, but that's a talk for CoupCon. 507 00:21:08,910 --> 00:21:11,850 And I'm looking at the default ones and I'm just like, 508 00:21:11,850 --> 00:21:14,760 wow, they're basically allowing everything. 509 00:21:14,760 --> 00:21:17,370 And there there's a requirement, there is a PSP, 510 00:21:17,370 --> 00:21:19,170 but you've done everything you can 511 00:21:19,170 --> 00:21:20,850 to disable anything useful in it. 512 00:21:20,850 --> 00:21:22,980 So, it's having the firewall where 513 00:21:22,980 --> 00:21:25,010 everything's left open to all the ports. 514 00:21:25,010 --> 00:21:26,880 What's the point of having a firewall? 515 00:21:26,880 --> 00:21:29,400 You're just checking a box that you have one at that point. 516 00:21:29,400 --> 00:21:31,260 So, we started going in and trying to figure out, 517 00:21:31,260 --> 00:21:33,479 okay, what can I narrow this down for? 518 00:21:33,479 --> 00:21:35,880 And I'm going in and out and we're launching it 519 00:21:35,880 --> 00:21:39,270 and tearing it down and just it's taking forever, 520 00:21:39,270 --> 00:21:42,390 because every time I try to take the little least amount 521 00:21:42,390 --> 00:21:46,730 of privileges away from it, it crashes and won't run. 522 00:21:46,730 --> 00:21:48,551 And so we're piecing this apart and it's like, 523 00:21:48,551 --> 00:21:51,150 well, really there is no reasonable PSP around this. 524 00:21:51,150 --> 00:21:53,014 It's doing all sorts of nasty things. 525 00:21:53,014 --> 00:21:56,670 It's bringing in directories from the root on the host, 526 00:21:56,670 --> 00:21:58,680 into the container on the inside, 527 00:21:58,680 --> 00:22:00,540 and then it's also running itself as root, 528 00:22:00,540 --> 00:22:03,420 which if you're in a container and you're mounting 529 00:22:03,420 --> 00:22:05,730 all the important directories inside your container 530 00:22:05,730 --> 00:22:08,610 and running as root, you're not really a container anymore. 531 00:22:08,610 --> 00:22:11,640 You're just a way to run code at that point. 532 00:22:11,640 --> 00:22:14,400 It's nullify as any of the security advantages. 533 00:22:14,400 --> 00:22:16,200 So, we're looking at this thinking, okay, 534 00:22:16,200 --> 00:22:18,420 well, at least we can try to make the network layer secure 535 00:22:18,420 --> 00:22:22,500 because they didn't support any network policy or MTLS. 536 00:22:22,500 --> 00:22:24,060 So, I started digging into that now, 537 00:22:24,060 --> 00:22:26,100 and just more of this weird behavior 538 00:22:26,100 --> 00:22:28,830 where the network policies are really difficult to write 539 00:22:28,830 --> 00:22:30,870 because every time it does something, 540 00:22:30,870 --> 00:22:33,630 it opens up a new port and fun fact 541 00:22:33,630 --> 00:22:34,740 that a lot of people don't know that 542 00:22:34,740 --> 00:22:36,060 when you're declaring your pod spec, 543 00:22:36,060 --> 00:22:37,950 you specify which ports you have. 544 00:22:37,950 --> 00:22:40,230 That's more of just road signs and information. 545 00:22:40,230 --> 00:22:42,690 Your pod can listen on anything that is absolutely 546 00:22:42,690 --> 00:22:44,821 not a fire- - That's not a firewall rule. 547 00:22:44,821 --> 00:22:46,110 - Yeah. (both laughing) 548 00:22:46,110 --> 00:22:47,970 We've had this conversation where they literally thought 549 00:22:47,970 --> 00:22:49,620 that those were some sort of firewall 550 00:22:49,620 --> 00:22:50,940 that was blocking traffic. 551 00:22:50,940 --> 00:22:53,280 It's really just a convenient shorthand 552 00:22:53,280 --> 00:22:54,510 to help you locate the port. 553 00:22:54,510 --> 00:22:56,970 So, we can't lock it down. 554 00:22:56,970 --> 00:23:00,519 Inside the container's a mess. The networking's a mess. 555 00:23:00,519 --> 00:23:02,970 And at some point we start talking about 556 00:23:02,970 --> 00:23:05,760 what is this pod doing and why is it literally root 557 00:23:05,760 --> 00:23:07,053 on every single host? 558 00:23:09,240 --> 00:23:11,160 So, we started digging into it a little bit 559 00:23:11,160 --> 00:23:13,560 and thinking about it and having had a conversation 560 00:23:13,560 --> 00:23:16,050 about the Kiali thing before we're like, okay, 561 00:23:16,050 --> 00:23:19,080 well, let's pick this apart and see how it's authenticating? 562 00:23:19,080 --> 00:23:21,180 Because we started noticing that we weren't seeing 563 00:23:21,180 --> 00:23:22,260 secrets in the cluster, 564 00:23:22,260 --> 00:23:24,509 which that was the same hint that we had for Kiali, 565 00:23:24,509 --> 00:23:26,160 that Kiali was up to no good. 566 00:23:26,160 --> 00:23:29,250 So, we're digging around and we're not seeing anything 567 00:23:29,250 --> 00:23:30,750 that's storing the persistent data 568 00:23:30,750 --> 00:23:33,567 to how would these pods authenticate each other? 569 00:23:33,567 --> 00:23:35,940 And if it worked with MTLS, we could kind of pretend 570 00:23:35,940 --> 00:23:37,290 that it was doing a good job 571 00:23:37,290 --> 00:23:39,990 because Istio can kind of fill that gap, 572 00:23:39,990 --> 00:23:41,788 but we couldn't get it to work with Istio, 573 00:23:41,788 --> 00:23:44,258 because of this weird network behavior. 574 00:23:44,258 --> 00:23:47,130 So, we're digging around and sure enough we found 575 00:23:47,130 --> 00:23:49,530 that they actually weren't authenticating each other. 576 00:23:49,530 --> 00:23:52,860 It's just a GRPC that's listing on a standard port 577 00:23:52,860 --> 00:23:55,143 that's not declared, and away it goes. 578 00:23:57,540 --> 00:24:02,069 - Okay. So, GRPC, for those who may not be aware 579 00:24:02,069 --> 00:24:06,150 is a remote procedure called Protocol from Google. 580 00:24:06,150 --> 00:24:07,760 It runs on top of HTTP/2. 581 00:24:07,760 --> 00:24:10,650 It uses protocol buffers to encode data. 582 00:24:10,650 --> 00:24:14,280 I love it. It's a very cool protocol. 583 00:24:14,280 --> 00:24:17,283 You should absolutely be using GRPC, like truthfully. 584 00:24:19,230 --> 00:24:21,480 One of the nice things about it is you don't have 585 00:24:21,480 --> 00:24:24,390 to write the server and client bits yourself. 586 00:24:24,390 --> 00:24:27,780 You write a protocol or protofile that defines 587 00:24:27,780 --> 00:24:29,250 this is what my API looks like? 588 00:24:29,250 --> 00:24:30,360 If you're familiar with swagger, 589 00:24:30,360 --> 00:24:32,850 it's a little bit like that, but then in addition, 590 00:24:32,850 --> 00:24:36,120 GRPC has plugins for all sorts of languages. 591 00:24:36,120 --> 00:24:40,050 So, you can run that plugin in your compiler 592 00:24:40,050 --> 00:24:41,670 and point it at your protofile 593 00:24:41,670 --> 00:24:44,370 and you actually get the server or the client. 594 00:24:44,370 --> 00:24:46,470 With the server, it expects you to have something 595 00:24:46,470 --> 00:24:48,930 that implements an actual interface that does the thing, 596 00:24:48,930 --> 00:24:50,850 but it gives you all the connective bits in between, 597 00:24:50,850 --> 00:24:51,990 and it's really nice. 598 00:24:51,990 --> 00:24:54,810 It's particularly nice if you're an endpoint developer 599 00:24:54,810 --> 00:24:56,730 and you wanna support people that are writing 600 00:24:56,730 --> 00:24:59,460 in different languages because you could write and Go, 601 00:24:59,460 --> 00:25:00,930 somebody else could write in Java. 602 00:25:00,930 --> 00:25:02,190 I don't know why they'd write in Java, 603 00:25:02,190 --> 00:25:03,630 but they could write in Java, 604 00:25:03,630 --> 00:25:05,280 and then they could just run the plugin 605 00:25:05,280 --> 00:25:08,193 and they get a Java client for your API. 606 00:25:09,780 --> 00:25:13,470 It's also really nice because it's easy to read. 607 00:25:13,470 --> 00:25:17,400 So, I was really excited when I looked at Longhorns repo 608 00:25:17,400 --> 00:25:18,400 and just saw these protocols in there. 609 00:25:18,400 --> 00:25:21,240 I was like, oh, I can read that. That's simple. 610 00:25:21,240 --> 00:25:23,880 So, I started pulling it apart and I saw this service 611 00:25:23,880 --> 00:25:26,730 that was called "Create Process." 612 00:25:26,730 --> 00:25:31,350 And I thought, well, I hope that the Longhorn developers 613 00:25:31,350 --> 00:25:35,250 have a different meaning of process than what I have. 614 00:25:35,250 --> 00:25:36,930 So, let me look at the parameters 615 00:25:36,930 --> 00:25:38,733 that get passed to this service. 616 00:25:38,733 --> 00:25:40,770 And there was fun little bits in there, 617 00:25:40,770 --> 00:25:45,770 like binary and ARGs, and I thought, 618 00:25:46,200 --> 00:25:50,910 surely this is not an API 619 00:25:50,910 --> 00:25:52,530 that allows anybody inside the cluster. 620 00:25:52,530 --> 00:25:54,690 'Cause remember cluster's networking perimeter, 621 00:25:54,690 --> 00:25:56,070 is the cluster. 622 00:25:56,070 --> 00:25:57,570 It's not at the name space. 623 00:25:57,570 --> 00:26:00,720 So, no ability to add a network policy. 624 00:26:00,720 --> 00:26:04,530 No ability to use Istio to do MTLS enforcements. 625 00:26:04,530 --> 00:26:07,170 No authentication. 626 00:26:07,170 --> 00:26:08,160 This can't be the case. 627 00:26:08,160 --> 00:26:09,870 They must be doing authentication somehow. 628 00:26:09,870 --> 00:26:12,330 Now long run manager does generate the pods itself 629 00:26:12,330 --> 00:26:13,500 and push 'em out there. 630 00:26:13,500 --> 00:26:15,030 So, it could be generating something as 631 00:26:15,030 --> 00:26:16,080 an environment variable, 632 00:26:16,080 --> 00:26:17,340 which wouldn't be the most secure thing, 633 00:26:17,340 --> 00:26:18,453 but it'd be okay-ish. 634 00:26:19,440 --> 00:26:23,793 No, no, it wasn't authenticated at all. 635 00:26:26,580 --> 00:26:28,710 I work on the West Coast and Will works on the East Coast. 636 00:26:28,710 --> 00:26:31,650 So, I have to wake up at 5:00 AM and start work. 637 00:26:31,650 --> 00:26:34,100 So, it's about 5:05 AM, and I was looking at this 638 00:26:35,551 --> 00:26:37,470 and I'm awake now and I reached out to Will, 639 00:26:37,470 --> 00:26:39,120 and we said, well, let's do a POC, 640 00:26:39,120 --> 00:26:41,640 like this can't be a thing. 641 00:26:41,640 --> 00:26:44,520 So, we started looking at it a little bit and I said, 642 00:26:44,520 --> 00:26:47,220 well, what can we do with this access? 643 00:26:47,220 --> 00:26:49,717 I mean, it's sandbox, it's a container. 644 00:26:49,717 --> 00:26:52,383 Well, we'll go back to what Will was talking about 645 00:26:52,383 --> 00:26:56,010 the file systems from the host that were being mounted. 646 00:26:56,010 --> 00:27:00,813 So, there was two, proc and Def. Not the best two. 647 00:27:04,680 --> 00:27:09,680 And Will said you know, in proc PID-1 route, 648 00:27:10,080 --> 00:27:13,980 that will always be the host's route file system. 649 00:27:13,980 --> 00:27:15,780 So, I wonder if we could just say something, 650 00:27:15,780 --> 00:27:20,780 that's just gonna touch something in host proc one route, 651 00:27:20,880 --> 00:27:23,684 let's get fun, Etsy, whatever. 652 00:27:23,684 --> 00:27:26,520 So, because this was written in with GRPC, 653 00:27:26,520 --> 00:27:29,490 it took me about 25 minutes to slap together 654 00:27:29,490 --> 00:27:32,610 a real quick POC and go and run this thing, 655 00:27:32,610 --> 00:27:35,940 and Will sitting in there, look at he's SSH into the node. 656 00:27:35,940 --> 00:27:39,510 And he's like, okay, I just did an LS on Etsy. 657 00:27:39,510 --> 00:27:42,360 Go, I go, I hit go. He refreshes. 658 00:27:42,360 --> 00:27:44,910 And there's a file that says this should not exist. 659 00:27:45,967 --> 00:27:47,613 (both laughing) Owned by route. 660 00:27:53,531 --> 00:27:55,050 So, we took that a little bit further. 661 00:27:55,050 --> 00:27:57,450 We developed an exploit it for it. 662 00:27:57,450 --> 00:28:00,480 It's called Rustler. It is available on GitHub. 663 00:28:00,480 --> 00:28:03,660 It basically allows you to point Rustler at a target 664 00:28:03,660 --> 00:28:07,500 that's running Longhorn and just say, do this for me please, 665 00:28:07,500 --> 00:28:09,000 and it does every time. 666 00:28:09,000 --> 00:28:09,900 It's very reliable 667 00:28:11,700 --> 00:28:14,730 Now to Longhorns credit, again, all the developers 668 00:28:14,730 --> 00:28:16,830 we worked with on this have been really amazing. 669 00:28:16,830 --> 00:28:17,700 We reached out to them. 670 00:28:17,700 --> 00:28:20,880 They immediately understood the problem 671 00:28:20,880 --> 00:28:22,180 and they worked to fix it. 672 00:28:23,580 --> 00:28:25,080 But before they did that, 673 00:28:25,080 --> 00:28:27,203 let's take a look at what we were able to do. 674 00:28:32,040 --> 00:28:35,906 So, I'm back on my reverse proxy from the previous demo. 675 00:28:35,906 --> 00:28:38,280 I need to find Longhorn. I need to find something. 676 00:28:38,280 --> 00:28:41,610 So, I'm gonna temporarily confuse broadcast with netmask. 677 00:28:41,610 --> 00:28:44,190 Ignore that and I'm gonna run a quick end map 678 00:28:44,190 --> 00:28:46,050 across the subnet that I'm working on, 679 00:28:46,050 --> 00:28:47,880 looking for anything, listening on 8,500, 680 00:28:47,880 --> 00:28:50,490 which is what instance manager listens on. 681 00:28:50,490 --> 00:28:52,740 It's not a big subnet, so it doesn't take long. 682 00:28:52,740 --> 00:28:54,510 There's two. I'll grab one. 683 00:28:54,510 --> 00:28:57,480 Remember it's running a Ubuntu, that's the base image. 684 00:28:57,480 --> 00:29:01,140 So, I have access to all of a Ubuntu 2004, 685 00:29:01,140 --> 00:29:02,790 that I can run commands against. 686 00:29:02,790 --> 00:29:04,740 So, I'm gonna start up listening again, 687 00:29:04,740 --> 00:29:06,240 and I'm just gonna do a really basic 688 00:29:06,240 --> 00:29:07,923 one liner bash reverse shell. 689 00:29:20,190 --> 00:29:24,803 Who am I? I'm Route! (audience applauding) 690 00:29:30,480 --> 00:29:33,510 So, going into the proc directory, 691 00:29:33,510 --> 00:29:37,353 if I change route to host proc one route, 692 00:29:38,970 --> 00:29:40,623 now I'm route on the host. 693 00:29:42,270 --> 00:29:43,920 And for the cube administrators out there, 694 00:29:43,920 --> 00:29:45,033 I can run cry control. 695 00:29:46,200 --> 00:29:48,660 So, basically, at this point, 696 00:29:48,660 --> 00:29:51,090 I can do whatever I want with your cluster. 697 00:29:51,090 --> 00:29:52,743 If this isn't enough fun for you, 698 00:29:54,000 --> 00:29:57,900 Longhorn has essentially two different modes 699 00:29:57,900 --> 00:29:58,733 that it's running in. 700 00:29:58,733 --> 00:30:00,870 There's like everything and then there's Longhorn manager, 701 00:30:00,870 --> 00:30:02,670 which Will was talking about before, 702 00:30:02,670 --> 00:30:04,500 which is creating a lot of pods. 703 00:30:04,500 --> 00:30:05,850 It's doing a lot of things. 704 00:30:05,850 --> 00:30:09,150 So, it has very permissive our back rules. 705 00:30:09,150 --> 00:30:11,640 It's allowed to do a lot of things. 706 00:30:11,640 --> 00:30:13,020 For some reason, it's allowed to do those 707 00:30:13,020 --> 00:30:14,580 in any name space it wants. 708 00:30:14,580 --> 00:30:17,220 It's not isolated to the Longhorn system name space. 709 00:30:17,220 --> 00:30:18,330 I don't know why that's the case. 710 00:30:18,330 --> 00:30:20,954 It doesn't seem to be necessary at all, but it is the case. 711 00:30:20,954 --> 00:30:24,930 Also, at least previously, even though 712 00:30:24,930 --> 00:30:26,940 there was two very different security modes of, 713 00:30:26,940 --> 00:30:27,990 I need to do a lot of things 714 00:30:27,990 --> 00:30:29,670 and I basically need to do nothing. 715 00:30:29,670 --> 00:30:32,610 There's one service account and that service account 716 00:30:32,610 --> 00:30:34,860 is shared across all of the pods. 717 00:30:34,860 --> 00:30:37,080 So at this point, running an instant manager, 718 00:30:37,080 --> 00:30:40,200 I am able to run inside that pod cube control 719 00:30:40,200 --> 00:30:41,580 and create pods. 720 00:30:41,580 --> 00:30:43,482 I can do fun things like read any secret 721 00:30:43,482 --> 00:30:47,250 in the cluster anywhere, create, read, update, delete, 722 00:30:47,250 --> 00:30:51,000 on almost every type of resource. 723 00:30:51,000 --> 00:30:55,260 Even if I didn't have that, Fleet still got my back, 724 00:30:55,260 --> 00:31:00,260 because Fleet's permissions are API versions, all of them. 725 00:31:01,050 --> 00:31:05,940 Resources, all of them. Verbs, all of them. 726 00:31:05,940 --> 00:31:08,943 So, anything that I want, I can do. 727 00:31:11,640 --> 00:31:13,890 - So, basically, once you have that little foothold 728 00:31:13,890 --> 00:31:14,970 in the cluster that we've gotten, 729 00:31:14,970 --> 00:31:17,040 and really this is just a one way to do it. 730 00:31:17,040 --> 00:31:18,231 But a lot of people don't think about, 731 00:31:18,231 --> 00:31:21,510 they think about the containers as a security mechanism 732 00:31:21,510 --> 00:31:25,050 and they assume that once you're applications in a container 733 00:31:25,050 --> 00:31:26,550 that's gonna hold back an attacker. 734 00:31:26,550 --> 00:31:27,870 But realistically, 735 00:31:27,870 --> 00:31:30,594 because name spaces are not a security boundary, 736 00:31:30,594 --> 00:31:33,330 and there's really nothing that's a firewall in this, 737 00:31:33,330 --> 00:31:36,030 Kubernetes makes service discovery really easy, 738 00:31:36,030 --> 00:31:39,000 which is why we were able to find Longhorn pods so quickly 739 00:31:39,000 --> 00:31:41,160 and there's nothing holding us back from talking to them. 740 00:31:41,160 --> 00:31:42,750 So, it's one of those things that 741 00:31:42,750 --> 00:31:43,830 when you're deploying a cluster, 742 00:31:43,830 --> 00:31:45,120 you need to be thinking about 743 00:31:45,120 --> 00:31:46,950 where are the actual security barriers, 744 00:31:46,950 --> 00:31:51,093 and where are there shadows of insert security barrier here? 745 00:31:51,960 --> 00:31:52,793 You can write. 746 00:31:52,793 --> 00:31:53,910 - Somebody's got this for me, right? 747 00:31:53,910 --> 00:31:54,827 - Yeah. Someday someone's gonna insert 748 00:31:54,827 --> 00:31:56,820 in the security policy that blocks this. 749 00:31:56,820 --> 00:31:59,160 But building out your cluster, 750 00:31:59,160 --> 00:32:00,706 you really need to think about that. 751 00:32:00,706 --> 00:32:03,810 Basically, once we're on the host, 752 00:32:03,810 --> 00:32:06,090 there's very little that we can't do on that host 753 00:32:06,090 --> 00:32:07,890 as far as getting back to the master nodes 754 00:32:07,890 --> 00:32:09,120 or the control plane stuff. 755 00:32:09,120 --> 00:32:10,620 And we don't even really need to do anything 756 00:32:10,620 --> 00:32:12,100 on the control plane, 757 00:32:12,100 --> 00:32:13,170 because we can launch literally any pod we want, 758 00:32:13,170 --> 00:32:16,140 including more incredibly vulnerable pods 759 00:32:16,140 --> 00:32:18,930 that mounts very dangerous directories into them. 760 00:32:18,930 --> 00:32:20,400 So, there's really nothing we can't do 761 00:32:20,400 --> 00:32:21,909 in the cluster at that point. 762 00:32:21,909 --> 00:32:24,870 - And if there are good pod security policies in place 763 00:32:24,870 --> 00:32:26,670 that prevent us from doing that, it's okay. 764 00:32:26,670 --> 00:32:28,380 We can deploy our own pod security policies 765 00:32:28,380 --> 00:32:29,411 that correct that for us. 766 00:32:29,411 --> 00:32:32,078 (Will laughing) 767 00:32:37,080 --> 00:32:39,270 All right, so some big takeaways. 768 00:32:39,270 --> 00:32:41,760 One thing that we do really want to emphasize 769 00:32:41,760 --> 00:32:44,520 is Kubernetes itself is pretty great. 770 00:32:44,520 --> 00:32:48,240 The Kubernetes developers are absolutely amazing. 771 00:32:48,240 --> 00:32:50,070 There's a ton of them and there's some 772 00:32:50,070 --> 00:32:51,660 of the most brilliant developers 773 00:32:51,660 --> 00:32:54,060 that I've ever had the pleasure of working with, 774 00:32:54,960 --> 00:32:56,970 but not everything in the Kubernetes ecosystem 775 00:32:56,970 --> 00:32:58,980 is written by those Kubernetes developers. 776 00:32:58,980 --> 00:33:01,980 And a lot of times to be really fair, these projects, 777 00:33:01,980 --> 00:33:05,100 they start off as one developer's idea, 778 00:33:05,100 --> 00:33:06,750 and that turns into a proof of concept, 779 00:33:06,750 --> 00:33:07,920 and then someone goes, 780 00:33:07,920 --> 00:33:09,990 cool, ship it, it's mission critical, let's go. 781 00:33:09,990 --> 00:33:12,602 And we don't necessarily rewind and sort of think rethink. 782 00:33:12,602 --> 00:33:15,753 Is it secure? Is it what we need it to be? 783 00:33:17,550 --> 00:33:19,830 I mean, you have some bullet points on that. 784 00:33:19,830 --> 00:33:21,570 - One of my things that I like to drive home with people is 785 00:33:21,570 --> 00:33:23,824 like good code does not equal good deployments. 786 00:33:23,824 --> 00:33:25,950 One of the advantages of Kubernetes being 787 00:33:25,950 --> 00:33:28,110 the ultimate shift left capability is 788 00:33:28,110 --> 00:33:30,450 that developers get to work on deployments. 789 00:33:30,450 --> 00:33:33,250 But there's also people that have been working in system 790 00:33:34,113 --> 00:33:36,720 administration and building all of this cluster stuff out. 791 00:33:36,720 --> 00:33:39,570 A lot of lessons learned about how to have least privilege 792 00:33:39,570 --> 00:33:41,730 and the developers aren't necessarily thinking about that. 793 00:33:41,730 --> 00:33:44,010 So, a lot of what times what happens is you'll see 794 00:33:44,010 --> 00:33:46,920 a perfectly good project that was actually implemented 795 00:33:46,920 --> 00:33:48,660 pretty well on the coding part. 796 00:33:48,660 --> 00:33:51,690 Then when you go to deploy it with the Helm chart, 797 00:33:51,690 --> 00:33:53,220 the NAMI has a lot of this. 798 00:33:53,220 --> 00:33:55,860 None of the defaults are anywhere near secure, 799 00:33:55,860 --> 00:33:57,600 and it's difficult to secure it 800 00:33:57,600 --> 00:33:58,830 because of the way things are written. 801 00:33:58,830 --> 00:34:01,230 So, just because you have a really secure project, 802 00:34:01,230 --> 00:34:03,810 like your MySQL, your Postgres that's up to date, 803 00:34:03,810 --> 00:34:06,810 doesn't mean that the deployment of that system 804 00:34:06,810 --> 00:34:08,700 can't shoot you in the foot. 805 00:34:08,700 --> 00:34:12,030 - So, not that there's any attackers in the audience today, 806 00:34:12,030 --> 00:34:14,550 but if there were, what do you do? 807 00:34:14,550 --> 00:34:16,285 You got your hands on a Kubernetes cluster. 808 00:34:16,285 --> 00:34:18,660 Maybe you found an insecure application 809 00:34:18,660 --> 00:34:20,490 that lets you get just a toe in. 810 00:34:20,490 --> 00:34:21,690 How do you pivot? 811 00:34:21,690 --> 00:34:24,300 Understand some of the common mistakes that people make? 812 00:34:24,300 --> 00:34:27,630 What are the assumptions that are just frankly inaccurate 813 00:34:27,630 --> 00:34:29,070 in a Kubernetes environment? 814 00:34:29,070 --> 00:34:31,744 So again, going back to service discovery, 815 00:34:31,744 --> 00:34:36,266 there is unfortunately commonly a presumption that networks 816 00:34:36,266 --> 00:34:39,270 are like that name spaces are network segmentation. 817 00:34:39,270 --> 00:34:40,290 They're not. 818 00:34:40,290 --> 00:34:43,080 So, look for people that are just have completely 819 00:34:43,080 --> 00:34:45,680 insecure APIs and don't be surprised if you find it. 820 00:34:46,590 --> 00:34:48,870 Secrets should exist somewhere. 821 00:34:48,870 --> 00:34:51,413 Somehow information needs to be shared across 822 00:34:51,413 --> 00:34:53,040 these different applications 823 00:34:53,040 --> 00:34:55,230 that need to be able to communicate with each other. 824 00:34:55,230 --> 00:34:56,430 If that doesn't exist, 825 00:34:56,430 --> 00:34:58,980 ask yourself how is this being secured? 826 00:34:58,980 --> 00:35:00,603 'Cause it might not be. 827 00:35:02,160 --> 00:35:05,910 Additionally, GitOps is super fun to play with. 828 00:35:05,910 --> 00:35:09,210 If you can get access to GitOps repo, 829 00:35:09,210 --> 00:35:10,770 you own not one cluster, 830 00:35:10,770 --> 00:35:13,908 but lots and lots of clusters in all likelihood. 831 00:35:13,908 --> 00:35:15,090 So, watch for that. 832 00:35:15,090 --> 00:35:16,980 And then finally for me, at least, 833 00:35:16,980 --> 00:35:19,830 don't forget to check the logs, I go back to it. 834 00:35:19,830 --> 00:35:20,882 Some developers whispers, 835 00:35:20,882 --> 00:35:23,433 other developers definitely are shouting. 836 00:35:25,560 --> 00:35:28,920 - I think one of the things that I wanna wrap up on here 837 00:35:28,920 --> 00:35:31,230 is that Kubernetes is pretty great 838 00:35:31,230 --> 00:35:32,670 and has all this great stuff in there, 839 00:35:32,670 --> 00:35:36,480 but a lot of the bad behaviors that people do 840 00:35:36,480 --> 00:35:38,310 on their local development machines, 841 00:35:38,310 --> 00:35:40,230 just has a tendency to get embedded 842 00:35:40,230 --> 00:35:42,090 into the deployment systems, 843 00:35:42,090 --> 00:35:44,790 and then end up through GitOps into the cluster. 844 00:35:44,790 --> 00:35:48,240 So, in no world is GitOps, I used to joke a lot 845 00:35:48,240 --> 00:35:50,940 with people that like devs or sec devs, 846 00:35:50,940 --> 00:35:53,310 is really just that we took the developers 847 00:35:53,310 --> 00:35:54,813 and fired all the ops team. 848 00:35:56,070 --> 00:35:58,170 There's a lot of lessons learned from traditional ops 849 00:35:58,170 --> 00:36:01,980 that can be applied to through DevOps type stuff, 850 00:36:01,980 --> 00:36:04,650 to the deployment of applications on Kubernetes, 851 00:36:04,650 --> 00:36:07,350 and we can't let the devs just take over the whole process 852 00:36:07,350 --> 00:36:09,250 because we forgot to hire an ops team. 853 00:36:11,580 --> 00:36:12,679 - So, before we go to questions, 854 00:36:12,679 --> 00:36:16,650 again, I mentioned that the full demo environment 855 00:36:16,650 --> 00:36:19,080 is automated and available. 856 00:36:19,080 --> 00:36:20,910 These slides are available too, by the way, 857 00:36:20,910 --> 00:36:23,790 any of the main tracks, if you go to media.defcon.org, 858 00:36:23,790 --> 00:36:24,960 you can get all these materials, 859 00:36:24,960 --> 00:36:27,061 the recording, all that's there. 860 00:36:27,061 --> 00:36:32,040 But feel free to go there to github.com/dagan/defcon30. 861 00:36:32,040 --> 00:36:34,230 I promise to really actually try to support folks 862 00:36:34,230 --> 00:36:35,310 for at least the next couple weeks. 863 00:36:35,310 --> 00:36:37,200 So, if you have problems working through any of this, 864 00:36:37,200 --> 00:36:40,083 open an issue, it should be pretty easy for folks to do. 865 00:36:40,920 --> 00:36:42,540 Also today we talked a lot about 866 00:36:42,540 --> 00:36:44,250 what can go wrong in Kubernetes? 867 00:36:44,250 --> 00:36:45,690 And if you're interested in knowing 868 00:36:45,690 --> 00:36:47,469 how to protect your clusters from that? 869 00:36:47,469 --> 00:36:49,230 We're gonna be doing a similar talk, 870 00:36:49,230 --> 00:36:52,260 but in the defense side at CoupCon, 871 00:36:52,260 --> 00:36:54,300 cloud NativeCOM in Detroit, in October. 872 00:36:54,300 --> 00:36:56,370 So, feel free to meet up with us there. 873 00:36:56,370 --> 00:36:58,050 With that being said, we have, 874 00:36:58,050 --> 00:37:00,540 I think just under 10 minutes or so left, 875 00:37:00,540 --> 00:37:02,370 so if anybody has any questions, 876 00:37:02,370 --> 00:37:04,140 there's a goon back there with the microphone. 877 00:37:04,140 --> 00:37:05,550 We'll do our best to answer 'em for you. 878 00:37:05,550 --> 00:37:07,957 Thank you very much. - Yeah. Thank you. 879 00:37:07,957 --> 00:37:11,124 (audience applauding) 880 00:37:15,210 --> 00:37:16,623 - Anybody with questions? 881 00:37:19,230 --> 00:37:20,530 I guess we covered it all. 882 00:37:23,850 --> 00:37:25,530 All right. Well, we'll be hanging around for a little bit 883 00:37:25,530 --> 00:37:27,060 and then headed out that door, 884 00:37:27,060 --> 00:37:29,130 so if anybody wants to ask us anything directly, 885 00:37:29,130 --> 00:37:30,150 again, thank you all for your time. 886 00:37:30,150 --> 00:37:32,483 - Thank you. Thanks for coming out.