1 00:00:00,000 --> 00:00:10,820 You did a great job. Thanks jurist. You're amazing. Why hello everybody. It's so uh it's 2 00:00:10,820 --> 00:00:16,180 so awesome to be here today. I'm excited to bring you uh one of my passion projects called 3 00:00:16,180 --> 00:00:22,220 Private Keys in Public Places. Before we get started I'll let you know who I am. My name is 4 00:00:22,220 --> 00:00:28,560 Tom Pohl. Uh I'm the penetration testing team manager for LMG security out of Missoula 5 00:00:28,560 --> 00:00:36,100 Montana. Uh I like to do things like play CTFs. Uh I I I'm a I you know a penetration tester 6 00:00:36,100 --> 00:00:41,620 by day, dream crusher at night. Um I've won a few CTFs. Uh I've won black badge here at 7 00:00:41,620 --> 00:00:47,740 DEF CON. I've uh won like gold badge at DOT CON and a couple at Wild West Hackfest. Three 8 00:00:47,740 --> 00:00:54,660 times at Circle City Con. But uh I am an evil twin. I'd like to point out uh hi Andy. My twin 9 00:00:54,660 --> 00:00:58,540 brother is uh this is his first DEF CON. He is here in the audience tonight. He's been 10 00:00:58,540 --> 00:01:02,020 here tonight. So if you see me around the con and I look like I don't know what I'm doing it's 11 00:01:02,020 --> 00:01:10,180 probably him. At any rate um and this is me hard at work. My wife took a picture that just doesn't 12 00:01:10,180 --> 00:01:14,300 exist uh of me uh in a hacker hoodie. I don't think I've ever worn a hoodie besides this picture 13 00:01:14,300 --> 00:01:21,320 in my life. Uh at any rate I want to give a shout out uh greets to uh Sec KC and Sec DSM. As 14 00:01:21,320 --> 00:01:27,980 Jera said. You know in Des Moines we like to call ourselves uh Sec KC's little brother because 15 00:01:27,980 --> 00:01:34,620 we kick the we kick him in the shins. The last time I saw jurists I was uh giving a talk at Sec 16 00:01:34,620 --> 00:01:41,880 KC and we had just deprived him of being able to mine cryptocurrency of the Sec KC coin. And uh 17 00:01:41,880 --> 00:01:49,700 but they used it to buy swag and and I gave him a paper wallet with 500 Sec KC coin in it so he 18 00:01:49,700 --> 00:01:56,060 could buy himself a little patch of swag at the Sec KC store. And then I gave paper wallets to 19 00:01:56,060 --> 00:01:57,880 everybody in the room because they're taped up. 20 00:01:57,980 --> 00:02:03,660 Under their seats. It was great. Anyways love those guys. Greets to the Sec KC. And I want to 21 00:02:03,660 --> 00:02:10,460 give a special thanks to my buddy Nick Stark my partner in crime thought crimes. And he's really 22 00:02:10,460 --> 00:02:15,180 the reason why I started looking for private keys. I will get into that in a little bit and 23 00:02:15,180 --> 00:02:21,400 also Sherry Davidoff. She couldn't be here today. My wife. She's amazing. This talk would not have 24 00:02:21,400 --> 00:02:26,400 existed without her because she's been urging me to create this for years. Literally years because 25 00:02:26,400 --> 00:02:27,960 I've been been at this for quite a while. So thank you very much. Thank you very much. 26 00:02:27,980 --> 00:02:33,860 Thank you so much my love. I hope you're watching. Uh so the roadmap. So we're gonna talk a 27 00:02:33,860 --> 00:02:39,320 little bit of background uh and then we're gonna get into 3 different uh examples of how I find 28 00:02:39,320 --> 00:02:45,520 private keys in public places. And then we'll get at the very end into some recommendations. So 29 00:02:45,520 --> 00:02:52,800 first of all why do we care about private keys? Right? It's not you know the hot sexy new uh 30 00:02:52,800 --> 00:02:57,340 vulnerability out there. In fact there's some really great projects out there that are uh extracting 31 00:02:57,980 --> 00:03:03,560 keys and and doing things but I don't I don't think it gets as much airtime and publicity as as 32 00:03:03,560 --> 00:03:08,560 it as it needs because this is a systemic problem and it's happening in the real world right? As 33 00:03:08,560 --> 00:03:14,940 attackers uh get into your networks uh they're looking for easy ways to gain more and more 34 00:03:14,940 --> 00:03:21,920 access and this is in using keys especially in the in cloud environments is definitely uh a way 35 00:03:21,920 --> 00:03:26,500 that people are do that the the bad guys are doing it right? So we really need to raise 36 00:03:26,500 --> 00:03:35,380 awareness for this. So once upon a time uh back in January 2020 imagine this is pre-COVID right? 37 00:03:35,480 --> 00:03:40,900 Nobody knew what COVID was back then. I was at a conference and my buddy Nick Stark he's over there 38 00:03:40,900 --> 00:03:47,340 in the audience. Hi Nick. I love you. Uh he was he he texted I think he texted me. I think you 39 00:03:47,340 --> 00:03:51,940 were texting. We weren't talking that day and he was angry at Netgear because he found a bunch of 40 00:03:51,940 --> 00:03:56,400 vulnerabilities. He's if you don't know Nick Stark look him up. He's like 41 00:03:56,500 --> 00:04:02,340 the hacker's hacker like he is amazing. And uh he found a bunch of problems on Netgear and they're 42 00:04:02,340 --> 00:04:06,600 trying to like bundle them all into one even though they're like totally different problems uh 43 00:04:06,600 --> 00:04:10,740 and so he called me being angry about it and I'm like trying to console him I'm like you know 44 00:04:10,740 --> 00:04:15,140 what here's how we're gonna do we're let's just go hack some shit. And I'm like you know I'm 45 00:04:15,140 --> 00:04:21,240 like no idea I'm like he's like what should we hack on? Hint hint. Netgear. Since he was angry 46 00:04:21,240 --> 00:04:26,180 at Netgear like let's just go start pulling random firmware and let's start exploring it and see 47 00:04:26,180 --> 00:04:34,320 if we can find anything. And uh we're about three firmwares in probably before uh uh looking 48 00:04:34,320 --> 00:04:40,780 and we found two different certificates that were publicly uh uh CA blast certificates with 49 00:04:40,780 --> 00:04:47,740 their associated private keys both for routerlogin.net and the mini app uh funjsq.com. Both 50 00:04:47,740 --> 00:04:53,340 from from reputable uh certificate authorities like the certificates there it's not expired and 51 00:04:53,340 --> 00:04:59,060 the private keys with it. Um and just to show you uh I actually took those keys put them on my own 52 00:04:59,060 --> 00:05:04,160 server I you know controlled the DNS for an environment went to it sure enough you get you 53 00:05:04,160 --> 00:05:09,420 know a happy lock icon the certificates are valid the whole deal because I controlled the 54 00:05:09,420 --> 00:05:13,800 certi- the uh the DNS. And it's like okay well you a lot of people are like whoa that's not a 55 00:05:13,800 --> 00:05:18,480 big deal is it? Uh and and actually it is there's a couple of guys that went out there and and 56 00:05:18,480 --> 00:05:23,320 wrote some really good articles about like actually this is a pretty big deal. You know and uh uh I'm 57 00:05:23,340 --> 00:05:29,260 uh I am uh a professional bad guy but we're a good guy. I'm helping you I'm out there helping 58 00:05:29,260 --> 00:05:33,280 people find vulnerabilities on their network right? But I was kinda you know thinking about 59 00:05:33,280 --> 00:05:39,320 like if I was to be a bad guy uh what what would I do right? Like say I was targeting an 60 00:05:39,320 --> 00:05:45,000 individual and I went to the same coffee shop they were in right? I could set up you know my fake 61 00:05:45,000 --> 00:05:50,140 routerlogin.net and if I knew they were running this Netgear product uh you know I could uh fake 62 00:05:53,340 --> 00:05:57,320 their website and then if they save their password when they log into their router I can get 63 00:05:57,320 --> 00:06:02,160 that password right? I can have them autofill that password and then do a autosubmit and then I'll 64 00:06:02,160 --> 00:06:08,520 get their password and then uh uh if uh um uh if they have remote administration at home then I 65 00:06:08,520 --> 00:06:12,740 could go log into their home router and play with them all day long or you know if they have the 66 00:06:12,740 --> 00:06:16,620 same password say we're gonna do a credential stuffing attack I could go then maybe log into 67 00:06:16,620 --> 00:06:21,900 some of their other services across the internet. Alright uh by the way I wanted to say if 68 00:06:21,900 --> 00:06:23,320 anybody's got questions. 69 00:06:23,340 --> 00:06:28,040 While I'm talking through this feel free to jump up I think there's a mic in the middle in the 70 00:06:28,040 --> 00:06:33,660 back and interrupt me cause I love interaction. In fact I brought with me some Montana hacker 71 00:06:33,660 --> 00:06:38,300 shot glasses so if you got a good question I will chuck these at you for a good question so 72 00:06:38,980 --> 00:06:46,580 there's a bounty on good questions here. Um alright so we tried to tell Netgear right? Uh it 73 00:06:46,580 --> 00:06:51,280 was what January 14th so really early in January was when we found it and we went you know we 74 00:06:51,280 --> 00:06:53,320 tried to contact Netgear actually in a couple different ways. We tried to contact Netgear 75 00:06:53,340 --> 00:06:57,780 in different ways. The most we had gotten was through their direct message on Twitter or X or 76 00:06:57,780 --> 00:07:03,300 whatever it's called now. Um and then uh so we decided that was like kind of a no go they 77 00:07:03,300 --> 00:07:06,640 weren't really wanting to talk to us so we reached out to bug crowd who ran their bug bounty 78 00:07:06,640 --> 00:07:11,840 program to try to establish communications. And uh they came back and they would not accept 79 00:07:11,840 --> 00:07:18,680 information about our vulnerability outside of their program. Which was really frustrating 80 00:07:18,680 --> 00:07:23,120 because uh here's why. Uh they require what's called 81 00:07:23,340 --> 00:07:27,180 responsible disclosure. So anybody who's familiar with bug bounty programs there's a couple 82 00:07:27,180 --> 00:07:32,380 different ways you you can uh uh disclose right? There's responsible disclosure and then there's 83 00:07:32,380 --> 00:07:39,280 coordinated disclosure right? They didn't uh they're uh in in I've got people that are very 84 00:07:39,280 --> 00:07:43,500 familiar with this where basically you go to the bug bounty program you report the vulnerability 85 00:07:43,980 --> 00:07:49,240 they pay you the money and then they never fix it. And guess what? You can't ever talk about it 86 00:07:49,240 --> 00:07:53,240 you can't force the issue you're bound by their terms and they never have to do anything about 87 00:07:53,340 --> 00:07:59,260 it. Basically you reported it and you have to walk away. Uh and uh so we we didn't want to go 88 00:07:59,260 --> 00:08:03,240 that route. We wanted to do coordinated disclosure. We're we're basically like hey we're 89 00:08:03,240 --> 00:08:06,960 gonna talk about this we want to set up a timeline we want to talk with you and and make 90 00:08:06,960 --> 00:08:12,060 sure it works with you. And they weren't having it. Uh so we went ahead and uh publicly 91 00:08:12,060 --> 00:08:17,020 disclosed within a few days and they fixed it within like a week. Uh and I actually April of uh 92 00:08:17,020 --> 00:08:20,840 2020 even Bruce Schneier came out and said that bug bounty programs were being used to buy 93 00:08:23,340 --> 00:08:27,620 uh sorry if if uh this impacted you but you know I think this is an important issue. I think 94 00:08:27,620 --> 00:08:34,040 more and more uh programs should do coordinated disclosure instead of responsible disclosure 95 00:08:34,040 --> 00:08:44,340 just so that we're forcing these vendors to fix their products. Alright. So what did we 96 00:08:44,340 --> 00:08:50,920 learn from Netgear? Uh number one uh private keys can be trivially found. Uh I literally we 97 00:08:50,920 --> 00:08:53,320 just did grep on the fi- you know we we experimented with it and we found out that the 98 00:08:53,340 --> 00:08:57,200 key was private. We extracted the file system into folders and we like recursively grepped for 99 00:08:57,200 --> 00:09:05,200 the word private in all caps and out came some keys. So we started to wonder. You know was 100 00:09:05,200 --> 00:09:09,820 this a fluke? Was this just like this is the one product out there that's got private keys in 101 00:09:09,820 --> 00:09:15,320 it? Uh and the answer uh oddly enough is no. This is this is not a this is not an isolated 102 00:09:15,320 --> 00:09:21,360 incident. Uh so we started to ask uh what else might have private keys? So let's take a look. At 103 00:09:21,360 --> 00:09:23,320 the time I was working for a software developer. Uh I was working for a software developer. Uh I was 104 00:09:23,340 --> 00:09:27,900 working for a software as a service company as I'd actually been spent most of my career on the 105 00:09:27,900 --> 00:09:36,080 blue team side uh building software as a service uh platform. And uh I I was a Fortinet customer 106 00:09:36,080 --> 00:09:41,720 and I'm like you know what? Let me go download the latest version of the software that runs all 107 00:09:41,720 --> 00:09:48,800 of my firewalls. Just to take a look to see what I can find. Guess what I found? Well first of 108 00:09:48,800 --> 00:09:53,320 all uh uh I was able to download uh uh I was able to download uh uh I was able to download uh uh 109 00:09:53,340 --> 00:10:01,080 the uh the the software and it was a VMDK right off their website and I extracted file system and 110 00:10:01,080 --> 00:10:07,160 I found that most of the stuff were there's uh most of the software is in these tar dot XZ files 111 00:10:07,740 --> 00:10:14,180 on the uh on the on the firmware. And I'm like okay cool. So it's an XZ X you know normally 112 00:10:14,180 --> 00:10:20,520 people don't use XZ uh compression but that's cool I'll go with it. Uh and even if you ran a file 113 00:10:20,520 --> 00:10:23,000 on it says their XZ compressed data. Uh 114 00:10:23,340 --> 00:10:26,540 So look, what happens when we go ahead and try to decompress it, right? 115 00:10:26,680 --> 00:10:29,300 Oh, XZ, sorry, compressed data is corrupt. 116 00:10:30,400 --> 00:10:31,940 Like, that's interesting. 117 00:10:33,600 --> 00:10:37,540 So it's, I mean, it's brand new firmware I just downloaded from their website. 118 00:10:37,620 --> 00:10:41,980 I extracted it through very solid techniques. 119 00:10:42,240 --> 00:10:44,180 I knew this stuff wasn't corrupted. 120 00:10:44,460 --> 00:10:49,960 And actually, I intentionally downloaded the x86 64-bit version of the software 121 00:10:49,960 --> 00:10:54,280 because I knew I wouldn't have any, you know, weird file system issues 122 00:10:54,280 --> 00:10:56,060 and things of that nature. 123 00:10:56,360 --> 00:11:00,360 And so when I started exploring this file system more, 124 00:11:00,980 --> 00:11:05,560 I saw that they had a couple utilities outside of those XZ files 125 00:11:05,560 --> 00:11:06,960 and a bunch of libraries. 126 00:11:07,300 --> 00:11:11,580 They had one called FTAR, and they had another utility called XZ. 127 00:11:11,920 --> 00:11:13,660 I'm like, this is interesting. 128 00:11:13,900 --> 00:11:19,940 You couldn't just run the XZ binaries or the FTAR binaries. 129 00:11:19,960 --> 00:11:22,820 Because they had their own libraries associated with it. 130 00:11:22,900 --> 00:11:29,360 But if I ran a chroot, if I ran their firmware in a chroot environment, 131 00:11:29,560 --> 00:11:32,380 mind you, I had my virtual machine I'm running this on 132 00:11:32,380 --> 00:11:35,260 is the same architecture as the version I downloaded. 133 00:11:35,420 --> 00:11:37,380 That's really important if you're going to do this at home. 134 00:11:38,040 --> 00:11:41,420 And so I ran their XZ binaries and their FTAR binaries. 135 00:11:41,560 --> 00:11:42,140 Guess what happened? 136 00:11:42,320 --> 00:11:46,200 I was able to decompress and extract the rest of the file system into files. 137 00:11:47,380 --> 00:11:48,560 For a shy class, 138 00:11:48,860 --> 00:11:49,860 why do you think they, 139 00:11:49,960 --> 00:11:50,220 why did they do that? 140 00:11:51,100 --> 00:11:51,520 Anybody? 141 00:11:52,440 --> 00:11:54,860 Security by security. 142 00:11:55,300 --> 00:11:55,640 Rick's in. 143 00:11:57,260 --> 00:11:57,680 Yes. 144 00:11:58,200 --> 00:11:58,860 All right. 145 00:11:58,960 --> 00:12:00,180 That's exactly right. 146 00:12:04,860 --> 00:12:05,700 All right. 147 00:12:05,800 --> 00:12:10,300 So, so, you know, now that we've extracted all these files out in the file system, 148 00:12:10,440 --> 00:12:13,120 we're like, there's got to be something good here because they've hidden it. 149 00:12:13,180 --> 00:12:14,360 They made it a little bit hard. 150 00:12:14,660 --> 00:12:16,540 So I play a lot of CTF. 151 00:12:16,560 --> 00:12:19,820 So far we're at like maybe a level 100 challenge, right? 152 00:12:19,960 --> 00:12:20,940 This stuff is not hard. 153 00:12:21,100 --> 00:12:29,240 So if, if I want to, I want to encourage everyone that if you do CTFs and, you know, 154 00:12:29,760 --> 00:12:35,520 hacking in the real world is generally easier than, than some of these CTF challenges. 155 00:12:35,880 --> 00:12:37,580 And I'm not, I'm not joking. 156 00:12:37,780 --> 00:12:43,120 I'm not doing anything that, that is really hard or complex, but I think it's overlooked. 157 00:12:43,320 --> 00:12:49,480 I think there's, there's, there's things that, that people don't realize that if you even, you know, remotely try to go look at, 158 00:12:49,480 --> 00:12:52,020 you're going to find some, some pretty terrifying things. 159 00:12:52,520 --> 00:12:53,100 All right. 160 00:12:53,140 --> 00:12:54,780 So we're going to use the same technique before. 161 00:12:56,080 --> 00:12:58,640 Strings on a fine, maybe a little bit of grep-r. 162 00:12:58,940 --> 00:12:59,280 Yeah. 163 00:12:59,400 --> 00:13:00,500 And guess what we find? 164 00:13:02,100 --> 00:13:05,920 Two publicly blessed CA or certificates blessed by CAs. 165 00:13:06,000 --> 00:13:08,380 These, this time, one was published by Apple. 166 00:13:08,540 --> 00:13:09,900 The other one from Google. 167 00:13:10,240 --> 00:13:11,800 Any idea what these are used for? 168 00:13:15,420 --> 00:13:16,300 Push notifications. 169 00:13:16,700 --> 00:13:18,540 Hey, here's a shot glass for the guy in the back. 170 00:13:18,580 --> 00:13:19,460 Oh wait, that's my buddy. 171 00:13:19,460 --> 00:13:19,680 Nick. 172 00:13:19,980 --> 00:13:21,240 Oh, overshot. 173 00:13:22,320 --> 00:13:25,900 My, uh, NFL career is, uh, is not going to take off. 174 00:13:26,780 --> 00:13:33,520 Um, yes, we found some push notification certificates in everyone's Fortinet firewall. 175 00:13:34,300 --> 00:13:35,940 Is that exciting or scary? 176 00:13:36,820 --> 00:13:38,320 It's pretty excitingly scary. 177 00:13:38,520 --> 00:13:39,200 Yeah, I know. 178 00:13:39,340 --> 00:13:45,440 Uh, so just to show you, so, uh, I'm like, really, these are really Apple certificates. 179 00:13:46,000 --> 00:13:46,820 And sure enough. 180 00:13:46,860 --> 00:13:49,240 So here's the, the output right here. 181 00:13:49,240 --> 00:13:54,780 Uh, from, uh, looking at the, the PEM certificate, the, the public, uh, part of the key or part 182 00:13:54,780 --> 00:13:56,060 of the public part of the certificate. 183 00:13:56,200 --> 00:14:01,620 And it's, uh, the CN from Apple worldwide developer relations certificate, uh, relations 184 00:14:01,620 --> 00:14:04,260 certificate, uh, right there, Apple. 185 00:14:04,460 --> 00:14:08,280 And, uh, and then, so you can, you know, cryptographically check to make sure that 186 00:14:08,280 --> 00:14:10,760 like, okay, I've got a public certificate and a private key. 187 00:14:10,840 --> 00:14:17,440 You can actually use open SSL, uh, uh, in order to, uh, look at, uh, look at the, the 188 00:14:17,440 --> 00:14:18,300 values from it. 189 00:14:18,300 --> 00:14:21,440 And sure enough, the, the certificate matched the private key. 190 00:14:21,540 --> 00:14:22,720 So it's a match. 191 00:14:22,900 --> 00:14:23,700 This is legit. 192 00:14:23,840 --> 00:14:28,720 I've got a private key to a certificate that was issued by Apple that I extracted from 193 00:14:28,720 --> 00:14:30,340 my Fortinet firewall software. 194 00:14:31,040 --> 00:14:31,940 Pretty exciting. 195 00:14:32,660 --> 00:14:33,280 All right. 196 00:14:33,320 --> 00:14:35,340 So what if I was an attacker, right? 197 00:14:35,420 --> 00:14:40,440 So, you know, uh, in their final response that will, uh, that they came out with, uh, 198 00:14:40,540 --> 00:14:45,520 they acknowledge that, uh, these certificates are actually used for both their Apple and 199 00:14:45,520 --> 00:14:47,300 Google cloud messaging services. 200 00:14:47,560 --> 00:14:48,040 So. 201 00:14:48,300 --> 00:14:54,840 So does this mean I could send push notifications to people's phones from as a, as Fortinet? 202 00:14:55,720 --> 00:14:56,120 Yeah. 203 00:14:56,140 --> 00:14:59,940 You know, when I figured this out pretty early on, that's why at the point that I'm like, 204 00:15:00,000 --> 00:15:00,920 this is bad. 205 00:15:01,120 --> 00:15:03,600 I had, I went ahead and reported it to Fortinet. 206 00:15:03,800 --> 00:15:07,080 I'd actually never worked with Fortinet and didn't know of anybody that had. 207 00:15:07,300 --> 00:15:09,100 So I gave them the benefit of the doubt. 208 00:15:09,380 --> 00:15:12,680 Um, and so, uh, what this is probably two weeks. 209 00:15:12,760 --> 00:15:16,880 I, it was literally two weeks after we had the net the year thing go on is when I first 210 00:15:16,880 --> 00:15:18,000 reported this incident. 211 00:15:18,000 --> 00:15:21,960 Uh, or this issue to, uh, to, to Fortinet. 212 00:15:22,520 --> 00:15:24,100 And I also reached out to Apple. 213 00:15:24,220 --> 00:15:28,220 I mean, I'm like, Hey Apple, I've got the private key to a certificate you guys issued. 214 00:15:28,600 --> 00:15:32,640 Uh, I, I felt that they should know, uh, also, and actually is, is interesting. 215 00:15:32,880 --> 00:15:38,060 Apple like responded back and be like, we need to know whose key you've got access to. 216 00:15:38,100 --> 00:15:41,820 I mean, they were like, tell us what, you know, tell me who your daddy is and what does 217 00:15:41,820 --> 00:15:42,040 he do? 218 00:15:42,140 --> 00:15:42,340 Right. 219 00:15:42,700 --> 00:15:45,940 Uh, and, uh, and I was a little out there. 220 00:15:45,940 --> 00:15:46,280 Luckily. 221 00:15:46,280 --> 00:15:50,980 You know, after I, I told Fortinet that I had reached out to Apple as well, you know, 222 00:15:50,980 --> 00:15:54,740 they self-reported themselves to Apple and, you know, it was all, all in a merry way. 223 00:15:55,060 --> 00:16:02,100 Um, and then, um, and then actually I was issued a CVE, uh, in February, uh, 2026, 224 00:16:02,100 --> 00:16:02,960 6 45. 225 00:16:03,260 --> 00:16:04,460 Every, anybody ever heard that one? 226 00:16:04,920 --> 00:16:05,340 No. 227 00:16:05,480 --> 00:16:08,240 If you still look, it's still actually not even, uh, allocated. 228 00:16:08,680 --> 00:16:15,280 Um, but they, they, they told me in April, April, 2020 that they're coming with a fix 229 00:16:15,280 --> 00:16:16,440 in 6.4.1. 230 00:16:16,540 --> 00:16:20,140 And actually I think it ended up in 6.4.2, but they wanted me to wait. 231 00:16:20,260 --> 00:16:24,900 You know, what's really interesting is these, this is more than just like I found a problem 232 00:16:24,900 --> 00:16:27,000 that, you know, firmware update can kind of fix. 233 00:16:27,140 --> 00:16:29,800 This is like an architectural issue in their infrastructure. 234 00:16:30,060 --> 00:16:30,320 Right. 235 00:16:30,420 --> 00:16:34,760 So no longer should, you know, should your firewall be able to talk directly to Apple 236 00:16:34,760 --> 00:16:35,800 and Google and send a push notification? 237 00:16:36,220 --> 00:16:36,420 No. 238 00:16:37,120 --> 00:16:39,920 They're, um, and so this is actually an architectural issue. 239 00:16:40,060 --> 00:16:43,800 So we, you know, we had discussions around, they need to come up with a solution. 240 00:16:43,800 --> 00:16:49,860 Then they had to push it out and then they wanted me to wait until, um, you know, a sufficient 241 00:16:49,860 --> 00:16:53,080 enough time had passed that enough customers had updated their firmware. 242 00:16:53,240 --> 00:16:54,320 I was like, totally cool. 243 00:16:54,440 --> 00:16:56,000 I, I'm right on board with you. 244 00:16:56,040 --> 00:16:58,080 You know, I'm a, I was a Fortinet customer at the time. 245 00:16:58,120 --> 00:16:59,400 I'm like totally on board with it. 246 00:17:01,340 --> 00:17:02,300 And then I waited. 247 00:17:02,600 --> 00:17:05,840 This was April, 2020 when I kind of last talked with them. 248 00:17:06,000 --> 00:17:09,400 Uh, and then I waited and I waited and I waited. 249 00:17:09,880 --> 00:17:13,680 And three years later, April of 2020. 250 00:17:13,800 --> 00:17:16,420 23 or sorry, February of 2023. 251 00:17:16,940 --> 00:17:20,540 Um, they finally issued the, the, uh, the advisory. 252 00:17:20,780 --> 00:17:27,240 It wasn't without pro I w I was poking and prodding them probably every, um, every, 253 00:17:27,320 --> 00:17:30,060 you know, three or four months, something like that. 254 00:17:30,220 --> 00:17:35,280 Uh, and they, uh, they'd never, they, they kept being like, oh yeah, I actually went 255 00:17:35,280 --> 00:17:38,460 through, uh, three piece or representatives. 256 00:17:38,780 --> 00:17:43,580 Like I ended up, you know, every time, you know, every few months a new person was running 257 00:17:43,580 --> 00:17:43,780 the program. 258 00:17:43,800 --> 00:17:48,320 Uh, and, uh, but they finally acknowledged it in February, 2023. 259 00:17:48,540 --> 00:17:52,220 I think they actually, it took them at least two, two and a half years to fix it. 260 00:17:52,420 --> 00:17:56,420 Uh, but to acknowledge it, it was, uh, February 16th, 2023. 261 00:17:56,760 --> 00:17:58,540 And you see, it's got a different CVE number. 262 00:17:58,700 --> 00:18:03,520 So what they actually, they were like internal first internal reasons that are, I don't understand. 263 00:18:03,760 --> 00:18:08,760 They had to remember the CVE to 2022, two, two, three Oh two, even though I reported 264 00:18:08,760 --> 00:18:09,800 it back in 2020. 265 00:18:10,060 --> 00:18:13,140 So that's an interesting story that I'm sure I will never hear about. 266 00:18:13,140 --> 00:18:18,800 Um, uh, you know, maybe it was too long between when it was reported and when they reported 267 00:18:18,800 --> 00:18:19,220 the fix. 268 00:18:19,360 --> 00:18:19,920 I don't know. 269 00:18:19,940 --> 00:18:21,280 And they could only go back so far. 270 00:18:21,920 --> 00:18:23,720 But wait, that's not all. 271 00:18:23,900 --> 00:18:29,560 So, uh, fast forward, I think it was actually January or February of this year. 272 00:18:29,660 --> 00:18:34,800 I was on, I was helping one of my, uh, penetration testers on his external penetration test. 273 00:18:35,000 --> 00:18:40,620 So we found a Fortinet firewall on the public internet that was acting as a VPN server. 274 00:18:40,740 --> 00:18:42,320 Very common in a lot of people's organizations. 275 00:18:42,320 --> 00:18:45,440 Uh, but we did a password guessing attack. 276 00:18:45,760 --> 00:18:50,720 Uh, and we found a username and password that worked, that let us, uh, kind of get 277 00:18:50,720 --> 00:18:53,180 past that first door, but they had MFA turned on. 278 00:18:53,220 --> 00:18:53,780 Good for them. 279 00:18:54,140 --> 00:18:57,800 Uh, and so we didn't have a good way of bypassing their MFA. 280 00:18:58,080 --> 00:19:01,240 So I thought, Hey, I, we're, we're Florida friends. 281 00:19:01,680 --> 00:19:06,660 Uh, we, I reached out and got the latest version of the software, the same, same software 282 00:19:06,660 --> 00:19:07,460 that I used to run. 283 00:19:07,940 --> 00:19:09,340 And I got the latest version. 284 00:19:09,420 --> 00:19:11,140 I was like, uh, because it was the software. 285 00:19:11,140 --> 00:19:12,940 That's where our, our, the version our client was running. 286 00:19:14,260 --> 00:19:18,100 And I, uh, decided to look through the firmware to see if I could find some kind of an MFA 287 00:19:18,100 --> 00:19:18,820 bypass, right? 288 00:19:18,940 --> 00:19:23,020 You know, what's great is since I, I, uh, manage the team, I don't get my own engagements. 289 00:19:23,500 --> 00:19:27,820 So I get to jump into all my coworkers engagements to kind of help them. 290 00:19:27,920 --> 00:19:31,220 And I like sometimes swing from the swing for the fences and, and sometimes it'll hit 291 00:19:31,220 --> 00:19:31,680 a home run. 292 00:19:31,820 --> 00:19:33,660 Sometimes I'll just do some interesting things. 293 00:19:33,900 --> 00:19:39,340 In this particular case, uh, I got sidetracked because as I was looking through the firmware, 294 00:19:39,460 --> 00:19:40,340 looking at all the code. 295 00:19:40,340 --> 00:19:46,520 I found the fix for the issue I reported three years ago, which is actually the preface, 296 00:19:46,520 --> 00:19:52,700 uh, preface to why they finally published, uh, the, their advisory is because I poked 297 00:19:52,700 --> 00:19:52,860 them. 298 00:19:52,940 --> 00:19:55,480 I'm like, Hey, I see you've got the fix in the software. 299 00:19:55,580 --> 00:19:59,440 I don't know exactly when it came out, but hopefully it's been long enough since it's 300 00:19:59,440 --> 00:20:03,300 been three years, uh, to go ahead and, uh, and talk about it. 301 00:20:03,480 --> 00:20:08,900 And, uh, and that's when, uh, but, but then the fix was really super interesting. 302 00:20:09,160 --> 00:20:10,280 So I go, uh, 303 00:20:10,340 --> 00:20:16,700 Most of the, the, the firmware on Fortinet is in a giant in it file, right? 304 00:20:16,800 --> 00:20:22,080 So it's kind of like busy box, uh, where you've got like a massive binary and then you link 305 00:20:22,080 --> 00:20:26,840 to it with LS, you link to it with RM and CP and bending on the file name that you're 306 00:20:26,840 --> 00:20:30,980 running it as it'll hit that executable differently in a run on a little bit different version 307 00:20:30,980 --> 00:20:31,500 of the program. 308 00:20:31,560 --> 00:20:34,720 Well, that's how Fortinet's firmware is mostly set up. 309 00:20:34,880 --> 00:20:40,320 And inside of that, that, that in it file, I found, uh, a URL program. 310 00:20:40,320 --> 00:20:42,020 It's called productapi.fortinet.com. 311 00:20:42,240 --> 00:20:44,360 And, uh, it's right there highlighted in black. 312 00:20:44,560 --> 00:20:48,120 Uh, and, and right above it it says iOS, iOS push data. 313 00:20:48,520 --> 00:20:54,180 And then above that you see, uh, you see, uh, fgt.crt and fgt.key. 314 00:20:54,320 --> 00:20:55,120 Guess what those are? 315 00:20:56,460 --> 00:20:57,360 Any ideas? 316 00:20:59,840 --> 00:21:00,320 Organizational private key. 317 00:21:00,320 --> 00:21:01,960 Organizational private key in the, in the green shirt? 318 00:21:02,220 --> 00:21:03,260 Oh, back there. 319 00:21:05,320 --> 00:21:05,800 Oh! 320 00:21:06,260 --> 00:21:08,360 Still not gonna be an NFL player. 321 00:21:08,500 --> 00:21:08,720 Sorry. 322 00:21:09,300 --> 00:21:10,080 Nice work. 323 00:21:10,980 --> 00:21:11,780 Uh, yeah. 324 00:21:11,960 --> 00:21:16,860 So this time, uh, this time I discovered that, uh, they have another private key. 325 00:21:17,020 --> 00:21:25,120 Uh, and so I went ahead, uh, I went ahead and, uh, and, uh, tried to talk to this productapi.fortinet.com. 326 00:21:26,360 --> 00:21:30,080 Um, and it, it gave me, you know, a handshake failure, right? 327 00:21:30,140 --> 00:21:31,660 So I just tried to curl against it. 328 00:21:31,740 --> 00:21:34,440 Uh, and it's like, nope, not gonna be able to talk to me. 329 00:21:35,160 --> 00:21:38,640 So then I'm like, well, this is probably a client certificate. 330 00:21:38,720 --> 00:21:41,800 To do client, uh, authentication against the URL. 331 00:21:42,020 --> 00:21:43,880 So sure enough, do the curl again. 332 00:21:44,800 --> 00:21:48,980 Pass it in the Fortinet, uh, issued certificate and private key. 333 00:21:49,320 --> 00:21:52,040 And, uh, yup, I can talk to the URL. 334 00:21:52,200 --> 00:21:55,200 So I didn't dig any deeper than this. 335 00:21:55,360 --> 00:21:58,840 So, uh, I, uh, I don't have anything more to report here. 336 00:21:58,960 --> 00:22:06,900 But let's just hope that the authentication between your firewall and their push notification proxy server has more authentication than just a private key that's on their firmware. 337 00:22:07,100 --> 00:22:07,620 I don't know. 338 00:22:07,620 --> 00:22:09,800 So that's where I left that. 339 00:22:10,120 --> 00:22:10,740 All right. 340 00:22:10,780 --> 00:22:13,960 So what did we learn from Fortinet now that, uh, for this example? 341 00:22:14,080 --> 00:22:16,360 You know, I'm convinced private keys are everywhere. 342 00:22:16,680 --> 00:22:19,720 And, uh, I'm only showing you three different examples today. 343 00:22:19,860 --> 00:22:22,500 I've got so many more, uh, in my library. 344 00:22:22,660 --> 00:22:23,440 It's, it's ridiculous. 345 00:22:23,820 --> 00:22:30,580 And, um, and I've also found that manufacturers may not prioritize this type of an issue, right? 346 00:22:30,680 --> 00:22:32,300 Especially when they're hard problems, right? 347 00:22:32,400 --> 00:22:34,860 This is in particular an architectural issue. 348 00:22:35,080 --> 00:22:37,600 They had to now stand up an entire site. 349 00:22:37,620 --> 00:22:40,960 On the Internet that all of their customers are now going to hit. 350 00:22:41,140 --> 00:22:44,700 I mean, this is a, this is kind of a, this is kind of a big change for them. 351 00:22:44,800 --> 00:22:48,920 And it can take some time, a really long time in this case, to actually address it. 352 00:22:49,340 --> 00:22:50,520 So we need to be careful. 353 00:22:52,000 --> 00:22:54,960 But then we kept finding private keys. 354 00:22:55,300 --> 00:23:02,880 So in my line of business, you know, we go kind of the extra mile to, uh, to, to find all the weaknesses in your organization. 355 00:23:03,140 --> 00:23:07,360 And private keys are very useful on penetration tests, right? 356 00:23:07,360 --> 00:23:11,000 So let's fast forward a little bit more to April of this year. 357 00:23:11,220 --> 00:23:14,840 Uh, my team was, uh, conducting an internal penetration test. 358 00:23:15,280 --> 00:23:25,300 Uh, you know, basically what happens on our internal penetration test is we'll, we'll take a box, uh, a server box that we run our own, uh, distribution, Linux distribution on. 359 00:23:25,440 --> 00:23:26,980 And we'll send it to the customer. 360 00:23:27,100 --> 00:23:33,940 And we'll have them plug it into the network, you know, and get us either on a user subnet or as if it's like plugged into a conference room. 361 00:23:34,200 --> 00:23:35,900 Basically, no level of access. 362 00:23:35,900 --> 00:23:43,400 And this thing will, it will find its way out of their network, connect back to our infrastructure, and allow us to pivot in to do our work to, to find, uh, vulnerabilities. 363 00:23:44,120 --> 00:23:52,840 And so, you know, one of the first things that we do, you know, is part of discovery on your network in an internal penetration test is look for, for, for low-hanging fruit. 364 00:23:52,920 --> 00:23:54,600 Look for, for vulnerabilities, right? 365 00:23:54,660 --> 00:23:55,800 We'll use a vulnerability scanner. 366 00:23:56,080 --> 00:24:05,800 And, uh, uh, my coworker who is running this internal penetration test, uh, he's like, the next day, he's like, after he ran scans for a day. 367 00:24:05,900 --> 00:24:15,760 He's like, hey, uh, I found, I got access to what was reported as a, uh, Palo Alto firewall, uh, with that, with default credentials. 368 00:24:15,880 --> 00:24:18,740 But I got into it and it does not look like a Palo Alto firewall. 369 00:24:19,020 --> 00:24:21,580 And guess what credentials he used to get into it with? 370 00:24:23,900 --> 00:24:25,440 Hey, he said admin, admin. 371 00:24:25,640 --> 00:24:26,100 That guy. 372 00:24:26,360 --> 00:24:26,780 Here we go. 373 00:24:27,420 --> 00:24:29,000 Sorry, that was the one I drank out of. 374 00:24:29,040 --> 00:24:30,100 You might want to watch that one. 375 00:24:32,440 --> 00:24:33,080 My bad. 376 00:24:33,240 --> 00:24:33,920 That was the last one. 377 00:24:34,300 --> 00:24:35,520 Uh, yeah. 378 00:24:35,900 --> 00:24:36,400 Admin, admin. 379 00:24:36,620 --> 00:24:38,400 So that's how we got into this thing. 380 00:24:38,500 --> 00:24:40,000 And we didn't, he didn't know what it was. 381 00:24:40,020 --> 00:24:40,960 I didn't know what it was. 382 00:24:41,640 --> 00:24:43,460 Um, it was kind of a mysterious device. 383 00:24:43,820 --> 00:24:46,680 It, it looked like it was running some version of like maybe a CentOS. 384 00:24:47,440 --> 00:24:56,020 Uh, looking through process lists, uh, looking through, uh, file systems, you know, it's, we, we found, we found that this is an interesting thing. 385 00:24:56,140 --> 00:24:59,100 So, uh, first of all, we started looking through process lists. 386 00:24:59,280 --> 00:25:05,820 And, uh, and you'll see here that we're, we, uh, we found the directory that was, uh, labeled, uh, Delta. 387 00:25:05,820 --> 00:25:06,380 And, uh, we, uh, we found the directory that was, uh, Dell Compellent. 388 00:25:07,060 --> 00:25:08,180 I'm like, Dell Compellent? 389 00:25:08,200 --> 00:25:09,460 That's the storage platform I'm on. 390 00:25:09,500 --> 00:25:10,540 Am I in a Compellent? 391 00:25:10,800 --> 00:25:13,420 I've never seen a Compellent before, but I've heard about them. 392 00:25:13,960 --> 00:25:15,620 It, it, but it wasn't a Compellent. 393 00:25:15,720 --> 00:25:20,580 I'm like, I'm looking like there's got to be a giant-ass hard drive around here that's hosting like their, their domain controller. 394 00:25:20,720 --> 00:25:21,780 Like, that's what I was looking for. 395 00:25:21,820 --> 00:25:21,980 Right? 396 00:25:22,020 --> 00:25:25,460 Like, I'm hunting for a way that I can forensically grab all your passwords. 397 00:25:25,680 --> 00:25:28,180 Or other user, other, other user credentials. 398 00:25:28,680 --> 00:25:29,460 But it wasn't. 399 00:25:29,580 --> 00:25:33,440 And, uh, but I did have some, some directories I could start looking at. 400 00:25:33,640 --> 00:25:35,800 And so, in one of these directories. 401 00:25:35,800 --> 00:25:40,260 I found, uh, a, a properties file that was called vsp.properties. 402 00:25:40,380 --> 00:25:42,280 I had, and then at the top it gave me a hint. 403 00:25:42,360 --> 00:25:48,040 It says Dell Compellent VMware vSphere plugin for VMware vCenter configuration parameters. 404 00:25:48,920 --> 00:25:50,540 Like, that sounds very fancy. 405 00:25:50,880 --> 00:25:54,100 Uh, and they had a password and a username. 406 00:25:54,380 --> 00:25:57,640 And the username was vSphere.localadministrator. 407 00:25:58,940 --> 00:26:01,440 So, this seemed like a high value target to me. 408 00:26:01,720 --> 00:26:05,180 Uh, uh, that, that, uh, but that password looked real odd. 409 00:26:05,260 --> 00:26:05,460 Right? 410 00:26:05,800 --> 00:26:08,060 It was all hex, uppercase, and numbers. 411 00:26:08,360 --> 00:26:13,340 And I'm like, that couldn't possibly be the password to their vCenter, could it? 412 00:26:14,160 --> 00:26:14,520 No. 413 00:26:14,600 --> 00:26:15,240 It wasn't. 414 00:26:15,340 --> 00:26:16,600 I went, I went to their vCenter. 415 00:26:16,740 --> 00:26:17,760 I tried to log in with it. 416 00:26:18,080 --> 00:26:21,820 And, uh, sure enough, it was not the, uh, I got invalid credentials. 417 00:26:22,120 --> 00:26:23,380 I'm like, okay, that's cool. 418 00:26:23,680 --> 00:26:28,700 So, that means that that password has to be able to be read by the Compellent. 419 00:26:28,780 --> 00:26:33,460 So, you know, it's like, so, this turns out it's a, it appears to be some kind of a connector 420 00:26:34,500 --> 00:26:35,780 between Dell Compellent and vSphere. 421 00:26:37,480 --> 00:26:39,320 So, this kind of sits in between. 422 00:26:39,460 --> 00:26:40,200 I don't know what it does. 423 00:26:40,300 --> 00:26:42,700 It probably, like, creates snapshots or provisions volumes. 424 00:26:42,960 --> 00:26:43,640 I have no idea. 425 00:26:43,760 --> 00:26:48,680 But for some reason, in the documentation, it says, you must give it administrator-level 426 00:26:48,680 --> 00:26:54,320 credentials to the Compellent connector in order to, to, you know, be the, the connector 427 00:26:54,320 --> 00:26:56,620 to your vSphere or your vCenter. 428 00:26:57,440 --> 00:27:01,700 And so, I needed to figure out, like, something's got to read and write that config. 429 00:27:01,900 --> 00:27:02,840 Let's figure this out. 430 00:27:02,960 --> 00:27:05,640 And so, sure enough, I found, uh, the install script. 431 00:27:05,760 --> 00:27:11,160 That was sitting on the box, uh, that referenced a vsp-config-util.jar. 432 00:27:11,480 --> 00:27:19,480 And they had a, a, a class name, uh, uh, referenced Dell, com.dell.compellent-vsp-config-util. 433 00:27:20,380 --> 00:27:23,460 Um, you know, there I, I enlarged it so you can see it a little bit. 434 00:27:23,540 --> 00:27:25,160 So, there's a, a jar file that we found. 435 00:27:25,340 --> 00:27:29,980 Uh, and then, uh, there's the class file that gets run with that jar file. 436 00:27:30,160 --> 00:27:34,060 And I'm like, well, that sounds like something that, you know, get configuration data. 437 00:27:34,060 --> 00:27:38,820 That thing's got to be able to somehow read that encrypted value and decrypt it, right? 438 00:27:39,800 --> 00:27:43,560 So, I, uh, I've actually been a Java developer for most of my career. 439 00:27:43,680 --> 00:27:46,280 I probably coded Java for about 25 years. 440 00:27:46,420 --> 00:27:48,840 I'm like, I'm not, I'm not shied away by this at all. 441 00:27:49,040 --> 00:27:50,580 And I've, you know, played a few CTS. 442 00:27:51,100 --> 00:27:55,240 I'm like, I, you know, understanding a jar file, all it is is a zip file. 443 00:27:55,380 --> 00:28:01,380 Uh, you can use a jar util or you, in this particular case, you just use, I just used unzip, unzip the jar file. 444 00:28:01,500 --> 00:28:03,840 And it, it takes every, so, uh, 445 00:28:04,060 --> 00:28:06,960 in Java, everything's compiled into different classes. 446 00:28:07,460 --> 00:28:10,800 So, uh, in, in, those classes are put into packages. 447 00:28:11,020 --> 00:28:14,020 All it is is fancy words for files and, files and directories. 448 00:28:14,320 --> 00:28:17,720 Uh, and I decided I'd just start going down the list. 449 00:28:17,820 --> 00:28:19,900 I started with, uh, confutil.class. 450 00:28:20,060 --> 00:28:23,300 I'm like, I can decompile that because it's an interpreted language. 451 00:28:23,680 --> 00:28:27,880 Or, uh, well, it's a compiled language that's interpreted by the Java, uh, interpreter. 452 00:28:28,180 --> 00:28:30,520 Uh, so I, uh, you can get decompilers. 453 00:28:30,620 --> 00:28:33,840 And I, I actually used, so, in a few different steps, like, 454 00:28:34,060 --> 00:28:37,780 I used, uh, the benef, uh, .org, uh, decompiler. 455 00:28:38,100 --> 00:28:39,580 Uh, it's pretty lightweight. 456 00:28:39,820 --> 00:28:41,340 Uh, easy to use. 457 00:28:41,480 --> 00:28:43,780 Uh, and it gives you some pretty nice looking output. 458 00:28:43,980 --> 00:28:48,560 So, all I did was download the, uh, benef.org, uh, decompiler. 459 00:28:48,760 --> 00:28:51,320 And I fed it that confutil.class. 460 00:28:51,500 --> 00:28:52,220 And guess what I got? 461 00:28:53,880 --> 00:28:54,560 Private key. 462 00:28:54,860 --> 00:28:55,700 Who said that? 463 00:28:56,380 --> 00:28:58,360 I wish I had another shot glass for you, man. 464 00:28:58,460 --> 00:28:59,720 I would throw it right at you. 465 00:29:00,020 --> 00:29:01,900 I got a whiskey bottle, but it's empty. 466 00:29:01,900 --> 00:29:04,060 Um, all right. 467 00:29:04,120 --> 00:29:06,480 So, sure enough, it's even commented. 468 00:29:06,640 --> 00:29:09,140 Like, uh, right above where it says encryption key, 469 00:29:09,300 --> 00:29:10,700 it says encrypted type AES. 470 00:29:11,060 --> 00:29:13,900 I'm like, they're even giving me hints as to what I need to do here. 471 00:29:14,020 --> 00:29:14,620 This is great. 472 00:29:16,260 --> 00:29:19,920 Uh, and so, uh, sure enough, there's a static AES key. 473 00:29:20,020 --> 00:29:22,620 So, since this is in the compiled jar file, 474 00:29:22,820 --> 00:29:26,220 that means this isn't a unique key by customer, right? 475 00:29:26,300 --> 00:29:28,040 Anybody that's got this product, 476 00:29:28,260 --> 00:29:31,880 their vCenter passwords are stored in the vCenter. 477 00:29:31,900 --> 00:29:34,040 It's in the, in this VSP properties file encrypted 478 00:29:34,380 --> 00:29:36,180 with the same key as any other customer. 479 00:29:36,320 --> 00:29:37,580 So it didn't matter, you know, 480 00:29:37,600 --> 00:29:40,300 so as long as you can get access to that VSP properties file, 481 00:29:40,460 --> 00:29:42,540 I'm gonna be able to get your password out of that, right? 482 00:29:42,880 --> 00:29:43,620 Well, maybe. 483 00:29:44,220 --> 00:29:47,620 Well, so, let's, let's, uh, step through how I got it, right? 484 00:29:47,720 --> 00:29:49,660 So I knew it was AES of some sort. 485 00:29:49,820 --> 00:29:51,000 Uh, I've got a key. 486 00:29:52,120 --> 00:29:53,540 I've got it out of the properties file. 487 00:29:53,680 --> 00:29:58,800 So, uh, what I did was, uh, I used a very common, uh, CTF tool. 488 00:29:59,000 --> 00:30:00,940 Any, any idea what tool I might have used? 489 00:30:02,360 --> 00:30:02,680 CyberChef. 490 00:30:02,720 --> 00:30:03,260 That's right. 491 00:30:03,380 --> 00:30:04,800 So I just went to CyberChef. 492 00:30:05,260 --> 00:30:06,940 I, uh, took the AES key. 493 00:30:07,040 --> 00:30:08,020 I haven't redacted here. 494 00:30:08,120 --> 00:30:09,300 I'm not that much of an asshole. 495 00:30:09,740 --> 00:30:12,420 I, uh, uh, you, if you guys want the key, you can go get it yourselves. 496 00:30:12,680 --> 00:30:18,860 Uh, but, uh, so I put the, the AES key, uh, into the AES decrypt method of CyberChef. 497 00:30:19,120 --> 00:30:21,700 And then I put the encrypted password in the input. 498 00:30:21,920 --> 00:30:24,040 And then I'm like, wait, I was digging through the source code. 499 00:30:24,120 --> 00:30:26,200 What did they use for initial, initialization factor? 500 00:30:26,380 --> 00:30:30,200 In fact, I was like, maybe this is the secret sauce to their private key. 501 00:30:30,420 --> 00:30:31,880 Because they didn't actually have it in there. 502 00:30:31,900 --> 00:30:37,580 I was trying to figure out, like, okay, how, so I just put in enough zeros until CyberChef was happy with it. 503 00:30:37,680 --> 00:30:38,640 And guess what popped out? 504 00:30:39,340 --> 00:30:40,380 Clear text password. 505 00:30:40,880 --> 00:30:48,600 So now, uh, I'm armed with, uh, I'm armed with a, uh, with a clear text username and password. 506 00:30:48,780 --> 00:30:50,380 I go back to the vCenter, and I'm in. 507 00:30:50,560 --> 00:30:50,980 Right? 508 00:30:51,160 --> 00:30:52,900 So at this point, you do the happy dance. 509 00:30:53,000 --> 00:30:55,320 You're like, oh, my God, I've got access to their vCenter. 510 00:30:56,800 --> 00:30:58,880 Uh, why, why is this important? 511 00:30:59,320 --> 00:31:01,880 Why is it important that I have full access? 512 00:31:01,900 --> 00:31:03,140 Why is it important that I have admin access to your vCenter? 513 00:31:04,480 --> 00:31:05,220 You fucked it up. 514 00:31:05,220 --> 00:31:05,820 You fucked it up, yeah. 515 00:31:05,860 --> 00:31:07,860 You know, you're running all the stuff, right? 516 00:31:07,940 --> 00:31:16,280 This is where, like, your domain controllers live and your servers and all the important infrastructure that runs your organization is generally accessed through vCenter. 517 00:31:16,380 --> 00:31:17,600 That's how vCenter works. 518 00:31:17,980 --> 00:31:26,120 Uh, so normally at this point, like, my go-to move when I get vCenter access, so I've, I've exploited vCenter in a lot of different ways. 519 00:31:26,240 --> 00:31:27,200 Thank you, Log4J. 520 00:31:27,320 --> 00:31:28,380 You're the gift that keeps on giving. 521 00:31:28,740 --> 00:31:29,180 Um. 522 00:31:29,180 --> 00:31:34,040 But normally what I'll do is I will, uh, snapshot your domain controller. 523 00:31:34,340 --> 00:31:47,760 Uh, and then I'll, uh, I'll download the, uh, the first 10 to 20 gigs of the C drive of your, of the hard drive of your domain controller or whatever hard drive is that you're storing the NTDS at it because it doesn't have to be your C drive. 524 00:31:48,140 --> 00:31:59,160 Uh, and then I will forensically recover, uh, all the NTDLM hashes from the NTDS.dit, uh, using the system registry hives, uh, and, and the vCenter. 525 00:31:59,160 --> 00:32:00,540 It does that, but it actually needs to have a special way of, uh, using that file in order to get all the hashes. 526 00:32:00,660 --> 00:32:01,380 And it's cool. 527 00:32:01,480 --> 00:32:14,480 Actually, if people are interested, I actually have a really cool technique how I can use a partial hard drive download to then convert it to qt cow too, and then, uh, uh, be able to mount it and, and access it. 528 00:32:14,540 --> 00:32:20,240 Because, uh, often, uh, when I'm doing these things, uh, you're not getting clean images, right. 529 00:32:20,300 --> 00:32:24,360 Sometimes you're, like, you want to read-only mount of an iSCSI or MFAS or something like that. 530 00:32:24,420 --> 00:32:29,080 And, like, even the, uh, uh, the registry hives, you're not, they're not going to be clean. 531 00:32:29,100 --> 00:32:29,140 Right. 532 00:32:29,160 --> 00:32:34,280 but some of them are. Like the last known good boot. So like I've like manually patched 533 00:32:34,280 --> 00:32:39,200 impact it so I can like go after the last known good boot to get the the boot key from the 534 00:32:39,200 --> 00:32:45,700 previous boot. It's likely still the same key to decrypt your boot key. Um at any rate but 535 00:32:45,700 --> 00:32:50,840 that's not what we did here. So we took it a step further. Uh so in this particular case uh 536 00:32:50,840 --> 00:32:56,400 it just you know people always love a good pento story right? Uh I uh I went ahead and you 537 00:32:56,400 --> 00:33:01,340 know I passed on the access to to my coworker who's working the the IPT and I decided I was 538 00:33:01,340 --> 00:33:04,600 gonna fuck around with some of their other stuff. They had a certificate authority on the 539 00:33:04,600 --> 00:33:10,140 network. And so I was uh I was working with you know we had another low level uh user on 540 00:33:10,140 --> 00:33:17,000 their network at this point. And I was uh I was uh you know messing with their certificate 541 00:33:17,000 --> 00:33:22,500 authority. And I definitely set off some alarms but my coworker was in the console of vCenter 542 00:33:22,500 --> 00:33:26,300 and he's checking out the consoles. He's checking out what servers he had access to. And 543 00:33:26,300 --> 00:33:32,920 apparently I must have set off an alert because uh their their domain administrator went to the 544 00:33:32,920 --> 00:33:41,680 CA box and opened up through vCenter the console that my buddy was sitting on. And so the the he 545 00:33:41,680 --> 00:33:45,240 logged into the console and he was looking around at the logs and seeing what was going on 546 00:33:45,240 --> 00:33:49,780 because I was you know messing with the certificate authority. My coworker was on the 547 00:33:49,780 --> 00:33:54,280 console of the same certificate authority when the domain administrator logged in. And then it 548 00:33:54,280 --> 00:33:55,980 stopped moving. He stopped typing stuff. 549 00:33:56,560 --> 00:34:01,940 And it looked like he walked away. So guess what my coworker did? Popped the PowerShell, 550 00:34:02,020 --> 00:34:05,620 created his own domain admin account and then closed it. So then we got domain administrator 551 00:34:05,620 --> 00:34:11,720 without even downloading uh the domain controller. We had open access to a uh a console. And 552 00:34:11,720 --> 00:34:16,720 then from there then we went ahead and uh and we dumped the uh NTDS that did across the network 553 00:34:16,720 --> 00:34:22,040 and then we we uh we won the day. That was the game over for for that organization. They 554 00:34:22,040 --> 00:34:26,280 learned a valuable lesson about uh managing consoles for sure. Um anyways so uh I'm gonna 555 00:34:26,280 --> 00:34:30,220 go ahead and get back to you guys. Great. So in this attack you know uh there's definitely 556 00:34:30,220 --> 00:34:35,480 uh multiple ways to exploit the the issue. So getting back to the Dell issue. Uh they actually 557 00:34:35,480 --> 00:34:39,540 have a couple different user accounts. Uh in fact Dell even came out and says oh well you 558 00:34:39,540 --> 00:34:42,760 need to change the credentials. Well there's actually two credentials on that box and they 559 00:34:42,760 --> 00:34:47,560 only talk about the root credential. And actually any level of access to their just Dell 560 00:34:47,560 --> 00:34:54,200 compelling uh connector uh will actually let you go and and get this uh get everything that I 561 00:34:54,200 --> 00:34:56,260 did here today. Uh and as well as you know I'm gonna go ahead and I'm gonna go ahead and 562 00:34:56,280 --> 00:35:00,180 do this. So if you ever if any company was doing any segregation of duties like maybe the 563 00:35:00,180 --> 00:35:03,820 administrator had access to upgrade the software and do this but they didn't have access to the 564 00:35:03,820 --> 00:35:07,180 vcenter password and they had somebody else put that in. Well the administrator then could 565 00:35:07,180 --> 00:35:12,080 then go retrieve that. So if you're trying to do any kind of segregation of duties with uh 566 00:35:12,080 --> 00:35:18,120 this connector piece it's it's it's a failed attempt. Um at any rate so that was that. Uh so 567 00:35:18,120 --> 00:35:23,320 and then we reported to Dell right? So in April it's about the 11th of April I emailed Dell 568 00:35:23,320 --> 00:35:25,680 actually I think it was at this point that Sherry's like 569 00:35:26,580 --> 00:35:31,620 you need to talk about this. And I'm like and like she's like DEFCON and Black Hat are coming 570 00:35:31,620 --> 00:35:37,700 up. Uh she's like you need to submit and it's more than 90 days so you need to report it and 571 00:35:37,700 --> 00:35:42,000 then you're gonna talk about it. I'm like okay I'll do it finally talk about it. Uh and so I 572 00:35:42,000 --> 00:35:48,060 emailed uh with Dell and and uh my first response back was from Dell was like oh well uh you 573 00:35:48,060 --> 00:35:51,620 need to open a support ticket with your customer number and this is actually a configuration 574 00:35:51,620 --> 00:35:55,500 issue on your part and have a nice day. And I'm like so does that mean you're cool with me 575 00:35:56,280 --> 00:36:00,340 doing this in August? And they're like whoa whoa whoa sorry we didn't mean to send you that. 576 00:36:00,540 --> 00:36:06,240 Uh yeah we take security seriously. Uh and and we're going to we're going to look into this and 577 00:36:06,240 --> 00:36:11,500 get back to you. Uh and you know and I gave them some more details in May you know a little bit 578 00:36:11,500 --> 00:36:15,620 more back and forth. And finally in the end of June I got an email update saying that they're 579 00:36:15,620 --> 00:36:20,700 going to fix it in November. And I'm like but I'm gonna talk about it in August I told you in 580 00:36:20,700 --> 00:36:25,200 April I'm gonna talk about it in August. And they uh they they've actually been pretty decent to 581 00:36:26,280 --> 00:36:30,280 me. That's why the other reason why I have a release key uh but you can go get it. I think I've 582 00:36:30,280 --> 00:36:34,100 given you enough of the of the road roadmap that you can go do this if you've got this in your 583 00:36:34,100 --> 00:36:38,900 organization. Uh but uh yeah they're coming out with a fix. Uh I think actually that whole 584 00:36:38,900 --> 00:36:44,980 storage platform is being uh deprecated. But if how many people in here run storage platforms as 585 00:36:44,980 --> 00:36:51,520 long as humanly possible? Right? Cause how hard is it to move storage platforms? It sucks. So I 586 00:36:51,520 --> 00:36:56,200 you know imagine this is going to be not you know the last time you know oh they deprecated it 587 00:36:56,280 --> 00:36:59,800 it's going away. Well it'll probably you know we will probably get five six more years out of it 588 00:36:59,800 --> 00:37:04,100 until the thing is not usable anymore. Uh cause so many people will use their storage platforms 589 00:37:04,510 --> 00:37:11,420 forever and ever. I know this cause I used to be you. Uh and uh you know there's a lot more uh 590 00:37:11,420 --> 00:37:17,340 things out here. Like I just gave you today these are just three examples of uh private keys in 591 00:37:17,340 --> 00:37:22,980 public places. Right? Um I mean I I've found private keys in remote login apps, in routers, 592 00:37:22,980 --> 00:37:26,260 cameras. My favorite I'm not ready to talk about this one yet. I'm not ready to talk about this one 593 00:37:26,280 --> 00:37:33,100 yet. Uh there's a camera out there that it has it doesn't have a private key. It's got all the 594 00:37:33,100 --> 00:37:40,500 components to make a private key. That they make the private key on the fly and then use that uh to 595 00:37:40,500 --> 00:37:48,600 do the communication with the cloud service. So if that's you guys I'm coming for you. Uh and and 596 00:37:48,600 --> 00:37:52,180 also I you know also printers. I get you know you can get keys out of printers. I actually have 597 00:37:52,180 --> 00:37:56,140 other ways other than just going through firmware to get keys out of printers. I gave a a talk 598 00:37:56,140 --> 00:38:00,780 actually on Wednesday at B-Sides on how I met your printer. Uh go ahead and uh look that one up if 599 00:38:00,780 --> 00:38:05,920 you're interested in and how I can elicit uh your printer just to give me it's password instead of 600 00:38:05,920 --> 00:38:12,460 doing it forensically. But this is a systemic problem. Right? Uh private keys uh in these public 601 00:38:12,460 --> 00:38:18,760 places like firmware is a is a big problem. Uh and uh I recommend you know firmware software 602 00:38:18,760 --> 00:38:25,040 developers the private keys don't belong in the binaries of your software. Right? This is a bad 603 00:38:25,040 --> 00:38:25,660 practice. 604 00:38:25,660 --> 00:38:32,380 You know if anything you know uniquely generated per device or whatever. Uh and also you know 605 00:38:32,940 --> 00:38:37,280 regular security audits and and and testing should be done to look for these secrets. Right? 606 00:38:37,380 --> 00:38:42,060 There's tools out there that will look at your source code and stop the build in your CICD 607 00:38:42,060 --> 00:38:47,420 process if you've got a private key hidden there. Cause developers will do it. I've seen it. I've 608 00:38:47,420 --> 00:38:51,840 done I've audited code. I've been in software development for most of my career. I know how you 609 00:38:51,840 --> 00:38:55,640 guys think. Uh and uh and also coordinated disclosure. I've been in software development for 610 00:38:55,640 --> 00:38:55,760 30 years. Right? So if you found you it's always good to talk about bug prevention. 611 00:38:55,760 --> 00:38:57,520 this topic that I think hashtag endangered another teacher program should be adopted. Right? No 612 00:38:57,520 --> 00:39:02,220 more hush money programs where it's like oh yeah we totally took care of that problem. We paid 613 00:39:02,220 --> 00:39:08,460 off the the guy that found it. Right? Like no you should you should fix your problems and 614 00:39:08,460 --> 00:39:15,080 and come clean about it in public releases. So any more questions? Any more comments? Uh if 615 00:39:15,080 --> 00:39:19,440 you wanna find me on the internet I'm uh Creatively Tom Pull. Actually it's funny I 616 00:39:19,440 --> 00:39:24,320 think my first device was DEFCON 20 and I I came in I think fourth place in a competition 617 00:39:24,320 --> 00:39:25,600 that year. And that's when Dark Tam- 618 00:39:25,640 --> 00:39:29,040 agent called me out in front of 25,000 hackers for being the only guy that goes to a hacker 619 00:39:29,040 --> 00:39:34,320 conference and uses his real name. Uh, but you can find me very creatively, uh, on social 620 00:39:34,320 --> 00:39:40,300 media as, uh, Tom Poole. Uh, I'm on both X or Twitter, whatever it is now, as well as, uh, 621 00:39:40,340 --> 00:39:45,700 the, on Mastodon. And you can find me on LinkedIn as well. Thank you much for your time today. 622 00:39:54,220 --> 00:39:58,120 If you want, if anybody wants to talk afterwards, I'm totally cool. Like, if you want to go to, like, 623 00:39:58,140 --> 00:40:02,260 the Chill Out Lounge or something and we, you know, sit and, and, uh, have some beers and, and 624 00:40:02,260 --> 00:40:03,860 talk more, I'd love to talk with you guys.