WEBVTT 00:00.000 --> 00:03.220 I like it. 00:05.180 --> 00:06.800 You see my botnet back there? 00:07.240 --> 00:09.980 I got buddhists connected to my botnet. 00:10.220 --> 00:11.400 I got hella buddhists. 00:11.400 --> 00:12.560 Hella buddhists. 00:12.620 --> 00:13.740 Buddhists. 00:13.840 --> 00:15.140 Hella buddhists. 00:15.140 --> 00:16.260 Buddhists. 00:16.340 --> 00:17.720 Hella buddhists. 00:17.820 --> 00:18.680 Buddhists. 00:18.680 --> 00:20.360 I got buddhists connected to my botnet. 00:20.360 --> 00:22.960 Hanging out on hack forums, best believe I got warrants. 00:22.960 --> 00:25.480 Telling all who listened that my botnet's not boring. 00:25.480 --> 00:28.220 Modified Mariah and I'm mixing in the minor. 00:28.220 --> 00:30.480 There's no surprise getting wrecked by a minor. 00:30.480 --> 00:33.020 Cover up his drop there, looking for me everywhere. 00:33.020 --> 00:35.520 No I'm really not there, places I will not share. 00:35.520 --> 00:38.100 Drop a couple backups, I know you won't find me. 00:38.100 --> 00:40.500 Buy a couple more smart balls, won't you kindly? 00:40.500 --> 00:43.060 Screening Captain Phillips, who the hell are you? 00:43.060 --> 00:45.580 Wonder why your toaster's connected to RU. 00:45.580 --> 00:47.960 Why your toaster's toasting, I'm using it for roasting. 00:47.960 --> 00:50.680 Some kid got busy boasting, now his modem's smoking. 00:50.680 --> 00:53.180 Keep the crypto flowing, I need my money now. 00:53.180 --> 00:55.600 But I gotta go, cause my mom said study now. 00:55.600 --> 00:58.040 I'm calling all the shots, and it's time for me to score. 00:58.040 --> 01:00.800 I got a couple spots, hit me up on Discord. 01:00.980 --> 01:03.180 Botnet, I got hella booters. 01:03.600 --> 01:05.700 Botnet, take over your rooters. 01:06.160 --> 01:08.400 Botnet, in your internet of things. 01:08.640 --> 01:10.840 Botnet, man this shit hella stings. 01:11.060 --> 01:13.480 Botnet, I got hella booters. 01:13.640 --> 01:15.980 Botnet, take over your rooters. 01:16.160 --> 01:18.540 Botnet, in your internet of things. 01:18.720 --> 01:21.040 Botnet, man this shit hella stings. 01:21.200 --> 01:23.620 Botnet, I got hella booters. 01:23.800 --> 01:26.100 Botnet, take over your rooters. 01:28.440 --> 01:34.360 Well, that was amazing. 01:35.980 --> 01:42.060 Well, thanks so much, Dave, for that amazing intro. 01:42.060 --> 01:50.060 I was not really sure what to think when I just got a PM saying that there was a theme song for my talk. 01:50.420 --> 01:51.960 So hi, everybody. 01:52.280 --> 01:53.880 Thanks for coming and hanging out. 01:53.880 --> 01:55.280 Let me... hold on. 01:55.860 --> 01:57.220 Making sure everybody can hear me, right? 01:57.220 --> 01:58.020 I'll chat. 01:58.020 --> 01:58.920 You can hear me? 01:59.820 --> 02:01.220 Say hi if you can. 02:01.260 --> 02:02.020 Just making sure. 02:02.020 --> 02:03.160 I've got a terrible microphone. 02:03.160 --> 02:03.380 All right. 02:03.380 --> 02:04.040 Cool, cool. 02:04.240 --> 02:04.760 Okay. 02:04.760 --> 02:06.140 So let's get going. 02:06.780 --> 02:08.680 So, yeah, this is my talk. 02:09.560 --> 02:11.160 I'll just get right into it. 02:11.160 --> 02:12.200 So who am I? 02:12.560 --> 02:16.760 I'm Ned Spooky, senior reverse engineer at Redacted Company. 02:16.760 --> 02:25.240 I primarily work on embedded devices, firmware, industrial control systems, and taking apart proprietary network protocols. 02:25.240 --> 02:28.880 You know me online as either Ned Spooky or you. 02:29.300 --> 02:36.800 And I contribute OSS tooling and other errata for 3rd Intel, RE, and offensive security. 02:37.320 --> 02:40.260 So, all right, we'll do a little background on this. 02:40.260 --> 02:42.100 So why do this talk? 02:42.400 --> 02:52.820 So I know that there's a lot of people that, you know, have seen IoT botnets, whether it be how you've been affected by it or see people talking about them online. 02:52.960 --> 03:01.020 And, you know, I had done a bit of my own research and I really kind of wanted to add, you know, a bit of perspective from my end on this. 03:01.240 --> 03:04.740 Because IoT botnets are definitely still incredibly prevalent. 03:05.400 --> 03:09.040 You know, we are all affected by them, whether we like it or not. 03:09.040 --> 03:18.980 You know, if your work slack is suddenly down because of somebody's fight on Xbox Live or Minecraft, you know, this is affecting you in some way. 03:19.040 --> 03:21.140 But I don't think a lot of people take them as seriously. 03:21.140 --> 03:31.580 A lot of people think it's like, you know, kiddy stuff, like script kiddy stuff, which is unfortunate because it definitely is something that is, you know, an issue that we all have to deal with. 03:31.940 --> 03:36.300 And so, I spent a good amount of time collecting malware sources. 03:36.300 --> 03:39.240 So I started doing this in about 2018. 03:39.520 --> 03:49.140 And I had been collecting and developing tools to analyze source code and analyze binaries, which I'll get into later. 03:49.140 --> 03:53.880 I studied a lot of the commonly exploited vulnerabilities and wanted to know more about, like, why they were so prevalent. 03:53.880 --> 04:00.360 Like, why can you have 4 million hacked routers on your botnet? 04:00.360 --> 04:02.460 You know, it's insane to me. 04:02.560 --> 04:10.960 So I wanted to inform others about the impact of their technology choices, specifically firmware devs and people who are also end consumers. 04:10.960 --> 04:14.180 And I also wanted to propose some ideas on how to address these. 04:14.660 --> 04:20.580 And so, I also, this talk, I should have given it about a year ago or so. 04:20.640 --> 04:23.380 It's just kind of been pushed under the back burner a bit. 04:23.380 --> 04:26.060 But I tried to update it as much as I could. 04:26.060 --> 04:30.400 And I also had to kind of cut out some of the parts, which I'll talk about as I go through the talk. 04:30.800 --> 04:32.220 So here's the outline real quick. 04:32.220 --> 04:34.380 So we're going to go over IoT botnet history. 04:34.840 --> 04:37.680 We're going to go over the actual botnet scene a little bit. 04:37.860 --> 04:41.600 Talk about the architecture of botnets and how they spread and propagate. 04:41.740 --> 04:47.600 And then we're going to talk about the firmware vulnerabilities that enable them and steps to move forward for vendors. 04:47.880 --> 04:50.840 So starting off just with IoT botnet history. 04:50.840 --> 04:52.700 So what is an IoT botnet? 04:52.700 --> 05:00.240 So if you're here watching for IoT Village, you probably are aware of IoT issues and botnets in general. 05:00.340 --> 05:11.520 But for those who haven't seen anything about them, they're basically a network of hacked IoT devices that are basically internet connected devices like routers, laptop boxes, webcams, your toaster. 05:11.800 --> 05:18.120 This little toaster on here has a little script on how to scrape Shodan for Linksys routers. 05:18.840 --> 05:24.760 But so they're used primarily for DDoS and they're sometimes used for crypto mining and also tunneling and proxying traffic. 05:25.140 --> 05:38.300 And so for this thing, I'm going to... this talk is kind of more of a cultural as it is technical and I kind of want to be able to kind of go through a lot of the confusing nomenclature that surrounds this because there's a ton of it. 05:38.580 --> 05:45.100 People call botnets by a million different names, whether you're talking to a researcher or talking to a person who develops them. 05:45.120 --> 05:47.080 There are a ton of different types. 05:47.080 --> 05:49.400 So let's kind of go through a little bit of the history here. 05:49.440 --> 06:02.800 So IoT botnets as they're known right now can be traced back to, I guess, 2014-ish when LizardSquad came out with the, I guess, the botnet, the malware with the most names available, like any malware. 06:03.000 --> 06:04.920 It's been termed a bashlight. 06:04.920 --> 06:12.300 It can also be called LulzBot, Torless, Lizkabob, LizardStressor, Ballpit, Gaffigat, and just a bajillion other names. 06:12.620 --> 06:18.740 And so it was spread by exploiting Shellshock vulnerabilities when Shellshock came out in BusyBox on a bunch of different devices. 06:18.740 --> 06:29.000 So there was... people were, you know, scanning the entire internet for Shellshock vulnerabilities because they were all over the place, but they were very, very common in a lot of IoT devices. 06:29.800 --> 06:39.040 And so when this was actually happening , though, there were actually a lot of different botnets or bots that were being distributed because there hadn't been as many sources as there are now. 06:39.180 --> 06:49.980 So you've heard of KITAN, which is like an IRC C2-based botnet that was spread a lot, as well as like just Perl bots that are just literally DDoS bots that are written in Perl. 06:50.300 --> 06:55.140 So the source code for this, though, was leaked in 2015, and a lot of people started to work on it. 06:55.180 --> 07:08.360 And so collectively, it's hard to choose one name for them, but collectively I would say that these would be categorized as QBot, which is unrelated, again, to the KAKBot malware, which people call QBot. 07:08.900 --> 07:17.300 And so, yeah, new devices that are still vulnerable to this exact same vulnerability appear online like newly to this day. 07:17.500 --> 07:30.760 And so fast forward a couple years, so MIRA came out in 2016, and so it was used in some famous DDoS attacks like the Dyne DDoS attacks, the ones on Brian Krebs and some other people. 07:30.760 --> 07:43.380 But it was leaked shortly after some of the bigger DDoS attacks happened, and people started to immediately use it because it was a lot more streamlined than the previous versions of DDoS malware. 07:43.380 --> 07:52.440 A lot of the stuff was in, you know, really simple one-file bots and servers, very, very basic stuff over telnet. 07:52.580 --> 07:55.140 So MIRA was a lot more streamlined. 07:55.140 --> 07:57.620 It was very modular, so there's different files. 07:57.620 --> 08:06.360 It made it easier for you to plug in new exploits into, and also made it easier for you to have access control for users that were coming on. 08:06.680 --> 08:09.920 And so it also had a bit better code. 08:09.920 --> 08:15.520 It was definitely still not the best, but it's a lot better than the previous code for LulzBot. 08:17.280 --> 08:22.200 It also had a SQL server on there, which made it running the server a lot easier for them. 08:22.760 --> 08:28.260 And so it seems like everybody has a botnet fork these days, since then. 08:28.320 --> 08:31.280 So there's other IoT botnets that are pretty major that have come out. 08:31.480 --> 08:46.760 A big one that you may have heard of was Satori or FBot or Okiru, and it's a pretty well-known MIRA fork that's a bit different from some of the other ones because a lot of them are kind of just very copy-pasted stack overflow questions fit into some Golang and C code. 08:47.420 --> 08:56.300 But so this one here had a bit... the person who was doing it definitely knew what they were doing a bit more than most people, and that person actually just went to jail recently. 08:57.400 --> 09:06.240 We've also seen BrickerBot, which is the botnet that would just basically infect and break IoT devices, and there's been a few iterations of it. 09:06.640 --> 09:12.520 There was one in 2017, and then there was one recently, I think it was like a 13-year-old kid or something like that that did it as well. 09:13.000 --> 09:24.260 A newer one, really interesting, is Kaiji, which is a Golang-based cross-compiled SSH brute-forcer, and it actually installs a root kit or tries to to establish persistence, which is really interesting. 09:24.260 --> 09:25.760 I'll get into that more later. 09:26.360 --> 09:27.960 AxisR is another one that I've seen. 09:27.960 --> 09:33.320 I just threw in there because I didn't hear anybody talking about that one, but it's more modular, still crappy. 09:33.880 --> 09:37.140 Then you have various Bitcoin miner botnets that you may have seen. 09:37.140 --> 09:44.360 It's harder to do Bitcoin mining on an IoT device because they don't have as much CPU power and no GPU, but they're still out there. 09:44.360 --> 09:57.600 And then I also did a write-up on some Mirai variants that are targeting FPGAs and some really exotic architectures, which I have a link in citations at the end here, or you can go on my website and see it. 09:58.200 --> 10:11.600 So yeah, botnet activity growth, I mean, like similarly to Qbot, Mirai just started popping up all over the place once it was leaked, and there became basically a huge marketplace for people who are trying to sell spots on the botnet, right? 10:11.600 --> 10:16.060 There's reseller markets, affiliate programs, and incentives for having it grow. 10:16.420 --> 10:32.340 And also booting itself, you know, DDoSing somebody, their home router, became a really common thing for people to actually, you know, try and do because it's just like a way to knock people offline, especially if you're mad at them in Call of Duty or something. 10:32.460 --> 10:37.080 And so yeah, it's basically just like the thing that people start to do. 10:37.080 --> 10:46.060 And so like, you know, though, as these things develop and grow, like a thousand monkeys at a thousand terminals will eventually take out the intranet, and that's kind of what's been happening. 10:47.120 --> 10:50.320 And so we'll get a little bit into the scene here. 10:50.320 --> 11:04.780 So the botnet scene at a glance, there's entire communities that are dedicated specifically to just one botnet or one botnet group, and they are usually talking on Discord, sometimes they're on forums or IRC. 11:05.120 --> 11:13.440 There used to be definitely more IRC back in the day where people would have C2s connected to IRC, but nowadays it's more Discord that people are talking. 11:13.540 --> 11:18.300 Advertising is done on literally every single social media platform you can think of. 11:18.300 --> 11:33.420 I think somebody found Pinterest that had somebody advertising botnets, but yeah, if you go on Instagram and just literally search botnet or Qbot or botnet setup or YouTube, you will find somebody advertising their latest slamming botnet. 11:34.060 --> 11:47.020 And so bootertime is generally sold people for DDoS or through a web panel or through a telnet interface, but that's like the main thing people are trying to do is just sell time on the botnet. 11:47.460 --> 12:02.460 And so you'll see here on the bottom, it might be a little small for some of you, but there's just some advertisements for different botnets and also some videos on how to boot people offline and how to do it using just an Android and best booters 2020. 12:02.740 --> 12:09.820 These all have hundreds of thousands of views too, so these are people that are really going hard with the advertising. 12:10.360 --> 12:14.680 So the sources, so I talked about the sources that have kind of been modified by people. 12:14.880 --> 12:24.460 They're usually distributed as Zips or RARs or whatever and they are sold for about $500 to $300 USD from just what I've seen. 12:25.060 --> 12:37.080 So the authors will typically change very little of the code base, usually just involves something simple, just changing the ASCII art or changing the variable names, something like ctrl f and replace. 12:37.480 --> 12:55.300 Sometimes they might even add a new exploit, which is always interesting, but exploits themselves to load bots are sometimes sold, but a lot of them are literally like you can google any part of the script and you will find the exploit db link of where they took it from. 12:55.300 --> 13:10.560 The ones that are sold though from exploit db or metasploit modules are usually backdoored and it's really funny they just have like a base64 blob that just like runs like import os and then just run this or import sys and run this or whatever. 13:10.880 --> 13:29.860 And so when I was going through and finding a lot of these sources though, I would find that when people would scam each other or rip off somebody or somebody had a fight with them, they would leak each other's source code, which is great for threat intel people and reverse engineers who want to figure out what's going on because they would just be like, 13:29.860 --> 13:38.100 oh hey, here's this person, here's everything they've done, here's their botnet and here's their code and you can just kind of scoop it up and take a look at it. 13:39.300 --> 13:47.000 And so selling spots, primary source revenue, as I said, they're typically sold in weekly, monthly, or lifetime plans. 13:47.000 --> 13:49.660 You can see a breakdown of plans over here. 13:49.700 --> 13:51.220 Pretty cheap too. 13:51.760 --> 14:05.880 The lifetime is always really funny to me because it really just means for the duration of the bot's lifetime, botnet's lifetime, and sometimes that does not last longer than the three days or a month depending on how bad their operation is. 14:06.560 --> 14:17.120 Some more enterprising people, people who are a bit more advanced might use a web stressor and they can sell access to that with users and everything for a web browser. 14:17.180 --> 14:21.660 There's been a few big web stressors that have been taken down and some that are still up. 14:22.060 --> 14:24.860 Web stressor source leaked volumes in the web stressor. 14:24.860 --> 14:30.240 There's so much surrounding that it adds a bit of abstraction to it that makes it harder to manage. 14:31.220 --> 14:37.900 And then yeah, finally, some people act as resellers and they get a cut of the sales over time. 14:38.980 --> 14:41.160 So who runs a botnet, right? 14:41.160 --> 14:52.500 IoT botnet operators, based on what I've seen in the scene, I guess, they're usually pretty young, high school age, sometimes college age. 14:52.500 --> 14:57.520 They're somewhat experienced with computers, but they're usually not developers. 14:57.940 --> 15:08.920 They learn a lot through YouTube and through text files, which I have a collection of in the GitHub that I'll explain in a little bit, which are just tutorials on how to set them up. 15:08.940 --> 15:20.660 Basically, how to spin up a rel box and how to actually... hold on one second... how to actually just do the basic things that compile with GCC. 15:21.460 --> 15:25.540 A lot of the times, though, they really have no clue what they're doing. 15:25.760 --> 15:35.860 So you'll see people who are trying to get support for different botnets and they're really confused about GCC or what access control is. 15:36.760 --> 15:41.180 But more sophisticated people might have a web stressor or an API, like I said before. 15:41.300 --> 15:47.960 People will use cryptocurrency instead of PayPal, which is very common for some reason, even though it's tied to your bank directly. 15:48.240 --> 15:54.000 In some cases, though, people will also use botnets for additional purposes like proxying traffic. 15:54.000 --> 16:04.920 So sometimes you'll see fly-by-night VPN operations that might be doing something shady like, you know, routing their traffic through routers and that's just their VPN somehow. 16:06.140 --> 16:09.560 So why run an IoT botnet? 16:09.560 --> 16:18.140 So, I mean, just as much as most malware is, there's a lot of similar reasons, but there's a lot of stuff that comes with the fact that there's a lot of younger kids involved in this. 16:18.340 --> 16:25.400 So they usually do it for either money, you know, because people can earn money from the sale of botnet spots. 16:25.400 --> 16:27.620 A lot of people do this for attention. 16:28.080 --> 16:34.440 People, you know, seek attention for stuff, even if it's not even a DDoS, and it's just like a regular production outage. 16:34.440 --> 16:43.760 Some people might say, oh, yeah, my group, we DDoS these people, and, you know, we're gonna extort them for money. 16:43.800 --> 16:51.660 And then you look at their status pages like, oh, yeah, sorry, we had a blip in, you know, updating this thing, and we're back now, which is always awesome. 16:51.940 --> 16:53.220 Supply and demand. 16:53.440 --> 16:57.400 It's definitely people who want, you know, to DDoS each other. 16:57.600 --> 17:03.900 And so that's, you know, definitely wanting to meet that demand is something that, you know, it's good for any young entrepreneur. 17:04.600 --> 17:06.000 Revenge is also big. 17:06.000 --> 17:14.100 I see a lot of people claiming that somebody, you know, docks them or DDoS them, and they want to get back at them by getting their IP and booting them offline. 17:14.460 --> 17:22.900 And then people are also inspired a lot by past attacks, because people have seen what actually happens if somebody DDoS and takes out, you know, the internet. 17:23.040 --> 17:25.680 They want to be, you know, doing that. 17:25.680 --> 17:29.620 And also, it's incredibly easy to set up an IoT botnet. 17:30.280 --> 17:35.180 So let's take a little bit of time to go over the architecture of DDoS botnets. 17:35.180 --> 17:46.520 So as I said before, earlier botnets use standalone bot files and C2 files that were just compiled with GCC or UCLibc for cross-compiling. 17:46.540 --> 17:51.780 They're very, very simple to set up and deploy. 17:51.900 --> 18:04.340 Some of them for C2 itself, they used like IRC for command and control, and they'd have IRC, like very bare bones IRC clients connected to their... or within their bots. 18:04.560 --> 18:13.860 But Mirai modernized it, and they have actually a C2 protocol that is used, and they have like a SQL backend for tracking bots and all that. 18:14.900 --> 18:23.060 So web stressors will use PHP and some other, I guess, API stuff for managing the bots. 18:23.060 --> 18:29.200 But it's definitely evolved a lot more than it used to be like five years ago, which is interesting to see. 18:29.960 --> 18:34.940 So the life cycle of a botnet is usually very, very short. 18:35.100 --> 18:39.460 You don't see them for very long, not going to be over a month or two. 18:40.020 --> 18:51.080 Basically, somebody will set up a C2 on like a lax VPS host, they'll scan for their own devices, they'll get some bots to their botnet, they'll advertise their spots, and then use it. 18:51.240 --> 18:53.400 And then the takedown goes one of two ways. 18:53.400 --> 19:07.540 Either somebody, like say bad package report, will tweet out their botnet to the web host, and somebody will notice it and get it taken down. 19:07.540 --> 19:14.340 Or somebody else's botnet will start kicking their bots from the system, and they won't be able to keep up and they'll lose power. 19:14.460 --> 19:16.060 But then it'll just keep happening again. 19:16.060 --> 19:18.620 This is just the same thing you see over and over again. 19:19.660 --> 19:21.120 Hold on one moment. 19:27.580 --> 19:28.600 Okay, that's cool. 19:28.600 --> 19:29.640 I didn't think it was the water. 19:30.500 --> 19:35.140 So this inevitably leads to a king of the hill game for botnets. 19:35.960 --> 19:37.360 They're very territorial. 19:37.360 --> 19:43.680 People are, you know, targeting one specific type of device with one specific vulnerability that they've coded into their variants. 19:43.880 --> 19:51.840 And then when somebody else gets the same idea, you know, they'll start hacking and doing things like that, getting their bots on there. 19:52.060 --> 20:05.140 Anybody who touches the devices usually already has root access, but they might have either like some weird file system, or there's no way to really reconfigure it, or they might not know how to reconfigure the device to kick everybody else out. 20:05.140 --> 20:10.460 But basically, every bot will only last as long as it can before somebody else takes its place. 20:10.460 --> 20:12.300 And also, there's really no repercussions for this. 20:12.300 --> 20:22.300 So everyone's just kind of slamming on different IoT devices, and pictured as an IoT operator, or botnet operator watching their bot count drop. 20:23.640 --> 20:27.300 So evasion is definitely an interesting aspect of this. 20:27.300 --> 20:32.180 So there's a lot of very simplistic evasion that you'll see here. 20:32.500 --> 20:42.540 This one up at the top here is somebody just renaming their process to DropBear, which is, I guess it works, but it's also used by everybody. 20:42.540 --> 20:46.300 So then everybody will just kill the DropBear process once they log on. 20:46.880 --> 20:52.520 But that's realistically, this is not to hide from, you know, any sort of firewall or any sort of AV or anything. 20:52.820 --> 20:56.540 It's only really used to evade other botnet operators. 20:56.680 --> 21:07.640 And so they'll do things like, you know, the process masking, they might learn about a different area of the file system that they can put a bot in, they might hide a backup bot and have like, potentially something like a cron job. 21:07.640 --> 21:12.240 It's always very, very primitive, and like, very, very bespoke. 21:13.020 --> 21:19.780 And I actually had a whole code review section that could have actually been an entire talk, but I had to cut it for time here. 21:19.780 --> 21:28.020 But there's a lot of very, very strange ways that people try to do evasion, which I would love to talk about at another date. 21:28.920 --> 21:41.360 And so bot killing, as I said before, people will do this, you'll take a look at this, if you can see it on the side here, here's a, you know, an array that's just full of a ton of different bot names. 21:41.360 --> 22:01.140 And every time they updated this botnet, which I have multiple versions of it, they would add more and more of these, but you'll see they'll do things like iterate from one to however many, try to kill every process that's called that, or every single version of this specific JackBind MIPS or whatever, 22:01.140 --> 22:02.120 TwoFace. 22:02.260 --> 22:07.320 So people will like, be aware of different botnets that are operating and what they name them, and then put them into their scripts. 22:07.320 --> 22:18.560 And it's like a cat and mouse game, because nobody can fit everything in there, otherwise their binaries are going to be full of strings that ultimately are going to get detected by people who are reverse engineering the malware. 22:19.860 --> 22:31.920 And so, not some, not most, nearly all bots and C2s, I would say all, have really, really silly vulnerabilities that make them incredibly easy to knock offline. 22:31.920 --> 22:40.560 And I don't really see too many of these techniques really utilized or advertised by people, but in my next slide, I'll show you something interesting, I guess. 22:40.760 --> 22:44.180 So here's my non-live demo for a C2 killer. 22:44.180 --> 22:46.680 So this is something that I found. 22:46.680 --> 22:53.440 I definitely was not the first person to find this, but it was part of my testing when I was testing out these different things when I was researching. 22:54.040 --> 22:58.780 It's incredibly easy to kill, you know, Mirai C2s. 22:58.780 --> 23:01.320 This is a, you know, take your screenshots or whatever. 23:01.320 --> 23:03.460 I already put this out on Twitter at some point. 23:03.620 --> 23:14.140 But yeah, this is, if you send this to either the admin port or to the heartbeat protocol port, it just, it just segfaults the Mirai C2. 23:14.140 --> 23:16.140 And I've never seen anybody fix this. 23:16.140 --> 23:18.460 This isn't every version of Mirai that I've seen. 23:18.600 --> 23:25.820 Some people had claimed that it was fixed, but I've still, after reviewing every single one of them that I could find, I've never seen that. 23:26.020 --> 23:31.720 And so, take a bit of a second, the last little bit on the botnet scene here. 23:31.960 --> 23:37.960 You know, when I was going through and doing this work, I ended up creating a tool to help me track all this stuff. 23:37.960 --> 23:39.460 And I put it out on GitHub. 23:39.520 --> 23:45.020 It hasn't been updated for a bit because it just, I've gotten too many other projects to do. 23:45.160 --> 23:53.800 But it's, it's a static analysis and classification tool for zip files and binaries and things like that. 23:53.800 --> 23:56.460 It just feeds it on to a big Elasticsearch database. 23:56.800 --> 23:58.880 I definitely, I'm taking the next week off of work. 23:58.880 --> 24:02.100 So I'm going to take some time to push my big update to this. 24:02.100 --> 24:04.120 But if you want to check it out, definitely do it. 24:04.120 --> 24:09.600 I have some new things in API, different symbol hashes and key extraction techniques in there. 24:09.780 --> 24:22.460 But it's just mainly for me for fast analysis because I was, you know, I had, it's basically feeding in either new source code or new bot binaries into this and just tracking them. 24:22.460 --> 24:24.260 But it works for other malware too. 24:24.680 --> 24:35.740 Also, if you want to, I have a Twitter that was deleted by Twitter for some reason, that was called Threatland, but that was the name of the project that I used to track all these sources. 24:35.740 --> 24:42.580 So I have like every Mirai and Qbot and other botnet, just even beyond IoT. 24:42.840 --> 24:50.580 I tracked them all in a big repo called TLBots and have a few other repos if you want to check them out for like fraud tools and stuff. 24:50.580 --> 24:57.540 But yeah, there's literally, clone that, there's like a gigabyte worth of zip files of every malware source code that I could find. 24:59.460 --> 25:02.240 Yeah, so now we're going to get into talking about vulnerabilities. 25:02.240 --> 25:17.740 And this is kind of, this aspect of it is a bit more about like stuff for devs, because I wanted to be able to give info for developers who are working on IoT devices to take all of what I just said there and put it into context for their actual security architecture. 25:18.440 --> 25:20.380 So let me take another sip of water. 25:28.500 --> 25:31.560 So okay, so we're here, we're peering into the void here. 25:31.720 --> 25:46.140 So here is, if you ever have gone on Green Noise, they have a lot of tags for different either vulnerabilities themselves of the people are scanning for, or just classes of, you know, malicious traffic. 25:46.140 --> 25:58.160 If you do a search just for Mirai, you'll see here, it's very, very tiny, but there's four and a half million results for unique devices that have been scanning with Mirai-like traffic. 25:58.160 --> 26:05.480 So that gives you a rough example of how many people are, or how many devices are actually infected and actively scanning. 26:05.740 --> 26:22.580 And then the other one is a showdown search for this hacked router help SOS had do password thing, which there are still, this is, I think that that hack happened like four years ago, and there's still 6,500 devices that have been hacked and have this hostname. 26:22.580 --> 26:25.480 So it's always heartwarming to see, I guess. 26:25.820 --> 26:28.580 So what types of vulns are exploited by these botnets? 26:28.580 --> 26:30.960 So it's always very basic stuff here. 26:30.960 --> 26:33.640 We're talking about weak auth and auth bypass. 26:33.640 --> 26:46.820 So there's either admin admin as the credentials, or there's like a page that you can, you know, run OS commands on that doesn't actually need a password to be, you know, interacted with. 26:46.820 --> 26:52.900 There's also command injection like shell shock and other really silly command injection stuff. 26:53.480 --> 27:02.000 There's also a lot of common exploits and specific services and libraries like the Realtek UPnP SDK, which had a vuln that was like in everything. 27:02.000 --> 27:04.400 There's so many different devices. 27:04.920 --> 27:13.680 GoAhead webs and ThinkPHP also have vulns that were in a lot of places, thousands and thousands of stuff were affected by GoAhead . 27:15.000 --> 27:30.440 And so more rare though, you'll see actual shellcode and binary exploits, which is always interesting to see because, you know, you'll have devices that, you know, are using like the same base address and they can just do a shellcode exploit very, very easily. 27:30.740 --> 27:43.400 But if they're not as common as, as you'd think, and I think it might be because people don't know how to code a shellcode or how to inject shellcode, like with an NC, like when they're writing their bots. 27:43.400 --> 27:46.460 So who knows, but you'll see them in bot loaders for sure. 27:46.640 --> 27:49.660 A lot of other vectors include previously compromised devices. 27:49.660 --> 28:05.720 So like if you, people sell lists of compromised devices for a specific category of devices, which there might, I don't actually know if there's any in my repo, but I have seen a bunch of them where people are basically just passing those things around. 28:06.100 --> 28:08.220 So we're looking at the most targeted devices here. 28:08.220 --> 28:19.340 So if you want to see, you know, what vulns are most leveraged by these botnets, I have a command table or a table here of, basically, I went through every source code that we could find. 28:19.340 --> 28:22.660 There's like several hundred unique source codes. 28:22.680 --> 28:25.640 And these are the main ones that people are using. 28:25.700 --> 28:32.060 A lot of them here don't actually have any like CVE or CPE or any vendor acknowledgement. 28:32.060 --> 28:40.720 So you can only really find them by kind of looking up what the traffic is or what the command injection attempt was on, you know, in your log files. 28:40.720 --> 28:44.440 Actually, more than half of these don't have any CVE at all. 28:44.440 --> 28:51.980 The AVTech one, which I think is being used in the IoT CTF right now, doesn't have a CVE or anything. 28:51.980 --> 28:55.560 And it's just a blog post that, you know, people have written about it. 28:55.560 --> 29:02.780 Same with like, you know, some of these Netgear ones, the Netgear DGN 1000, that's a huge one that people have exploited. 29:02.780 --> 29:11.320 I've never seen a CVE for, you know, the HNAP, Vacron, like Zyxel stuff, even actually go-ahead websites have some. 29:11.760 --> 29:22.300 But here, these aren't even SSH or telnet brute force stuff, this is like, like, a lot of the stuff here, like buffer overflow, or like command injection, and some of these aren't even being tracked by anybody. 29:22.700 --> 29:30.240 So when a new exploit comes out, though, bot scanners will really just immediately start trying to load bots with like whatever PFC people have. 29:30.300 --> 29:34.580 And it's usually IoT bots, and it's very annoying. 29:35.220 --> 29:37.420 So the infections spill over from that. 29:37.420 --> 29:47.260 So like these malware families are, you know, running on a super diverse array of architectures, like there's every architecture you can think of has a Mirai variant for it at this point. 29:48.040 --> 29:53.080 Because of cross compiling, but this means that this can affect other hosts that aren't IoT. 29:53.220 --> 30:08.580 And so people will try to use and they'll try to get Mirai onto things like web servers using like Drupal get in or Apache struts, or, you know, CouchDB or whatever thing is running, they're going to try to do that to have that be the scanner as well. 30:08.860 --> 30:11.840 So these sort of infections spillover is really common. 30:11.840 --> 30:17.560 And you'll see, sometimes like IoT botnets are using Drupal get in and you're like, what router is running Drupal? 30:17.560 --> 30:19.780 It's because they're trying to get onto everything. 30:21.060 --> 30:25.140 And so why are these devices so easy to exploit? 30:25.680 --> 30:31.320 And so we've talked, we were talking about this in the last talk here about, you know, supply chain issues. 30:31.320 --> 30:35.320 And so there's, it's very difficult to validate supply chain is a big one. 30:35.320 --> 30:43.860 There's vulnerable software and libraries that people use, they might not be able to change or have the people to even, you know, make the changes for it. 30:43.920 --> 30:46.980 Easy to guess default passwords is a huge one. 30:47.100 --> 30:57.560 Devices by default, doing port forwarding, listening on the internet, giant list of phone devices are passed around, which makes it even easier for people who don't know what they're doing to just start exploiting. 30:57.560 --> 31:02.640 And then it all comes down to insufficient or non-existent security practices and development. 31:03.000 --> 31:07.600 And so we're gonna get a little bit into firmware bones now and security practices. 31:07.600 --> 31:16.860 So you'll see there was a awesome talk, I think two years ago, Shmoo Khan, about firmware bones by CITL. 31:17.660 --> 31:25.320 And so there's a lot of stuff like vendor security practices on a binary level are like, they're, you know, almost non-existent. 31:25.320 --> 31:34.760 And there's even regression analysis to show that firmware is actually becoming worse and having more vulnerabilities introduced to them in a 15 year data set, which is insane to me. 31:35.440 --> 31:39.620 So you see here, here's like every vendor that they had looked at. 31:39.620 --> 31:52.740 And you see anything that's closer to the edge here is going to have more of these things like stack guards or non-executable stack or railroad, ASLR, things that are closer to the edge are scoring higher. 31:52.740 --> 31:55.940 And actually that means that more binaries have these mitigations in place. 31:55.940 --> 32:01.500 But if you can see, there's very, very few that actually have anything on the graph. 32:02.020 --> 32:06.300 And then the ones that do, they only have like one or two, and there's very few. 32:06.740 --> 32:07.880 It's kind of sad. 32:07.880 --> 32:15.920 You want this whole, all these things to be blue, all the way blue, and there's like lines of blue, which is really disheartening here. 32:17.380 --> 32:21.100 And so why is firmware so difficult to maintain? 32:21.100 --> 32:23.060 So there's so many reasons for it. 32:23.060 --> 32:27.400 And I used to have done firmware development before for embedded devices. 32:27.760 --> 32:36.500 And even the experience that I had doing that is still... I could see all the echoes of this throughout the process, right? 32:36.500 --> 32:39.340 Because re-architecting cost is a huge thing. 32:39.340 --> 32:43.540 Cost is usually the biggest factor for why things aren't changing. 32:43.660 --> 32:46.540 But you can also be locked into a vendor contract. 32:46.540 --> 32:48.700 You can be locked into like a middleware contract. 32:48.700 --> 32:57.200 So you can only use drivers for this one piece of your kit, and you have to use it for a certain period of time. 32:57.200 --> 33:00.900 You might have unsupported chips or hardware to work with. 33:01.000 --> 33:02.960 Another huge thing is outdated toolchains. 33:02.960 --> 33:08.020 You might be using some toolchain from like 2005, and that's how you build everything in 2020. 33:08.520 --> 33:12.900 There's also a lot of things like hardware restraints, which sometimes you might not... 33:12.900 --> 33:17.180 like your hardware itself might not support the SSL version that you need to. 33:17.180 --> 33:30.560 That was an issue that I've had to deal with before trying to figure out how you can, you know, jerry-rig a new SSL encryption scheme and support newer versions of TLS in firmware that's, you know, 15 years old. 33:30.740 --> 33:37.580 Sometimes you also need to maintain backwards compatibility, which is a big thing to make this stuff really... it makes it very, very hard. 33:37.580 --> 33:40.320 You have to include stuff that you might not want to. 33:40.740 --> 33:47.180 A lot of stuff though is a lack of dependable updates for users to update their devices. 33:47.680 --> 33:57.320 So even if you have, you know, all the other things in place here, it's sometimes people don't have like a way to actually update the devices without some complicated process. 33:57.680 --> 34:02.540 Poor communication channels even tell people about vulnerabilities is also a big thing. 34:02.620 --> 34:09.740 And vendors might not have any sort of like channels for reporting bugs or telling people about bugs as well. 34:09.740 --> 34:19.580 And then, you know, lack of modern security measures like secure boot or binary hardening we talked about before or code signing are not going to really be in place. 34:19.580 --> 34:28.800 And it's hard to get those things back into your, you know, pipeline if you have to do a bunch of testing and you only have a couple people working on the thing. 34:29.400 --> 34:33.360 And so why do we see a lot of this older stuff working? 34:33.360 --> 34:44.840 So there's some sometimes you'll actually see Qbots or Kytenbots or even Perlbots trying to exploit stuff in your and, you know, if you download binary. 34:45.480 --> 34:49.920 And it's because the vulnerabilities are still there, right? 34:49.920 --> 34:55.080 And so this is something that I had actually to steal an analogy from Mudge, right? 34:55.080 --> 35:03.320 In mining, there's indicator minerals that can prove that there are other things that you, what's it called? 35:03.320 --> 35:05.940 There's like a, say you're looking for like diamonds or something. 35:07.040 --> 35:14.980 There's like indicator materials that prove that this might be there, the thing that you're looking for might be there. 35:15.020 --> 35:32.500 And so the security vulnerabilities that we're seeing are showing that there are not as many security practices that are being followed, which means that the older vulnerabilities are still going to be able to work, right? 35:32.500 --> 35:41.340 So like we're seeing like, you know, there's still command injection here in 2020, and then you can still run a Qbot or a Perlbot on this device. 35:41.340 --> 35:50.360 This means that there's really not anything that's going into the actual process of making the devices any more secure. 35:50.640 --> 36:02.100 And what's interesting is this is rare in other classes of malware , like say for desktop computers, because there's no patch really that you can apply to one specific device or whatever. 36:02.480 --> 36:16.560 So each time that there's a new vuln that comes out, there's all these new devices that are added to the pool, but there's still all the routers from 2014 that had shell shock vulnerabilities in them, and DVRs that still have auth bypass in them, and those are all just getting added to the pool. 36:16.560 --> 36:22.920 So people are just trying the same old techniques and they're still getting the actual devices that they would have exploited, you know, before. 36:23.060 --> 36:26.760 So it's kind of frustrating. 36:27.500 --> 36:34.420 So moving forward, this is the big thing for vendors and people who are, you know, developers of firmware and embedded devices. 36:34.420 --> 36:38.600 So what can we actually do to solve any of these problems here? 36:38.600 --> 36:55.480 So we can only really fix them by having better development practices for security by meeting the developers and the vendors where they're at, because we want them to be, you know, on top of their game actually doing, you know, the work that we'd like them to put in so that our, 36:55.480 --> 37:03.520 you know, toaster isn't, you know, DDoSing somebody because of some Mirai variant that's run by, like, a 14-year-old kid. 37:04.420 --> 37:08.700 So we have to, like, actually talk to them, talk to vendors the way that they... 37:08.700 --> 37:13.680 things that they already know in their... the way that they're already developing things. 37:13.740 --> 37:25.320 So for vendors, I guess my big advice here is to invest in developer training and to establish best practices and create security testing pipelines and encourage researchers to actually find bones and disclose them properly. 37:25.320 --> 37:39.980 We can mitigate some of the existing bones by encouraging safer use of IoT devices, but it only works so much because... imagine trying to explain to your parents how to, you know, turn off port forwarding on their... on their router, right? 37:39.980 --> 37:49.540 Like, they're not really going to understand it in the way that you might... I mean, they might, but they... you know, it's sometimes hard to get end users to actually follow your guidelines at all. 37:49.540 --> 37:53.120 They might not even be able to be aware of it. 37:53.240 --> 37:56.480 But that's one specific way that we can mitigate that. 37:57.040 --> 38:01.060 Establishing best practices, though, is a thing I wanted to highlight for a second here. 38:01.340 --> 38:04.360 So, like, auditing your development cycle itself. 38:04.520 --> 38:21.040 It definitely depends on what you're building, and you have to tailor it to that and be able to audit and say, hey, yeah, we are using C, we are using, you know, this tool chain, we're using either GCC, or, you know, UC Lib C, or whatever, to, you know, 38:21.040 --> 38:22.240 develop our firmware. 38:22.440 --> 38:25.520 But there are there are best practices for these things. 38:25.520 --> 38:43.620 So, OWASP, something that I had when I was doing firmware stuff, I had made developers, I had them look at these cheat sheets for OWASP on, like, toolchain hardening on, like, input sanitization for web apps, and, you know, how to do other things, you know, 38:43.620 --> 38:45.400 the best practices that there are. 38:45.400 --> 38:47.340 And there's tons of different ones like this. 38:47.340 --> 38:53.400 I just post to OWASP because it's, you know, very accessible for a lot of people, and it's free. 38:53.800 --> 38:56.180 Another big thing is CIS benchmarks. 38:56.460 --> 39:03.500 You know, there's, depending on what you're building, there are benchmarks for security that you can follow, which are super duper useful. 39:03.780 --> 39:05.140 You know, you can even automate that. 39:05.140 --> 39:07.740 I had done, like, Ansible CIS benchmarks before. 39:07.980 --> 39:10.440 You can build those into toolchains pretty easily. 39:10.840 --> 39:16.900 And then also, if you really need to, hire a consultant to come in and do all this work with you and work through it with your team. 39:16.960 --> 39:20.000 That's definitely a huge thing for vendors. 39:20.820 --> 39:24.540 Vulnerable disclosure, though, is probably my favorite one to talk about and the biggest one. 39:24.700 --> 39:31.100 So, when you do find people that are actually poking at your stuff, allow them to disclose vulnerabilities. 39:31.100 --> 39:34.900 Please, if you are a vendor and you're listening, there are ways now. 39:34.900 --> 39:35.680 It's 2020. 39:35.680 --> 39:37.340 You can't have a VDP . 39:37.340 --> 39:38.840 You can't have a bug bounty. 39:38.840 --> 39:42.460 You have to go through the channel, the proper channels, and make sure it's right for you. 39:42.460 --> 39:44.860 But there are a ton of resources. 39:44.860 --> 39:49.760 Disclose.io has, like, really good legal language and other things, resources for vendors. 39:50.100 --> 39:58.400 But yeah, people who do IRT research sometimes either get no response or they get, you know, a summons, you know, get sued for something. 39:59.040 --> 40:02.000 Establish security contact, though, and listen to email. 40:02.000 --> 40:13.020 There's security.txt is a really easy way on your vendor website to just have a email that somebody who has an issue or a vulnerability can talk to and not feel like they're, you know, trying to chase you down. 40:13.020 --> 40:18.700 Because, like, how many times do you see people on Twitter going, hey, does anybody have a vendor contact for, like, this company? 40:18.700 --> 40:20.320 And, like, nobody responds. 40:20.500 --> 40:29.760 It's like, if we have to go on Twitter to ask about this, not only does it draw more attention to your, you know, the vulnerability, but it also makes you look bad. 40:29.760 --> 40:32.740 So definitely, you know, keep up with that stuff. 40:33.020 --> 40:36.760 Work with researchers, too, because people who are bringing stuff up to you, they want to help you. 40:36.760 --> 40:43.720 If somebody wants to just use your device or crypto mining or, you know, to DDoS, you know, some kid on Minecraft, they're not going to tell you about it. 40:43.720 --> 40:47.460 So if you have a researcher that's here and talking to you, they want to help you. 40:47.460 --> 40:50.120 You should definitely, you know, heed their advice. 40:50.440 --> 40:54.340 And have some open channel with your customers, too, to have the word out about vulnerabilities. 40:54.340 --> 41:02.960 Like, if you either you do an internal thing or you submit to CVEs and then you post them on Twitter, whatever you do, just have something so people can know, update their devices. 41:03.460 --> 41:06.900 And ultimately, all these are elements of a vulnerability disclosure program. 41:06.900 --> 41:14.660 So if you put this all together, you have, you know, the baseline for what you need to actually have one, which is awesome. 41:15.040 --> 41:22.700 And so I've done a little quick question out there on Twitter, which you can see, I have the link here. 41:22.820 --> 41:26.160 But these are some community suggestions for what vendors can do. 41:26.160 --> 41:33.280 You know, everything from automatic updates to, I like the make a security in named persons problem. 41:33.360 --> 41:40.180 So like having say, like, I don't know, like Sherry has to deal with the, you know, firmware bones that come in. 41:40.180 --> 41:42.780 So, you know, talk to her if there's an issue. 41:43.560 --> 41:53.220 You know, other things like minimizing attack surfaces, code signing, all of this stuff is going to be part of other, you know, best practices that you're going to have to implement. 41:53.220 --> 41:58.720 But these are all, you know, elements of things that might be good for you to consider moving forward. 41:59.100 --> 42:03.400 Default settings definitely should be sane, and with security in mind. 42:03.400 --> 42:05.800 And also, don't reinvent the wheel. 42:06.460 --> 42:09.500 So final thoughts here, I got two minutes left, just about. 42:09.500 --> 42:12.620 So we want to make it less easy for people to run botnets, right? 42:12.620 --> 42:15.240 Overall, the supply is already there. 42:15.240 --> 42:16.440 The demand is great. 42:16.440 --> 42:17.540 Everything is set up. 42:17.540 --> 42:22.920 People can off the shelf, you know, get several thousand bots on botnet in an afternoon. 42:23.500 --> 42:25.960 Botnets are definitely getting a lot smarter. 42:26.140 --> 42:29.340 People are using the messy landscape of this to take control of it. 42:29.340 --> 42:34.460 Like if you see with the Kaiji botnet, which is definitely a lot more advanced than previous stuff. 42:34.520 --> 42:37.940 People are going to be using this for more nefarious purposes. 42:37.960 --> 42:51.900 And because there's so much going on in this space, it's very hard to pick out who is either nation state trying to get access into your router, or just some random kid who has no idea what they're doing. 42:52.200 --> 42:54.760 And new devices and architectures are always being targeted. 42:54.760 --> 42:57.800 As I said, FPGAs, there's tons of new stuff. 42:57.800 --> 42:59.600 You can read that analysis that I wrote. 43:00.180 --> 43:07.660 If you don't act soon, like the new products that you put out are already going to be dead on arrival and exploitable once they come out. 43:08.440 --> 43:13.920 And so yeah, the Q&A for this is going to be done in the DEF CON Discord, as you might have seen in the Twitch chat. 43:13.920 --> 43:18.080 So if you have questions, or you can always hit me up on Twitter, at netspooky. 43:18.740 --> 43:20.560 My DMs are open there. 43:20.860 --> 43:22.760 I got some shout outs real quick. 43:22.800 --> 43:29.340 Shout out to the Safari Zone crew, which is the people that have always been there to look for weird stuff on the internet. 43:29.620 --> 43:31.580 Hopefully we'll have a zine coming out soon. 43:32.020 --> 43:34.960 Threatland, everybody who helped out with that project to collect sources. 43:35.040 --> 43:37.660 And of course, the entire community of Thug Crowd. 43:37.820 --> 43:47.640 A special shout out to Hermit for helping me go through so much of this and be such an awesome person for, you know, looking at logs and other weird stuff. 43:48.500 --> 43:54.460 I like being able to tell me about a lot of interesting things that she has found. 43:54.580 --> 44:00.640 Andrew Morris, thanks so much for letting me use your data set too before it was even publicly available. 44:01.120 --> 44:06.340 Thanks to Mudge for coming in hot with some hot takes for me when I was building this talk. 44:06.920 --> 44:14.640 Check out Ilya's IoT Village talk on emulating IoT devices and malware, because we actually did a lot of that when I was writing this talk. 44:14.780 --> 44:17.080 And then thanks also to Dave for that theme song. 44:17.600 --> 44:18.560 So I'll have slides out. 44:18.560 --> 44:19.180 I'll tweet them out. 44:19.180 --> 44:20.860 Just follow me on Twitter and you'll see. 44:21.220 --> 44:23.120 I have citations if you want to read them. 44:23.120 --> 44:24.700 But yeah, thanks, everybody.