WEBVTT 00:00.000 --> 00:01.440 All right. 00:04.500 --> 00:11.100 All right, all right, all right. 00:11.100 --> 00:12.440 What's up, y'all? 00:13.140 --> 00:18.640 All right, welcome to my talk, IOT under the microscope here virtually. 00:18.640 --> 00:20.220 It's kind of disappointing. 00:20.380 --> 00:28.620 I'm looking at a screen here instead of all you in the in the audience, but hopefully we'll get through this and we'll have a good time doing it. 00:29.180 --> 00:33.140 I want to talk about vulnerability trends in the supply chain. 00:33.380 --> 00:39.500 I've got some very interesting things, I think, that we found in our in our data set size. 00:39.500 --> 00:43.440 So hopefully you guys will learn something here and we'll have a little fun while doing it. 00:43.440 --> 00:44.180 All right. 00:44.540 --> 00:45.240 Okay. 00:45.760 --> 00:47.140 So who am I? 00:47.140 --> 00:48.540 I'm Parker Wixel. 00:48.860 --> 00:51.360 I'm born and raised here in Columbus, Ohio. 00:51.360 --> 01:06.460 I've got 25 years industry experience on cybersecurity, on software development, full stack development, last 10 years of which have really been focused on cybersecurity research and product development. 01:06.460 --> 01:12.540 I was a contributor and developer of open source security projects like AFL Unicorn. 01:12.540 --> 01:25.000 It's a fuzzing framework for emulated binaries and then Patchwork, which is a static compilation of Linux kernels or patching of Linux kernels for debugging purposes. 01:26.040 --> 01:29.200 Like it was mentioned, I'm a senior engineer at Finite State. 01:29.200 --> 01:42.140 We're an IoT cybersecurity firm dealing with a lot of the topics that we're talking about here and all my data sets and stuff like that come from our Finite State repos and some of the products that we're working on here. 01:42.140 --> 01:51.860 I'm also a database lecturer over at The Ohio State University, looking to kick off yet another fall there, and then I'm a composer and a musician. 01:52.080 --> 01:54.300 Don't hold that against me. 01:54.300 --> 02:04.940 I realized early on that there's not a lot of money in music, so here I am on just another passion of mine, computers. 02:04.980 --> 02:07.840 All right, so why is this talk relevant to your interests? 02:07.840 --> 02:17.580 We're going to be talking about supply chain trends, vulnerable and not vulnerable, vulnerability standards and reporting, and then some firmware statistics and observations. 02:18.140 --> 02:35.520 Probably about the first half of the talk is going to be delving into the background of what supply chains are, some of the ways that we have to talk about vulnerabilities, what the supply chain introduces as far as vulnerabilities and visibility into those, 02:35.520 --> 02:47.340 and then probably the last half will be delving into the fun numbers, probably why you came to see this talk, but nevertheless, hopefully we learned something in both parts. 02:47.480 --> 03:06.480 So our data set that I'm pulling from right now, we do have partnerships with some private industry partners, so we do not include all of our private repos and stuff like that, but for this particular talk, I've got about 7 million files, represents about 50,000 03:06.480 --> 03:14.960 firmware images, 10,000 distinct product lines, and 150 different vendors. 03:14.960 --> 03:33.080 This is different architectures, different operating systems, you know, a lot of them are Linux based, some of them are RTOS, obviously, and we're hitting all the different verticals that you usually hear about in these talks, medical devices, critical infrastructure, 03:33.080 --> 03:38.340 security devices, home routers, Alexa, you know, whatever. 03:38.340 --> 03:51.500 We've got a bunch of different types of products in our data set, so hopefully the statistics that we talk about here on the second half, keep this in mind, it's fun to be able to troll a data set of this size. 03:52.140 --> 04:04.560 So let's take a step back, let's talk about the supply chain, let's talk about the problems that are introduced as a part of the supply chain and what the problems are, maybe even go into some of the solutions. 04:05.400 --> 04:13.580 So if you are a manufacturer, XYZ, of a security camera, that security camera is running some sort of firmware on it. 04:15.040 --> 04:16.560 There's hardware and there's firmware. 04:16.560 --> 04:19.740 The firmware is actually software that's written for that hardware device. 04:19.740 --> 04:27.760 We like to think of it as firmware because it's kind of baked in, it's usually not as fluid or as dynamic as software tends to be in, say, a PC or whatever. 04:27.760 --> 04:35.060 But the camera still has a full processor memory architecture, it's a full computer running in that thing. 04:35.060 --> 04:43.660 So if you can take advantage of that or take over that product from a vulnerability standpoint, you've accessed a whole computer's worth of resources. 04:43.660 --> 04:54.840 You have hardware components that go in the thing, you have drivers that talk to those hardwares, operating systems, libraries, apps, you name it, they're all just the same, except they're called firmware. 04:54.840 --> 05:01.200 The problem is on a security camera like this or any kind of IoT device is you're going to have multiple vendors. 05:01.200 --> 05:09.500 So if you're a company XYZ, you're making this product, those hardware components may come from various different vendors. 05:09.500 --> 05:19.900 And then there may be other vendors like vendor A who talks to your underlying camera optics and can put some image recognition on top of it or whatever else like that. 05:19.900 --> 05:23.840 Vendor B may be some support libraries or whatever. 05:24.180 --> 05:33.820 The real problem comes in is not only do you have to track vendor A, vendor B, vendor C, and what all they're putting onto your device, which quite frankly is not always the case. 05:33.820 --> 05:35.980 There's not always full disclosure there. 05:35.980 --> 05:44.320 But then vendor A and vendor B may rely on an open source library somewhere, vendor X, that you don't even know about. 05:44.320 --> 05:50.280 They may or may not even know that they're using it, depending on if their developers have reported on it. 05:50.560 --> 06:07.940 And the thing that makes that even worse is that vendor X library, say it's the same low-level image processing library, it may be version 1.1 in vendor A and vendor B in their libraries that they're including on your device has 1.5. 06:07.940 --> 06:15.160 Maybe vendor A's 1.1 version of vendor X has a vulnerability in it and the 1.5 version doesn't. 06:15.160 --> 06:19.480 How do you know what all component libraries you have in this simple camera? 06:19.480 --> 06:23.940 This is a full computer running on your network, right? 06:25.300 --> 06:36.160 So Donald Rumsfeld, when he was Secretary of Defense, brought this notion together that information can be divided into three categories, he said at the time. 06:36.160 --> 06:40.500 Known knowns, known unknowns, unknown unknowns, all right? 06:40.500 --> 06:45.220 And so we take that kind of approach towards IoT vulnerabilities. 06:46.180 --> 06:52.800 Our known knowns are vulnerabilities that have explicitly been discovered through scanning and testing on our devices. 06:52.800 --> 06:57.460 So we've tested it, we know that there's a vulnerability, we patch it or whatever like that. 06:57.460 --> 06:59.640 So that's our known knowns, all right? 06:59.640 --> 07:13.500 Our known unknowns are newly created software versions or even just upgraded versions or whatever that we've pulled in libraries or whatever that we don't have any kind of application testing behind yet, right? 07:13.500 --> 07:17.800 So who knows what is going on under the hood there? 07:17.800 --> 07:21.660 So we know the device, we just don't know if there's any vulnerabilities there. 07:22.460 --> 07:36.440 And then in the last categorization of the three, unknown unknowns are vulnerabilities that are in your camera or your device that you don't know and that nobody else knows. 07:36.440 --> 07:38.760 And these are what we're calling zero days, all right? 07:38.760 --> 07:50.380 So these zero days or not zero days because we haven't even discovered them, like say Ripple 20 was just discovered a month or two ago. 07:50.380 --> 07:53.600 Before then, it was an unknown unknown. 07:53.600 --> 07:58.520 It was in all these different devices, but nobody knew it was there. 07:58.520 --> 08:01.860 But the weakness was still there waiting to be discovered, all right? 08:02.180 --> 08:08.100 So there's an awful lot of work to be done in discovering zero days. 08:08.100 --> 08:18.980 As a security researcher, we know that trying to protect yourself and get through all that is really a challenge trying to find out what is vulnerable or not. 08:19.020 --> 08:24.540 So for this talk, we're going to talk about known or unknown knowns. 08:24.540 --> 08:34.700 This is a fourth dimension that we like to talk about, and it's comprising that which we intentionally refuse to acknowledge that we know or we don't like to know, okay? 08:34.700 --> 08:42.740 So that there are vulnerabilities that are known to exist but have not been associated with all the systems that are actually affected. 08:42.740 --> 08:53.420 So we know all these CVEs are against this open SSL library, but we don't know if that SSL library is in our device, so we're just going to kind of ignore it for now, all right? 08:54.140 --> 09:14.700 So this unknown knowns is where we can do an awful lot on our part as manufacturers or security researchers to ferret out and to discover and to patch before other actors out there find those same vulnerabilities and test them against your same device, 09:14.700 --> 09:15.440 okay? 09:15.940 --> 09:21.220 So all of that can be done through a software bill of materials. 09:21.220 --> 09:32.820 So a software bill of materials or an SBOM is there's a bill of materials that usually comes in the manufacturing world that they're very used to of all the components that make up certain devices. 09:32.820 --> 09:43.480 So if you're getting big printing presses or whatever else like that, you know all the pieces that make up that printing press so you can plan maintenance, you know what the cost is going to be up front. 09:43.580 --> 09:53.180 So as a industry, software industry, IoT industry, we should have the same thing, the software bill of materials, but we don't, all right? 09:53.180 --> 10:01.700 So manufacturers don't know all their components, all the different chips, system on chips that are running inside their devices, all that kind of stuff. 10:01.700 --> 10:12.940 Vendors don't know all their suppliers and their suppliers' suppliers because a lot of times vendors will make a product, but then they'll ship it off to a manufacturer to actually make for them. 10:13.220 --> 10:19.140 And so how do you validate that, you know, that things are exactly as you designed them or whatever? 10:19.140 --> 10:38.820 And then consumers, those of us who put these devices in our critical infrastructure, into our security systems, into our monitoring software, our monitoring networks, even in our homes, we put these devices into our networks, but we don't know all the software that's running on those devices. 10:38.820 --> 11:02.140 So the analogy is like, if you found a laptop out in the parking lot of your company or whatever, would you bring that laptop in, fire it up, boot it up, and plug it into your critical infrastructure security system network just to poke around on the laptop and see what's running on it? 11:02.280 --> 11:19.040 And I would hope that most of you or all of you would say no, there's not a chance that you would ever do that, because we know that those are full computer systems with operating systems that can be compromised with viruses, malicious software, etc. 11:19.040 --> 11:24.280 But yet with that exact same mentality, we don't apply to IoT devices. 11:24.280 --> 11:40.460 We'll take a camera that we do not know all the supply, all the software that's running on it, and all the weaknesses that might be inherent on those softwares, and we'll take that camera and we'll plug that same camera into and talk to it on our critical infrastructure network. 11:40.460 --> 11:41.220 All right? 11:41.220 --> 11:44.680 So we don't have any way to enumerate this. 11:44.680 --> 11:49.240 So how do we generate one of these software build materials reliably? 11:49.240 --> 11:51.740 How do we keep track of all those? 11:52.000 --> 12:00.080 Say it was even possible to keep track of all the components that are in there, how do we validate one of our devices against one? 12:00.080 --> 12:15.900 So we as manufacturers may even develop a device, and we might know exactly what we want on it, but if we send it away and it comes back, who's to say that whoever made that device for us put our firmware on there exactly as it was intended, and they didn't slip something else in? 12:16.300 --> 12:32.500 The other thing would be, and this has happened, is if you as a consumer have a device and you want to update it to the latest update patch, say there's a security vulnerability and you want to patch it, you go to the manufacturer or the vendor's site and you download the update for that firmware, 12:32.500 --> 12:37.980 and you flash your device with that firmware, how do you know that software wasn't compromised? 12:37.980 --> 12:54.880 We have seen in the industry places where upload servers have been compromised by malicious actors, and custom firmware has been placed in there as updates, and customers have downloaded malicious updates to their devices, which might have been perfectly fine in the first place, 12:54.880 --> 12:57.060 but now are running malicious software. 12:57.060 --> 13:06.040 So not only generating a software bill of materials, but validating against a bill of materials is critical and being able to do that. 13:06.400 --> 13:08.800 So let's shift away. 13:09.660 --> 13:21.760 One thing I'd like to mention about company commitment and stuff like that is, I just read Microsoft has a product, Azure Sphere, that they're developing that's a secure IoT chip, and it's one way of approaching it. 13:21.760 --> 13:27.180 Hey, let's control the ecosystem from the beginning and secure it down. 13:27.180 --> 13:30.080 And they've made commitments to the software bill of materials. 13:30.080 --> 13:37.840 So it would be nice if more companies could be able to control their environment, such as Microsoft has the luxury to do. 13:38.060 --> 13:48.880 And I think these kind of commitments are going to be our way forward, is following practices like this, generating our own software bill of materials, taking a hard look at what's running on that. 13:48.880 --> 13:49.620 All right. 13:49.940 --> 13:52.420 So let's switch to today. 13:52.420 --> 13:55.220 We have these devices, we don't know what's running on them. 13:55.220 --> 14:02.020 Let's look at the CVE, CPE reporting mechanisms that we have for finding vulnerabilities in our software. 14:02.020 --> 14:02.540 Okay. 14:02.540 --> 14:05.460 So CVE is common vulnerability exposures. 14:06.000 --> 14:13.760 It's a system that provides reference methods for publicly known information security vulnerabilities and exposures. 14:13.760 --> 14:13.980 Okay. 14:13.980 --> 14:15.960 So it's been around a while. 14:16.000 --> 14:20.440 MITRE is the one who came up with that and helped maintain that. 14:20.440 --> 14:25.120 So these are all the vulnerabilities that are discovered and reported for public knowledge. 14:25.120 --> 14:25.640 All right. 14:25.640 --> 14:26.180 So that's great. 14:26.180 --> 14:28.380 We have a central place to report vulnerabilities. 14:28.400 --> 14:32.480 As well, we have a national vulnerability database with the U.S. 14:32.480 --> 14:36.740 government that keeps track of CPEs or common platform enumeration. 14:36.740 --> 14:40.600 So this is a structure naming schemes for naming products. 14:40.600 --> 14:40.840 Okay. 14:40.840 --> 14:44.800 This is systems, softwares, software packages, et cetera. 14:45.680 --> 14:49.120 So there's a common way to put all this together. 14:49.120 --> 15:04.960 The only problem is, is that we have these frameworks for enumerating these kinds of vulnerabilities or these products, but there's not a lot of adherence or, sorry, hard regulations around how we use this. 15:04.960 --> 15:09.760 It's a very flexible system and it works fairly well when treated well. 15:09.760 --> 15:24.140 But there's a lot of inconsistency across the whole space about how we use CPEs, how we report them, how we link them to products that they apply to, et cetera. 15:24.140 --> 15:26.480 So let's look at some of these. 15:26.480 --> 15:27.180 All right. 15:27.780 --> 15:30.440 Are CPEs just complete products? 15:30.440 --> 15:32.000 Is it your whole camera system? 15:32.000 --> 15:35.020 Is that whole device one product, a CPE? 15:35.020 --> 15:38.540 Are the component systems in there, your optics and stuff like that? 15:38.540 --> 15:42.140 Are the system on chips that are running on there, is that a CPE? 15:42.140 --> 15:45.440 Should that have its own product entry into that database? 15:45.840 --> 15:47.280 Are there libraries within? 15:47.280 --> 15:49.100 You're using OpenSSL. 15:49.120 --> 15:50.160 That's great. 15:50.160 --> 15:53.000 But what about libcrypto that lives within it? 15:53.020 --> 15:55.640 Could that be separate from OpenSSL? 15:55.640 --> 15:57.260 Could that be its own product? 15:58.360 --> 16:03.260 So there's an awful lot of questions that we need to answer with all that. 16:03.260 --> 16:10.960 So if we look at something like OpenSSL, it's a commercial-grade secure sockets layer. 16:11.320 --> 16:13.820 It also has cryptographic libraries in it. 16:13.820 --> 16:26.760 If you go to the NVD, the database, and look up OpenSSL, because you want to find out what product it relates to, because maybe you found a vulnerability, OpenSSL brings back 405 results. 16:26.940 --> 16:31.440 Now, every single version of that software is going to be another result. 16:31.440 --> 16:37.080 So 405 is not necessarily all the different types of OpenSSL, but there are several. 16:37.080 --> 16:40.940 So here's a few examples that come back with OpenSSL. 16:41.080 --> 16:45.200 So the very top one is usually what we think of as OpenSSL. 16:45.200 --> 16:53.660 It's a C++ library that is compiled into a lot of Linux systems and stuff with OpenSSL included. 16:54.340 --> 16:57.280 0.9.7 was particularly vulnerable. 16:57.280 --> 17:02.560 There's a lot of different CVEs on there, 0.9.8, 1.0.1, et cetera. 17:02.560 --> 17:04.600 So there's a lot of different versions of that. 17:04.600 --> 17:15.160 Those star fields, we'll just kind of gloss over for now, but those are further ways that you can enumerate or specify what specific product this is if there's a lot of different versions. 17:15.700 --> 17:18.560 Betas, alphas, different platforms, et cetera. 17:18.620 --> 17:21.880 But what are all these other CVEs or CPEs that we see out there? 17:21.880 --> 17:25.620 We see this Calderon PyOpenSSL. 17:25.620 --> 17:27.920 So the first part is our vendor. 17:27.920 --> 17:30.120 The second part is the actual product. 17:30.300 --> 17:32.360 So we have this PyOpenSSL. 17:32.360 --> 17:38.420 I guess we can make a guess that that would be a Python binding for OpenSSL or a Python library. 17:38.780 --> 17:42.920 Then we have Lua OpenSSL on the next line. 17:42.940 --> 17:45.000 So we guess maybe that's Lua script. 17:45.200 --> 17:46.980 But then look at the next two. 17:46.980 --> 17:50.080 We have Node OpenSSL. 17:50.340 --> 18:01.380 And if you go all the way down to the Node.js, that's the target software that it's targeting, not the language, but the target software, which is it's written for Node.js, right? 18:01.520 --> 18:04.060 So it's a JavaScript module written for Node. 18:04.480 --> 18:10.200 But the very next line, OpenSSL.js project OpenSSL.js. 18:10.460 --> 18:16.160 So you have Node OpenSSL and then you have OpenSSL.js and you have Node.js at the end. 18:16.200 --> 18:22.100 So now you have someone who's writing for Node that has their own way of specifying. 18:22.100 --> 18:23.680 But you have two different competing libraries. 18:23.680 --> 18:25.160 Which one is which? 18:25.160 --> 18:28.920 And then you go down to the last one, OpenSSL project OpenSSL. 18:28.920 --> 18:33.320 Well, now that looks an awful lot like the first one, OpenSSL. 18:33.520 --> 18:35.180 So which one is that? 18:35.180 --> 18:38.440 Well, the hard part is we really don't know. 18:38.620 --> 18:45.460 And if you dig into the metadata about this and you actually go to the web page, the title doesn't help you very much. 18:45.640 --> 18:47.980 OpenSSL project OpenSSL. 18:49.360 --> 18:54.800 So we go into the references and in the change log, we see a GitHub reference there. 18:54.800 --> 18:57.040 And in there we have a Rust OpenSSL. 18:57.040 --> 19:01.000 And if you go in there and you look at these different things, you have these keywords of Rust. 19:01.160 --> 19:03.300 Ah, that's for the Rust language. 19:03.300 --> 19:03.900 Okay? 19:03.940 --> 19:12.140 So finding qualified prod platforms, sorry, CPEs, can be a real challenge. 19:12.140 --> 19:19.560 The other thing that's a real challenge here is we don't know between CPEs what relates to each other. 19:19.580 --> 19:30.300 Does this Rust language depend on OpenSSL of a certain version, the C, C++ version of the first line, to be a certain version, right? 19:30.300 --> 19:36.980 Does 0.9.2 map to 0.9.7 or 0.9.7A or 1.01? 19:36.980 --> 19:40.160 We don't have these kinds of interrelations. 19:40.160 --> 19:50.600 So not only is generating an SBOM difficult, but developing a body of ground truth around an SBOM is extremely difficult. 19:50.600 --> 19:57.000 And it's not a problem that you think of because you think it should be obvious, but who owns that? 19:57.140 --> 20:08.360 Is it the company's responsibility to add their platform to the CPE database so that if vulnerabilities are found against it, that there's reliable data there? 20:08.360 --> 20:11.660 Or is it the CVEs who find the vulnerabilities? 20:11.660 --> 20:16.980 Is it their job to correctly find and identify the platforms if they're not there? 20:17.780 --> 20:22.340 So that's just a little background here on the CPE system. 20:22.580 --> 20:26.200 They're good systems to start off with, but we need more on top of that. 20:26.200 --> 20:27.740 We need some ground truth. 20:27.740 --> 20:29.780 We need some ways to relate this. 20:29.780 --> 20:35.280 By the way, there are some projects that are starting to put these together, but it's hard. 20:35.280 --> 20:44.000 Do you scrape the apt get repos for what components are in OpenSSL or HTTP servers? 20:44.000 --> 20:47.380 Certain HTTP servers rely on OpenSSL. 20:47.380 --> 20:49.040 Which versions go with which? 20:49.040 --> 20:55.600 Well, if you're installing it, you know because you look at the readme for the HTTP installer, right? 20:55.600 --> 21:02.740 And it'll tell you which versions you have, but we don't have any way of systematically obtaining any of that information. 21:03.140 --> 21:09.580 Okay, so let's go to a specific example of a vulnerability and let's look at the supply chain here. 21:09.580 --> 21:11.600 We'll go with an old example. 21:11.600 --> 21:14.260 This is from four years ago. 21:14.600 --> 21:33.280 I'm not going to try to butcher Robert's last name, but Darkonius, he released a write-up on a router backdoor that he had discovered originating from a version of OpenWrt, an open source router operating system from at that point was 10 years ago, was 10 years in the past. 21:33.280 --> 21:41.880 So there was this backdoor hash that's down in the code below, 1D, 680, whatever. 21:41.880 --> 21:49.220 If you entered in that on the command line, you immediately got a root shell and you can see the relevant code there. 21:49.680 --> 21:58.620 The problem is, is this was just in an OpenWrt version that was 10 years old from, I guess 2006, right? 21:58.620 --> 22:20.720 But the write-up that he had was this was not on an OpenWrt router, this was on a commercial router that had included OpenWrt and its libraries and were using components such as this component script called logon.sh as a part of their operating system. 22:20.760 --> 22:21.320 All right? 22:22.020 --> 22:25.600 So Darkonius found the backdoor in one or two sets of devices. 22:25.600 --> 22:33.160 Him and, and if you read through the comments, they, different people were chiming in, oh, I just tested it against this and I found it on this and this. 22:33.160 --> 22:39.220 So we found, you know, in the comments, three, four, maybe five models of devices listed. 22:40.340 --> 22:48.680 But how is Darkonius supposed to know and go out and find, is it his responsibility to go find every single device that this relates to? 22:48.680 --> 22:58.500 Is it even his obligation to go to the manufacturer of this device and tell him, I found this in there. 22:58.520 --> 23:07.700 If you go and you look for this CVE, you don't find this CVE in the database. 23:07.760 --> 23:18.400 If you go even further to the CPE, the platform for this device that he looked for, you don't even find CPEs that are related to this specific device. 23:18.680 --> 23:20.400 So, and this is not a small device. 23:20.400 --> 23:26.020 This is a, this is a, this is a device that made its rounds and you can, you can find it different places. 23:26.800 --> 23:29.120 So whose responsibility is it? 23:30.640 --> 23:39.560 In our dataset, when we've looked at all of our firmware just sitting around, we just did a string search for that magic hash. 23:39.860 --> 23:47.100 And lo and behold, we found 3,810 files that had this hash in it. 23:47.100 --> 23:50.260 And it wasn't just the hash, it was the full login script. 23:50.260 --> 23:55.120 But notice the file name, CLI2, factoryboot.sh, login.sh. 23:55.120 --> 24:15.880 So the original login.sh was a part of the original package, but we found the actual login.sh encapsulated in a binary called a command line interface, a CLI, or even two that was used by web servers and stuff on IoT devices that still ran the same code from 2006 that nobody, 24:15.880 --> 24:18.640 I'm sure nobody knows is in there. 24:18.640 --> 24:21.500 The manufacturer doesn't even know it's there anymore. 24:22.700 --> 24:26.700 And the thing is, is that three major vendor companies have this. 24:26.800 --> 24:31.180 44 different product models and 281 versions of this firmware. 24:31.180 --> 24:38.260 So this open wrt login.sh, this hash, has made its way into many, many, many different vendors. 24:38.260 --> 24:41.980 This isn't just one person accidentally downloading it and putting it in. 24:41.980 --> 24:49.200 This is the supply chain of people taking from A, who takes from X, who takes from Y, and it multiplies. 24:49.200 --> 24:51.040 So this is the heart of the problem. 24:51.040 --> 24:56.080 This is why we're in the bind that we're in, is the supply chain. 24:56.520 --> 25:01.460 We'll do one more example before we get into the more juicy stuff, the statistics that we found. 25:02.600 --> 25:07.280 This latest CVE, CPE that we're talking about is Ripple 20. 25:07.280 --> 25:09.080 I'm sure you guys have heard of it. 25:09.080 --> 25:12.180 19 zero-day vulnerabilities amplified by the supply chain. 25:12.180 --> 25:13.980 Its title says it all. 25:13.980 --> 25:19.220 And this was two months ago that this was or so was reported. 25:19.220 --> 25:28.380 And according to their white paper on it, this affects hundreds of millions of devices or more, includes multiple remote code execution vulnerabilities. 25:28.380 --> 25:31.860 So that's the worst kind of vulnerability that you'd want to have. 25:31.940 --> 25:39.680 Many other major international vendors suspected of being vulnerable in medical, transportation, industrial control systems, etc. 25:39.680 --> 25:48.620 So my previous example kind of laid the groundwork for how something like Ripple 20 exists. 25:49.000 --> 25:49.540 All right. 25:49.540 --> 25:54.260 So let's just take one of the CVEs as an example that they reported. 25:54.260 --> 25:54.740 All right. 25:54.740 --> 26:00.940 So this was published June, two months ago, June 17th, 2020. 26:01.320 --> 26:10.020 The description of it, the tracked TCP IP stack before this version allows remote code execution related to IPv4 tunneling. 26:10.020 --> 26:10.400 Okay. 26:10.400 --> 26:12.540 So this is a vulnerability . 26:12.540 --> 26:13.380 They said it's bad. 26:13.380 --> 26:24.340 There's a base score of 10 critical on here because the remote code execution and how easy it is to theoretically get access to this TCP IP stack. 26:24.420 --> 26:39.960 If we look at the CPE that is related to all CVEs have all the CPEs that are related to them, we find these four TREC versions are all rolled up underneath this TREC stack CPE. 26:40.040 --> 26:59.460 The problem is, is when we look for this TREC stack and we go back to or we click on the actual record in the CPE database, we find a quick info is that the CPE for this TREC stack was created almost a month after the vulnerability was discovered. 26:59.960 --> 27:00.240 All right? 27:00.240 --> 27:03.620 So there was no product for TREC TCP. 27:03.620 --> 27:12.380 We had to invent, not we, but the vulnerability researchers had to invent the CPE that even belonged to the TREC stack, right? 27:13.240 --> 27:16.500 And then we go back to our original problem. 27:17.160 --> 27:27.800 If you have this TREC library and you're trying to figure out where all it's where all it's at in the world, you've got to kind of do some detective work. 27:27.800 --> 27:32.180 And they did some detective work because we don't have any CPE to CPE relationships. 27:32.180 --> 27:34.720 And even if we did, this CPE didn't even exist. 27:34.720 --> 27:37.520 Nobody even knew that TREC TCP exists. 27:37.600 --> 27:52.420 So I kind of imagine reading through the paper about this, I almost imagine Luis from Ant-Man 2, when he's being interviewed, where is Scott, you know? 27:52.480 --> 27:56.640 And I kind of imagine him talking about, you know, hey, Luis, where is TREC? 27:56.640 --> 27:59.520 And he goes, hey, man, see, that's complicated. 27:59.540 --> 28:02.320 See, TREC, there's some smart guys, but they need some help, right? 28:02.320 --> 28:05.840 So they go to Elmec Systems, and they're like, hey, we need some developers. 28:05.840 --> 28:09.500 And Elmec Systems is like, yeah, we think you've got it going on, so let's get together. 28:09.500 --> 28:12.420 And so they started working together but then something happened. 28:12.420 --> 28:14.060 They're like, nah, homie, we're split. 28:14.060 --> 28:14.700 We're out of here. 28:14.700 --> 28:18.500 So TREC goes off on their own way, and Elmec Systems go off on their own way. 28:18.500 --> 28:20.920 And TREC be like, we got the TREC TCP IP stack. 28:20.920 --> 28:23.620 We're going to market all this in the United States. 28:23.620 --> 28:27.180 And Elmec Systems are like, no, homie, we're no longer Elmec Systems. 28:27.180 --> 28:28.240 We're Zook and Elmec. 28:28.240 --> 28:30.600 And we got the CASAGO TCP IP. 28:30.600 --> 28:33.440 I know it sounds like yours, but it's better because it's been renamed. 28:33.480 --> 28:37.300 And you think you're hot? 28:37.680 --> 28:41.920 Dang, we got all of Asia that we're going through. 28:41.920 --> 28:48.100 Meanwhile, you got security researchers in the middle of the night going, hey, can you just tell me where TREC is? 28:48.100 --> 28:50.600 And TREC is like, hmm, see, it's complicated. 28:50.600 --> 28:51.940 And they go to Zook and Elmec. 28:51.940 --> 28:58.700 And see, they're just like, where's this CASAGO at? 28:58.700 --> 29:01.900 And we can't even figure it out because the legal system is cutting us off. 29:01.900 --> 29:08.960 Whereas TREC is back here going, let's see, I got an uncle's girlfriend, boyfriend who had a relative named HP. 29:09.100 --> 29:12.540 And it's kind of like the Rona virus, right? 29:12.680 --> 29:14.500 Like, where's the contact on this? 29:15.200 --> 29:16.100 Tracing. 29:16.100 --> 29:20.680 And their uncle's second cousin, once removed, is named Aruba. 29:20.680 --> 29:22.480 And I think we were at a party together. 29:22.480 --> 29:27.460 And meanwhile, Zook and Elmec is like, and JSOF is just like, where is this? 29:27.460 --> 29:30.640 And Zook and Elmec and TREC and all these people can't find out. 29:30.640 --> 29:32.740 And they're like, that's what I'm trying to say. 29:32.740 --> 29:33.880 It's complicated. 29:36.600 --> 29:37.300 Right? 29:37.300 --> 29:37.840 I'm sorry. 29:37.840 --> 29:39.100 I probably butchered that. 29:39.100 --> 29:53.060 But I can imagine industrial control systems and industrial plants that have CYP13 investigators or people coming in. 29:53.060 --> 30:01.020 I can see their first slide to the CYP13 auditors are going to be this picture of Luis and two words, it's complicated. 30:01.300 --> 30:01.540 Right? 30:01.540 --> 30:02.380 Because it is. 30:02.380 --> 30:08.740 Trying to find out where this TREC TCP stack was, according to this paper, pretty nuts. 30:08.740 --> 30:09.340 Okay? 30:09.420 --> 30:13.700 So, anyways, it was developed somewhere in the 90s to 2000. 30:13.960 --> 30:17.400 It was included in a large quantity of firmware for devices. 30:17.400 --> 30:19.220 There's been various patches to it. 30:19.220 --> 30:25.240 Some of the Ripple CVEs were patched in versions as far back as 2009. 30:25.540 --> 30:27.460 So, you're a vendor. 30:27.940 --> 30:30.880 What products of yours contain this TCP IP stack? 30:30.880 --> 30:31.940 How do you know? 30:31.940 --> 30:39.220 How do you update a device when a patch is present or even know what version of the TREC stack you used to have in your device? 30:39.220 --> 30:41.980 How serious is the threat to your device? 30:42.000 --> 30:42.620 Right? 30:42.680 --> 30:45.960 Maybe it's not that problematic. 30:45.960 --> 30:48.260 Maybe it was just included but never used. 30:48.260 --> 30:50.580 So, reproducing CVEs are problematic. 30:50.580 --> 30:55.540 The actual attack vectors are really hard to classify in these individual products. 30:55.540 --> 31:01.380 You potentially waste time from vendors on trivial vulnerabilities that really aren't practical. 31:02.020 --> 31:11.200 So, really, the bottom line for vendors are, how much money do you spend trying to find out whether you have to develop firmware patch for software that you don't even know is in your device? 31:11.200 --> 31:14.900 Or how likely is the software that it's even going to be exploitable? 31:14.900 --> 31:15.400 Right? 31:15.460 --> 31:18.100 So, it's a hard, hard question. 31:18.420 --> 31:23.240 So, I'll come back to the software building materials here in a little bit. 31:23.240 --> 31:28.580 But just for right now, let's make a little shift over into our database. 31:28.620 --> 31:32.500 And let's look at some of the vulnerability statistics that we have here. 31:32.940 --> 31:35.320 Hopefully, you guys find this enlightening. 31:35.320 --> 31:36.280 I certainly did. 31:36.280 --> 31:41.120 This is just kind of represents a slice of the industry. 31:41.120 --> 31:42.340 We have medical devices. 31:42.340 --> 31:46.880 We have information control systems gear. 31:46.880 --> 31:53.060 We have all sorts of different sectors represented and verticals in our data sector. 31:53.060 --> 31:54.280 So, here we go. 31:54.320 --> 31:54.680 All right. 31:54.680 --> 31:59.440 So, Etsy Resolve popular name servers, 127.001. 31:59.440 --> 32:01.620 Not surprising is one of the top ones. 32:01.620 --> 32:03.640 We got some Google addresses there. 32:03.640 --> 32:07.220 88.844. 32:07.220 --> 32:12.640 And then the next three down there that you see 168 are all Asian DNSs. 32:12.780 --> 32:16.420 And then you've got some strange noise in there. 32:16.480 --> 32:17.920 192.168. 32:17.920 --> 32:19.560 Someone.com. 32:19.560 --> 32:20.600 A couple others. 32:21.260 --> 32:26.740 So, you know, if you have all these devices and they're going to the U.S. 32:26.740 --> 32:32.900 and they have Asian domain servers, you're going to have a performance hit or vice versa. 32:32.900 --> 32:36.560 How do you know where these products are going and which name servers are hitting? 32:36.560 --> 32:40.560 I saw some forum posts about 192.168.0.7. 32:40.560 --> 32:49.400 Maybe the people who made Etsy Resolve had it on test and they were just copying and pasting from a forum into this file. 32:49.400 --> 32:58.200 Someone.com, that actually was one vendor, but it's on a whole slew of firmware for all sorts of different products in their product line. 32:58.200 --> 33:11.100 So you can see not only is that illustrative that someone has just put that in as an example and they found that example somewhere, but it's in all their Etsy results and they just copy and paste that file into all their other firmware, right? 33:11.100 --> 33:15.260 So where's the checks and balances on what's actually running in there? 33:15.880 --> 33:19.660 Even stuff like corp.ubnt.com. 33:19.660 --> 33:25.340 Corp.ubnt.com doesn't exist, but it's running on a device as a resolve. 33:25.420 --> 33:32.860 So if someone decides to turn up that host name and then start serving requests on there, you could have some interesting results, right? 33:32.880 --> 33:34.060 Here's another one. 33:34.060 --> 33:37.100 TTY count in Etsy secure TTY. 33:37.100 --> 33:46.760 So if you look at the bottom, Etsy secure TTY file allows you to specify which TTY devices a root user is allowed to log in on. 33:46.760 --> 33:49.340 So let's just make one thing clear. 33:49.340 --> 33:56.880 Once you have an IoT device in production, you probably shouldn't be logging into that device at all. 33:56.880 --> 33:57.240 All right? 33:57.240 --> 34:05.820 So most normal operation shouldn't allow anybody from the outside to ever be tunneling into that device remotely. 34:06.420 --> 34:08.860 Certainly not a root user, okay? 34:08.860 --> 34:13.580 So I want to highlight this 196, 148. 34:13.580 --> 34:25.080 So what that means is that these counts are the number of shells, or sorry, the number of TTYs listed as approved TTYs that root can log into. 34:25.080 --> 34:40.220 So 1,486 firmware have file has an Etsy secure TTY in it with 196 approved TTY locations for root to log in on. 34:40.680 --> 34:45.080 All right, so I tracked this one down because this file is just like, where is this anomaly from? 34:45.080 --> 34:53.980 I found a Yocto project, which is a way to programmatically or to easily generate a Linux operating system custom build for your project. 34:54.240 --> 34:58.080 And this particular file that I found was from 2013. 34:58.080 --> 35:09.580 So here is a supply chain, someone who's building and wants a secure TTY, they pull down this file that's pretty permissive, probably for development purposes, and not really supposed to be for production. 35:09.720 --> 35:13.140 And yet here it makes its way into all these devices. 35:13.220 --> 35:15.500 So here's the supply chain in action. 35:15.500 --> 35:17.700 All right, number of valid login shells. 35:17.700 --> 35:20.300 Okay, so this is an Etsy shells. 35:20.300 --> 35:30.040 This is the number of acceptable different types of shells that you can log in from, and most have one. 35:30.040 --> 35:38.380 So again, whether you should have a shell that people can log into or not, that's questionable, but at least a lot of them, majority of them, have only one. 35:38.380 --> 35:41.880 But then we have this 10 login shells. 35:41.880 --> 35:53.620 And I guess what they want by this, a little tongue in cheek, is, you know, if their hackers are coming in and are used to have all their key bindings in their environment files for bash, they want to make it easier for them. 35:53.620 --> 35:55.340 So they're just going to throw in bash in there. 35:55.340 --> 35:58.700 And if you're ash or zsh, we got you covered. 35:58.700 --> 36:00.740 You can log in however you want. 36:00.960 --> 36:05.060 Again, a little tongue in cheek, but that's what that number represents. 36:05.580 --> 36:20.880 An additional thing here that was kind of concerning is when I got delved into the actual types of devices that this was, it was over 60 of these were models of patient monitoring health systems. 36:21.040 --> 36:37.840 Okay, so over 60 different models, which represents however many thousands or hundreds of thousands of health systems, patient monitors that are sitting bedside, have 10 or so login shells that somebody could log in with, right? 36:37.840 --> 36:40.640 These are just configuration examples in the supply chain. 36:40.640 --> 36:46.580 I'm not saying these are actual vulnerabilities, they're just interesting anomalies from the supply chain. 36:46.580 --> 36:48.820 Okay, so here's a questionable config. 36:49.260 --> 36:50.560 We're about out of time. 36:50.560 --> 36:52.060 So we're just going to go through these quick. 36:52.360 --> 36:59.060 Firmware where the root user has a login shell, 15,345. 36:59.120 --> 37:04.420 All right, so 30% of all of our firmware that we looked at has some sort of login shell. 37:04.420 --> 37:08.840 Whereas if you don't want the root to log in, use sbin login, guess how many? 37:08.840 --> 37:11.140 Over under one. 37:11.140 --> 37:14.140 We found one firmware that actually had sbin no longer in there. 37:14.140 --> 37:16.200 Okay, firmware with keys and authorized keys. 37:16.200 --> 37:26.720 Now this is a bad back because this basically says if someone SSHs into your box, if your key matches their key in the authorized keys, you get to log in without any password. 37:26.720 --> 37:28.440 So this is a classic backdoor. 37:28.440 --> 37:32.560 Whether it was purposefully put there or not, what's the over under? 37:32.980 --> 37:35.040 175, 175 firmware. 37:35.040 --> 37:38.840 Remember this is firmware, however many devices this represents. 37:39.240 --> 37:43.040 This is 12 different vendors with these keys. 37:43.040 --> 37:43.760 All right. 37:43.760 --> 37:54.180 And over 29, these were really interesting known hosts to see where these kind of came from and the different medical facilities and different places that had known hosts coming from them. 37:54.180 --> 38:01.340 Clam AV, only four firmware had any kind of open antivirus software. 38:01.340 --> 38:04.860 Firmware running HTTPD with mod auto index. 38:05.000 --> 38:17.800 So this auto index is a way to auto index files on your web browser or your web server when there isn't any kind of match for the HTML name. 38:17.800 --> 38:21.220 So it's a great way to kind of troll through your directories. 38:21.220 --> 38:27.120 So these are, again, things that are questionable that are in our configs. 38:27.120 --> 38:30.100 Firmware starting TTY and init tabs. 38:30.100 --> 38:36.180 So quite a few firmware are starting TTY, but not that many distinct files. 38:36.180 --> 38:45.920 So here we're seeing amplification in the supply chain of here's 63 files, but these 63 files are found in 4,729 devices. 38:46.140 --> 38:47.680 Here's a particularly bad one. 38:47.680 --> 38:50.520 Firmware with PHP to default to display errors. 38:51.120 --> 38:52.600 332 firmware. 38:52.600 --> 39:04.720 So using SQL injection or any kind of command injection, I love when developers leave on their display errors, because when I made a mistake, it tells me exactly what's there. 39:04.720 --> 39:08.400 Problem is, PHP by default allows display errors. 39:08.400 --> 39:10.320 You have to explicitly turn it off. 39:10.320 --> 39:17.120 So maybe we should be making software that doesn't have defaults that are vulnerable like this in their default configuration. 39:17.640 --> 39:18.240 All right. 39:18.240 --> 39:19.180 Next part. 39:19.180 --> 39:22.160 Firmware with insecure default DES encryption. 39:22.160 --> 39:26.960 So if you don't specify the type of encryption for your passwords, you're going to use DES, which is a bit weaker. 39:27.260 --> 39:29.660 318 firmware, 29 files. 39:29.660 --> 39:36.600 But even worse, there were firmware that specified the flag to turn on MD5 as their password tripped. 39:36.600 --> 39:37.600 It was 12 firmware. 39:37.600 --> 39:40.100 Fortunately, it's small, but there's still 12 firmware. 39:40.100 --> 39:43.220 I have no idea where those are at, but they're out there. 39:43.220 --> 39:46.420 Firmware with a master.password with no password on root. 39:46.420 --> 39:53.760 So in the password file, there is no over, under, on 100 devices, 10 devices, a million devices. 39:53.760 --> 39:54.840 Now only seven. 39:54.840 --> 39:58.960 But still, one file made its way in with no password on the root. 39:59.380 --> 40:01.360 MySQL binding to 000. 40:01.600 --> 40:06.240 So listen to the world and allow people to go right to your MySQL. 40:06.260 --> 40:07.700 26 devices. 40:08.100 --> 40:09.480 Firmware with Redis, the same way. 40:09.480 --> 40:10.520 Default world bind. 40:10.520 --> 40:14.060 So if you don't specify a bind by default, it will allow you in. 40:14.060 --> 40:14.660 Three. 40:14.660 --> 40:15.540 It's only one file. 40:15.540 --> 40:19.100 But still, three firmware allow you into their Redis by default. 40:20.200 --> 40:22.420 Average number of unsafe function calls per firmware. 40:22.420 --> 40:39.200 So this one is any unsafe function calls is stuff like stir copy, mem copy, all those unbounded ones like ripple 20 found and stuff like malicious ways to use standard allocation functions without checking links and stuff like that. 40:39.200 --> 40:41.840 How many do you think in per firmware? 40:42.320 --> 40:48.300 1,500 and some average unsafe function calls per firmware. 40:48.340 --> 40:48.820 Okay? 40:48.820 --> 40:51.520 Number of firmware with unsafe function calls? 40:51.520 --> 40:52.600 23,000. 40:52.600 --> 40:53.880 That's over 50%. 40:53.880 --> 40:55.660 And those are the ones that we've named. 40:55.660 --> 41:04.220 We haven't even trolled through those with function hashing and stuff to find the unlabeled ones that also call these. 41:04.220 --> 41:04.720 All right? 41:04.780 --> 41:05.340 Firmware. 41:05.340 --> 41:06.500 Here's the last one. 41:06.500 --> 41:09.120 Firmware exporting NFS mounts to the world. 41:09.120 --> 41:09.860 All right? 41:09.940 --> 41:11.400 Over, under on this one? 41:11.720 --> 41:12.460 42. 41:12.460 --> 41:25.760 There are four files that contribute to 42 firmware mounts to the world, such as slash root, mount a star, so you can as root write to root as user ID zero. 41:25.980 --> 41:30.580 So these are all questionable configs that have made it their way, just fun things that I found trolling through. 41:30.580 --> 41:38.100 And finally, what would a talk like this be without talking about the common passwords and all that? 41:38.280 --> 41:51.320 So you see here amplified here, again, is the supply chain where you have the file counts of all of this, sort of the light orange, and the dark orange is how many firmware use those files in them. 41:51.600 --> 41:54.060 And the number one being admin. 41:54.060 --> 41:56.460 It's only in five files in the password file. 41:56.460 --> 42:07.220 But those five password files are found in 470 firmware, which may represent hundreds of thousands of devices where we've cracked the password admin. 42:07.220 --> 42:13.100 I'm going to give you ten to one guess what the password is going to be for that admin user. 42:13.420 --> 42:14.000 All right. 42:14.000 --> 42:18.340 So anyways, in conclusion, it's all about the software bill of materials. 42:18.340 --> 42:21.080 Drive towards generating and validating these. 42:21.080 --> 42:22.340 This is the path forward. 42:22.340 --> 42:30.980 Manufacturing, implementing software inventory systems, developers being diligent and reporting all the components used in their systems. 42:30.980 --> 42:40.280 Consumers having a high demand or having a high standard in transparency and favor companies and buying from companies with that info as well. 42:40.480 --> 42:46.200 Policymakers who help guide these unified standards and reporting and formats. 42:46.580 --> 42:52.280 And our security community here developing tools to assist in the automation of the inventory. 42:52.480 --> 42:55.980 If we're doing the same thing now in five years, we're going behind. 42:55.980 --> 42:57.540 We should be using machine learning. 42:57.540 --> 43:11.000 We should be using tools and other things to help us inventory and validate these things, not just on the front end, but on the back end to validate patches and updates are what they say they are and they contain what they think they contain. 43:11.180 --> 43:17.020 So thanks to all my team at finite state and a lot of good research that's been in the industry. 43:17.980 --> 43:27.160 Finally, obviously the question and answer on this session are going to the discord, but I've had a fun time talking to you all. 43:27.480 --> 43:28.680 Peace out. 43:28.680 --> 43:32.440 Hope you enjoy the rest of DEF CON and IoT Village. 43:32.440 --> 43:33.740 Thank you so much for having me.