All right. All right, all right, all right. What's up, y'all? All right, welcome to my
talk, IOT under the microscope here virtually. It's kind of disappointing. I'm
looking at a screen here instead of all you in the in the audience, but hopefully
we'll get through this and we'll have a good time doing it. I want to talk about
vulnerability trends in the supply chain. I've got some very interesting things, I
think, that we found in our in our data set size. So hopefully you guys will learn
something here and we'll have a little fun while doing it. All right. Okay. So who
am I? I'm Parker Wixel. I'm born and raised here in Columbus, Ohio. I've got 25
years industry experience on cybersecurity, on software development,
full stack development, last 10 years of which have really been focused on
cybersecurity research and product development. I was a contributor and
developer of open source security projects like AFL Unicorn. It's a fuzzing
framework for emulated binaries and then Patchwork, which is a static
compilation of Linux kernels or patching of Linux kernels for debugging
purposes. Like it was mentioned, I'm a senior engineer at Finite State. We're an
IoT cybersecurity firm dealing with a lot of the topics that we're talking
about here and all my data sets and stuff like that come from our Finite
State repos and some of the products that we're working on here. I'm also a
database lecturer over at The Ohio State University, looking to kick off yet
another fall there, and then I'm a composer and a musician. Don't hold that
against me. I realized early on that there's not a lot of money in music, so
here I am on just another passion of mine, computers. All right, so why is this
talk relevant to your interests? We're going to be talking about supply chain
trends, vulnerable and not vulnerable, vulnerability standards and reporting,
and then some firmware statistics and observations. Probably about the first
half of the talk is going to be delving into the background of what supply chains
are, some of the ways that we have to talk about vulnerabilities, what the
supply chain introduces as far as vulnerabilities and visibility into those,
and then probably the last half will be delving into the fun numbers, probably
why you came to see this talk, but nevertheless, hopefully we learned
something in both parts. So our data set that I'm pulling from right now, we do
have partnerships with some private industry partners, so we do not include
all of our private repos and stuff like that, but for this particular talk, I've
got about 7 million files, represents about 50,000 firmware images, 10,000
distinct product lines, and 150 different vendors. This is different
architectures, different operating systems, you know, a lot of them are Linux
based, some of them are RTOS, obviously, and we're hitting all the different
verticals that you usually hear about in these talks, medical devices, critical
infrastructure, security devices, home routers, Alexa, you know, whatever. We've
got a bunch of different types of products in our data set, so hopefully
the statistics that we talk about here on the second half, keep this in mind,
it's fun to be able to troll a data set of this size. So let's take a step back,
let's talk about the supply chain, let's talk about the problems that are
introduced as a part of the supply chain and what the problems are, maybe even go
into some of the solutions. So if you are a manufacturer, XYZ, of a security
camera, that security camera is running some sort of firmware on it. There's
hardware and there's firmware. The firmware is actually software that's
written for that hardware device. We like to think of it as firmware because it's
kind of baked in, it's usually not as fluid or as dynamic as software tends to
be in, say, a PC or whatever. But the camera still has a full processor memory
architecture, it's a full computer running in that thing. So if you can take
advantage of that or take over that product from a vulnerability standpoint,
you've accessed a whole computer's worth of resources. You have hardware
components that go in the thing, you have drivers that talk to those hardwares,
operating systems, libraries, apps, you name it, they're all just the same,
except they're called firmware. The problem is on a security camera like
this or any kind of IoT device is you're going to have multiple vendors. So if
you're a company XYZ, you're making this product, those hardware components
may come from various different vendors. And then there may be other vendors
like vendor A who talks to your underlying camera optics and can put some
image recognition on top of it or whatever else like that. Vendor B may be
some support libraries or whatever. The real problem comes in is not only do
you have to track vendor A, vendor B, vendor C, and what all they're putting
onto your device, which quite frankly is not always the case. There's not always
full disclosure there. But then vendor A and vendor B may rely on an open source
library somewhere, vendor X, that you don't even know about. They may or may
not even know that they're using it, depending on if their developers have
reported on it. And the thing that makes that even worse is that vendor X
library, say it's the same low-level image processing library, it may be
version 1.1 in vendor A and vendor B in their libraries that they're including
on your device has 1.5. Maybe vendor A's 1.1 version of vendor X has a
vulnerability in it and the 1.5 version doesn't. How do you know what all
component libraries you have in this simple camera? This is a full computer
running on your network, right? So Donald Rumsfeld, when he was Secretary of
Defense, brought this notion together that information can be divided into
three categories, he said at the time. Known knowns, known unknowns, unknown
unknowns, all right? And so we take that kind of approach towards IoT
vulnerabilities. Our known knowns are vulnerabilities that have explicitly been
discovered through scanning and testing on our devices. So we've tested it, we
know that there's a vulnerability, we patch it or whatever like that. So that's
our known knowns, all right? Our known unknowns are newly created software
versions or even just upgraded versions or whatever that we've pulled in
libraries or whatever that we don't have any kind of application testing behind
yet, right? So who knows what is going on under the hood there? So we know the
device, we just don't know if there's any vulnerabilities there. And then in
the last categorization of the three, unknown unknowns are vulnerabilities
that are in your camera or your device that you don't know and that nobody else
knows. And these are what we're calling zero days, all right? So these zero days
or not zero days because we haven't even discovered them, like say
Ripple 20 was just discovered a month or two ago. Before then, it was an unknown
unknown. It was in all these different devices, but nobody knew it was there. But
the weakness was still there waiting to be discovered, all right? So there's an
awful lot of work to be done in discovering zero days. As a security
researcher, we know that trying to protect yourself and get through all that
is really a challenge trying to find out what is vulnerable or not. So for this
talk, we're going to talk about known or unknown knowns. This is a fourth
dimension that we like to talk about, and it's comprising that which we
intentionally refuse to acknowledge that we know or we don't like to know, okay? So
that there are vulnerabilities that are known to exist but have not been
associated with all the systems that are actually affected. So we know all
these CVEs are against this open SSL library, but we don't know if that SSL
library is in our device, so we're just going to kind of ignore it for now, all
right? So this unknown knowns is where we can do an awful lot on our part as
manufacturers or security researchers to ferret out and to discover and to patch
before other actors out there find those same vulnerabilities and test them
against your same device, okay? So all of that can be done through a software
bill of materials. So a software bill of materials or an SBOM is there's a bill
of materials that usually comes in the manufacturing world that they're very
used to of all the components that make up certain devices. So if you're getting
big printing presses or whatever else like that, you know all the pieces that make
up that printing press so you can plan maintenance, you know what the cost is
going to be up front. So as a industry, software industry, IoT industry, we
should have the same thing, the software bill of materials, but we don't, all
right? So manufacturers don't know all their components, all the different
chips, system on chips that are running inside their devices, all that kind of
stuff. Vendors don't know all their suppliers and their suppliers' suppliers
because a lot of times vendors will make a product, but then they'll ship it off
to a manufacturer to actually make for them. And so how do you validate that,
you know, that things are exactly as you designed them or whatever? And then
consumers, those of us who put these devices in our critical infrastructure,
into our security systems, into our monitoring software, our monitoring
networks, even in our homes, we put these devices into our networks, but we don't
know all the software that's running on those devices. So the analogy is like, if
you found a laptop out in the parking lot of your company or whatever, would
you bring that laptop in, fire it up, boot it up, and plug it into your
critical infrastructure security system network just to poke around on the
laptop and see what's running on it? And I would hope that most of you or all of
you would say no, there's not a chance that you would ever do that, because we
know that those are full computer systems with operating systems that can
be compromised with viruses, malicious software, etc. But yet with that exact
same mentality, we don't apply to IoT devices. We'll take a camera that we do
not know all the supply, all the software that's running on it, and all the
weaknesses that might be inherent on those softwares, and we'll take that
camera and we'll plug that same camera into and talk to it on our critical
infrastructure network. All right? So we don't have any way to enumerate this.
So how do we generate one of these software build materials reliably? How do
we keep track of all those? Say it was even possible to keep track of all the
components that are in there, how do we validate one of our devices against one?
So we as manufacturers may even develop a device, and we might know exactly what
we want on it, but if we send it away and it comes back, who's to say that whoever
made that device for us put our firmware on there exactly as it was intended, and
they didn't slip something else in? The other thing would be, and this has
happened, is if you as a consumer have a device and you want to update it to the
latest update patch, say there's a security vulnerability and you want to
patch it, you go to the manufacturer or the vendor's site and you download the
update for that firmware, and you flash your device with that firmware, how do
you know that software wasn't compromised? We have seen in the industry
places where upload servers have been compromised by malicious actors, and
custom firmware has been placed in there as updates, and customers have
downloaded malicious updates to their devices, which might have been perfectly
fine in the first place, but now are running malicious software. So not only
generating a software bill of materials, but validating against a bill of
materials is critical and being able to do that. So let's shift away. One thing
I'd like to mention about company commitment and stuff like that is, I just
read Microsoft has a product, Azure Sphere, that they're developing that's a
secure IoT chip, and it's one way of approaching it. Hey, let's control the
ecosystem from the beginning and secure it down. And they've made commitments to
the software bill of materials. So it would be nice if more companies could be
able to control their environment, such as Microsoft has the luxury to do. And I
think these kind of commitments are going to be our way forward, is following
practices like this, generating our own software bill of materials, taking a
hard look at what's running on that. All right. So let's switch to today. We have
these devices, we don't know what's running on them. Let's look at the CVE,
CPE reporting mechanisms that we have for finding vulnerabilities in our
software. Okay. So CVE is common vulnerability exposures. It's a system
that provides reference methods for publicly known information security
vulnerabilities and exposures. Okay. So it's been around a while. MITRE is the
one who came up with that and helped maintain that. So these are all the
vulnerabilities that are discovered and reported for public knowledge. All right.
So that's great. We have a central place to report vulnerabilities. As well, we
have a national vulnerability database with the U.S. government that keeps track
of CPEs or common platform enumeration. So this is a structure naming schemes
for naming products. Okay. This is systems, softwares, software packages,
et cetera. So there's a common way to put all this together. The only problem
is, is that we have these frameworks for enumerating these kinds of
vulnerabilities or these products, but there's not a lot of adherence or,
sorry, hard regulations around how we use this. It's a very flexible system and
it works fairly well when treated well. But there's a lot of inconsistency
across the whole space about how we use CPEs, how we report them, how we link
them to products that they apply to, et cetera. So let's look at some of these.
All right. Are CPEs just complete products? Is it your whole camera system?
Is that whole device one product, a CPE? Are the component systems in there,
your optics and stuff like that? Are the system on chips that are running on there,
is that a CPE? Should that have its own product entry into that database?
Are there libraries within? You're using OpenSSL. That's great. But what about
libcrypto that lives within it? Could that be separate from OpenSSL? Could that be
its own product? So there's an awful lot of questions that we need to answer with
all that. So if we look at something like OpenSSL, it's a commercial-grade secure
sockets layer. It also has cryptographic libraries in it. If you go to the NVD,
the database, and look up OpenSSL, because you want to find out what product it
relates to, because maybe you found a vulnerability, OpenSSL brings back 405 results.
Now, every single version of that software is going to be another result. So 405 is not
necessarily all the different types of OpenSSL, but there are several. So here's a few examples
that come back with OpenSSL. So the very top one is usually what we think of as OpenSSL.
It's a C++ library that is compiled into a lot of Linux systems and stuff with OpenSSL
included. 0.9.7 was particularly vulnerable. There's a lot of different CVEs on there,
0.9.8, 1.0.1, et cetera. So there's a lot of different versions of that. Those star
fields, we'll just kind of gloss over for now, but those are further ways that you can
enumerate or specify what specific product this is if there's a lot of different versions.
Betas, alphas, different platforms, et cetera. But what are all these other CVEs or CPEs that
we see out there? We see this Calderon PyOpenSSL. So the first part is our vendor. The second
part is the actual product. So we have this PyOpenSSL. I guess we can make a guess that
that would be a Python binding for OpenSSL or a Python library. Then we have Lua OpenSSL
on the next line. So we guess maybe that's Lua script. But then look at the next two.
We have Node OpenSSL. And if you go all the way down to the Node.js, that's the target
software that it's targeting, not the language, but the target software, which is it's written
for Node.js, right? So it's a JavaScript module written for Node. But the very next line,
OpenSSL.js project OpenSSL.js. So you have Node OpenSSL and then you have OpenSSL.js
and you have Node.js at the end. So now you have someone who's writing for Node that has
their own way of specifying. But you have two different competing libraries. Which one
is which? And then you go down to the last one, OpenSSL project OpenSSL. Well, now that
looks an awful lot like the first one, OpenSSL. So which one is that? Well, the hard part
is we really don't know. And if you dig into the metadata about this and you actually go
to the web page, the title doesn't help you very much. OpenSSL project OpenSSL. So we
go into the references and in the change log, we see a GitHub reference there. And in there
we have a Rust OpenSSL. And if you go in there and you look at these different things, you
have these keywords of Rust. Ah, that's for the Rust language. Okay? So finding qualified
prod platforms, sorry, CPEs, can be a real challenge. The other thing that's a real challenge
here is we don't know between CPEs what relates to each other. Does this Rust language depend
on OpenSSL of a certain version, the C, C++ version of the first line, to be a certain
version, right? Does 0.9.2 map to 0.9.7 or 0.9.7A or 1.01? We don't have these kinds
of interrelations. So not only is generating an SBOM difficult, but developing a body of
ground truth around an SBOM is extremely difficult. And it's not a problem that you think of because
you think it should be obvious, but who owns that? Is it the company's responsibility to
add their platform to the CPE database so that if vulnerabilities are found against it, that
there's reliable data there? Or is it the CVEs who find the vulnerabilities? Is it their job
to correctly find and identify the platforms if they're not there? So that's just a little
background here on the CPE system. They're good systems to start off with, but we need more on
top of that. We need some ground truth. We need some ways to relate this. By the way, there are
some projects that are starting to put these together, but it's hard. Do you scrape the apt
get repos for what components are in OpenSSL or HTTP servers? Certain HTTP servers rely on OpenSSL.
Which versions go with which? Well, if you're installing it, you know because you look at the
readme for the HTTP installer, right? And it'll tell you which versions you have, but we don't have
any way of systematically obtaining any of that information. Okay, so let's go to a specific
example of a vulnerability and let's look at the supply chain here. We'll go with an old example.
This is from four years ago. I'm not going to try to butcher Robert's last name,
but Darkonius, he released a write-up on a router backdoor that he had discovered
originating from a version of OpenWrt, an open source router operating system from at that point
was 10 years ago, was 10 years in the past. So there was this backdoor hash that's down in the
code below, 1D, 680, whatever. If you entered in that on the command line, you immediately got a
root shell and you can see the relevant code there. The problem is, is this was just in an
OpenWrt version that was 10 years old from, I guess 2006, right? But the write-up that he had
was this was not on an OpenWrt router, this was on a commercial router that had included
OpenWrt and its libraries and were using components such as this component script called
logon.sh as a part of their operating system. All right? So Darkonius found the backdoor in
one or two sets of devices. Him and, and if you read through the comments, they, different people
were chiming in, oh, I just tested it against this and I found it on this and this. So we found,
you know, in the comments, three, four, maybe five models of devices listed. But how is Darkonius
supposed to know and go out and find, is it his responsibility to go find every single device that
this relates to? Is it even his obligation to go to the manufacturer of this device and tell him,
I found this in there. If you go and you look for this CVE, you don't find this CVE in the database.
If you go even further to the CPE, the platform for this device that he looked for, you don't
even find CPEs that are related to this specific device. So, and this is not a small device. This
is a, this is a, this is a device that made its rounds and you can, you can find it different
places. So whose responsibility is it? In our dataset, when we've looked at all of our firmware
just sitting around, we just did a string search for that magic hash. And lo and behold, we found
3,810 files that had this hash in it. And it wasn't just the hash, it was the full login script.
But notice the file name, CLI2, factoryboot.sh, login.sh. So the original login.sh was a part of
the original package, but we found the actual login.sh encapsulated in a binary called a
command line interface, a CLI, or even two that was used by web servers and stuff on IoT devices
that still ran the same code from 2006 that nobody, I'm sure nobody knows is in there. The
manufacturer doesn't even know it's there anymore. And the thing is, is that three major vendor
companies have this. 44 different product models and 281 versions of this firmware. So this open
wrt login.sh, this hash, has made its way into many, many, many different vendors. This isn't
just one person accidentally downloading it and putting it in. This is the supply chain of people
taking from A, who takes from X, who takes from Y, and it multiplies. So this is the heart of the
problem. This is why we're in the bind that we're in, is the supply chain. We'll do one more example
before we get into the more juicy stuff, the statistics that we found. This latest CVE, CPE
that we're talking about is Ripple 20. I'm sure you guys have heard of it. 19 zero-day vulnerabilities
amplified by the supply chain. Its title says it all. And this was two months ago that this was
or so was reported. And according to their white paper on it, this affects hundreds of millions of
devices or more, includes multiple remote code execution vulnerabilities. So that's the worst
kind of vulnerability that you'd want to have. Many other major international vendors suspected
of being vulnerable in medical, transportation, industrial control systems, etc. So my previous
example kind of laid the groundwork for how something like Ripple 20 exists. All right.
So let's just take one of the CVEs as an example that they reported. All right. So this was published
June, two months ago, June 17th, 2020. The description of it, the tracked TCP IP stack
before this version allows remote code execution related to IPv4 tunneling. Okay. So this is a
vulnerability. They said it's bad. There's a base score of 10 critical on here because the remote
code execution and how easy it is to theoretically get access to this TCP IP stack. If we look at
the CPE that is related to all CVEs have all the CPEs that are related to them, we find these four
TREC versions are all rolled up underneath this TREC stack CPE. The problem is, is when we look
for this TREC stack and we go back to or we click on the actual record in the CPE database, we find
a quick info is that the CPE for this TREC stack was created almost a month after the vulnerability
was discovered. All right? So there was no product for TREC TCP. We had to invent, not we, but the
vulnerability researchers had to invent the CPE that even belonged to the TREC stack, right?
And then we go back to our original problem. If you have this TREC library and you're trying
to figure out where all it's where all it's at in the world, you've got to kind of do some
detective work. And they did some detective work because we don't have any CPE to CPE
relationships. And even if we did, this CPE didn't even exist. Nobody even knew that TREC TCP
exists. So I kind of imagine reading through the paper about this, I almost imagine
Luis from Ant-Man 2, when he's being interviewed, where is Scott, you know? And I kind of imagine
him talking about, you know, hey, Luis, where is TREC? And he goes, hey, man, see, that's
complicated. See, TREC, there's some smart guys, but they need some help, right? So they go to
Elmec Systems, and they're like, hey, we need some developers. And Elmec Systems is like, yeah,
we think you've got it going on, so let's get together. And so they started working together
but then something happened. They're like, nah, homie, we're split. We're out of here. So TREC
goes off on their own way, and Elmec Systems go off on their own way. And TREC be like,
we got the TREC TCP IP stack. We're going to market all this in the United States. And Elmec
Systems are like, no, homie, we're no longer Elmec Systems. We're Zook and Elmec. And we got the
CASAGO TCP IP. I know it sounds like yours, but it's better because it's been renamed. And you
think you're hot? Dang, we got all of Asia that we're going through. Meanwhile, you got
security researchers in the middle of the night going, hey, can you just tell me where TREC is?
And TREC is like, hmm, see, it's complicated. And they go to Zook and Elmec. And see, they're
just like, where's this CASAGO at? And we can't even figure it out because the legal system is
cutting us off. Whereas TREC is back here going, let's see, I got an uncle's girlfriend, boyfriend
who had a relative named HP. And it's kind of like the Rona virus, right? Like, where's the
contact on this? Tracing. And their uncle's second cousin, once removed, is named Aruba.
And I think we were at a party together. And meanwhile, Zook and Elmec is like,
and JSOF is just like, where is this? And Zook and Elmec and TREC and all these people can't
find out. And they're like, that's what I'm trying to say. It's complicated.
Right? I'm sorry. I probably butchered that. But I can imagine industrial control systems
and industrial plants that have CYP13 investigators or people coming in. I can see their first slide
to the CYP13 auditors are going to be this picture of Luis and two words, it's complicated.
Right? Because it is. Trying to find out where this TREC TCP stack was, according to this paper,
pretty nuts. Okay? So, anyways, it was developed somewhere in the 90s to 2000. It was included in
a large quantity of firmware for devices. There's been various patches to it. Some of the Ripple
CVEs were patched in versions as far back as 2009. So, you're a vendor. What products of yours
contain this TCP IP stack? How do you know? How do you update a device when a patch is present or
even know what version of the TREC stack you used to have in your device? How serious is the threat
to your device? Right? Maybe it's not that problematic. Maybe it was just included but
never used. So, reproducing CVEs are problematic. The actual attack vectors are really hard to
classify in these individual products. You potentially waste time from vendors on trivial
vulnerabilities that really aren't practical. So, really, the bottom line for vendors are,
how much money do you spend trying to find out whether you have to develop firmware patch for
software that you don't even know is in your device? Or how likely is the software that it's
even going to be exploitable? Right? So, it's a hard, hard question. So, I'll come back to the
software building materials here in a little bit. But just for right now, let's make a little shift
over into our database. And let's look at some of the vulnerability statistics that we have here.
Hopefully, you guys find this enlightening. I certainly did. This is just kind of represents
a slice of the industry. We have medical devices. We have information control systems
gear. We have all sorts of different sectors represented and verticals in our data sector.
So, here we go. All right. So, Etsy Resolve popular name servers, 127.001. Not surprising
is one of the top ones. We got some Google addresses there. 88.844. And then the next three
down there that you see 168 are all Asian DNSs. And then you've got some strange noise in there.
192.168. Someone.com. A couple others. So, you know, if you have all these devices and they're
going to the U.S. and they have Asian domain servers, you're going to have a performance hit
or vice versa. How do you know where these products are going and which name servers are
hitting? I saw some forum posts about 192.168.0.7. Maybe the people who made Etsy Resolve had it on
test and they were just copying and pasting from a forum into this file. Someone.com,
that actually was one vendor, but it's on a whole slew of firmware for all sorts of
different products in their product line. So you can see not only is that illustrative that
someone has just put that in as an example and they found that example somewhere,
but it's in all their Etsy results and they just copy and paste that file into all their other
firmware, right? So where's the checks and balances on what's actually running in there?
Even stuff like corp.ubnt.com. Corp.ubnt.com doesn't exist, but it's running on a device as
a resolve. So if someone decides to turn up that host name and then start serving requests on
there, you could have some interesting results, right? Here's another one. TTY count in Etsy
secure TTY. So if you look at the bottom, Etsy secure TTY file allows you to specify which TTY
devices a root user is allowed to log in on. So let's just make one thing clear. Once you have an
IoT device in production, you probably shouldn't be logging into that device at all. All right?
So most normal operation shouldn't allow anybody from the outside to ever be tunneling into that
device remotely. Certainly not a root user, okay? So I want to highlight this 196, 148. So what that
means is that these counts are the number of shells, or sorry, the number of TTYs listed as
approved TTYs that root can log into. So 1,486 firmware have file has an Etsy secure TTY in it
with 196 approved TTY locations for root to log in on. All right, so I tracked this one down because
this file is just like, where is this anomaly from? I found a Yocto project, which is a way to
programmatically or to easily generate a Linux operating system custom build for your project.
And this particular file that I found was from 2013. So here is a supply chain, someone who's
building and wants a secure TTY, they pull down this file that's pretty permissive, probably for
development purposes, and not really supposed to be for production. And yet here it makes its way
into all these devices. So here's the supply chain in action. All right, number of valid login shells.
Okay, so this is an Etsy shells. This is the number of acceptable different types of
shells that you can log in from, and most have one. So again, whether you should have a shell that
people can log into or not, that's questionable, but at least a lot of them, majority of them, have
only one. But then we have this 10 login shells. And I guess what they want by this, a little tongue
in cheek, is, you know, if their hackers are coming in and are used to have all their key bindings in
their environment files for bash, they want to make it easier for them. So they're just going to throw
in bash in there. And if you're ash or zsh, we got you covered. You can log in however you want.
Again, a little tongue in cheek, but that's what that number represents. An additional thing here
that was kind of concerning is when I got delved into the actual types of devices that this was,
it was over 60 of these were models of patient monitoring health systems. Okay, so over 60
different models, which represents however many thousands or hundreds of thousands of
health systems, patient monitors that are sitting bedside, have 10 or so login shells that somebody
could log in with, right? These are just configuration examples in the supply chain. I'm
not saying these are actual vulnerabilities, they're just interesting anomalies from the supply chain.
Okay, so here's a questionable config. We're about out of time. So we're just going to go
through these quick. Firmware where the root user has a login shell, 15,345. All right, so 30%
of all of our firmware that we looked at has some sort of login shell. Whereas if you don't want the
root to log in, use sbin login, guess how many? Over under one. We found one firmware that actually
had sbin no longer in there. Okay, firmware with keys and authorized keys. Now this is a bad back
because this basically says if someone SSHs into your box, if your key matches their key
in the authorized keys, you get to log in without any password. So this is a classic backdoor.
Whether it was purposefully put there or not, what's the over under? 175, 175 firmware. Remember
this is firmware, however many devices this represents. This is 12 different vendors with
these keys. All right. And over 29, these were really interesting known hosts to see where these
kind of came from and the different medical facilities and different places that had known
hosts coming from them. Clam AV, only four firmware had any kind of open antivirus software.
Firmware running HTTPD with mod auto index. So this auto index is a way to auto index files
on your web browser or your web server when there isn't any kind of match for the HTML name. So it's
a great way to kind of troll through your directories. So these are, again, things that are
questionable that are in our configs. Firmware starting TTY and init tabs. So quite a few
firmware are starting TTY, but not that many distinct files. So here we're seeing amplification
in the supply chain of here's 63 files, but these 63 files are found in 4,729 devices.
Here's a particularly bad one. Firmware with PHP to default to display errors. 332 firmware. So
using SQL injection or any kind of command injection, I love when developers leave on
their display errors, because when I made a mistake, it tells me exactly what's there.
Problem is, PHP by default allows display errors. You have to explicitly turn it off.
So maybe we should be making software that doesn't have defaults that are vulnerable like this in
their default configuration. All right. Next part. Firmware with insecure default DES encryption. So
if you don't specify the type of encryption for your passwords, you're going to use DES,
which is a bit weaker. 318 firmware, 29 files. But even worse, there were firmware that specified
the flag to turn on MD5 as their password tripped. It was 12 firmware. Fortunately,
it's small, but there's still 12 firmware. I have no idea where those are at,
but they're out there. Firmware with a master.password with no password on root. So in
the password file, there is no over, under, on 100 devices, 10 devices, a million devices. Now
only seven. But still, one file made its way in with no password on the root. MySQL binding to
000. So listen to the world and allow people to go right to your MySQL. 26 devices. Firmware
with Redis, the same way. Default world bind. So if you don't specify a bind by default, it will
allow you in. Three. It's only one file. But still, three firmware allow you into their Redis by
default. Average number of unsafe function calls per firmware. So this one is any unsafe function
calls is stuff like stir copy, mem copy, all those unbounded ones like ripple 20 found and stuff
like malicious ways to use standard allocation functions without checking links and stuff like
that. How many do you think in per firmware? 1,500 and some average unsafe function calls
per firmware. Okay? Number of firmware with unsafe function calls? 23,000. That's over 50%. And those
are the ones that we've named. We haven't even trolled through those with function hashing and
stuff to find the unlabeled ones that also call these. All right? Firmware. Here's the last one.
Firmware exporting NFS mounts to the world. All right? Over, under on this one? 42. There are
four files that contribute to 42 firmware mounts to the world, such as slash root, mount a star,
so you can as root write to root as user ID zero. So these are all questionable configs that have
made it their way, just fun things that I found trolling through. And finally, what would a talk
like this be without talking about the common passwords and all that? So you see here amplified
here, again, is the supply chain where you have the file counts of all of this, sort of the light
orange, and the dark orange is how many firmware use those files in them. And the number one being
admin. It's only in five files in the password file. But those five password files are found in
470 firmware, which may represent hundreds of thousands of devices where we've cracked the
password admin. I'm going to give you ten to one guess what the password is going to be for that
admin user. All right. So anyways, in conclusion, it's all about the software bill of materials.
Drive towards generating and validating these. This is the path forward. Manufacturing,
implementing software inventory systems, developers being diligent and reporting all
the components used in their systems. Consumers having a high demand or having a high standard
in transparency and favor companies and buying from companies with that info as well. Policymakers
who help guide these unified standards and reporting and formats. And our security community
here developing tools to assist in the automation of the inventory. If we're doing the same thing now
in five years, we're going behind. We should be using machine learning. We should be using tools
and other things to help us inventory and validate these things, not just on the front end, but on the
back end to validate patches and updates are what they say they are and they contain what they think
they contain. So thanks to all my team at finite state and a lot of good research that's been in
the industry. Finally, obviously the question and answer on this session are going to the discord,
but I've had a fun time talking to you all. Peace out. Hope you enjoy the rest of DEF CON
and IoT Village. Thank you so much for having me.