start	end	text
0	14740	All right. All right, all right, all right. What's up, y'all? All right, welcome to my
14740	20800	talk, IOT under the microscope here virtually. It's kind of disappointing. I'm
20800	25720	looking at a screen here instead of all you in the in the audience, but hopefully
25720	30620	we'll get through this and we'll have a good time doing it. I want to talk about
30620	36240	vulnerability trends in the supply chain. I've got some very interesting things, I
36240	41040	think, that we found in our in our data set size. So hopefully you guys will learn
41040	46480	something here and we'll have a little fun while doing it. All right. Okay. So who
46480	52300	am I? I'm Parker Wixel. I'm born and raised here in Columbus, Ohio. I've got 25
52300	58040	years industry experience on cybersecurity, on software development,
58040	63240	full stack development, last 10 years of which have really been focused on
63240	68240	cybersecurity research and product development. I was a contributor and
68240	73120	developer of open source security projects like AFL Unicorn. It's a fuzzing
73120	79680	framework for emulated binaries and then Patchwork, which is a static
79680	83860	compilation of Linux kernels or patching of Linux kernels for debugging
83860	89680	purposes. Like it was mentioned, I'm a senior engineer at Finite State. We're an
89680	95040	IoT cybersecurity firm dealing with a lot of the topics that we're talking
95040	98400	about here and all my data sets and stuff like that come from our Finite
98400	102960	State repos and some of the products that we're working on here. I'm also a
102960	107980	database lecturer over at The Ohio State University, looking to kick off yet
107980	113160	another fall there, and then I'm a composer and a musician. Don't hold that
113160	121140	against me. I realized early on that there's not a lot of money in music, so
121140	126160	here I am on just another passion of mine, computers. All right, so why is this
126160	129560	talk relevant to your interests? We're going to be talking about supply chain
129560	134600	trends, vulnerable and not vulnerable, vulnerability standards and reporting,
134600	139500	and then some firmware statistics and observations. Probably about the first
139500	145000	half of the talk is going to be delving into the background of what supply chains
145000	150060	are, some of the ways that we have to talk about vulnerabilities, what the
150060	155520	supply chain introduces as far as vulnerabilities and visibility into those,
155520	160500	and then probably the last half will be delving into the fun numbers, probably
160500	164640	why you came to see this talk, but nevertheless, hopefully we learned
164640	172120	something in both parts. So our data set that I'm pulling from right now, we do
172120	177960	have partnerships with some private industry partners, so we do not include
177960	182540	all of our private repos and stuff like that, but for this particular talk, I've
182540	189520	got about 7 million files, represents about 50,000 firmware images, 10,000
189520	196380	distinct product lines, and 150 different vendors. This is different
196380	200480	architectures, different operating systems, you know, a lot of them are Linux
200480	206660	based, some of them are RTOS, obviously, and we're hitting all the different
206660	211980	verticals that you usually hear about in these talks, medical devices, critical
211980	218840	infrastructure, security devices, home routers, Alexa, you know, whatever. We've
218840	222480	got a bunch of different types of products in our data set, so hopefully
222480	227900	the statistics that we talk about here on the second half, keep this in mind,
227900	234300	it's fun to be able to troll a data set of this size. So let's take a step back,
234300	238280	let's talk about the supply chain, let's talk about the problems that are
238280	242600	introduced as a part of the supply chain and what the problems are, maybe even go
242600	249560	into some of the solutions. So if you are a manufacturer, XYZ, of a security
249560	255520	camera, that security camera is running some sort of firmware on it. There's
255520	258260	hardware and there's firmware. The firmware is actually software that's
258260	261460	written for that hardware device. We like to think of it as firmware because it's
261460	265320	kind of baked in, it's usually not as fluid or as dynamic as software tends to
265320	272580	be in, say, a PC or whatever. But the camera still has a full processor memory
272580	275760	architecture, it's a full computer running in that thing. So if you can take
275760	280200	advantage of that or take over that product from a vulnerability standpoint,
280200	284260	you've accessed a whole computer's worth of resources. You have hardware
284260	287760	components that go in the thing, you have drivers that talk to those hardwares,
287760	292680	operating systems, libraries, apps, you name it, they're all just the same,
292680	297500	except they're called firmware. The problem is on a security camera like
297500	301460	this or any kind of IoT device is you're going to have multiple vendors. So if
301460	306260	you're a company XYZ, you're making this product, those hardware components
306260	311120	may come from various different vendors. And then there may be other vendors
311120	316960	like vendor A who talks to your underlying camera optics and can put some
316960	320700	image recognition on top of it or whatever else like that. Vendor B may be
320700	326400	some support libraries or whatever. The real problem comes in is not only do
326400	329440	you have to track vendor A, vendor B, vendor C, and what all they're putting
329440	334380	onto your device, which quite frankly is not always the case. There's not always
334380	340340	full disclosure there. But then vendor A and vendor B may rely on an open source
340340	345240	library somewhere, vendor X, that you don't even know about. They may or may
345240	348900	not even know that they're using it, depending on if their developers have
348900	353900	reported on it. And the thing that makes that even worse is that vendor X
353900	360160	library, say it's the same low-level image processing library, it may be
360160	366020	version 1.1 in vendor A and vendor B in their libraries that they're including
366020	372380	on your device has 1.5. Maybe vendor A's 1.1 version of vendor X has a
372380	377040	vulnerability in it and the 1.5 version doesn't. How do you know what all
377040	380980	component libraries you have in this simple camera? This is a full computer
381540	389880	running on your network, right? So Donald Rumsfeld, when he was Secretary of
389880	394060	Defense, brought this notion together that information can be divided into
394060	398960	three categories, he said at the time. Known knowns, known unknowns, unknown
398960	403800	unknowns, all right? And so we take that kind of approach towards IoT
403800	410060	vulnerabilities. Our known knowns are vulnerabilities that have explicitly been
410060	414180	discovered through scanning and testing on our devices. So we've tested it, we
414180	417780	know that there's a vulnerability, we patch it or whatever like that. So that's
417780	423300	our known knowns, all right? Our known unknowns are newly created software
423300	428360	versions or even just upgraded versions or whatever that we've pulled in
428360	432620	libraries or whatever that we don't have any kind of application testing behind
432620	438760	yet, right? So who knows what is going on under the hood there? So we know the
438760	443080	device, we just don't know if there's any vulnerabilities there. And then in
443080	449500	the last categorization of the three, unknown unknowns are vulnerabilities
449500	456040	that are in your camera or your device that you don't know and that nobody else
456040	460140	knows. And these are what we're calling zero days, all right? So these zero days
460140	466580	or not zero days because we haven't even discovered them, like say
466580	472980	Ripple 20 was just discovered a month or two ago. Before then, it was an unknown
472980	478600	unknown. It was in all these different devices, but nobody knew it was there. But
478600	484000	the weakness was still there waiting to be discovered, all right? So there's an
484000	488600	awful lot of work to be done in discovering zero days. As a security
488600	491940	researcher, we know that trying to protect yourself and get through all that
491940	500420	is really a challenge trying to find out what is vulnerable or not. So for this
500420	505280	talk, we're going to talk about known or unknown knowns. This is a fourth
505280	509600	dimension that we like to talk about, and it's comprising that which we
509600	515100	intentionally refuse to acknowledge that we know or we don't like to know, okay? So
515100	519120	that there are vulnerabilities that are known to exist but have not been
519120	523680	associated with all the systems that are actually affected. So we know all
523680	528440	these CVEs are against this open SSL library, but we don't know if that SSL
528440	532700	library is in our device, so we're just going to kind of ignore it for now, all
532700	540300	right? So this unknown knowns is where we can do an awful lot on our part as
540300	545460	manufacturers or security researchers to ferret out and to discover and to patch
545460	552520	before other actors out there find those same vulnerabilities and test them
552520	560060	against your same device, okay? So all of that can be done through a software
560060	565720	bill of materials. So a software bill of materials or an SBOM is there's a bill
565720	568980	of materials that usually comes in the manufacturing world that they're very
568980	573980	used to of all the components that make up certain devices. So if you're getting
573980	578500	big printing presses or whatever else like that, you know all the pieces that make
578500	582260	up that printing press so you can plan maintenance, you know what the cost is
582260	588540	going to be up front. So as a industry, software industry, IoT industry, we
588540	592900	should have the same thing, the software bill of materials, but we don't, all
592900	597400	right? So manufacturers don't know all their components, all the different
597400	601420	chips, system on chips that are running inside their devices, all that kind of
601420	605080	stuff. Vendors don't know all their suppliers and their suppliers' suppliers
605080	610020	because a lot of times vendors will make a product, but then they'll ship it off
610020	614880	to a manufacturer to actually make for them. And so how do you validate that,
614880	619720	you know, that things are exactly as you designed them or whatever? And then
619720	624880	consumers, those of us who put these devices in our critical infrastructure,
624880	629480	into our security systems, into our monitoring software, our monitoring
629480	636200	networks, even in our homes, we put these devices into our networks, but we don't
636200	641160	know all the software that's running on those devices. So the analogy is like, if
641160	648620	you found a laptop out in the parking lot of your company or whatever, would
648620	655060	you bring that laptop in, fire it up, boot it up, and plug it into your
655060	660100	critical infrastructure security system network just to poke around on the
660100	665200	laptop and see what's running on it? And I would hope that most of you or all of
665200	669900	you would say no, there's not a chance that you would ever do that, because we
669900	674060	know that those are full computer systems with operating systems that can
674060	680680	be compromised with viruses, malicious software, etc. But yet with that exact
680680	686520	same mentality, we don't apply to IoT devices. We'll take a camera that we do
686520	690380	not know all the supply, all the software that's running on it, and all the
690380	693840	weaknesses that might be inherent on those softwares, and we'll take that
693840	698740	camera and we'll plug that same camera into and talk to it on our critical
698740	704680	infrastructure network. All right? So we don't have any way to enumerate this.
704680	709520	So how do we generate one of these software build materials reliably? How do
709520	715300	we keep track of all those? Say it was even possible to keep track of all the
715300	720080	components that are in there, how do we validate one of our devices against one?
720080	725440	So we as manufacturers may even develop a device, and we might know exactly what
725440	729280	we want on it, but if we send it away and it comes back, who's to say that whoever
729280	733800	made that device for us put our firmware on there exactly as it was intended, and
733800	738800	they didn't slip something else in? The other thing would be, and this has
738800	743700	happened, is if you as a consumer have a device and you want to update it to the
743700	746540	latest update patch, say there's a security vulnerability and you want to
746540	750940	patch it, you go to the manufacturer or the vendor's site and you download the
750940	755080	update for that firmware, and you flash your device with that firmware, how do
755080	759680	you know that software wasn't compromised? We have seen in the industry
759680	765420	places where upload servers have been compromised by malicious actors, and
765420	769740	custom firmware has been placed in there as updates, and customers have
769740	773500	downloaded malicious updates to their devices, which might have been perfectly
773500	778280	fine in the first place, but now are running malicious software. So not only
778280	782560	generating a software bill of materials, but validating against a bill of
782560	790340	materials is critical and being able to do that. So let's shift away. One thing
790340	794300	I'd like to mention about company commitment and stuff like that is, I just
794300	798500	read Microsoft has a product, Azure Sphere, that they're developing that's a
798500	804460	secure IoT chip, and it's one way of approaching it. Hey, let's control the
804460	808960	ecosystem from the beginning and secure it down. And they've made commitments to
808960	813220	the software bill of materials. So it would be nice if more companies could be
813220	819040	able to control their environment, such as Microsoft has the luxury to do. And I
819040	823000	think these kind of commitments are going to be our way forward, is following
823000	827360	practices like this, generating our own software bill of materials, taking a
827360	833300	hard look at what's running on that. All right. So let's switch to today. We have
833300	836780	these devices, we don't know what's running on them. Let's look at the CVE,
836780	842020	CPE reporting mechanisms that we have for finding vulnerabilities in our
842020	849560	software. Okay. So CVE is common vulnerability exposures. It's a system
849560	852300	that provides reference methods for publicly known information security
852300	857060	vulnerabilities and exposures. Okay. So it's been around a while. MITRE is the
857060	861420	one who came up with that and helped maintain that. So these are all the
861420	865640	vulnerabilities that are discovered and reported for public knowledge. All right.
865640	869660	So that's great. We have a central place to report vulnerabilities. As well, we
869660	873980	have a national vulnerability database with the U.S. government that keeps track
873980	878780	of CPEs or common platform enumeration. So this is a structure naming schemes
878780	883840	for naming products. Okay. This is systems, softwares, software packages,
883840	890140	et cetera. So there's a common way to put all this together. The only problem
890140	896520	is, is that we have these frameworks for enumerating these kinds of
896520	900860	vulnerabilities or these products, but there's not a lot of adherence or,
900860	906680	sorry, hard regulations around how we use this. It's a very flexible system and
906680	911620	it works fairly well when treated well. But there's a lot of inconsistency
912380	920620	across the whole space about how we use CPEs, how we report them, how we link
920620	926480	them to products that they apply to, et cetera. So let's look at some of these.
926480	932000	All right. Are CPEs just complete products? Is it your whole camera system?
932000	936760	Is that whole device one product, a CPE? Are the component systems in there,
936760	941160	your optics and stuff like that? Are the system on chips that are running on there,
941160	945440	is that a CPE? Should that have its own product entry into that database?
945840	950900	Are there libraries within? You're using OpenSSL. That's great. But what about
950900	955980	libcrypto that lives within it? Could that be separate from OpenSSL? Could that be
955980	962600	its own product? So there's an awful lot of questions that we need to answer with
962600	969560	all that. So if we look at something like OpenSSL, it's a commercial-grade secure
969560	975720	sockets layer. It also has cryptographic libraries in it. If you go to the NVD,
975720	979820	the database, and look up OpenSSL, because you want to find out what product it
979820	986760	relates to, because maybe you found a vulnerability, OpenSSL brings back 405 results.
986940	992740	Now, every single version of that software is going to be another result. So 405 is not
992740	998540	necessarily all the different types of OpenSSL, but there are several. So here's a few examples
998540	1005200	that come back with OpenSSL. So the very top one is usually what we think of as OpenSSL.
1005200	1012660	It's a C++ library that is compiled into a lot of Linux systems and stuff with OpenSSL
1012660	1019000	included. 0.9.7 was particularly vulnerable. There's a lot of different CVEs on there,
1019000	1025060	0.9.8, 1.0.1, et cetera. So there's a lot of different versions of that. Those star
1025060	1029440	fields, we'll just kind of gloss over for now, but those are further ways that you can
1029440	1035160	enumerate or specify what specific product this is if there's a lot of different versions.
1035700	1041240	Betas, alphas, different platforms, et cetera. But what are all these other CVEs or CPEs that
1041240	1048180	we see out there? We see this Calderon PyOpenSSL. So the first part is our vendor. The second
1048180	1054280	part is the actual product. So we have this PyOpenSSL. I guess we can make a guess that
1054280	1061120	that would be a Python binding for OpenSSL or a Python library. Then we have Lua OpenSSL
1061120	1066980	on the next line. So we guess maybe that's Lua script. But then look at the next two.
1066980	1073520	We have Node OpenSSL. And if you go all the way down to the Node.js, that's the target
1073520	1079540	software that it's targeting, not the language, but the target software, which is it's written
1079540	1085840	for Node.js, right? So it's a JavaScript module written for Node. But the very next line,
1085840	1093960	OpenSSL.js project OpenSSL.js. So you have Node OpenSSL and then you have OpenSSL.js
1093960	1100980	and you have Node.js at the end. So now you have someone who's writing for Node that has
1100980	1104080	their own way of specifying. But you have two different competing libraries. Which one
1104080	1109440	is which? And then you go down to the last one, OpenSSL project OpenSSL. Well, now that
1109440	1116400	looks an awful lot like the first one, OpenSSL. So which one is that? Well, the hard part
1116400	1121840	is we really don't know. And if you dig into the metadata about this and you actually go
1121840	1130080	to the web page, the title doesn't help you very much. OpenSSL project OpenSSL. So we
1130080	1135320	go into the references and in the change log, we see a GitHub reference there. And in there
1135320	1139420	we have a Rust OpenSSL. And if you go in there and you look at these different things, you
1139420	1146640	have these keywords of Rust. Ah, that's for the Rust language. Okay? So finding qualified
1146640	1153620	prod platforms, sorry, CPEs, can be a real challenge. The other thing that's a real challenge
1153620	1162220	here is we don't know between CPEs what relates to each other. Does this Rust language depend
1162220	1169160	on OpenSSL of a certain version, the C, C++ version of the first line, to be a certain
1169160	1179020	version, right? Does 0.9.2 map to 0.9.7 or 0.9.7A or 1.01? We don't have these kinds
1179020	1186680	of interrelations. So not only is generating an SBOM difficult, but developing a body of
1186680	1193700	ground truth around an SBOM is extremely difficult. And it's not a problem that you think of because
1193700	1199800	you think it should be obvious, but who owns that? Is it the company's responsibility to
1200380	1206320	add their platform to the CPE database so that if vulnerabilities are found against it, that
1206320	1212620	there's reliable data there? Or is it the CVEs who find the vulnerabilities? Is it their job
1212620	1219700	to correctly find and identify the platforms if they're not there? So that's just a little
1219700	1225740	background here on the CPE system. They're good systems to start off with, but we need more on
1225740	1230600	top of that. We need some ground truth. We need some ways to relate this. By the way, there are
1230600	1236480	some projects that are starting to put these together, but it's hard. Do you scrape the apt
1236480	1247380	get repos for what components are in OpenSSL or HTTP servers? Certain HTTP servers rely on OpenSSL.
1247380	1252120	Which versions go with which? Well, if you're installing it, you know because you look at the
1252120	1258080	readme for the HTTP installer, right? And it'll tell you which versions you have, but we don't have
1258080	1265940	any way of systematically obtaining any of that information. Okay, so let's go to a specific
1265940	1271600	example of a vulnerability and let's look at the supply chain here. We'll go with an old example.
1271600	1277700	This is from four years ago. I'm not going to try to butcher Robert's last name,
1277700	1283000	but Darkonius, he released a write-up on a router backdoor that he had discovered
1283880	1290800	originating from a version of OpenWrt, an open source router operating system from at that point
1290800	1297600	was 10 years ago, was 10 years in the past. So there was this backdoor hash that's down in the
1297600	1306300	code below, 1D, 680, whatever. If you entered in that on the command line, you immediately got a
1306300	1312520	root shell and you can see the relevant code there. The problem is, is this was just in an
1313240	1321560	OpenWrt version that was 10 years old from, I guess 2006, right? But the write-up that he had
1321560	1327480	was this was not on an OpenWrt router, this was on a commercial router that had included
1328700	1333560	OpenWrt and its libraries and were using components such as this component script called
1333560	1344220	logon.sh as a part of their operating system. All right? So Darkonius found the backdoor in
1344220	1349660	one or two sets of devices. Him and, and if you read through the comments, they, different people
1349660	1354120	were chiming in, oh, I just tested it against this and I found it on this and this. So we found,
1354120	1362440	you know, in the comments, three, four, maybe five models of devices listed. But how is Darkonius
1362440	1367480	supposed to know and go out and find, is it his responsibility to go find every single device that
1367480	1376720	this relates to? Is it even his obligation to go to the manufacturer of this device and tell him,
1376720	1387700	I found this in there. If you go and you look for this CVE, you don't find this CVE in the database.
1387760	1394600	If you go even further to the CPE, the platform for this device that he looked for, you don't
1394600	1400440	even find CPEs that are related to this specific device. So, and this is not a small device. This
1400440	1405200	is a, this is a, this is a device that made its rounds and you can, you can find it different
1405200	1413760	places. So whose responsibility is it? In our dataset, when we've looked at all of our firmware
1413760	1421240	just sitting around, we just did a string search for that magic hash. And lo and behold, we found
1421240	1430260	3,810 files that had this hash in it. And it wasn't just the hash, it was the full login script.
1430260	1437820	But notice the file name, CLI2, factoryboot.sh, login.sh. So the original login.sh was a part of
1437820	1444300	the original package, but we found the actual login.sh encapsulated in a binary called a
1444300	1451320	command line interface, a CLI, or even two that was used by web servers and stuff on IoT devices
1451320	1458900	that still ran the same code from 2006 that nobody, I'm sure nobody knows is in there. The
1458900	1465200	manufacturer doesn't even know it's there anymore. And the thing is, is that three major vendor
1465200	1471820	companies have this. 44 different product models and 281 versions of this firmware. So this open
1472600	1478620	wrt login.sh, this hash, has made its way into many, many, many different vendors. This isn't
1478620	1483920	just one person accidentally downloading it and putting it in. This is the supply chain of people
1484460	1490500	taking from A, who takes from X, who takes from Y, and it multiplies. So this is the heart of the
1490500	1498000	problem. This is why we're in the bind that we're in, is the supply chain. We'll do one more example
1498000	1504760	before we get into the more juicy stuff, the statistics that we found. This latest CVE, CPE
1504760	1510580	that we're talking about is Ripple 20. I'm sure you guys have heard of it. 19 zero-day vulnerabilities
1510580	1516700	amplified by the supply chain. Its title says it all. And this was two months ago that this was
1516700	1523060	or so was reported. And according to their white paper on it, this affects hundreds of millions of
1523060	1529180	devices or more, includes multiple remote code execution vulnerabilities. So that's the worst
1529180	1534640	kind of vulnerability that you'd want to have. Many other major international vendors suspected
1534640	1541720	of being vulnerable in medical, transportation, industrial control systems, etc. So my previous
1541720	1549540	example kind of laid the groundwork for how something like Ripple 20 exists. All right.
1549540	1555960	So let's just take one of the CVEs as an example that they reported. All right. So this was published
1556600	1564440	June, two months ago, June 17th, 2020. The description of it, the tracked TCP IP stack
1564440	1572140	before this version allows remote code execution related to IPv4 tunneling. Okay. So this is a
1572140	1576340	vulnerability. They said it's bad. There's a base score of 10 critical on here because the remote
1576340	1585740	code execution and how easy it is to theoretically get access to this TCP IP stack. If we look at
1585740	1593820	the CPE that is related to all CVEs have all the CPEs that are related to them, we find these four
1593820	1603280	TREC versions are all rolled up underneath this TREC stack CPE. The problem is, is when we look
1603280	1610620	for this TREC stack and we go back to or we click on the actual record in the CPE database, we find
1610700	1617880	a quick info is that the CPE for this TREC stack was created almost a month after the vulnerability
1617880	1626560	was discovered. All right? So there was no product for TREC TCP. We had to invent, not we, but the
1626560	1632380	vulnerability researchers had to invent the CPE that even belonged to the TREC stack, right?
1633240	1640560	And then we go back to our original problem. If you have this TREC library and you're trying
1640560	1647080	to figure out where all it's where all it's at in the world, you've got to kind of do some
1647080	1651020	detective work. And they did some detective work because we don't have any CPE to CPE
1651600	1656180	relationships. And even if we did, this CPE didn't even exist. Nobody even knew that TREC TCP
1656720	1664140	exists. So I kind of imagine reading through the paper about this, I almost imagine
1665420	1673520	Luis from Ant-Man 2, when he's being interviewed, where is Scott, you know? And I kind of imagine
1673520	1678760	him talking about, you know, hey, Luis, where is TREC? And he goes, hey, man, see, that's
1678760	1682920	complicated. See, TREC, there's some smart guys, but they need some help, right? So they go to
1682920	1687120	Elmec Systems, and they're like, hey, we need some developers. And Elmec Systems is like, yeah,
1687120	1690860	we think you've got it going on, so let's get together. And so they started working together
1690860	1695100	but then something happened. They're like, nah, homie, we're split. We're out of here. So TREC
1695100	1699120	goes off on their own way, and Elmec Systems go off on their own way. And TREC be like,
1699120	1704080	we got the TREC TCP IP stack. We're going to market all this in the United States. And Elmec
1704080	1708900	Systems are like, no, homie, we're no longer Elmec Systems. We're Zook and Elmec. And we got the
1708900	1713740	CASAGO TCP IP. I know it sounds like yours, but it's better because it's been renamed. And you
1713740	1722580	think you're hot? Dang, we got all of Asia that we're going through. Meanwhile, you got
1722580	1728100	security researchers in the middle of the night going, hey, can you just tell me where TREC is?
1728100	1733700	And TREC is like, hmm, see, it's complicated. And they go to Zook and Elmec. And see, they're
1733700	1741020	just like, where's this CASAGO at? And we can't even figure it out because the legal system is
1741020	1745860	cutting us off. Whereas TREC is back here going, let's see, I got an uncle's girlfriend, boyfriend
1745860	1753460	who had a relative named HP. And it's kind of like the Rona virus, right? Like, where's the
1753460	1760680	contact on this? Tracing. And their uncle's second cousin, once removed, is named Aruba.
1760680	1764960	And I think we were at a party together. And meanwhile, Zook and Elmec is like,
1764960	1770060	and JSOF is just like, where is this? And Zook and Elmec and TREC and all these people can't
1770060	1773880	find out. And they're like, that's what I'm trying to say. It's complicated.
1776600	1782780	Right? I'm sorry. I probably butchered that. But I can imagine industrial control systems
1782780	1794960	and industrial plants that have CYP13 investigators or people coming in. I can see their first slide
1794960	1801020	to the CYP13 auditors are going to be this picture of Luis and two words, it's complicated.
1801300	1807700	Right? Because it is. Trying to find out where this TREC TCP stack was, according to this paper,
1807700	1815180	pretty nuts. Okay? So, anyways, it was developed somewhere in the 90s to 2000. It was included in
1815380	1819800	a large quantity of firmware for devices. There's been various patches to it. Some of the Ripple
1819800	1829140	CVEs were patched in versions as far back as 2009. So, you're a vendor. What products of yours
1829140	1834160	contain this TCP IP stack? How do you know? How do you update a device when a patch is present or
1834160	1840420	even know what version of the TREC stack you used to have in your device? How serious is the threat
1840420	1847120	to your device? Right? Maybe it's not that problematic. Maybe it was just included but
1847120	1852820	never used. So, reproducing CVEs are problematic. The actual attack vectors are really hard to
1852820	1858760	classify in these individual products. You potentially waste time from vendors on trivial
1858760	1865140	vulnerabilities that really aren't practical. So, really, the bottom line for vendors are,
1865140	1869180	how much money do you spend trying to find out whether you have to develop firmware patch for
1869180	1873460	software that you don't even know is in your device? Or how likely is the software that it's
1873460	1880120	even going to be exploitable? Right? So, it's a hard, hard question. So, I'll come back to the
1880120	1885600	software building materials here in a little bit. But just for right now, let's make a little shift
1885600	1892500	over into our database. And let's look at some of the vulnerability statistics that we have here.
1892940	1897600	Hopefully, you guys find this enlightening. I certainly did. This is just kind of represents
1897780	1906120	a slice of the industry. We have medical devices. We have information control systems
1906120	1913060	gear. We have all sorts of different sectors represented and verticals in our data sector.
1913060	1920500	So, here we go. All right. So, Etsy Resolve popular name servers, 127.001. Not surprising
1920500	1928780	is one of the top ones. We got some Google addresses there. 88.844. And then the next three
1928780	1936420	down there that you see 168 are all Asian DNSs. And then you've got some strange noise in there.
1936480	1945320	192.168. Someone.com. A couple others. So, you know, if you have all these devices and they're
1945320	1951760	going to the U.S. and they have Asian domain servers, you're going to have a performance hit
1951760	1955840	or vice versa. How do you know where these products are going and which name servers are
1955840	1965340	hitting? I saw some forum posts about 192.168.0.7. Maybe the people who made Etsy Resolve had it on
1965340	1970520	test and they were just copying and pasting from a forum into this file. Someone.com,
1970520	1976540	that actually was one vendor, but it's on a whole slew of firmware for all sorts of
1976540	1981220	different products in their product line. So you can see not only is that illustrative that
1981220	1985220	someone has just put that in as an example and they found that example somewhere,
1985220	1989940	but it's in all their Etsy results and they just copy and paste that file into all their other
1989940	1995260	firmware, right? So where's the checks and balances on what's actually running in there?
1995880	2004280	Even stuff like corp.ubnt.com. Corp.ubnt.com doesn't exist, but it's running on a device as
2004420	2010260	a resolve. So if someone decides to turn up that host name and then start serving requests on
2010260	2016020	there, you could have some interesting results, right? Here's another one. TTY count in Etsy
2016020	2022680	secure TTY. So if you look at the bottom, Etsy secure TTY file allows you to specify which TTY
2022680	2030280	devices a root user is allowed to log in on. So let's just make one thing clear. Once you have an
2030280	2037240	IoT device in production, you probably shouldn't be logging into that device at all. All right?
2037240	2044240	So most normal operation shouldn't allow anybody from the outside to ever be tunneling into that
2044240	2054300	device remotely. Certainly not a root user, okay? So I want to highlight this 196, 148. So what that
2054300	2061880	means is that these counts are the number of shells, or sorry, the number of TTYs listed as
2061880	2072900	approved TTYs that root can log into. So 1,486 firmware have file has an Etsy secure TTY in it
2072900	2082440	with 196 approved TTY locations for root to log in on. All right, so I tracked this one down because
2082440	2088200	this file is just like, where is this anomaly from? I found a Yocto project, which is a way to
2088200	2093980	programmatically or to easily generate a Linux operating system custom build for your project.
2094240	2100340	And this particular file that I found was from 2013. So here is a supply chain, someone who's
2100340	2106100	building and wants a secure TTY, they pull down this file that's pretty permissive, probably for
2106100	2110940	development purposes, and not really supposed to be for production. And yet here it makes its way
2110940	2117700	into all these devices. So here's the supply chain in action. All right, number of valid login shells.
2117700	2125940	Okay, so this is an Etsy shells. This is the number of acceptable different types of
2125940	2132120	shells that you can log in from, and most have one. So again, whether you should have a shell that
2132120	2137660	people can log into or not, that's questionable, but at least a lot of them, majority of them, have
2137660	2144860	only one. But then we have this 10 login shells. And I guess what they want by this, a little tongue
2144860	2150040	in cheek, is, you know, if their hackers are coming in and are used to have all their key bindings in
2150040	2154480	their environment files for bash, they want to make it easier for them. So they're just going to throw
2154480	2160740	in bash in there. And if you're ash or zsh, we got you covered. You can log in however you want.
2160960	2167620	Again, a little tongue in cheek, but that's what that number represents. An additional thing here
2167620	2175060	that was kind of concerning is when I got delved into the actual types of devices that this was,
2175060	2183500	it was over 60 of these were models of patient monitoring health systems. Okay, so over 60
2183500	2187640	different models, which represents however many thousands or hundreds of thousands of
2188200	2196060	health systems, patient monitors that are sitting bedside, have 10 or so login shells that somebody
2196060	2200680	could log in with, right? These are just configuration examples in the supply chain. I'm
2200680	2206580	not saying these are actual vulnerabilities, they're just interesting anomalies from the supply chain.
2206580	2211200	Okay, so here's a questionable config. We're about out of time. So we're just going to go
2211200	2220480	through these quick. Firmware where the root user has a login shell, 15,345. All right, so 30%
2220480	2225540	of all of our firmware that we looked at has some sort of login shell. Whereas if you don't want the
2225540	2232480	root to log in, use sbin login, guess how many? Over under one. We found one firmware that actually
2232480	2237120	had sbin no longer in there. Okay, firmware with keys and authorized keys. Now this is a bad back
2237120	2243180	because this basically says if someone SSHs into your box, if your key matches their key
2243180	2248440	in the authorized keys, you get to log in without any password. So this is a classic backdoor.
2248440	2255340	Whether it was purposefully put there or not, what's the over under? 175, 175 firmware. Remember
2255340	2261500	this is firmware, however many devices this represents. This is 12 different vendors with
2261500	2267700	these keys. All right. And over 29, these were really interesting known hosts to see where these
2267700	2272500	kind of came from and the different medical facilities and different places that had known
2272500	2281340	hosts coming from them. Clam AV, only four firmware had any kind of open antivirus software.
2281340	2291200	Firmware running HTTPD with mod auto index. So this auto index is a way to auto index files
2291200	2297940	on your web browser or your web server when there isn't any kind of match for the HTML name. So it's
2298020	2304320	a great way to kind of troll through your directories. So these are, again, things that are
2304320	2311280	questionable that are in our configs. Firmware starting TTY and init tabs. So quite a few
2311280	2317500	firmware are starting TTY, but not that many distinct files. So here we're seeing amplification
2317500	2325920	in the supply chain of here's 63 files, but these 63 files are found in 4,729 devices.
2326140	2332700	Here's a particularly bad one. Firmware with PHP to default to display errors. 332 firmware. So
2333900	2340300	using SQL injection or any kind of command injection, I love when developers leave on
2340300	2344720	their display errors, because when I made a mistake, it tells me exactly what's there.
2344720	2350320	Problem is, PHP by default allows display errors. You have to explicitly turn it off.
2350320	2355640	So maybe we should be making software that doesn't have defaults that are vulnerable like this in
2355640	2362240	their default configuration. All right. Next part. Firmware with insecure default DES encryption. So
2362240	2365760	if you don't specify the type of encryption for your passwords, you're going to use DES,
2365760	2372120	which is a bit weaker. 318 firmware, 29 files. But even worse, there were firmware that specified
2372120	2378080	the flag to turn on MD5 as their password tripped. It was 12 firmware. Fortunately,
2378080	2381860	it's small, but there's still 12 firmware. I have no idea where those are at,
2381860	2386600	but they're out there. Firmware with a master.password with no password on root. So in
2386600	2394060	the password file, there is no over, under, on 100 devices, 10 devices, a million devices. Now
2394060	2400460	only seven. But still, one file made its way in with no password on the root. MySQL binding to
2400460	2408280	000. So listen to the world and allow people to go right to your MySQL. 26 devices. Firmware
2408280	2413060	with Redis, the same way. Default world bind. So if you don't specify a bind by default, it will
2413060	2418320	allow you in. Three. It's only one file. But still, three firmware allow you into their Redis by
2418600	2425020	default. Average number of unsafe function calls per firmware. So this one is any unsafe function
2425020	2432560	calls is stuff like stir copy, mem copy, all those unbounded ones like ripple 20 found and stuff
2432560	2438580	like malicious ways to use standard allocation functions without checking links and stuff like
2438580	2446700	that. How many do you think in per firmware? 1,500 and some average unsafe function calls
2446700	2454540	per firmware. Okay? Number of firmware with unsafe function calls? 23,000. That's over 50%. And those
2454540	2458660	are the ones that we've named. We haven't even trolled through those with function hashing and
2458660	2466500	stuff to find the unlabeled ones that also call these. All right? Firmware. Here's the last one.
2466500	2472800	Firmware exporting NFS mounts to the world. All right? Over, under on this one? 42. There are
2472800	2480540	four files that contribute to 42 firmware mounts to the world, such as slash root, mount a star,
2481040	2487620	so you can as root write to root as user ID zero. So these are all questionable configs that have
2487620	2492100	made it their way, just fun things that I found trolling through. And finally, what would a talk
2492100	2500120	like this be without talking about the common passwords and all that? So you see here amplified
2500120	2506160	here, again, is the supply chain where you have the file counts of all of this, sort of the light
2506160	2513460	orange, and the dark orange is how many firmware use those files in them. And the number one being
2513460	2518660	admin. It's only in five files in the password file. But those five password files are found in
2521100	2526480	470 firmware, which may represent hundreds of thousands of devices where we've cracked the
2526480	2531840	password admin. I'm going to give you ten to one guess what the password is going to be for that
2531840	2538340	admin user. All right. So anyways, in conclusion, it's all about the software bill of materials.
2538340	2543260	Drive towards generating and validating these. This is the path forward. Manufacturing,
2543260	2549200	implementing software inventory systems, developers being diligent and reporting all
2549200	2554660	the components used in their systems. Consumers having a high demand or having a high standard
2554660	2561080	in transparency and favor companies and buying from companies with that info as well. Policymakers
2561080	2568080	who help guide these unified standards and reporting and formats. And our security community
2568080	2573900	here developing tools to assist in the automation of the inventory. If we're doing the same thing now
2573900	2579060	in five years, we're going behind. We should be using machine learning. We should be using tools
2579060	2584500	and other things to help us inventory and validate these things, not just on the front end, but on the
2584500	2589060	back end to validate patches and updates are what they say they are and they contain what they think
2589060	2595800	they contain. So thanks to all my team at finite state and a lot of good research that's been in
2595800	2604080	the industry. Finally, obviously the question and answer on this session are going to the discord,
2604080	2611140	but I've had a fun time talking to you all. Peace out. Hope you enjoy the rest of DEF CON
2611140	2613740	and IoT Village. Thank you so much for having me.