1
00:00:00,000 --> 00:00:01,440
All right.

2
00:00:04,500 --> 00:00:11,100
All right, all right, all right.

3
00:00:11,100 --> 00:00:12,440
What's up, y'all?

4
00:00:13,140 --> 00:00:18,640
All right, welcome to my talk, IOT under the microscope here virtually.

5
00:00:18,640 --> 00:00:20,220
It's kind of disappointing.

6
00:00:20,380 --> 00:00:28,620
I'm looking at a screen here instead of all you in the in the audience, but hopefully we'll get through this and we'll have a good time doing it.

7
00:00:29,180 --> 00:00:33,140
I want to talk about vulnerability trends in the supply chain.

8
00:00:33,380 --> 00:00:39,500
I've got some very interesting things, I think, that we found in our in our data set size.

9
00:00:39,500 --> 00:00:43,440
So hopefully you guys will learn something here and we'll have a little fun while doing it.

10
00:00:43,440 --> 00:00:44,180
All right.

11
00:00:44,540 --> 00:00:45,240
Okay.

12
00:00:45,760 --> 00:00:47,140
So who am I?

13
00:00:47,140 --> 00:00:48,540
I'm Parker Wixel.

14
00:00:48,860 --> 00:00:51,360
I'm born and raised here in Columbus, Ohio.

15
00:00:51,360 --> 00:01:06,460
I've got 25 years industry experience on cybersecurity, on software development, full stack development, last 10 years of which have really been focused on cybersecurity research and product development.

16
00:01:06,460 --> 00:01:12,540
I was a contributor and developer of open source security projects like AFL Unicorn.

17
00:01:12,540 --> 00:01:25,000
It's a fuzzing framework for emulated binaries and then Patchwork, which is a static compilation of Linux kernels or patching of Linux kernels for debugging purposes.

18
00:01:26,040 --> 00:01:29,200
Like it was mentioned, I'm a senior engineer at Finite State.

19
00:01:29,200 --> 00:01:42,140
We're an IoT cybersecurity firm dealing with a lot of the topics that we're talking about here and all my data sets and stuff like that come from our Finite State repos and some of the products that we're working on here.

20
00:01:42,140 --> 00:01:51,860
I'm also a database lecturer over at The Ohio State University, looking to kick off yet another fall there, and then I'm a composer and a musician.

21
00:01:52,080 --> 00:01:54,300
Don't hold that against me.

22
00:01:54,300 --> 00:02:04,940
I realized early on that there's not a lot of money in music, so here I am on just another passion of mine, computers.

23
00:02:04,980 --> 00:02:07,840
All right, so why is this talk relevant to your interests?

24
00:02:07,840 --> 00:02:17,580
We're going to be talking about supply chain trends, vulnerable and not vulnerable, vulnerability standards and reporting, and then some firmware statistics and observations.

25
00:02:18,140 --> 00:02:35,520
Probably about the first half of the talk is going to be delving into the background of what supply chains are, some of the ways that we have to talk about vulnerabilities, what the supply chain introduces as far as vulnerabilities and visibility into those,

26
00:02:35,520 --> 00:02:47,340
and then probably the last half will be delving into the fun numbers, probably why you came to see this talk, but nevertheless, hopefully we learned something in both parts.

27
00:02:47,480 --> 00:03:06,480
So our data set that I'm pulling from right now, we do have partnerships with some private industry partners, so we do not include all of our private repos and stuff like that, but for this particular talk, I've got about 7 million files, represents about 50,000

28
00:03:06,480 --> 00:03:14,960
firmware images, 10,000 distinct product lines, and 150 different vendors.

29
00:03:14,960 --> 00:03:33,080
This is different architectures, different operating systems, you know, a lot of them are Linux based, some of them are RTOS, obviously, and we're hitting all the different verticals that you usually hear about in these talks, medical devices, critical infrastructure,

30
00:03:33,080 --> 00:03:38,340
security devices, home routers, Alexa, you know, whatever.

31
00:03:38,340 --> 00:03:51,500
We've got a bunch of different types of products in our data set, so hopefully the statistics that we talk about here on the second half, keep this in mind, it's fun to be able to troll a data set of this size.

32
00:03:52,140 --> 00:04:04,560
So let's take a step back, let's talk about the supply chain, let's talk about the problems that are introduced as a part of the supply chain and what the problems are, maybe even go into some of the solutions.

33
00:04:05,400 --> 00:04:13,580
So if you are a manufacturer, XYZ, of a security camera, that security camera is running some sort of firmware on it.

34
00:04:15,040 --> 00:04:16,560
There's hardware and there's firmware.

35
00:04:16,560 --> 00:04:19,740
The firmware is actually software that's written for that hardware device.

36
00:04:19,740 --> 00:04:27,760
We like to think of it as firmware because it's kind of baked in, it's usually not as fluid or as dynamic as software tends to be in, say, a PC or whatever.

37
00:04:27,760 --> 00:04:35,060
But the camera still has a full processor memory architecture, it's a full computer running in that thing.

38
00:04:35,060 --> 00:04:43,660
So if you can take advantage of that or take over that product from a vulnerability standpoint, you've accessed a whole computer's worth of resources.

39
00:04:43,660 --> 00:04:54,840
You have hardware components that go in the thing, you have drivers that talk to those hardwares, operating systems, libraries, apps, you name it, they're all just the same, except they're called firmware.

40
00:04:54,840 --> 00:05:01,200
The problem is on a security camera like this or any kind of IoT device is you're going to have multiple vendors.

41
00:05:01,200 --> 00:05:09,500
So if you're a company XYZ, you're making this product, those hardware components may come from various different vendors.

42
00:05:09,500 --> 00:05:19,900
And then there may be other vendors like vendor A who talks to your underlying camera optics and can put some image recognition on top of it or whatever else like that.

43
00:05:19,900 --> 00:05:23,840
Vendor B may be some support libraries or whatever.

44
00:05:24,180 --> 00:05:33,820
The real problem comes in is not only do you have to track vendor A, vendor B, vendor C, and what all they're putting onto your device, which quite frankly is not always the case.

45
00:05:33,820 --> 00:05:35,980
There's not always full disclosure there.

46
00:05:35,980 --> 00:05:44,320
But then vendor A and vendor B may rely on an open source library somewhere, vendor X, that you don't even know about.

47
00:05:44,320 --> 00:05:50,280
They may or may not even know that they're using it, depending on if their developers have reported on it.

48
00:05:50,560 --> 00:06:07,940
And the thing that makes that even worse is that vendor X library, say it's the same low-level image processing library, it may be version 1.1 in vendor A and vendor B in their libraries that they're including on your device has 1.5.

49
00:06:07,940 --> 00:06:15,160
Maybe vendor A's 1.1 version of vendor X has a vulnerability in it and the 1.5 version doesn't.

50
00:06:15,160 --> 00:06:19,480
How do you know what all component libraries you have in this simple camera?

51
00:06:19,480 --> 00:06:23,940
This is a full computer running on your network, right?

52
00:06:25,300 --> 00:06:36,160
So Donald Rumsfeld, when he was Secretary of Defense, brought this notion together that information can be divided into three categories, he said at the time.

53
00:06:36,160 --> 00:06:40,500
Known knowns, known unknowns, unknown unknowns, all right?

54
00:06:40,500 --> 00:06:45,220
And so we take that kind of approach towards IoT vulnerabilities.

55
00:06:46,180 --> 00:06:52,800
Our known knowns are vulnerabilities that have explicitly been discovered through scanning and testing on our devices.

56
00:06:52,800 --> 00:06:57,460
So we've tested it, we know that there's a vulnerability, we patch it or whatever like that.

57
00:06:57,460 --> 00:06:59,640
So that's our known knowns, all right?

58
00:06:59,640 --> 00:07:13,500
Our known unknowns are newly created software versions or even just upgraded versions or whatever that we've pulled in libraries or whatever that we don't have any kind of application testing behind yet, right?

59
00:07:13,500 --> 00:07:17,800
So who knows what is going on under the hood there?

60
00:07:17,800 --> 00:07:21,660
So we know the device, we just don't know if there's any vulnerabilities there.

61
00:07:22,460 --> 00:07:36,440
And then in the last categorization of the three, unknown unknowns are vulnerabilities that are in your camera or your device that you don't know and that nobody else knows.

62
00:07:36,440 --> 00:07:38,760
And these are what we're calling zero days, all right?

63
00:07:38,760 --> 00:07:50,380
So these zero days or not zero days because we haven't even discovered them, like say Ripple 20 was just discovered a month or two ago.

64
00:07:50,380 --> 00:07:53,600
Before then, it was an unknown unknown.

65
00:07:53,600 --> 00:07:58,520
It was in all these different devices, but nobody knew it was there.

66
00:07:58,520 --> 00:08:01,860
But the weakness was still there waiting to be discovered, all right?

67
00:08:02,180 --> 00:08:08,100
So there's an awful lot of work to be done in discovering zero days.

68
00:08:08,100 --> 00:08:18,980
As a security researcher, we know that trying to protect yourself and get through all that is really a challenge trying to find out what is vulnerable or not.

69
00:08:19,020 --> 00:08:24,540
So for this talk, we're going to talk about known or unknown knowns.

70
00:08:24,540 --> 00:08:34,700
This is a fourth dimension that we like to talk about, and it's comprising that which we intentionally refuse to acknowledge that we know or we don't like to know, okay?

71
00:08:34,700 --> 00:08:42,740
So that there are vulnerabilities that are known to exist but have not been associated with all the systems that are actually affected.

72
00:08:42,740 --> 00:08:53,420
So we know all these CVEs are against this open SSL library, but we don't know if that SSL library is in our device, so we're just going to kind of ignore it for now, all right?

73
00:08:54,140 --> 00:09:14,700
So this unknown knowns is where we can do an awful lot on our part as manufacturers or security researchers to ferret out and to discover and to patch before other actors out there find those same vulnerabilities and test them against your same device,

74
00:09:14,700 --> 00:09:15,440
okay?

75
00:09:15,940 --> 00:09:21,220
So all of that can be done through a software bill of materials.

76
00:09:21,220 --> 00:09:32,820
So a software bill of materials or an SBOM is there's a bill of materials that usually comes in the manufacturing world that they're very used to of all the components that make up certain devices.

77
00:09:32,820 --> 00:09:43,480
So if you're getting big printing presses or whatever else like that, you know all the pieces that make up that printing press so you can plan maintenance, you know what the cost is going to be up front.

78
00:09:43,580 --> 00:09:53,180
So as a industry, software industry, IoT industry, we should have the same thing, the software bill of materials, but we don't, all right?

79
00:09:53,180 --> 00:10:01,700
So manufacturers don't know all their components, all the different chips, system on chips that are running inside their devices, all that kind of stuff.

80
00:10:01,700 --> 00:10:12,940
Vendors don't know all their suppliers and their suppliers' suppliers because a lot of times vendors will make a product, but then they'll ship it off to a manufacturer to actually make for them.

81
00:10:13,220 --> 00:10:19,140
And so how do you validate that, you know, that things are exactly as you designed them or whatever?

82
00:10:19,140 --> 00:10:38,820
And then consumers, those of us who put these devices in our critical infrastructure, into our security systems, into our monitoring software, our monitoring networks, even in our homes, we put these devices into our networks, but we don't know all the software that's running on those devices.

83
00:10:38,820 --> 00:11:02,140
So the analogy is like, if you found a laptop out in the parking lot of your company or whatever, would you bring that laptop in, fire it up, boot it up, and plug it into your critical infrastructure security system network just to poke around on the laptop and see what's running on it?

84
00:11:02,280 --> 00:11:19,040
And I would hope that most of you or all of you would say no, there's not a chance that you would ever do that, because we know that those are full computer systems with operating systems that can be compromised with viruses, malicious software, etc.

85
00:11:19,040 --> 00:11:24,280
But yet with that exact same mentality, we don't apply to IoT devices.

86
00:11:24,280 --> 00:11:40,460
We'll take a camera that we do not know all the supply, all the software that's running on it, and all the weaknesses that might be inherent on those softwares, and we'll take that camera and we'll plug that same camera into and talk to it on our critical infrastructure network.

87
00:11:40,460 --> 00:11:41,220
All right?

88
00:11:41,220 --> 00:11:44,680
So we don't have any way to enumerate this.

89
00:11:44,680 --> 00:11:49,240
So how do we generate one of these software build materials reliably?

90
00:11:49,240 --> 00:11:51,740
How do we keep track of all those?

91
00:11:52,000 --> 00:12:00,080
Say it was even possible to keep track of all the components that are in there, how do we validate one of our devices against one?

92
00:12:00,080 --> 00:12:15,900
So we as manufacturers may even develop a device, and we might know exactly what we want on it, but if we send it away and it comes back, who's to say that whoever made that device for us put our firmware on there exactly as it was intended, and they didn't slip something else in?

93
00:12:16,300 --> 00:12:32,500
The other thing would be, and this has happened, is if you as a consumer have a device and you want to update it to the latest update patch, say there's a security vulnerability and you want to patch it, you go to the manufacturer or the vendor's site and you download the update for that firmware,

94
00:12:32,500 --> 00:12:37,980
and you flash your device with that firmware, how do you know that software wasn't compromised?

95
00:12:37,980 --> 00:12:54,880
We have seen in the industry places where upload servers have been compromised by malicious actors, and custom firmware has been placed in there as updates, and customers have downloaded malicious updates to their devices, which might have been perfectly fine in the first place,

96
00:12:54,880 --> 00:12:57,060
but now are running malicious software.

97
00:12:57,060 --> 00:13:06,040
So not only generating a software bill of materials, but validating against a bill of materials is critical and being able to do that.

98
00:13:06,400 --> 00:13:08,800
So let's shift away.

99
00:13:09,660 --> 00:13:21,760
One thing I'd like to mention about company commitment and stuff like that is, I just read Microsoft has a product, Azure Sphere, that they're developing that's a secure IoT chip, and it's one way of approaching it.

100
00:13:21,760 --> 00:13:27,180
Hey, let's control the ecosystem from the beginning and secure it down.

101
00:13:27,180 --> 00:13:30,080
And they've made commitments to the software bill of materials.

102
00:13:30,080 --> 00:13:37,840
So it would be nice if more companies could be able to control their environment, such as Microsoft has the luxury to do.

103
00:13:38,060 --> 00:13:48,880
And I think these kind of commitments are going to be our way forward, is following practices like this, generating our own software bill of materials, taking a hard look at what's running on that.

104
00:13:48,880 --> 00:13:49,620
All right.

105
00:13:49,940 --> 00:13:52,420
So let's switch to today.

106
00:13:52,420 --> 00:13:55,220
We have these devices, we don't know what's running on them.

107
00:13:55,220 --> 00:14:02,020
Let's look at the CVE, CPE reporting mechanisms that we have for finding vulnerabilities in our software.

108
00:14:02,020 --> 00:14:02,540
Okay.

109
00:14:02,540 --> 00:14:05,460
So CVE is common vulnerability exposures.

110
00:14:06,000 --> 00:14:13,760
It's a system that provides reference methods for publicly known information security vulnerabilities and exposures.

111
00:14:13,760 --> 00:14:13,980
Okay.

112
00:14:13,980 --> 00:14:15,960
So it's been around a while.

113
00:14:16,000 --> 00:14:20,440
MITRE is the one who came up with that and helped maintain that.

114
00:14:20,440 --> 00:14:25,120
So these are all the vulnerabilities that are discovered and reported for public knowledge.

115
00:14:25,120 --> 00:14:25,640
All right.

116
00:14:25,640 --> 00:14:26,180
So that's great.

117
00:14:26,180 --> 00:14:28,380
We have a central place to report vulnerabilities.

118
00:14:28,400 --> 00:14:32,480
As well, we have a national vulnerability database with the U.S.

119
00:14:32,480 --> 00:14:36,740
government that keeps track of CPEs or common platform enumeration.

120
00:14:36,740 --> 00:14:40,600
So this is a structure naming schemes for naming products.

121
00:14:40,600 --> 00:14:40,840
Okay.

122
00:14:40,840 --> 00:14:44,800
This is systems, softwares, software packages, et cetera.

123
00:14:45,680 --> 00:14:49,120
So there's a common way to put all this together.

124
00:14:49,120 --> 00:15:04,960
The only problem is, is that we have these frameworks for enumerating these kinds of vulnerabilities or these products, but there's not a lot of adherence or, sorry, hard regulations around how we use this.

125
00:15:04,960 --> 00:15:09,760
It's a very flexible system and it works fairly well when treated well.

126
00:15:09,760 --> 00:15:24,140
But there's a lot of inconsistency across the whole space about how we use CPEs, how we report them, how we link them to products that they apply to, et cetera.

127
00:15:24,140 --> 00:15:26,480
So let's look at some of these.

128
00:15:26,480 --> 00:15:27,180
All right.

129
00:15:27,780 --> 00:15:30,440
Are CPEs just complete products?

130
00:15:30,440 --> 00:15:32,000
Is it your whole camera system?

131
00:15:32,000 --> 00:15:35,020
Is that whole device one product, a CPE?

132
00:15:35,020 --> 00:15:38,540
Are the component systems in there, your optics and stuff like that?

133
00:15:38,540 --> 00:15:42,140
Are the system on chips that are running on there, is that a CPE?

134
00:15:42,140 --> 00:15:45,440
Should that have its own product entry into that database?

135
00:15:45,840 --> 00:15:47,280
Are there libraries within?

136
00:15:47,280 --> 00:15:49,100
You're using OpenSSL.

137
00:15:49,120 --> 00:15:50,160
That's great.

138
00:15:50,160 --> 00:15:53,000
But what about libcrypto that lives within it?

139
00:15:53,020 --> 00:15:55,640
Could that be separate from OpenSSL?

140
00:15:55,640 --> 00:15:57,260
Could that be its own product?

141
00:15:58,360 --> 00:16:03,260
So there's an awful lot of questions that we need to answer with all that.

142
00:16:03,260 --> 00:16:10,960
So if we look at something like OpenSSL, it's a commercial-grade secure sockets layer.

143
00:16:11,320 --> 00:16:13,820
It also has cryptographic libraries in it.

144
00:16:13,820 --> 00:16:26,760
If you go to the NVD, the database, and look up OpenSSL, because you want to find out what product it relates to, because maybe you found a vulnerability, OpenSSL brings back 405 results.

145
00:16:26,940 --> 00:16:31,440
Now, every single version of that software is going to be another result.

146
00:16:31,440 --> 00:16:37,080
So 405 is not necessarily all the different types of OpenSSL, but there are several.

147
00:16:37,080 --> 00:16:40,940
So here's a few examples that come back with OpenSSL.

148
00:16:41,080 --> 00:16:45,200
So the very top one is usually what we think of as OpenSSL.

149
00:16:45,200 --> 00:16:53,660
It's a C++ library that is compiled into a lot of Linux systems and stuff with OpenSSL included.

150
00:16:54,340 --> 00:16:57,280
0.9.7 was particularly vulnerable.

151
00:16:57,280 --> 00:17:02,560
There's a lot of different CVEs on there, 0.9.8, 1.0.1, et cetera.

152
00:17:02,560 --> 00:17:04,600
So there's a lot of different versions of that.

153
00:17:04,600 --> 00:17:15,160
Those star fields, we'll just kind of gloss over for now, but those are further ways that you can enumerate or specify what specific product this is if there's a lot of different versions.

154
00:17:15,700 --> 00:17:18,560
Betas, alphas, different platforms, et cetera.

155
00:17:18,620 --> 00:17:21,880
But what are all these other CVEs or CPEs that we see out there?

156
00:17:21,880 --> 00:17:25,620
We see this Calderon PyOpenSSL.

157
00:17:25,620 --> 00:17:27,920
So the first part is our vendor.

158
00:17:27,920 --> 00:17:30,120
The second part is the actual product.

159
00:17:30,300 --> 00:17:32,360
So we have this PyOpenSSL.

160
00:17:32,360 --> 00:17:38,420
I guess we can make a guess that that would be a Python binding for OpenSSL or a Python library.

161
00:17:38,780 --> 00:17:42,920
Then we have Lua OpenSSL on the next line.

162
00:17:42,940 --> 00:17:45,000
So we guess maybe that's Lua script.

163
00:17:45,200 --> 00:17:46,980
But then look at the next two.

164
00:17:46,980 --> 00:17:50,080
We have Node OpenSSL.

165
00:17:50,340 --> 00:18:01,380
And if you go all the way down to the Node.js, that's the target software that it's targeting, not the language, but the target software, which is it's written for Node.js, right?

166
00:18:01,520 --> 00:18:04,060
So it's a JavaScript module written for Node.

167
00:18:04,480 --> 00:18:10,200
But the very next line, OpenSSL.js project OpenSSL.js.

168
00:18:10,460 --> 00:18:16,160
So you have Node OpenSSL and then you have OpenSSL.js and you have Node.js at the end.

169
00:18:16,200 --> 00:18:22,100
So now you have someone who's writing for Node that has their own way of specifying.

170
00:18:22,100 --> 00:18:23,680
But you have two different competing libraries.

171
00:18:23,680 --> 00:18:25,160
Which one is which?

172
00:18:25,160 --> 00:18:28,920
And then you go down to the last one, OpenSSL project OpenSSL.

173
00:18:28,920 --> 00:18:33,320
Well, now that looks an awful lot like the first one, OpenSSL.

174
00:18:33,520 --> 00:18:35,180
So which one is that?

175
00:18:35,180 --> 00:18:38,440
Well, the hard part is we really don't know.

176
00:18:38,620 --> 00:18:45,460
And if you dig into the metadata about this and you actually go to the web page, the title doesn't help you very much.

177
00:18:45,640 --> 00:18:47,980
OpenSSL project OpenSSL.

178
00:18:49,360 --> 00:18:54,800
So we go into the references and in the change log, we see a GitHub reference there.

179
00:18:54,800 --> 00:18:57,040
And in there we have a Rust OpenSSL.

180
00:18:57,040 --> 00:19:01,000
And if you go in there and you look at these different things, you have these keywords of Rust.

181
00:19:01,160 --> 00:19:03,300
Ah, that's for the Rust language.

182
00:19:03,300 --> 00:19:03,900
Okay?

183
00:19:03,940 --> 00:19:12,140
So finding qualified prod platforms, sorry, CPEs, can be a real challenge.

184
00:19:12,140 --> 00:19:19,560
The other thing that's a real challenge here is we don't know between CPEs what relates to each other.

185
00:19:19,580 --> 00:19:30,300
Does this Rust language depend on OpenSSL of a certain version, the C, C++ version of the first line, to be a certain version, right?

186
00:19:30,300 --> 00:19:36,980
Does 0.9.2 map to 0.9.7 or 0.9.7A or 1.01?

187
00:19:36,980 --> 00:19:40,160
We don't have these kinds of interrelations.

188
00:19:40,160 --> 00:19:50,600
So not only is generating an SBOM difficult, but developing a body of ground truth around an SBOM is extremely difficult.

189
00:19:50,600 --> 00:19:57,000
And it's not a problem that you think of because you think it should be obvious, but who owns that?

190
00:19:57,140 --> 00:20:08,360
Is it the company's responsibility to add their platform to the CPE database so that if vulnerabilities are found against it, that there's reliable data there?

191
00:20:08,360 --> 00:20:11,660
Or is it the CVEs who find the vulnerabilities?

192
00:20:11,660 --> 00:20:16,980
Is it their job to correctly find and identify the platforms if they're not there?

193
00:20:17,780 --> 00:20:22,340
So that's just a little background here on the CPE system.

194
00:20:22,580 --> 00:20:26,200
They're good systems to start off with, but we need more on top of that.

195
00:20:26,200 --> 00:20:27,740
We need some ground truth.

196
00:20:27,740 --> 00:20:29,780
We need some ways to relate this.

197
00:20:29,780 --> 00:20:35,280
By the way, there are some projects that are starting to put these together, but it's hard.

198
00:20:35,280 --> 00:20:44,000
Do you scrape the apt get repos for what components are in OpenSSL or HTTP servers?

199
00:20:44,000 --> 00:20:47,380
Certain HTTP servers rely on OpenSSL.

200
00:20:47,380 --> 00:20:49,040
Which versions go with which?

201
00:20:49,040 --> 00:20:55,600
Well, if you're installing it, you know because you look at the readme for the HTTP installer, right?

202
00:20:55,600 --> 00:21:02,740
And it'll tell you which versions you have, but we don't have any way of systematically obtaining any of that information.

203
00:21:03,140 --> 00:21:09,580
Okay, so let's go to a specific example of a vulnerability and let's look at the supply chain here.

204
00:21:09,580 --> 00:21:11,600
We'll go with an old example.

205
00:21:11,600 --> 00:21:14,260
This is from four years ago.

206
00:21:14,600 --> 00:21:33,280
I'm not going to try to butcher Robert's last name, but Darkonius, he released a write-up on a router backdoor that he had discovered originating from a version of OpenWrt, an open source router operating system from at that point was 10 years ago, was 10 years in the past.

207
00:21:33,280 --> 00:21:41,880
So there was this backdoor hash that's down in the code below, 1D, 680, whatever.

208
00:21:41,880 --> 00:21:49,220
If you entered in that on the command line, you immediately got a root shell and you can see the relevant code there.

209
00:21:49,680 --> 00:21:58,620
The problem is, is this was just in an OpenWrt version that was 10 years old from, I guess 2006, right?

210
00:21:58,620 --> 00:22:20,720
But the write-up that he had was this was not on an OpenWrt router, this was on a commercial router that had included OpenWrt and its libraries and were using components such as this component script called logon.sh as a part of their operating system.

211
00:22:20,760 --> 00:22:21,320
All right?

212
00:22:22,020 --> 00:22:25,600
So Darkonius found the backdoor in one or two sets of devices.

213
00:22:25,600 --> 00:22:33,160
Him and, and if you read through the comments, they, different people were chiming in, oh, I just tested it against this and I found it on this and this.

214
00:22:33,160 --> 00:22:39,220
So we found, you know, in the comments, three, four, maybe five models of devices listed.

215
00:22:40,340 --> 00:22:48,680
But how is Darkonius supposed to know and go out and find, is it his responsibility to go find every single device that this relates to?

216
00:22:48,680 --> 00:22:58,500
Is it even his obligation to go to the manufacturer of this device and tell him, I found this in there.

217
00:22:58,520 --> 00:23:07,700
If you go and you look for this CVE, you don't find this CVE in the database.

218
00:23:07,760 --> 00:23:18,400
If you go even further to the CPE, the platform for this device that he looked for, you don't even find CPEs that are related to this specific device.

219
00:23:18,680 --> 00:23:20,400
So, and this is not a small device.

220
00:23:20,400 --> 00:23:26,020
This is a, this is a, this is a device that made its rounds and you can, you can find it different places.

221
00:23:26,800 --> 00:23:29,120
So whose responsibility is it?

222
00:23:30,640 --> 00:23:39,560
In our dataset, when we've looked at all of our firmware just sitting around, we just did a string search for that magic hash.

223
00:23:39,860 --> 00:23:47,100
And lo and behold, we found 3,810 files that had this hash in it.

224
00:23:47,100 --> 00:23:50,260
And it wasn't just the hash, it was the full login script.

225
00:23:50,260 --> 00:23:55,120
But notice the file name, CLI2, factoryboot.sh, login.sh.

226
00:23:55,120 --> 00:24:15,880
So the original login.sh was a part of the original package, but we found the actual login.sh encapsulated in a binary called a command line interface, a CLI, or even two that was used by web servers and stuff on IoT devices that still ran the same code from 2006 that nobody,

227
00:24:15,880 --> 00:24:18,640
I'm sure nobody knows is in there.

228
00:24:18,640 --> 00:24:21,500
The manufacturer doesn't even know it's there anymore.

229
00:24:22,700 --> 00:24:26,700
And the thing is, is that three major vendor companies have this.

230
00:24:26,800 --> 00:24:31,180
44 different product models and 281 versions of this firmware.

231
00:24:31,180 --> 00:24:38,260
So this open wrt login.sh, this hash, has made its way into many, many, many different vendors.

232
00:24:38,260 --> 00:24:41,980
This isn't just one person accidentally downloading it and putting it in.

233
00:24:41,980 --> 00:24:49,200
This is the supply chain of people taking from A, who takes from X, who takes from Y, and it multiplies.

234
00:24:49,200 --> 00:24:51,040
So this is the heart of the problem.

235
00:24:51,040 --> 00:24:56,080
This is why we're in the bind that we're in, is the supply chain.

236
00:24:56,520 --> 00:25:01,460
We'll do one more example before we get into the more juicy stuff, the statistics that we found.

237
00:25:02,600 --> 00:25:07,280
This latest CVE, CPE that we're talking about is Ripple 20.

238
00:25:07,280 --> 00:25:09,080
I'm sure you guys have heard of it.

239
00:25:09,080 --> 00:25:12,180
19 zero-day vulnerabilities amplified by the supply chain.

240
00:25:12,180 --> 00:25:13,980
Its title says it all.

241
00:25:13,980 --> 00:25:19,220
And this was two months ago that this was or so was reported.

242
00:25:19,220 --> 00:25:28,380
And according to their white paper on it, this affects hundreds of millions of devices or more, includes multiple remote code execution vulnerabilities.

243
00:25:28,380 --> 00:25:31,860
So that's the worst kind of vulnerability that you'd want to have.

244
00:25:31,940 --> 00:25:39,680
Many other major international vendors suspected of being vulnerable in medical, transportation, industrial control systems, etc.

245
00:25:39,680 --> 00:25:48,620
So my previous example kind of laid the groundwork for how something like Ripple 20 exists.

246
00:25:49,000 --> 00:25:49,540
All right.

247
00:25:49,540 --> 00:25:54,260
So let's just take one of the CVEs as an example that they reported.

248
00:25:54,260 --> 00:25:54,740
All right.

249
00:25:54,740 --> 00:26:00,940
So this was published June, two months ago, June 17th, 2020.

250
00:26:01,320 --> 00:26:10,020
The description of it, the tracked TCP IP stack before this version allows remote code execution related to IPv4 tunneling.

251
00:26:10,020 --> 00:26:10,400
Okay.

252
00:26:10,400 --> 00:26:12,540
So this is a vulnerability .

253
00:26:12,540 --> 00:26:13,380
They said it's bad.

254
00:26:13,380 --> 00:26:24,340
There's a base score of 10 critical on here because the remote code execution and how easy it is to theoretically get access to this TCP IP stack.

255
00:26:24,420 --> 00:26:39,960
If we look at the CPE that is related to all CVEs have all the CPEs that are related to them, we find these four TREC versions are all rolled up underneath this TREC stack CPE.

256
00:26:40,040 --> 00:26:59,460
The problem is, is when we look for this TREC stack and we go back to or we click on the actual record in the CPE database, we find a quick info is that the CPE for this TREC stack was created almost a month after the vulnerability was discovered.

257
00:26:59,960 --> 00:27:00,240
All right?

258
00:27:00,240 --> 00:27:03,620
So there was no product for TREC TCP.

259
00:27:03,620 --> 00:27:12,380
We had to invent, not we, but the vulnerability researchers had to invent the CPE that even belonged to the TREC stack, right?

260
00:27:13,240 --> 00:27:16,500
And then we go back to our original problem.

261
00:27:17,160 --> 00:27:27,800
If you have this TREC library and you're trying to figure out where all it's where all it's at in the world, you've got to kind of do some detective work.

262
00:27:27,800 --> 00:27:32,180
And they did some detective work because we don't have any CPE to CPE relationships.

263
00:27:32,180 --> 00:27:34,720
And even if we did, this CPE didn't even exist.

264
00:27:34,720 --> 00:27:37,520
Nobody even knew that TREC TCP exists.

265
00:27:37,600 --> 00:27:52,420
So I kind of imagine reading through the paper about this, I almost imagine Luis from Ant-Man 2, when he's being interviewed, where is Scott, you know?

266
00:27:52,480 --> 00:27:56,640
And I kind of imagine him talking about, you know, hey, Luis, where is TREC?

267
00:27:56,640 --> 00:27:59,520
And he goes, hey, man, see, that's complicated.

268
00:27:59,540 --> 00:28:02,320
See, TREC, there's some smart guys, but they need some help, right?

269
00:28:02,320 --> 00:28:05,840
So they go to Elmec Systems, and they're like, hey, we need some developers.

270
00:28:05,840 --> 00:28:09,500
And Elmec Systems is like, yeah, we think you've got it going on, so let's get together.

271
00:28:09,500 --> 00:28:12,420
And so they started working together but then something happened.

272
00:28:12,420 --> 00:28:14,060
They're like, nah, homie, we're split.

273
00:28:14,060 --> 00:28:14,700
We're out of here.

274
00:28:14,700 --> 00:28:18,500
So TREC goes off on their own way, and Elmec Systems go off on their own way.

275
00:28:18,500 --> 00:28:20,920
And TREC be like, we got the TREC TCP IP stack.

276
00:28:20,920 --> 00:28:23,620
We're going to market all this in the United States.

277
00:28:23,620 --> 00:28:27,180
And Elmec Systems are like, no, homie, we're no longer Elmec Systems.

278
00:28:27,180 --> 00:28:28,240
We're Zook and Elmec.

279
00:28:28,240 --> 00:28:30,600
And we got the CASAGO TCP IP.

280
00:28:30,600 --> 00:28:33,440
I know it sounds like yours, but it's better because it's been renamed.

281
00:28:33,480 --> 00:28:37,300
And you think you're hot?

282
00:28:37,680 --> 00:28:41,920
Dang, we got all of Asia that we're going through.

283
00:28:41,920 --> 00:28:48,100
Meanwhile, you got security researchers in the middle of the night going, hey, can you just tell me where TREC is?

284
00:28:48,100 --> 00:28:50,600
And TREC is like, hmm, see, it's complicated.

285
00:28:50,600 --> 00:28:51,940
And they go to Zook and Elmec.

286
00:28:51,940 --> 00:28:58,700
And see, they're just like, where's this CASAGO at?

287
00:28:58,700 --> 00:29:01,900
And we can't even figure it out because the legal system is cutting us off.

288
00:29:01,900 --> 00:29:08,960
Whereas TREC is back here going, let's see, I got an uncle's girlfriend, boyfriend who had a relative named HP.

289
00:29:09,100 --> 00:29:12,540
And it's kind of like the Rona virus, right?

290
00:29:12,680 --> 00:29:14,500
Like, where's the contact on this?

291
00:29:15,200 --> 00:29:16,100
Tracing.

292
00:29:16,100 --> 00:29:20,680
And their uncle's second cousin, once removed, is named Aruba.

293
00:29:20,680 --> 00:29:22,480
And I think we were at a party together.

294
00:29:22,480 --> 00:29:27,460
And meanwhile, Zook and Elmec is like, and JSOF is just like, where is this?

295
00:29:27,460 --> 00:29:30,640
And Zook and Elmec and TREC and all these people can't find out.

296
00:29:30,640 --> 00:29:32,740
And they're like, that's what I'm trying to say.

297
00:29:32,740 --> 00:29:33,880
It's complicated.

298
00:29:36,600 --> 00:29:37,300
Right?

299
00:29:37,300 --> 00:29:37,840
I'm sorry.

300
00:29:37,840 --> 00:29:39,100
I probably butchered that.

301
00:29:39,100 --> 00:29:53,060
But I can imagine industrial control systems and industrial plants that have CYP13 investigators or people coming in.

302
00:29:53,060 --> 00:30:01,020
I can see their first slide to the CYP13 auditors are going to be this picture of Luis and two words, it's complicated.

303
00:30:01,300 --> 00:30:01,540
Right?

304
00:30:01,540 --> 00:30:02,380
Because it is.

305
00:30:02,380 --> 00:30:08,740
Trying to find out where this TREC TCP stack was, according to this paper, pretty nuts.

306
00:30:08,740 --> 00:30:09,340
Okay?

307
00:30:09,420 --> 00:30:13,700
So, anyways, it was developed somewhere in the 90s to 2000.

308
00:30:13,960 --> 00:30:17,400
It was included in a large quantity of firmware for devices.

309
00:30:17,400 --> 00:30:19,220
There's been various patches to it.

310
00:30:19,220 --> 00:30:25,240
Some of the Ripple CVEs were patched in versions as far back as 2009.

311
00:30:25,540 --> 00:30:27,460
So, you're a vendor.

312
00:30:27,940 --> 00:30:30,880
What products of yours contain this TCP IP stack?

313
00:30:30,880 --> 00:30:31,940
How do you know?

314
00:30:31,940 --> 00:30:39,220
How do you update a device when a patch is present or even know what version of the TREC stack you used to have in your device?

315
00:30:39,220 --> 00:30:41,980
How serious is the threat to your device?

316
00:30:42,000 --> 00:30:42,620
Right?

317
00:30:42,680 --> 00:30:45,960
Maybe it's not that problematic.

318
00:30:45,960 --> 00:30:48,260
Maybe it was just included but never used.

319
00:30:48,260 --> 00:30:50,580
So, reproducing CVEs are problematic.

320
00:30:50,580 --> 00:30:55,540
The actual attack vectors are really hard to classify in these individual products.

321
00:30:55,540 --> 00:31:01,380
You potentially waste time from vendors on trivial vulnerabilities that really aren't practical.

322
00:31:02,020 --> 00:31:11,200
So, really, the bottom line for vendors are, how much money do you spend trying to find out whether you have to develop firmware patch for software that you don't even know is in your device?

323
00:31:11,200 --> 00:31:14,900
Or how likely is the software that it's even going to be exploitable?

324
00:31:14,900 --> 00:31:15,400
Right?

325
00:31:15,460 --> 00:31:18,100
So, it's a hard, hard question.

326
00:31:18,420 --> 00:31:23,240
So, I'll come back to the software building materials here in a little bit.

327
00:31:23,240 --> 00:31:28,580
But just for right now, let's make a little shift over into our database.

328
00:31:28,620 --> 00:31:32,500
And let's look at some of the vulnerability statistics that we have here.

329
00:31:32,940 --> 00:31:35,320
Hopefully, you guys find this enlightening.

330
00:31:35,320 --> 00:31:36,280
I certainly did.

331
00:31:36,280 --> 00:31:41,120
This is just kind of represents a slice of the industry.

332
00:31:41,120 --> 00:31:42,340
We have medical devices.

333
00:31:42,340 --> 00:31:46,880
We have information control systems gear.

334
00:31:46,880 --> 00:31:53,060
We have all sorts of different sectors represented and verticals in our data sector.

335
00:31:53,060 --> 00:31:54,280
So, here we go.

336
00:31:54,320 --> 00:31:54,680
All right.

337
00:31:54,680 --> 00:31:59,440
So, Etsy Resolve popular name servers, 127.001.

338
00:31:59,440 --> 00:32:01,620
Not surprising is one of the top ones.

339
00:32:01,620 --> 00:32:03,640
We got some Google addresses there.

340
00:32:03,640 --> 00:32:07,220
88.844.

341
00:32:07,220 --> 00:32:12,640
And then the next three down there that you see 168 are all Asian DNSs.

342
00:32:12,780 --> 00:32:16,420
And then you've got some strange noise in there.

343
00:32:16,480 --> 00:32:17,920
192.168.

344
00:32:17,920 --> 00:32:19,560
Someone.com.

345
00:32:19,560 --> 00:32:20,600
A couple others.

346
00:32:21,260 --> 00:32:26,740
So, you know, if you have all these devices and they're going to the U.S.

347
00:32:26,740 --> 00:32:32,900
and they have Asian domain servers, you're going to have a performance hit or vice versa.

348
00:32:32,900 --> 00:32:36,560
How do you know where these products are going and which name servers are hitting?

349
00:32:36,560 --> 00:32:40,560
I saw some forum posts about 192.168.0.7.

350
00:32:40,560 --> 00:32:49,400
Maybe the people who made Etsy Resolve had it on test and they were just copying and pasting from a forum into this file.

351
00:32:49,400 --> 00:32:58,200
Someone.com, that actually was one vendor, but it's on a whole slew of firmware for all sorts of different products in their product line.

352
00:32:58,200 --> 00:33:11,100
So you can see not only is that illustrative that someone has just put that in as an example and they found that example somewhere, but it's in all their Etsy results and they just copy and paste that file into all their other firmware, right?

353
00:33:11,100 --> 00:33:15,260
So where's the checks and balances on what's actually running in there?

354
00:33:15,880 --> 00:33:19,660
Even stuff like corp.ubnt.com.

355
00:33:19,660 --> 00:33:25,340
Corp.ubnt.com doesn't exist, but it's running on a device as a resolve.

356
00:33:25,420 --> 00:33:32,860
So if someone decides to turn up that host name and then start serving requests on there, you could have some interesting results, right?

357
00:33:32,880 --> 00:33:34,060
Here's another one.

358
00:33:34,060 --> 00:33:37,100
TTY count in Etsy secure TTY.

359
00:33:37,100 --> 00:33:46,760
So if you look at the bottom, Etsy secure TTY file allows you to specify which TTY devices a root user is allowed to log in on.

360
00:33:46,760 --> 00:33:49,340
So let's just make one thing clear.

361
00:33:49,340 --> 00:33:56,880
Once you have an IoT device in production, you probably shouldn't be logging into that device at all.

362
00:33:56,880 --> 00:33:57,240
All right?

363
00:33:57,240 --> 00:34:05,820
So most normal operation shouldn't allow anybody from the outside to ever be tunneling into that device remotely.

364
00:34:06,420 --> 00:34:08,860
Certainly not a root user, okay?

365
00:34:08,860 --> 00:34:13,580
So I want to highlight this 196, 148.

366
00:34:13,580 --> 00:34:25,080
So what that means is that these counts are the number of shells, or sorry, the number of TTYs listed as approved TTYs that root can log into.

367
00:34:25,080 --> 00:34:40,220
So 1,486 firmware have file has an Etsy secure TTY in it with 196 approved TTY locations for root to log in on.

368
00:34:40,680 --> 00:34:45,080
All right, so I tracked this one down because this file is just like, where is this anomaly from?

369
00:34:45,080 --> 00:34:53,980
I found a Yocto project, which is a way to programmatically or to easily generate a Linux operating system custom build for your project.

370
00:34:54,240 --> 00:34:58,080
And this particular file that I found was from 2013.

371
00:34:58,080 --> 00:35:09,580
So here is a supply chain, someone who's building and wants a secure TTY, they pull down this file that's pretty permissive, probably for development purposes, and not really supposed to be for production.

372
00:35:09,720 --> 00:35:13,140
And yet here it makes its way into all these devices.

373
00:35:13,220 --> 00:35:15,500
So here's the supply chain in action.

374
00:35:15,500 --> 00:35:17,700
All right, number of valid login shells.

375
00:35:17,700 --> 00:35:20,300
Okay, so this is an Etsy shells.

376
00:35:20,300 --> 00:35:30,040
This is the number of acceptable different types of shells that you can log in from, and most have one.

377
00:35:30,040 --> 00:35:38,380
So again, whether you should have a shell that people can log into or not, that's questionable, but at least a lot of them, majority of them, have only one.

378
00:35:38,380 --> 00:35:41,880
But then we have this 10 login shells.

379
00:35:41,880 --> 00:35:53,620
And I guess what they want by this, a little tongue in cheek, is, you know, if their hackers are coming in and are used to have all their key bindings in their environment files for bash, they want to make it easier for them.

380
00:35:53,620 --> 00:35:55,340
So they're just going to throw in bash in there.

381
00:35:55,340 --> 00:35:58,700
And if you're ash or zsh, we got you covered.

382
00:35:58,700 --> 00:36:00,740
You can log in however you want.

383
00:36:00,960 --> 00:36:05,060
Again, a little tongue in cheek, but that's what that number represents.

384
00:36:05,580 --> 00:36:20,880
An additional thing here that was kind of concerning is when I got delved into the actual types of devices that this was, it was over 60 of these were models of patient monitoring health systems.

385
00:36:21,040 --> 00:36:37,840
Okay, so over 60 different models, which represents however many thousands or hundreds of thousands of health systems, patient monitors that are sitting bedside, have 10 or so login shells that somebody could log in with, right?

386
00:36:37,840 --> 00:36:40,640
These are just configuration examples in the supply chain.

387
00:36:40,640 --> 00:36:46,580
I'm not saying these are actual vulnerabilities, they're just interesting anomalies from the supply chain.

388
00:36:46,580 --> 00:36:48,820
Okay, so here's a questionable config.

389
00:36:49,260 --> 00:36:50,560
We're about out of time.

390
00:36:50,560 --> 00:36:52,060
So we're just going to go through these quick.

391
00:36:52,360 --> 00:36:59,060
Firmware where the root user has a login shell, 15,345.

392
00:36:59,120 --> 00:37:04,420
All right, so 30% of all of our firmware that we looked at has some sort of login shell.

393
00:37:04,420 --> 00:37:08,840
Whereas if you don't want the root to log in, use sbin login, guess how many?

394
00:37:08,840 --> 00:37:11,140
Over under one.

395
00:37:11,140 --> 00:37:14,140
We found one firmware that actually had sbin no longer in there.

396
00:37:14,140 --> 00:37:16,200
Okay, firmware with keys and authorized keys.

397
00:37:16,200 --> 00:37:26,720
Now this is a bad back because this basically says if someone SSHs into your box, if your key matches their key in the authorized keys, you get to log in without any password.

398
00:37:26,720 --> 00:37:28,440
So this is a classic backdoor.

399
00:37:28,440 --> 00:37:32,560
Whether it was purposefully put there or not, what's the over under?

400
00:37:32,980 --> 00:37:35,040
175, 175 firmware.

401
00:37:35,040 --> 00:37:38,840
Remember this is firmware, however many devices this represents.

402
00:37:39,240 --> 00:37:43,040
This is 12 different vendors with these keys.

403
00:37:43,040 --> 00:37:43,760
All right.

404
00:37:43,760 --> 00:37:54,180
And over 29, these were really interesting known hosts to see where these kind of came from and the different medical facilities and different places that had known hosts coming from them.

405
00:37:54,180 --> 00:38:01,340
Clam AV, only four firmware had any kind of open antivirus software.

406
00:38:01,340 --> 00:38:04,860
Firmware running HTTPD with mod auto index.

407
00:38:05,000 --> 00:38:17,800
So this auto index is a way to auto index files on your web browser or your web server when there isn't any kind of match for the HTML name.

408
00:38:17,800 --> 00:38:21,220
So it's a great way to kind of troll through your directories.

409
00:38:21,220 --> 00:38:27,120
So these are, again, things that are questionable that are in our configs.

410
00:38:27,120 --> 00:38:30,100
Firmware starting TTY and init tabs.

411
00:38:30,100 --> 00:38:36,180
So quite a few firmware are starting TTY, but not that many distinct files.

412
00:38:36,180 --> 00:38:45,920
So here we're seeing amplification in the supply chain of here's 63 files, but these 63 files are found in 4,729 devices.

413
00:38:46,140 --> 00:38:47,680
Here's a particularly bad one.

414
00:38:47,680 --> 00:38:50,520
Firmware with PHP to default to display errors.

415
00:38:51,120 --> 00:38:52,600
332 firmware.

416
00:38:52,600 --> 00:39:04,720
So using SQL injection or any kind of command injection, I love when developers leave on their display errors, because when I made a mistake, it tells me exactly what's there.

417
00:39:04,720 --> 00:39:08,400
Problem is, PHP by default allows display errors.

418
00:39:08,400 --> 00:39:10,320
You have to explicitly turn it off.

419
00:39:10,320 --> 00:39:17,120
So maybe we should be making software that doesn't have defaults that are vulnerable like this in their default configuration.

420
00:39:17,640 --> 00:39:18,240
All right.

421
00:39:18,240 --> 00:39:19,180
Next part.

422
00:39:19,180 --> 00:39:22,160
Firmware with insecure default DES encryption.

423
00:39:22,160 --> 00:39:26,960
So if you don't specify the type of encryption for your passwords, you're going to use DES, which is a bit weaker.

424
00:39:27,260 --> 00:39:29,660
318 firmware, 29 files.

425
00:39:29,660 --> 00:39:36,600
But even worse, there were firmware that specified the flag to turn on MD5 as their password tripped.

426
00:39:36,600 --> 00:39:37,600
It was 12 firmware.

427
00:39:37,600 --> 00:39:40,100
Fortunately, it's small, but there's still 12 firmware.

428
00:39:40,100 --> 00:39:43,220
I have no idea where those are at, but they're out there.

429
00:39:43,220 --> 00:39:46,420
Firmware with a master.password with no password on root.

430
00:39:46,420 --> 00:39:53,760
So in the password file, there is no over, under, on 100 devices, 10 devices, a million devices.

431
00:39:53,760 --> 00:39:54,840
Now only seven.

432
00:39:54,840 --> 00:39:58,960
But still, one file made its way in with no password on the root.

433
00:39:59,380 --> 00:40:01,360
MySQL binding to 000.

434
00:40:01,600 --> 00:40:06,240
So listen to the world and allow people to go right to your MySQL.

435
00:40:06,260 --> 00:40:07,700
26 devices.

436
00:40:08,100 --> 00:40:09,480
Firmware with Redis, the same way.

437
00:40:09,480 --> 00:40:10,520
Default world bind.

438
00:40:10,520 --> 00:40:14,060
So if you don't specify a bind by default, it will allow you in.

439
00:40:14,060 --> 00:40:14,660
Three.

440
00:40:14,660 --> 00:40:15,540
It's only one file.

441
00:40:15,540 --> 00:40:19,100
But still, three firmware allow you into their Redis by default.

442
00:40:20,200 --> 00:40:22,420
Average number of unsafe function calls per firmware.

443
00:40:22,420 --> 00:40:39,200
So this one is any unsafe function calls is stuff like stir copy, mem copy, all those unbounded ones like ripple 20 found and stuff like malicious ways to use standard allocation functions without checking links and stuff like that.

444
00:40:39,200 --> 00:40:41,840
How many do you think in per firmware?

445
00:40:42,320 --> 00:40:48,300
1,500 and some average unsafe function calls per firmware.

446
00:40:48,340 --> 00:40:48,820
Okay?

447
00:40:48,820 --> 00:40:51,520
Number of firmware with unsafe function calls?

448
00:40:51,520 --> 00:40:52,600
23,000.

449
00:40:52,600 --> 00:40:53,880
That's over 50%.

450
00:40:53,880 --> 00:40:55,660
And those are the ones that we've named.

451
00:40:55,660 --> 00:41:04,220
We haven't even trolled through those with function hashing and stuff to find the unlabeled ones that also call these.

452
00:41:04,220 --> 00:41:04,720
All right?

453
00:41:04,780 --> 00:41:05,340
Firmware.

454
00:41:05,340 --> 00:41:06,500
Here's the last one.

455
00:41:06,500 --> 00:41:09,120
Firmware exporting NFS mounts to the world.

456
00:41:09,120 --> 00:41:09,860
All right?

457
00:41:09,940 --> 00:41:11,400
Over, under on this one?

458
00:41:11,720 --> 00:41:12,460
42.

459
00:41:12,460 --> 00:41:25,760
There are four files that contribute to 42 firmware mounts to the world, such as slash root, mount a star, so you can as root write to root as user ID zero.

460
00:41:25,980 --> 00:41:30,580
So these are all questionable configs that have made it their way, just fun things that I found trolling through.

461
00:41:30,580 --> 00:41:38,100
And finally, what would a talk like this be without talking about the common passwords and all that?

462
00:41:38,280 --> 00:41:51,320
So you see here amplified here, again, is the supply chain where you have the file counts of all of this, sort of the light orange, and the dark orange is how many firmware use those files in them.

463
00:41:51,600 --> 00:41:54,060
And the number one being admin.

464
00:41:54,060 --> 00:41:56,460
It's only in five files in the password file.

465
00:41:56,460 --> 00:42:07,220
But those five password files are found in 470 firmware, which may represent hundreds of thousands of devices where we've cracked the password admin.

466
00:42:07,220 --> 00:42:13,100
I'm going to give you ten to one guess what the password is going to be for that admin user.

467
00:42:13,420 --> 00:42:14,000
All right.

468
00:42:14,000 --> 00:42:18,340
So anyways, in conclusion, it's all about the software bill of materials.

469
00:42:18,340 --> 00:42:21,080
Drive towards generating and validating these.

470
00:42:21,080 --> 00:42:22,340
This is the path forward.

471
00:42:22,340 --> 00:42:30,980
Manufacturing, implementing software inventory systems, developers being diligent and reporting all the components used in their systems.

472
00:42:30,980 --> 00:42:40,280
Consumers having a high demand or having a high standard in transparency and favor companies and buying from companies with that info as well.

473
00:42:40,480 --> 00:42:46,200
Policymakers who help guide these unified standards and reporting and formats.

474
00:42:46,580 --> 00:42:52,280
And our security community here developing tools to assist in the automation of the inventory.

475
00:42:52,480 --> 00:42:55,980
If we're doing the same thing now in five years, we're going behind.

476
00:42:55,980 --> 00:42:57,540
We should be using machine learning.

477
00:42:57,540 --> 00:43:11,000
We should be using tools and other things to help us inventory and validate these things, not just on the front end, but on the back end to validate patches and updates are what they say they are and they contain what they think they contain.

478
00:43:11,180 --> 00:43:17,020
So thanks to all my team at finite state and a lot of good research that's been in the industry.

479
00:43:17,980 --> 00:43:27,160
Finally, obviously the question and answer on this session are going to the discord, but I've had a fun time talking to you all.

480
00:43:27,480 --> 00:43:28,680
Peace out.

481
00:43:28,680 --> 00:43:32,440
Hope you enjoy the rest of DEF CON and IoT Village.

482
00:43:32,440 --> 00:43:33,740
Thank you so much for having me.