1
00:00:00,000 --> 00:00:06,600
Hello, everyone. And, yeah, I see more people than I expected. That's kind of surprising

2
00:00:06,940 --> 00:00:13,760
for the BMC talk because it means people know what the BMC is. So, well, first of all, thank

3
00:00:13,760 --> 00:00:20,080
you. My name is Adam and together with Alex, we have a pleasure to be here. So, thanks

4
00:00:20,080 --> 00:00:25,400
DEF CON organizer for allowing us to be here today, first time in the row as a speaker.

5
00:00:25,400 --> 00:00:30,940
And thanks for you, great audience, that you know what the BMC is. So, really, thank you,

6
00:00:30,960 --> 00:00:38,680
guys. So, what is our talk about? Our talk is about breaking BMC, the forgotten

7
00:00:38,680 --> 00:00:46,360
kingdom. So, yeah, it's a niche topic. But apparently it's not as niche as I want it to be.

8
00:00:47,860 --> 00:00:52,440
So, to tell us about ourselves, we don't want to spend too much time here because it's boring

9
00:00:52,440 --> 00:00:55,380
stuff. Both of us, me and Alex, we're going to be talking about the BMC. We're going to be

10
00:00:55,380 --> 00:01:00,260
working in NVIDIA doing this research. And everything when we're done, it was apparently as

11
00:01:00,260 --> 00:01:05,560
part of our work at NVIDIA. We don't want to spend too long here. There's some private

12
00:01:05,560 --> 00:01:10,360
contact information to us. You can always poke us through the e-mail or through the Twitter. If

13
00:01:10,360 --> 00:01:14,840
you have any questions, just feel free to shoot us in the message. And there's some short bio.

14
00:01:15,040 --> 00:01:20,840
But it doesn't really matter. What matters, this is not only the research for two of us, but

15
00:01:20,840 --> 00:01:25,360
for entire NVIDIA offensive security research team which I'm currently leading. So, we're

16
00:01:25,380 --> 00:01:28,980
going to be talking about the BMC. So, big kudos also to Jared, Max, and Nicola who also did

17
00:01:28,980 --> 00:01:34,840
this research together with us. So, first in first, what is BMC? What I expect you to know, it's

18
00:01:34,840 --> 00:01:40,920
a baseboard management controller. It's a very special service processor which you can see.

19
00:01:41,060 --> 00:01:47,400
So, that is an old picture from 2006. We didn't have time to take a new one. So, we just make a

20
00:01:47,400 --> 00:01:54,180
small adjustment to look at more modern. So, there is 2022 here. But physically it doesn't change

21
00:01:55,380 --> 00:02:03,140
anything. And we did this research in 2022 for one of the BMC from one of the vendor which we are

22
00:02:03,140 --> 00:02:09,940
not going to mention, but it's visible in the picture. And so, what physically BMC does, it's

23
00:02:09,940 --> 00:02:15,600
apparently used to control the data centers, especially the hosts. And the reason why, because

24
00:02:15,600 --> 00:02:21,360
you can imagine there could be like 100 of the host machine on the data center, 1,000 host

25
00:02:21,360 --> 00:02:25,220
machine, and sometimes even 100 of the 1,000 of the machine. Some of them are not even

26
00:02:25,380 --> 00:02:29,740
accessible physically because they might be somewhere in the water. So, it's very difficult to go

27
00:02:29,740 --> 00:02:35,280
there and do anything physically. Sometimes they still do. So, you can still go physically to this

28
00:02:35,280 --> 00:02:39,400
machine and do whatever you need to do. But, of course, as you can imagine, it's not a scalable

29
00:02:40,800 --> 00:02:45,640
solution. So, that's why we do have BMCs that can do it for us remotely. They can do it

30
00:02:46,600 --> 00:02:52,740
everything what you can physically do by yourself. So, it's essentially a specialized service

31
00:02:52,740 --> 00:02:55,360
processor that can monitor the physical system. So, that's why we have BMCs that can monitor the physical

32
00:02:55,380 --> 00:03:01,820
state of the computer, not only computer, but any network server, hardware, and they're using

33
00:03:01,820 --> 00:03:06,800
sensors, essentially, to be able to communicate to the administrator of this device. What is also

34
00:03:06,800 --> 00:03:11,960
interesting, they allow you to do a full control of the machine which they monitor. So, you

35
00:03:11,960 --> 00:03:18,260
could have something like a KVM capability. So, you can go remotely to the BMC and you can then

36
00:03:18,260 --> 00:03:22,880
reconfigure the host. You can go to the BIOS. You can change the BIOS settings if you want to

37
00:03:22,880 --> 00:03:25,360
remotely. You can do whatever you want, essentially. So, you can do whatever you want, essentially.

38
00:03:25,380 --> 00:03:31,660
Even more, you can flash the entire firmware of the controlled device. And what is even more

39
00:03:31,660 --> 00:03:37,000
funny, you can do it even when the device is powered off. So, BMC, as you can see, is a very

40
00:03:37,000 --> 00:03:43,960
powerful service processor. So, that's why it's a very interesting target for the attacker

41
00:03:43,960 --> 00:03:50,100
because it gives, at first, tons of the benefits. And also, it falls into the category like hack

42
00:03:50,100 --> 00:03:55,280
one, rule them all. You just hack one BMC and you can have control over entire data

43
00:03:55,280 --> 00:04:00,940
center which BMC physically controls. So, it's like a great and juicy target. And Alex and I

44
00:04:00,940 --> 00:04:09,280
will talk about the first issue which we want to cover here. BMC speaks IPMI protocol. And when

45
00:04:09,280 --> 00:04:13,640
you encounter a system that speaks IPMI, what would you check for? Of course, for a hash leak

46
00:04:13,640 --> 00:04:21,200
that has been known since 2013. So, what is this? This is a screenshot from a manual that

47
00:04:21,200 --> 00:04:25,260
describes the IPMI standard. And this is an explanation of the IPMI standard. And this is an

48
00:04:25,280 --> 00:04:30,380
explanation of how the authentication works in IPMI. So, you see, here, there are two

49
00:04:30,380 --> 00:04:35,820
messages. One of those messages is from the administrator console to BMC. And the

50
00:04:35,820 --> 00:04:43,420
administrator is only supposed to send the username to the BMC. BMC looks at the username and

51
00:04:44,000 --> 00:04:49,140
calculates the H magical language are some data which includes the username. So, the

52
00:04:49,140 --> 00:04:55,260
password for the user that is about to be authenticated is used as a key to the H device encryption. So if

53
00:04:55,280 --> 00:05:02,540
HMAC calculation. So this HMAC calculation result is sent back to the administrator.

54
00:05:03,160 --> 00:05:10,800
So if an attacker knows the correct username that exists in a database for BMC users,

55
00:05:12,240 --> 00:05:19,420
he can receive the hash calculated and the password will be used as a key to this HMAC.

56
00:05:20,280 --> 00:05:25,520
And this has been known since 2003 where this standard has been published.

57
00:05:26,320 --> 00:05:34,760
So you might think is it fixed by now in 2023? No, actually it's not because it's in the standard.

58
00:05:35,020 --> 00:05:42,020
So if you speak IPMI, you have to do the same way. The authentication process looks exactly the same.

59
00:05:42,180 --> 00:05:47,660
And we implemented the script just to test this and we found out that

60
00:05:49,420 --> 00:05:55,500
actually the system that we were looking at conforms to the same standard. But we had a

61
00:05:55,500 --> 00:06:02,660
problem while figuring out the correct username that we can use because, you know, if we don't

62
00:06:02,660 --> 00:06:08,760
know the username, any username that exists in the user database, we will not get this

63
00:06:09,440 --> 00:06:17,100
hash back. So we had a problem here. We don't have a valid username. How can we guess it or how can we

64
00:06:18,600 --> 00:06:19,400
find out?

65
00:06:19,420 --> 00:06:27,200
This correct username, at least one. So you see the server always responds with an unauthorized

66
00:06:27,200 --> 00:06:40,840
name for whatever name it's right. So we had this idea of looking for some issues with incorrect

67
00:06:41,660 --> 00:06:47,200
processing of this user sensitive data. For example, the username. And we found out a bug

68
00:06:47,200 --> 00:06:49,400
that is actually present on this server. And we found out a bug that is actually present on this

69
00:06:49,400 --> 00:06:55,380
system. And the CVs for this bug are issued by AMI and NVIDIA. You see it here.

70
00:06:56,880 --> 00:07:03,360
So what exactly is happening? So when a user sends a username to the BMC

71
00:07:03,360 --> 00:07:09,620
in order to authenticate, the length of the username is also sent in the same message.

72
00:07:09,960 --> 00:07:19,380
And we had an idea that maybe the server processes this username in non-constant time.

73
00:07:19,400 --> 00:07:25,020
For example, if it compares the username with some database entries, maybe it uses mem compare.

74
00:07:25,220 --> 00:07:32,540
Who knows? And we implemented a script that measures the time that it takes for a certain

75
00:07:32,540 --> 00:07:43,500
message to be processed by the server, by the BMC. And eventually, we got this result.

76
00:07:43,940 --> 00:07:45,740
Let me show you the demo.

77
00:07:52,120 --> 00:07:57,160
So we're sending a bunch of message ones, which are defined in the standard.

78
00:07:57,420 --> 00:08:04,340
And we are basically doing a brute force for the printable characters as user names,

79
00:08:04,480 --> 00:08:10,940
available for user names. But we use this time in Oracle to greatly speed up the brute force.

80
00:08:11,360 --> 00:08:19,360
You see, we begin with measuring the time that it takes for the server to process the user names. We

81
00:08:19,360 --> 00:08:27,340
are only one character long. And for all like 200 possible values for this character, we measure

82
00:08:27,340 --> 00:08:34,480
the time that it takes for the server to process it. And we find the longest time. So we find the

83
00:08:34,480 --> 00:08:40,400
symbol that takes the longest time to process. And it is very likely that this is the correct

84
00:08:40,400 --> 00:08:45,260
symbol, the first symbol of the user name. Because when the mem compare advances to the next symbol,

85
00:08:45,360 --> 00:08:48,120
it takes a little bit more time to process. So this is...

86
00:08:48,120 --> 00:08:56,600
This is why we can use the first guest symbol and advance to the next one. So essentially we do

87
00:08:56,600 --> 00:09:02,480
this in a sequence and we don't brute force the whole key space. We just move from one symbol to

88
00:09:02,480 --> 00:09:10,760
another. And do it a few times in a row just to get better data to measure the response time.

89
00:09:10,980 --> 00:09:17,920
But essentially this works really well on local networks. And you see here, we were

90
00:09:18,120 --> 00:09:24,060
able to guess the right user name. And eventually when we send this user name to authentication

91
00:09:24,060 --> 00:09:30,560
server, we get back the hash, the H mark calculation result. And we just brute force it using

92
00:09:31,120 --> 00:09:37,700
normal methods offline. So this attack can be happening completely offline. So here we use a

93
00:09:37,700 --> 00:09:44,040
word list and we are able to guess the right password. This is the explanation why this works.

94
00:09:44,120 --> 00:09:48,080
So by this moment, we didn't have access to any binaries for the

95
00:09:48,120 --> 00:09:55,400
BMC. We didn't have any access to source code. So this is just a picture that explains to you why

96
00:09:55,400 --> 00:10:01,140
this attack works. You see this mem compare. This is the usual mem compare from libc. It does not

97
00:10:01,140 --> 00:10:06,100
execute in a constant time. This is why it can be used as a timing oracle here. So if you look up

98
00:10:06,100 --> 00:10:11,660
the man page for mem compare, it says don't use it for sensitive data. And this is sensitive data.

99
00:10:12,660 --> 00:10:17,960
So now we have access to the BMC. We guess the user name and the password. So we

100
00:10:18,120 --> 00:10:25,580
only got one user information from the BMC. But we can talk to it. We can issue a request that

101
00:10:25,580 --> 00:10:26,320
requires authentication.

102
00:10:29,820 --> 00:10:35,940
So, sure, as you can see, to put a bit more to the story, we just saw a BMC. We know it's there. We

103
00:10:35,940 --> 00:10:42,160
know there is a known bug from 2006 with the CV from 2013, which we didn't expect to work. But it

104
00:10:42,160 --> 00:10:47,520
works. So interesting. We wanted to log in. It was just pure guess. Like, okay, let's try timing

105
00:10:48,120 --> 00:10:53,820
why not? Like many devices are vulnerable and turns out we have CVE. So, okay, we log

106
00:10:53,820 --> 00:10:58,100
in, one of the devices which VMC managed looks like this is a screenshot of the web interface.

107
00:10:58,480 --> 00:11:03,060
So looks cool. So why are we even speaking more about that? Like, okay, we have access

108
00:11:03,060 --> 00:11:08,840
to VMC, so what? So that's exactly the question. What to do next? And we have a couple of ideas.

109
00:11:09,180 --> 00:11:14,860
We could stop presentation here, but we did not and we did a bit more research. And the

110
00:11:14,860 --> 00:11:20,120
reason why, because at first we wanted to understand a lot of stuff about VMC which we did

111
00:11:20,120 --> 00:11:24,220
not at first. Why this timing protocol at first worked? Because we did not know why this

112
00:11:24,220 --> 00:11:28,440
timing given works. We wanted to find the VMC image, we wanted to reverse this VMC image and

113
00:11:28,440 --> 00:11:33,080
to be able to understand why this timing attack works at first place. And to be able to do that,

114
00:11:33,200 --> 00:11:39,060
we need to have somehow access to the VMC image, yes? We also thought, okay, if the timing

115
00:11:39,060 --> 00:11:44,500
protocol works, maybe there is more bugs, yes? Why not? So we need to have image to be able to

116
00:11:44,500 --> 00:11:50,660
find more bugs there. And also, again, we did not know too much at the time about internal

117
00:11:50,660 --> 00:11:56,340
architecture of the VMC. So we wanted to learn more, like, what is apparently there inside of this

118
00:11:56,340 --> 00:12:01,820
VMC excluding this timing attacks which we found. And also we wanted to analyze, okay, if maybe

119
00:12:01,820 --> 00:12:06,840
there is some other bugs which are good to exploit, what are the difficulties of doing that?

120
00:12:06,980 --> 00:12:11,560
Like, are there any kind of defense or hardening or mitigation techniques? What is the state of

121
00:12:11,560 --> 00:12:14,480
heart of the security of the VMC image? And to be able to do that, we need to have some

122
00:12:14,500 --> 00:12:22,580
how get this VMC image, yes? So how do you get the VMC image? Yes, that's the question. So we

123
00:12:22,580 --> 00:12:29,040
had an idea, like, VMC still has a firmware, yes? So this firmware must be somehow updated. So if

124
00:12:29,040 --> 00:12:34,640
people update the firmware, how many of people, like, delete the updates after they update the

125
00:12:34,640 --> 00:12:39,620
firmware? So we had this idea, maybe some of the host machine will have the, you know, not

126
00:12:39,620 --> 00:12:44,480
deleted updates for the firmware. So let's get now, because we're having the VMC, let's get the

127
00:12:44,500 --> 00:12:50,520
VMC access to the VMC, and we know that the VMC has, like, capabilities of KVM. Let's go from the

128
00:12:50,520 --> 00:12:55,720
VMC to the host and grab the updates for the VMC image, and then we can have ability to analyze

129
00:12:55,720 --> 00:13:01,860
that. So the question, how do we go from the VMC to the host and get the shell? So, of course,

130
00:13:01,880 --> 00:13:06,760
it's not as difficult because we have KVM capabilities, so let's go very old school techniques,

131
00:13:06,920 --> 00:13:13,740
change the bootloader commands and put in it binbash. Of course, sometimes might not that feature

132
00:13:13,740 --> 00:13:14,040
works.

133
00:13:14,040 --> 00:13:18,780
Because you might have secure boot and maybe the commands are assigned, et cetera, but again, we

134
00:13:18,780 --> 00:13:24,540
are in the KVM. So why not to go to the BIOS if there is no password and disable secure boot? We

135
00:13:24,540 --> 00:13:30,020
are in VMC. So there is a lot of opportunities to do so, so we went to the least difficult path,

136
00:13:30,220 --> 00:13:35,540
just changed the command line, binbash, and we end up there. So, yeah, we have shell. And the

137
00:13:35,540 --> 00:13:40,520
disk somehow is not encrypted. Surprise. So we are able now to go and see what is in the host

138
00:13:40,520 --> 00:13:44,020
machine. But then we also realize we are also able to, you know, we are able to, you know,

139
00:13:44,020 --> 00:13:48,180
to blade in the host machine and to mount whatever image we want. So even if the binbash won't work,

140
00:13:48,260 --> 00:13:53,600
you can do a booting attack, disable secure boot and just load our own image if you want to.

141
00:13:53,820 --> 00:13:59,160
But we combine two of these features, apparently. We just put in binbash, half a shell, and we

142
00:13:59,160 --> 00:14:04,500
just mount our tool set to be able to run the commands on the host machine which are not

143
00:14:04,500 --> 00:14:09,160
installed on the specific host. So what commands we wanted to run? So we found the VMC image,

144
00:14:09,320 --> 00:14:14,000
but we don't know how this firmware image looks like, what is inside. So we wrote a new VMC image,

145
00:14:14,000 --> 00:14:20,700
custom tool to be able to pass this image. As you can see here, we found interesting stuff,

146
00:14:20,920 --> 00:14:28,080
like there is essentially model bootloader, model JFFS2, there is model in CRAMFS and one of

147
00:14:28,080 --> 00:14:32,480
them is a root partition as you can see here. And this root partition starts from the

148
00:14:32,480 --> 00:14:39,640
address 2500,000. So what we did, we just very simple stuff. We know exactly how the header

149
00:14:39,640 --> 00:14:43,720
looks like. We know what is inside of this firmware because we just passed that. So we

150
00:14:43,720 --> 00:14:50,180
just do DD, dump all of this byte data and remote into the loopback and now we can able to

151
00:14:50,180 --> 00:14:58,100
browse the BMC and analyze that. So what did we find there? We found at first that BMC includes

152
00:14:58,100 --> 00:15:04,620
the IPMI server binary and libraries. And so what we saw in this interface on the screenshot

153
00:15:04,620 --> 00:15:10,460
which I showed you before, is just a very small subset of all of the APIs which are capabilities

154
00:15:10,460 --> 00:15:13,580
and functionalities which BMC physically can provide.

155
00:15:13,720 --> 00:15:18,620
So it's a very small subset of the feature. And then we started to analyze on the BMC, there's

156
00:15:18,620 --> 00:15:23,460
like a tons of other protocols which are apparently communicating with the BMC and you can poke.

157
00:15:23,720 --> 00:15:29,200
So then we found out that you can just directly send the IPMI messages to the BMC and bypass

158
00:15:29,200 --> 00:15:33,200
all of these web interfaces which they have, which they expose, and you have like a tons

159
00:15:33,200 --> 00:15:36,620
of the functionality. There is tons of the hidden API which is accessible, especially

160
00:15:36,620 --> 00:15:40,960
if you have a username and password, that you can just poke. So as you can imagine, the

161
00:15:40,960 --> 00:15:42,500
attack surface is pretty huge.

162
00:15:43,720 --> 00:15:48,160
So we pull a trigger to find out, let's find out more bugs. And this is one of the bugs

163
00:15:48,160 --> 00:15:49,560
which now Alex will cover as well.

164
00:15:51,720 --> 00:15:57,120
Yeah, we started to poke to this firmware image and found another issue that is identified

165
00:15:57,120 --> 00:16:03,280
by these CVs by MI and NVIDIA. Essentially, we found out that the user database that is

166
00:16:03,280 --> 00:16:10,700
stored and that is implemented in this BMC is managed by Redis database. And what is

167
00:16:10,700 --> 00:16:13,320
more interesting, the passwords,

168
00:16:13,720 --> 00:16:18,700
are encrypted in this database. So it's stored in an encrypted state, but the keys which

169
00:16:18,700 --> 00:16:24,320
are used to encrypt this password are stored right next to the passwords. So it doesn't

170
00:16:24,320 --> 00:16:31,560
make much sense, but the question is, why do they even have this obfuscated passwords

171
00:16:31,560 --> 00:16:37,900
in the database? Why do not hash the passwords instead? So the answer to that is the same

172
00:16:37,900 --> 00:16:43,700
issue with IPMI protocol. They need these plain text passwords to be able to encrypt

173
00:16:43,720 --> 00:16:44,420
the passwords. They need these plain text passwords to be able to authenticate users.

174
00:16:44,940 --> 00:16:53,260
That's why. So we found a really nice API that can be accessed from the host, which is a local

175
00:16:53,260 --> 00:17:01,420
access, considered local, which is not accessible remotely. So this API allows you to do

176
00:17:01,420 --> 00:17:09,600
specific requests to this Redis database. So we analyzed the image and found the correct keys,

177
00:17:09,780 --> 00:17:13,700
the names of the keys that has to be accessed in order to read this password. So we found a lot of

178
00:17:13,720 --> 00:17:17,360
passwords and the user names in the keys, and we implemented the proof of concept that just

179
00:17:17,360 --> 00:17:23,520
accesses this user database through this API and reads the encrypted passwords and then the keys

180
00:17:23,520 --> 00:17:31,120
from the same database and decrypts the passwords. So at this point, we got a full user database

181
00:17:31,120 --> 00:17:40,800
for the BMC, so user names and the passwords in a decrypted form. So we proceeded to analyze

182
00:17:40,800 --> 00:17:43,700
the firmware. We found that the user names and the passwords in a decrypted form. So we

183
00:17:43,720 --> 00:17:49,720
found another issue, which is also interesting, which I will talk about. This is also

184
00:17:49,720 --> 00:17:57,600
identified by these CVEs from IMI and NVIDIA. So there is a server that is actually an IPMI

185
00:17:57,600 --> 00:18:03,800
server that listens on a socket on the BMC site, and services all these requests that come

186
00:18:03,800 --> 00:18:13,700
remotely. And this particular API allows you to read any virtual memory via this server. So this

187
00:18:13,720 --> 00:18:21,740
is within the context of this process, of the server process. So there is special command

188
00:18:21,740 --> 00:18:30,600
that prepares some data, so it has to be issued before you access this API, but also you only

189
00:18:30,600 --> 00:18:34,900
need to have the user name and password, the usual user name and password for the BMC in order

190
00:18:34,900 --> 00:18:39,560
to issue this command. You cannot do it without any authentication. But at this point, we

191
00:18:39,560 --> 00:18:43,700
already have the user name and password, so we can just proceed. As you see here, we have

192
00:18:43,720 --> 00:18:51,400
no checks whatsoever about the region or the size of the data that we want to read. You can

193
00:18:51,400 --> 00:18:56,640
just read anything from this context of this IPMI server. And this is how we found out that,

194
00:18:56,660 --> 00:19:03,460
for example, the server always uses the same base, I mean, it's executed at the same base

195
00:19:04,120 --> 00:19:11,340
every time you boot the BMC, so there is no SLR. And you see here, there are no checks for the

196
00:19:11,340 --> 00:19:11,680
sizes.

197
00:19:13,720 --> 00:19:19,060
And this is the counterpart of this read API. This one writes arbitrary data to this context

198
00:19:19,060 --> 00:19:25,780
of this IPMI server process. So we can use it to write any data or code that we want to this

199
00:19:25,780 --> 00:19:32,340
server process. So that is very useful if you want to build a shell code inside the process.

200
00:19:33,020 --> 00:19:40,000
So at this point, we could use this API to execute any code we want in the context of this

201
00:19:40,000 --> 00:19:42,840
IPMI server process. This is how you...

202
00:19:43,720 --> 00:19:49,240
The actual override happens. You see there is a usual mem copy operation, also without any

203
00:19:49,240 --> 00:19:55,640
checks for the address and the size. And this is the proof of concept for this particular issue.

204
00:19:56,520 --> 00:20:05,840
First, we identify that the server process is actually... Is located at 10,000 hex, so there is

205
00:20:05,840 --> 00:20:11,620
no SLR, but we know the base of this module, so we can just read arbitrary data from it. And we

206
00:20:11,620 --> 00:20:13,700
are interested in reading special errors. So we can just read arbitrary data from it, and we are interested in reading special errors.

207
00:20:13,720 --> 00:20:17,760
We can also see that the array of the configuration array for this features available in this

208
00:20:17,760 --> 00:20:24,060
server, we... Which are controlled by this G-core features array. And in particular, we are

209
00:20:24,060 --> 00:20:31,620
interested in this offset 310... 310 hex. And if we enable the... If we set

210
00:20:31,620 --> 00:20:38,600
a bit at this location, the server now thinks that certain functionality has been enabled.

211
00:20:38,820 --> 00:20:43,340
And this functionality allows you to download arbitrary files from BMC.

212
00:20:43,460 --> 00:20:43,700
So, that is...

213
00:20:43,720 --> 00:20:47,460
By default this is disabled but since we have a write primitive we can now enable it.

214
00:20:47,580 --> 00:20:52,440
And we use this functionality to test and and download ETC Shadow.

215
00:20:52,700 --> 00:20:59,180
Why not? And you see that with dot dot slash dot dot slash string we were able to do it.

216
00:20:59,460 --> 00:21:03,640
So Adam will talk about about this download functionality in detail.

217
00:21:05,560 --> 00:21:09,960
So yeah, first No SLR to recap and the second thing what we wanted to recap so

218
00:21:09,960 --> 00:21:12,060
So we found now read and write capabilities

219
00:21:12,060 --> 00:21:15,580
to entire IPMI process and BMC server,

220
00:21:15,760 --> 00:21:17,240
so essentially we can enable and disable

221
00:21:17,240 --> 00:21:18,220
whatever feature we want.

222
00:21:18,480 --> 00:21:20,060
Of course, why wanna do it?

223
00:21:20,140 --> 00:21:21,580
Because you can still exploit that,

224
00:21:21,720 --> 00:21:23,380
but yeah, we just wanted to find out

225
00:21:23,380 --> 00:21:24,700
what other hidden features we have.

226
00:21:25,160 --> 00:21:26,920
So we identified this CV,

227
00:21:27,820 --> 00:21:29,500
which is assigned by AMI and NVIDIA,

228
00:21:29,620 --> 00:21:31,160
which as Alex mentioned,

229
00:21:31,940 --> 00:21:34,620
the feature itself is nothing wrong with the feature, yes?

230
00:21:34,780 --> 00:21:37,340
You have a feature who allows you to download just a file.

231
00:21:37,480 --> 00:21:38,900
Sometimes you might want to do it, why not?

232
00:21:38,900 --> 00:21:41,660
But the way how the program implemented that,

233
00:21:41,720 --> 00:21:44,060
it's again, old school 90s way

234
00:21:44,060 --> 00:21:45,800
of implementing the one in the file.

235
00:21:46,040 --> 00:21:47,300
So as you can see here,

236
00:21:47,880 --> 00:21:49,640
if you at first enable this feature,

237
00:21:49,860 --> 00:21:50,620
this is the offset,

238
00:21:51,560 --> 00:21:52,800
then this feature is available,

239
00:21:53,040 --> 00:21:53,720
and if it's available,

240
00:21:53,980 --> 00:21:56,380
they construct the path you can see on the bottom here.

241
00:21:56,540 --> 00:21:58,060
So the get file path is static,

242
00:21:58,320 --> 00:22:00,720
but the file path, but file name,

243
00:22:00,880 --> 00:22:02,780
directly comes from the package from the user.

244
00:22:03,080 --> 00:22:04,540
So it's coming from the user,

245
00:22:04,640 --> 00:22:06,020
of course, you just do very old school

246
00:22:06,020 --> 00:22:08,000
dash dash dot, dash dot slash,

247
00:22:08,140 --> 00:22:08,620
dash dot slash,

248
00:22:08,620 --> 00:22:10,700
and that's how we achieve ability

249
00:22:10,700 --> 00:22:13,300
to be able to download an NRB trial file.

250
00:22:13,500 --> 00:22:15,580
So that's what is exploited here in the bottom.

251
00:22:16,340 --> 00:22:18,320
So yeah, it's interesting way.

252
00:22:19,040 --> 00:22:22,440
And even more, when we wanted to put more about this feature,

253
00:22:22,540 --> 00:22:24,620
like how did they implemented that, we also find out,

254
00:22:24,640 --> 00:22:26,320
you can also choose the offset from where you

255
00:22:26,320 --> 00:22:27,360
want to start leading the file.

256
00:22:27,500 --> 00:22:29,400
So if the file is too big, you can just

257
00:22:29,400 --> 00:22:31,320
jump to specific offset.

258
00:22:31,440 --> 00:22:34,140
This is exactly fsig, and it's without any validation,

259
00:22:34,300 --> 00:22:35,520
just come from the user.

260
00:22:35,840 --> 00:22:37,220
So it's not only reading the file,

261
00:22:37,340 --> 00:22:38,480
but also you can choose which portion

262
00:22:38,620 --> 00:22:40,020
of the file precisely you can read.

263
00:22:40,660 --> 00:22:43,240
And another bug which we found is identified

264
00:22:43,240 --> 00:22:47,200
by these CV numbers, by AMI and NVIDIA.

265
00:22:47,340 --> 00:22:51,940
And this bug is exactly in the SNMP protocol,

266
00:22:52,500 --> 00:22:54,760
like functionality for SNMP.

267
00:22:54,920 --> 00:22:56,080
We call it SNMP injection.

268
00:22:56,440 --> 00:22:59,400
So what happens, SNMP is enabled by default,

269
00:22:59,580 --> 00:23:01,200
so we didn't need to use read write primitive

270
00:23:01,200 --> 00:23:01,980
to enable this feature.

271
00:23:02,320 --> 00:23:04,180
And it allows you to change the configuration

272
00:23:04,180 --> 00:23:05,240
of SNMP server.

273
00:23:05,460 --> 00:23:08,520
But whenever you send a new package that you want to reload,

274
00:23:08,620 --> 00:23:10,120
or change the SNMP configuration,

275
00:23:10,840 --> 00:23:13,560
the way how they implemented this logic,

276
00:23:13,660 --> 00:23:16,380
it's kind of unique for the developers, C developers.

277
00:23:16,680 --> 00:23:21,920
So they create a file, new SNMP configuration file from C,

278
00:23:22,020 --> 00:23:26,660
using this path of commands, like SNMP5 and system.

279
00:23:27,330 --> 00:23:29,220
So every line of the configuration

280
00:23:29,670 --> 00:23:32,920
is dynamically created by executing system command.

281
00:23:33,530 --> 00:23:37,700
And that's a unique way of creating configuration file.

282
00:23:38,640 --> 00:23:41,940
So we were surprised, but what is more interesting,

283
00:23:42,120 --> 00:23:45,160
let's see what are the arguments for these system commands.

284
00:23:45,300 --> 00:23:52,060
Yes, and we found out that these read only community variables,

285
00:23:52,360 --> 00:23:54,980
but also LW community, these two variables there,

286
00:23:55,160 --> 00:23:58,360
are essentially directly read from the user,

287
00:23:58,500 --> 00:23:59,440
from the package, yes.

288
00:23:59,860 --> 00:24:01,220
So, and it's enabled by default.

289
00:24:01,520 --> 00:24:04,780
So essentially what we do, you can do just very simple,

290
00:24:04,880 --> 00:24:06,400
again, 90s command injection.

291
00:24:06,780 --> 00:24:08,200
So we just implemented,

292
00:24:08,200 --> 00:24:09,400
this is a proof of concept,

293
00:24:09,580 --> 00:24:12,540
and we didn't know if this bug works or not.

294
00:24:12,740 --> 00:24:14,560
So we combined with the previous exploit

295
00:24:14,560 --> 00:24:15,960
of downloading arbitrary file.

296
00:24:16,280 --> 00:24:18,740
So we just did command injection,

297
00:24:19,040 --> 00:24:20,480
who's supposed to create a new file,

298
00:24:20,640 --> 00:24:22,640
and then we use previous bug to download

299
00:24:22,640 --> 00:24:23,740
if this file is created.

300
00:24:24,120 --> 00:24:28,940
And so here we inject the command echo hi tmp a,

301
00:24:29,780 --> 00:24:32,320
and then first thing what we wanted to check,

302
00:24:32,460 --> 00:24:36,080
how the new configuration for SNMP configuration works.

303
00:24:36,300 --> 00:24:37,880
So because if the command is injected,

304
00:24:37,880 --> 00:24:40,320
this all command and LW command

305
00:24:40,320 --> 00:24:41,620
is supposed to have null attached.

306
00:24:41,960 --> 00:24:42,820
So we download this file,

307
00:24:42,920 --> 00:24:44,920
and you can see there's no arguments for these two.

308
00:24:45,020 --> 00:24:46,540
So likely our command was injected.

309
00:24:46,880 --> 00:24:49,160
And then we download our tmp a file,

310
00:24:49,340 --> 00:24:51,500
using this previous bug, and yes, it exists.

311
00:24:51,760 --> 00:24:54,680
So essentially, you have command injection now.

312
00:24:55,080 --> 00:24:57,800
And the next bug will be covered by Alex.

313
00:24:58,020 --> 00:24:59,340
Oh, before we go there.

314
00:24:59,500 --> 00:25:03,100
So essentially, you don't physically need to write

315
00:25:03,100 --> 00:25:05,860
like very complicated exploit and proof of concept.

316
00:25:06,020 --> 00:25:07,300
The reason why, because we have,

317
00:25:07,300 --> 00:25:08,680
we have from the GitHub APMI tool.

318
00:25:09,040 --> 00:25:12,740
And APMI tool allows you just to be normally executed

319
00:25:12,740 --> 00:25:15,400
as a feature, that's how you run this command,

320
00:25:15,600 --> 00:25:16,900
and then you will be able to exploit

321
00:25:17,280 --> 00:25:18,800
that specific command injection bug.

322
00:25:18,940 --> 00:25:20,700
You just run the tool with this command,

323
00:25:20,820 --> 00:25:22,800
and you have command injection, so yay.

324
00:25:23,820 --> 00:25:25,460
And this bug will be covered by Alex.

325
00:25:27,200 --> 00:25:29,660
Yeah, this is also identified by these TVs,

326
00:25:29,740 --> 00:25:30,760
by my NVIDIA.

327
00:25:31,740 --> 00:25:33,620
This is another shell injection

328
00:25:33,620 --> 00:25:35,980
in another component of the BMC.

329
00:25:36,080 --> 00:25:36,720
So the SNMP

330
00:25:36,720 --> 00:25:39,300
was not accessible, the SNMP config was not accessible

331
00:25:39,300 --> 00:25:42,820
from the web UI that is available to administrator

332
00:25:42,820 --> 00:25:43,820
when it logs in.

333
00:25:43,940 --> 00:25:45,540
But this one is enabled.

334
00:25:45,760 --> 00:25:50,380
So you can set NTP servers that will be used by BMC

335
00:25:50,380 --> 00:25:51,900
to set the time on the device.

336
00:25:53,000 --> 00:25:58,000
So there is also an IPMI counterpart for this API.

337
00:25:58,200 --> 00:26:01,680
So you can call this API to set primary

338
00:26:01,680 --> 00:26:02,960
and secondary NTP servers,

339
00:26:03,120 --> 00:26:04,700
or enable and disable this functionality,

340
00:26:06,720 --> 00:26:06,860
if you want.

341
00:26:07,020 --> 00:26:09,600
So if you enable the functionality,

342
00:26:09,980 --> 00:26:12,580
you can set the, let's say, primary NTP server

343
00:26:12,580 --> 00:26:14,500
with this API.

344
00:26:14,840 --> 00:26:18,960
But the way they parse these addresses

345
00:26:18,960 --> 00:26:20,480
for the servers is interesting.

346
00:26:21,380 --> 00:26:25,040
Yeah, this is where you set this NTP server.

347
00:26:26,140 --> 00:26:29,960
Yeah, so you see here, if this functionality is enabled,

348
00:26:31,000 --> 00:26:35,740
line 42, the code proceeds to just print

349
00:26:36,720 --> 00:26:39,940
the full string that you specify as an address

350
00:26:39,940 --> 00:26:42,440
for the primary server to this buffer,

351
00:26:42,640 --> 00:26:44,940
and later it is saved in the configuration.

352
00:26:45,300 --> 00:26:50,640
And when the code is about to read this configuration,

353
00:26:50,860 --> 00:26:52,960
this is how it is implemented.

354
00:26:53,500 --> 00:26:56,900
They read this string without any checks

355
00:26:56,900 --> 00:26:58,880
to the local buffer on stack,

356
00:26:59,120 --> 00:27:01,240
and then they do printf operation

357
00:27:01,240 --> 00:27:04,100
to build a command line string,

358
00:27:04,840 --> 00:27:06,700
which is supposed to be, you know, a string,

359
00:27:06,700 --> 00:27:09,000
which is supposed to be passed to this NTP date tool.

360
00:27:09,400 --> 00:27:11,960
So NTP date will be executed with the arguments

361
00:27:11,960 --> 00:27:15,600
that basically are user-controlled, or attacker-controlled.

362
00:27:16,360 --> 00:27:19,240
So here you have another kind of shell injection.

363
00:27:19,600 --> 00:27:22,540
There is no validation for this server address.

364
00:27:23,080 --> 00:27:26,020
You can pass any string that will be processed

365
00:27:26,020 --> 00:27:29,080
as a server address, and the NTP date will be executed

366
00:27:29,080 --> 00:27:32,640
with this string attached to the command line.

367
00:27:32,840 --> 00:27:34,980
So this is very easy case

368
00:27:36,020 --> 00:27:37,380
for a shell injection.

369
00:27:38,660 --> 00:27:41,980
And it just executes this command with NTP date.

370
00:27:43,740 --> 00:27:46,760
So what we did, we implemented a proof of concept

371
00:27:46,760 --> 00:27:49,720
that calls this API and sets the server,

372
00:27:49,820 --> 00:27:54,780
primary NTP address of server for this string.

373
00:27:55,420 --> 00:27:59,600
So you see 127.0.0.1, and then echo test,

374
00:27:59,740 --> 00:28:02,160
so we write some data to the just temporary file.

375
00:28:02,340 --> 00:28:04,960
And then we read this file using our, you know,

376
00:28:04,980 --> 00:28:05,980
our download file exploit.

377
00:28:06,260 --> 00:28:09,560
And just to see if it was created or not.

378
00:28:09,700 --> 00:28:12,820
And actually, we see that the file is created.

379
00:28:13,020 --> 00:28:17,720
So when this NTP functionality is restarted,

380
00:28:17,760 --> 00:28:19,600
the new config file is generated,

381
00:28:19,840 --> 00:28:24,380
and this NTP date tool is executed with our string.

382
00:28:24,780 --> 00:28:27,720
And you see that we were able to download TMP test file.

383
00:28:27,880 --> 00:28:32,180
This is how we can now execute arbitrary shell commands

384
00:28:32,180 --> 00:28:33,180
on the BMC side.

385
00:28:33,180 --> 00:28:36,480
Or we can use this to write arbitrary files

386
00:28:36,480 --> 00:28:37,420
on the BMC side.

387
00:28:42,600 --> 00:28:46,200
So we, now we were thinking what can be done else?

388
00:28:46,760 --> 00:28:49,900
How can we attack the host in any interesting way?

389
00:28:50,120 --> 00:28:52,140
So we had full access to BMC.

390
00:28:52,200 --> 00:28:53,640
We can execute arbitrary code.

391
00:28:53,820 --> 00:28:55,380
We can inject shell commands,

392
00:28:55,640 --> 00:28:58,620
read and write any files on BMC side.

393
00:28:58,800 --> 00:29:00,760
So we found this interesting bug

394
00:29:00,760 --> 00:29:02,680
that is not related to BMC anymore.

395
00:29:02,680 --> 00:29:03,940
It is more related to the host.

396
00:29:04,400 --> 00:29:06,420
This is identified by the CVS.

397
00:29:07,340 --> 00:29:10,480
So there is a API on the BMC side

398
00:29:10,480 --> 00:29:13,700
that can be used to perform the update operations

399
00:29:13,700 --> 00:29:16,340
on the host flash, the spy flash of the host.

400
00:29:16,500 --> 00:29:20,300
So we can talk to BMC and ask it to please flash

401
00:29:20,300 --> 00:29:24,660
this piece of data to this location on the host flash.

402
00:29:25,780 --> 00:29:27,680
This is how they implemented it.

403
00:29:27,980 --> 00:29:32,320
So the host spy is just mounted as a block device.

404
00:29:32,320 --> 00:29:32,540
Here.

405
00:29:32,820 --> 00:29:35,700
And they open it as a file and just seek to a location

406
00:29:35,700 --> 00:29:37,860
that we want and write the data.

407
00:29:38,840 --> 00:29:40,980
Or this code just reads the data.

408
00:29:41,120 --> 00:29:44,020
OK, so first we want to be sure that we

409
00:29:44,020 --> 00:29:45,700
are going to overwrite the location

410
00:29:45,700 --> 00:29:47,180
that we want to overwrite.

411
00:29:47,300 --> 00:29:48,820
So we read the data from that location.

412
00:29:49,060 --> 00:29:51,660
We see is this the right place.

413
00:29:51,840 --> 00:29:54,500
And if it's all good, we can write data to it.

414
00:29:54,520 --> 00:29:56,020
Or erase it first and then write.

415
00:29:56,860 --> 00:30:00,700
This is the counterpart that performs the write operation

416
00:30:00,700 --> 00:30:01,440
to this flash.

417
00:30:02,420 --> 00:30:06,680
This is just the usual write call, write API.

418
00:30:07,020 --> 00:30:08,760
So again, there is no validation.

419
00:30:09,040 --> 00:30:10,980
We can overwrite any location on the host flash.

420
00:30:12,020 --> 00:30:13,420
What can we do with this?

421
00:30:13,660 --> 00:30:17,180
We can install persistent implants on the host

422
00:30:17,180 --> 00:30:19,980
by using this functionality.

423
00:30:20,360 --> 00:30:21,220
And this is what we did.

424
00:30:21,400 --> 00:30:27,700
We just read the location 37000 on the host flash.

425
00:30:27,860 --> 00:30:32,200
And we wanted to see some NVRAM structures on this host.

426
00:30:32,320 --> 00:30:33,540
Flash at that location.

427
00:30:33,780 --> 00:30:35,080
And this is what we actually get.

428
00:30:35,500 --> 00:30:44,640
We see that there is NVRAM structures, the NVRAM variable

429
00:30:44,640 --> 00:30:46,580
structures on this data.

430
00:30:46,820 --> 00:30:48,320
So this is the correct place.

431
00:30:48,440 --> 00:30:52,420
This is where the NVRAM starts on this host spy flash.

432
00:30:52,680 --> 00:30:54,000
Then we erase this data.

433
00:30:54,120 --> 00:30:56,840
There is another API on this VMC

434
00:30:56,840 --> 00:30:59,940
that we can use to erase this page of spy flash.

435
00:31:00,100 --> 00:31:02,240
And then we write our updated data.

436
00:31:02,320 --> 00:31:02,920
And then we write our updated data to this spy flash.

437
00:31:03,140 --> 00:31:06,220
And here you see, we write, hello, DEVCOM.

438
00:31:06,500 --> 00:31:06,980
Hello.

439
00:31:07,500 --> 00:31:11,500
We were able to install persistent malware

440
00:31:11,500 --> 00:31:12,920
on this host flash.

441
00:31:14,140 --> 00:31:17,860
Now you cannot just reset the machine to a non-good state.

442
00:31:18,020 --> 00:31:19,340
You have to reflash the firmware first.

443
00:31:22,200 --> 00:31:26,060
So yeah, now I think we broke VMC pretty well.

444
00:31:26,300 --> 00:31:27,100
We have common injection.

445
00:31:27,540 --> 00:31:29,960
We can do spy flash write read.

446
00:31:30,200 --> 00:31:31,640
We can download any file.

447
00:31:32,320 --> 00:31:34,960
We do see that the bugs from 2006 still works.

448
00:31:35,140 --> 00:31:36,500
So why even bother more?

449
00:31:37,400 --> 00:31:40,940
So we thought, OK, let's find some holy grail of the bugs.

450
00:31:41,180 --> 00:31:44,540
We do not allow, does not require some authentication

451
00:31:44,540 --> 00:31:45,220
at all.

452
00:31:45,360 --> 00:31:46,180
And we found one.

453
00:31:46,340 --> 00:31:50,140
And this is identified by this CV number by AMI and NVIDIA.

454
00:31:50,260 --> 00:31:51,760
So we found remote stack overflow

455
00:31:51,760 --> 00:31:53,280
in the pre-authentication phase.

456
00:31:53,820 --> 00:31:57,560
And yeah, knowing the state of the previous bug,

457
00:31:57,700 --> 00:31:59,420
we expected how difficult it will

458
00:31:59,420 --> 00:32:01,540
be to exploit remotely stack overflow

459
00:32:01,540 --> 00:32:03,040
in the pre-authentication phase.

460
00:32:03,680 --> 00:32:05,300
So at first, the bug is here.

461
00:32:06,080 --> 00:32:08,400
There is a telemetry shared library.

462
00:32:08,700 --> 00:32:10,140
Essentially, if you authenticate, you

463
00:32:10,140 --> 00:32:12,800
could have a telemetry enabled, essentially meaning, OK,

464
00:32:12,920 --> 00:32:17,580
let's log in to the VMC, which user name, et cetera.

465
00:32:17,960 --> 00:32:19,460
So yeah, that's a feature.

466
00:32:19,960 --> 00:32:22,360
This feature, the way how they implement it

467
00:32:22,360 --> 00:32:26,300
when the telemetry calls, essentially, it's here.

468
00:32:26,420 --> 00:32:28,000
You can see there is a string and copy.

469
00:32:28,600 --> 00:32:30,760
And as a parameter to string and copy,

470
00:32:30,760 --> 00:32:31,700
it took the local buffer.

471
00:32:31,860 --> 00:32:33,740
Local buffer is a static buffer on the stack.

472
00:32:34,420 --> 00:32:36,460
They get the user name from the package.

473
00:32:36,720 --> 00:32:39,900
And the length of the user name is also coming from the package.

474
00:32:40,280 --> 00:32:41,920
Because why not?

475
00:32:42,480 --> 00:32:45,540
And essentially, directly after the string copy,

476
00:32:45,700 --> 00:32:46,900
there is a function pointer, which

477
00:32:46,900 --> 00:32:48,060
is coming from the shell library.

478
00:32:48,560 --> 00:32:50,960
And it's called immediately after the overflow.

479
00:32:51,300 --> 00:32:55,080
And yeah, so we thought, how difficult

480
00:32:55,080 --> 00:32:59,220
is to be able to exploit the stack overflow in 2023?

481
00:32:59,600 --> 00:33:00,280
We found out.

482
00:33:00,280 --> 00:33:03,460
There is no stack canaries, to our surprise.

483
00:33:03,820 --> 00:33:08,420
So yeah, and then we saw, OK, what about other mitigations?

484
00:33:08,740 --> 00:33:10,680
So the stack is also executable.

485
00:33:11,860 --> 00:33:13,580
And there is no ASLR.

486
00:33:15,220 --> 00:33:18,160
So yeah, but we thought, it will be too easy,

487
00:33:18,200 --> 00:33:20,060
maybe, to exploit just return address.

488
00:33:20,440 --> 00:33:22,100
So there is a function pointer.

489
00:33:22,300 --> 00:33:25,380
So let's try to see where the function pointer is declared.

490
00:33:25,840 --> 00:33:27,080
So this is the local buffer.

491
00:33:27,240 --> 00:33:29,560
It's 16 bytes long for the user name.

492
00:33:29,560 --> 00:33:30,520
So it's not too long.

493
00:33:31,280 --> 00:33:34,240
And apparently, after the buffer, there is no function pointer.

494
00:33:34,360 --> 00:33:36,980
Function pointer is above the buffer.

495
00:33:37,120 --> 00:33:38,500
So we couldn't use this cool technique

496
00:33:38,500 --> 00:33:39,620
to write the function pointer.

497
00:33:39,820 --> 00:33:42,260
So we need to switch to the standard stack overflow

498
00:33:43,800 --> 00:33:46,380
and just write the return address on the stack.

499
00:33:46,720 --> 00:33:48,980
And because of no ASLR executable stack,

500
00:33:49,300 --> 00:33:52,740
it gives us a bit less relaxed time.

501
00:33:53,400 --> 00:33:56,080
And yes, this is exactly in the pre-authentication phase.

502
00:33:56,260 --> 00:33:58,940
So this is a slide which Alex explained at the beginning, when

503
00:33:58,940 --> 00:34:01,400
you just send the first package with the user name.

504
00:34:01,500 --> 00:34:03,780
This user name length just can cause your stack overflow

505
00:34:03,780 --> 00:34:05,060
in the pre-authentication phase.

506
00:34:05,400 --> 00:34:07,700
So yeah, we pull the trigger, and we wrote the exploit.

507
00:34:07,840 --> 00:34:08,140
Why not?

508
00:34:08,420 --> 00:34:11,960
So this is exactly the authentication phase.

509
00:34:12,060 --> 00:34:13,600
As you can see, we send a message

510
00:34:13,600 --> 00:34:15,440
with a very long user name.

511
00:34:15,660 --> 00:34:16,500
And that's a reply.

512
00:34:16,680 --> 00:34:18,540
We're supposed to say you're authenticated or not.

513
00:34:18,660 --> 00:34:19,600
And that's a reply instead.

514
00:34:19,800 --> 00:34:22,800
We just overwrite the messages and the logic on the server.

515
00:34:23,000 --> 00:34:24,100
And we just say, hello, DEF CON.

516
00:34:24,440 --> 00:34:26,680
So yes, that's pre-authentication stack overflow.

517
00:34:34,620 --> 00:34:35,060
Thanks.

518
00:34:36,120 --> 00:34:38,780
So in fact, we found many more bugs.

519
00:34:39,060 --> 00:34:41,840
This is just a few of them to our surprise.

520
00:34:41,960 --> 00:34:43,200
Yes, who would expect that?

521
00:34:43,500 --> 00:34:46,680
And so that's all of the CVs which we found.

522
00:34:46,840 --> 00:34:48,680
There is around 20 of them.

523
00:34:48,800 --> 00:34:51,520
But we only covered, during this talk, these bugs.

524
00:34:51,780 --> 00:34:54,100
So there's so much more to tell you

525
00:34:54,100 --> 00:34:56,760
about the state of the security there and the bugs

526
00:34:56,760 --> 00:34:57,260
which we found.

527
00:34:58,940 --> 00:35:01,080
And just a summary to recap what we saw.

528
00:35:01,240 --> 00:35:03,740
So at first, BMC, of course, is a very critical part

529
00:35:03,740 --> 00:35:05,200
of the security of the entire ecosystem.

530
00:35:06,280 --> 00:35:09,120
And breaking BMC, unfortunately, or fortunately,

531
00:35:09,440 --> 00:35:12,720
depends who you ask, falls into the category of break one

532
00:35:12,720 --> 00:35:13,760
to rule them all.

533
00:35:13,920 --> 00:35:16,260
You just have one component, BMC,

534
00:35:16,300 --> 00:35:18,500
and you have access to all of the host servers.

535
00:35:18,660 --> 00:35:19,760
So it's a nice feature.

536
00:35:20,160 --> 00:35:23,660
And unfortunately, BMC didn't get too much attention

537
00:35:23,660 --> 00:35:25,480
from the security research community,

538
00:35:25,700 --> 00:35:28,700
which is kind of reflected in the current state.

539
00:35:28,740 --> 00:35:28,920
OK.

540
00:35:28,920 --> 00:35:29,780
So the security of it.

541
00:35:30,120 --> 00:35:32,360
What I mean, essentially, we're still

542
00:35:32,360 --> 00:35:35,040
missing a lot of modern mitigation and hardening,

543
00:35:35,140 --> 00:35:37,120
which are expected to have in 2023.

544
00:35:37,860 --> 00:35:40,500
Stack overflows shouldn't be as easily exploitable.

545
00:35:40,820 --> 00:35:43,780
Stack cookies are kind of standard for some decades,

546
00:35:43,880 --> 00:35:44,380
at least.

547
00:35:44,760 --> 00:35:47,960
And yeah, some of the essential vulnerabilities

548
00:35:47,960 --> 00:35:49,660
would still fall into the category,

549
00:35:49,780 --> 00:35:51,900
let's exploit something like in the 90s, yes?

550
00:35:52,560 --> 00:35:54,040
So what we can do with this?

551
00:35:54,340 --> 00:35:57,560
First thing first, probably some kind of recommendation

552
00:35:57,560 --> 00:35:58,620
of secure coding.

553
00:35:58,920 --> 00:36:00,140
We practice supposed to be stabilized

554
00:36:00,140 --> 00:36:03,240
when we develop such kind of firmware, some kind of SDLC

555
00:36:03,240 --> 00:36:03,520
process.

556
00:36:04,240 --> 00:36:07,140
Also, heavy fuzzing could help finding such kind of flow

557
00:36:07,140 --> 00:36:07,800
hanging through it.

558
00:36:07,900 --> 00:36:09,840
So we also ask ourselves, how is it

559
00:36:09,840 --> 00:36:11,800
possible that this pre-authenticated stack

560
00:36:11,800 --> 00:36:13,260
overflow was not found before?

561
00:36:13,500 --> 00:36:16,960
Because a 16 bytes long buffer doesn't

562
00:36:16,960 --> 00:36:18,780
seem like too difficult to overflow.

563
00:36:19,100 --> 00:36:21,700
But apparently, the tool on the user side

564
00:36:22,380 --> 00:36:24,160
limits you how long username you can get.

565
00:36:24,360 --> 00:36:28,060
So the tool just was limiting up to 16 bytes, not the server.

566
00:36:28,280 --> 00:36:28,460
So.

567
00:36:28,920 --> 00:36:29,860
For you, that's why.

568
00:36:31,580 --> 00:36:33,400
Also, switching to memory-safe language

569
00:36:33,400 --> 00:36:37,540
could help and remove such kind of basic memory safety issue.

570
00:36:37,860 --> 00:36:41,060
But there is a catch, which we describe in this talk.

571
00:36:41,240 --> 00:36:42,660
Many of these bugs will not be affected

572
00:36:42,660 --> 00:36:44,000
by the memory-safe language, yes?

573
00:36:44,020 --> 00:36:46,460
The command injection, the downloading of the file,

574
00:36:46,600 --> 00:36:49,600
path traversal, it's a state of the security,

575
00:36:49,800 --> 00:36:51,160
not the state of the language, yes?

576
00:36:52,080 --> 00:36:54,380
And so not everything can be solved by that.

577
00:36:54,620 --> 00:36:57,920
And of course, investing in the security research could help,

578
00:36:57,920 --> 00:37:00,860
and it's recommended to improve overall state of the BMC.

579
00:37:01,820 --> 00:37:04,660
And as an acknowledge, we would like to thank NVIDIA, of course,

580
00:37:04,800 --> 00:37:06,860
especially offensive security research team,

581
00:37:07,000 --> 00:37:09,400
like Jared, Max, and Nikola, because they

582
00:37:09,400 --> 00:37:10,400
did this research with us.

583
00:37:10,840 --> 00:37:12,700
Server folks that allows us to touch

584
00:37:12,700 --> 00:37:14,960
this very expensive hardware, and they were not

585
00:37:14,960 --> 00:37:16,260
scared to give it our hand.

586
00:37:16,440 --> 00:37:18,820
We were scared to touch them, but they don't, so thanks.

587
00:37:19,520 --> 00:37:23,360
And product security team for handling all of this coordination

588
00:37:23,360 --> 00:37:25,300
disclosure, thank you very much.

589
00:37:25,440 --> 00:37:26,180
And everyone.

590
00:37:26,400 --> 00:37:27,760
And of course, Ami.

591
00:37:27,760 --> 00:37:29,640
Ami for allowing us doing this amazing research.

592
00:37:30,100 --> 00:37:32,380
So if you have any question, feel free to shoot us.

593
00:37:32,460 --> 00:37:33,080
We'll try to answer.

594
00:37:57,760 --> 00:38:01,280
I think when a mega-transitized CV, the user

595
00:38:01,280 --> 00:38:05,380
2023 and it's like a spot for BMC which probably

596
00:38:05,380 --> 00:38:06,220
are way over .

597
00:38:06,760 --> 00:38:08,760
So not only by single .

598
00:38:09,300 --> 00:38:11,040
Why is it different than the CS?

599
00:38:11,140 --> 00:38:12,460
How it's connected to something?

600
00:38:14,120 --> 00:38:15,160
SEBASTIAN KONSTANTINOSZEWSKI.

601
00:38:15,540 --> 00:38:19,520
OK, so the answer is because we analyze this BMC first

602
00:38:19,520 --> 00:38:22,240
on our product, so we wanted to have NVIDIA customer

603
00:38:22,240 --> 00:38:23,540
secure as fast as possible.

604
00:38:23,680 --> 00:38:26,400
So we release the CV faster and do some changes by ourselves.

605
00:38:26,400 --> 00:38:32,200
but in the same time we need to go to vendor and do coordination disclosure and it was

606
00:38:32,200 --> 00:38:37,040
taking a bit more time for them to patch as a reference code which could be moved to

607
00:38:37,040 --> 00:38:42,240
everybody who uses BMC. So yeah, there's a time difference between we did the patches and

608
00:38:42,240 --> 00:38:47,620
before the vendor did the patches in the reference code. So yeah, that's the reasons the

609
00:38:47,620 --> 00:38:48,160
difference here.

610
00:38:56,840 --> 00:39:01,420
So basically, you kind of like a patch for your own device, but then you can have their

611
00:39:01,420 --> 00:39:07,140
medical or basically when the CD is issued, it's basically patched for them. Let's say one

612
00:39:07,140 --> 00:39:12,100
vendor released an advisory, they say it should be patched, so maybe it will be additional

613
00:39:12,100 --> 00:39:16,320
tools for the researchers to go through different things. And then we can see if the boxes

614
00:39:16,320 --> 00:39:21,240
are kind of basically fine. How we can suggest to the vendors in such a way that the boxes

615
00:39:21,240 --> 00:39:25,840
really need to be patched and to react to things like blocks and clouds and shortcuts

616
00:39:27,900 --> 00:39:33,800
. It's a good question, just to repeat. So the question is, because there's

617
00:39:33,800 --> 00:39:38,720
time difference between when we found the bug and when NVIDIA fixed the issues and between when

618
00:39:38,720 --> 00:39:43,680
the army fixed the issues and we released the advisor a bit faster to make the protected

619
00:39:43,680 --> 00:39:51,920
customers also earlier, the malicious actor could get our like patches and trying to reverse

620
00:39:51,920 --> 00:39:55,820
engineer to find out if other vendors are also vulnerable. So I don't think it's a good

621
00:39:55,840 --> 00:40:02,440
answer for that. It's a supply chain issue. So there are start-ups who can address the

622
00:40:02,440 --> 00:40:09,760
supply chain and security of the state. And they could help. But, yeah, that's essentially the

623
00:40:09,760 --> 00:40:15,500
industry has some problems which needed reaction which we cannot solve by ourselves and we

624
00:40:15,500 --> 00:40:19,920
need the entire community to work together to make it more secure. And, yes, that's a valid

625
00:40:19,920 --> 00:40:22,020
point and I 100% agree with that statement.

626
00:40:28,400 --> 00:40:32,580
Do you have an idea about how many devices this might affect?

627
00:40:34,020 --> 00:40:34,980
A lot.

628
00:40:36,620 --> 00:40:38,420
Do you have an idea about how many?

629
00:40:40,200 --> 00:40:49,100
No. Let's say no. But, yeah, it's also a valid point. Unfortunately, because BMC is

630
00:40:50,060 --> 00:40:53,760
essentially affecting like a lot of vendors, so, yeah, the impact is high, yes.

631
00:40:59,300 --> 00:41:00,140
.

632
00:41:04,560 --> 00:41:09,820
We try not to touch Redfish. But it's a very good question.

633
00:41:14,660 --> 00:41:15,500
.

634
00:41:19,080 --> 00:41:20,760
Correct. Yes.

635
00:41:21,020 --> 00:41:21,440
.

636
00:41:23,440 --> 00:41:24,280
Correct.

637
00:41:24,660 --> 00:41:24,720
.

638
00:41:25,840 --> 00:41:31,440
Correct. And also requires them to not only patch but also force clients to flush the

639
00:41:31,440 --> 00:41:31,800
firmware.

640
00:41:35,620 --> 00:41:36,260
.

641
00:41:37,160 --> 00:41:37,800
.

642
00:41:42,080 --> 00:41:42,720
Yeah.

643
00:41:43,620 --> 00:41:48,480
The question is how do we interface with the device when we do this timing attack? By

644
00:41:48,480 --> 00:41:50,340
sending UDP packets remotely.

645
00:41:50,600 --> 00:41:51,220
.

646
00:41:53,360 --> 00:41:54,000
.

647
00:41:55,180 --> 00:41:55,820
.

648
00:41:55,820 --> 00:41:55,840
Yeah.

649
00:41:55,860 --> 00:42:00,740
In fact, if you look for the demo, it was not like a single shot. We repeated the same

650
00:42:02,240 --> 00:42:08,040
multiple times and get an average time because latency, et cetera. But it's practical. It

651
00:42:08,040 --> 00:42:08,180
works.

652
00:42:14,440 --> 00:42:20,440
So the question is why they do not run Linux but RTOS, correct? Other way around? Why

653
00:42:20,440 --> 00:42:25,420
they run Linux and not the RTOS? Yeah, I don't know. .

654
00:42:26,440 --> 00:42:32,060
I have no idea. We apparently do not have much visibility of what the vendor does. Apparently,

655
00:42:32,220 --> 00:42:35,940
if you think about the industry of the BMC, there is not as many solutions that you can

656
00:42:35,940 --> 00:42:41,220
choose. You can either use OpenBMC which requires you to essentially set up a team, hire people

657
00:42:41,220 --> 00:42:46,780
who do manage to be like entire process of creating the firmware, the BMC firmware, and

658
00:42:46,780 --> 00:42:52,540
then you can do more influence on the security side or you can go to the vendor and the vendor

659
00:42:52,540 --> 00:42:58,580
just gives you the image and mostly it is out of your hands what you can do there. You

660
00:42:58,580 --> 00:43:04,620
can ask the vendor to do some changes or alignments, but you are pretty limited how much you can

661
00:43:04,620 --> 00:43:10,260
influence that. This was one of the reasons why we apparently pulled the trigger and said,

662
00:43:10,400 --> 00:43:13,260
okay, let's analyze and find out what did we get.

663
00:43:23,340 --> 00:43:27,620
So, the question is, are there any firmware security tools which we recommend? Yes.

664
00:43:30,340 --> 00:43:39,180
Fuzzing. I'm not sure. Alex? Yeah, I don't know, to be honest. But, yes.

665
00:43:43,800 --> 00:43:49,400
For most of these vulnerabilities, is the initial vector, are there more than a few

666
00:43:49,400 --> 00:43:52,220
additional vectors from the BMC management network?

667
00:43:57,100 --> 00:44:02,180
Yes. So, for us, it was, we did slightly opposite, yes, because we saw there is a BMC.

668
00:44:02,460 --> 00:44:08,700
We didn't have any idea how to, you know, be able to get into this BMC. So, our research

669
00:44:08,700 --> 00:44:12,440
was apparently, okay, we know we can poke it. So, let's try to log in. We cannot log

670
00:44:12,440 --> 00:44:18,560
in. Let's try timing attack. And then so on, so on, so on. But, yes, there is many interfaces,

671
00:44:19,320 --> 00:44:22,520
especially to the APMI, but most of them require authentication. There are a lot of them that

672
00:44:22,540 --> 00:44:25,700
require authentication. So, that's why user name finding will be the best, like, entry

673
00:44:25,700 --> 00:44:33,340
point. But, yes. And there is also a local interface from the host. So, host can talk

674
00:44:33,340 --> 00:44:47,220
to BMC through KCS interfaces. Some API are not read. So, the question is, what's the

675
00:44:47,220 --> 00:44:52,520
interface that we can use to access BMC? So, some API that BMC provides, you know,

676
00:44:52,540 --> 00:44:58,660
interfaces are not reachable remotely. So, some of them have to be reached from the host

677
00:44:58,660 --> 00:45:04,280
locally. And this is why we need to have host access. So, we got the shell on the host first.

678
00:45:04,760 --> 00:45:10,280
Yeah. I like this Redis database with the surprise API give me all of your database. It was

679
00:45:10,280 --> 00:45:22,520
through the host. It's a pretty nice feature for attacker, I guess. Okay. So, now we have

680
00:45:22,540 --> 00:45:33,500
to release the PSCs. Unfortunately, we cannot release the PSCs. At least for now. Okay. Thank

681
00:45:33,500 --> 00:45:34,120
you very much again.