Hello, everyone. And, yeah, I see more people than I expected. That's kind of surprising
for the BMC talk because it means people know what the BMC is. So, well, first of all, thank
you. My name is Adam and together with Alex, we have a pleasure to be here. So, thanks
DEF CON organizer for allowing us to be here today, first time in the row as a speaker.
And thanks for you, great audience, that you know what the BMC is. So, really, thank you,
guys. So, what is our talk about? Our talk is about breaking BMC, the forgotten
kingdom. So, yeah, it's a niche topic. But apparently it's not as niche as I want it to be.
So, to tell us about ourselves, we don't want to spend too much time here because it's boring
stuff. Both of us, me and Alex, we're going to be talking about the BMC. We're going to be
working in NVIDIA doing this research. And everything when we're done, it was apparently as
part of our work at NVIDIA. We don't want to spend too long here. There's some private
contact information to us. You can always poke us through the e-mail or through the Twitter. If
you have any questions, just feel free to shoot us in the message. And there's some short bio.
But it doesn't really matter. What matters, this is not only the research for two of us, but
for entire NVIDIA offensive security research team which I'm currently leading. So, we're
going to be talking about the BMC. So, big kudos also to Jared, Max, and Nicola who also did
this research together with us. So, first in first, what is BMC? What I expect you to know, it's
a baseboard management controller. It's a very special service processor which you can see.
So, that is an old picture from 2006. We didn't have time to take a new one. So, we just make a
small adjustment to look at more modern. So, there is 2022 here. But physically it doesn't change
anything. And we did this research in 2022 for one of the BMC from one of the vendor which we are
not going to mention, but it's visible in the picture. And so, what physically BMC does, it's
apparently used to control the data centers, especially the hosts. And the reason why, because
you can imagine there could be like 100 of the host machine on the data center, 1,000 host
machine, and sometimes even 100 of the 1,000 of the machine. Some of them are not even
accessible physically because they might be somewhere in the water. So, it's very difficult to go
there and do anything physically. Sometimes they still do. So, you can still go physically to this
machine and do whatever you need to do. But, of course, as you can imagine, it's not a scalable
solution. So, that's why we do have BMCs that can do it for us remotely. They can do it
everything what you can physically do by yourself. So, it's essentially a specialized service
processor that can monitor the physical system. So, that's why we have BMCs that can monitor the physical
state of the computer, not only computer, but any network server, hardware, and they're using
sensors, essentially, to be able to communicate to the administrator of this device. What is also
interesting, they allow you to do a full control of the machine which they monitor. So, you
could have something like a KVM capability. So, you can go remotely to the BMC and you can then
reconfigure the host. You can go to the BIOS. You can change the BIOS settings if you want to
remotely. You can do whatever you want, essentially. So, you can do whatever you want, essentially.
Even more, you can flash the entire firmware of the controlled device. And what is even more
funny, you can do it even when the device is powered off. So, BMC, as you can see, is a very
powerful service processor. So, that's why it's a very interesting target for the attacker
because it gives, at first, tons of the benefits. And also, it falls into the category like hack
one, rule them all. You just hack one BMC and you can have control over entire data
center which BMC physically controls. So, it's like a great and juicy target. And Alex and I
will talk about the first issue which we want to cover here. BMC speaks IPMI protocol. And when
you encounter a system that speaks IPMI, what would you check for? Of course, for a hash leak
that has been known since 2013. So, what is this? This is a screenshot from a manual that
describes the IPMI standard. And this is an explanation of the IPMI standard. And this is an
explanation of how the authentication works in IPMI. So, you see, here, there are two
messages. One of those messages is from the administrator console to BMC. And the
administrator is only supposed to send the username to the BMC. BMC looks at the username and
calculates the H magical language are some data which includes the username. So, the
password for the user that is about to be authenticated is used as a key to the H device encryption. So if
HMAC calculation. So this HMAC calculation result is sent back to the administrator.
So if an attacker knows the correct username that exists in a database for BMC users,
he can receive the hash calculated and the password will be used as a key to this HMAC.
And this has been known since 2003 where this standard has been published.
So you might think is it fixed by now in 2023? No, actually it's not because it's in the standard.
So if you speak IPMI, you have to do the same way. The authentication process looks exactly the same.
And we implemented the script just to test this and we found out that
actually the system that we were looking at conforms to the same standard. But we had a
problem while figuring out the correct username that we can use because, you know, if we don't
know the username, any username that exists in the user database, we will not get this
hash back. So we had a problem here. We don't have a valid username. How can we guess it or how can we
find out?
This correct username, at least one. So you see the server always responds with an unauthorized
name for whatever name it's right. So we had this idea of looking for some issues with incorrect
processing of this user sensitive data. For example, the username. And we found out a bug
that is actually present on this server. And we found out a bug that is actually present on this
system. And the CVs for this bug are issued by AMI and NVIDIA. You see it here.
So what exactly is happening? So when a user sends a username to the BMC
in order to authenticate, the length of the username is also sent in the same message.
And we had an idea that maybe the server processes this username in non-constant time.
For example, if it compares the username with some database entries, maybe it uses mem compare.
Who knows? And we implemented a script that measures the time that it takes for a certain
message to be processed by the server, by the BMC. And eventually, we got this result.
Let me show you the demo.
So we're sending a bunch of message ones, which are defined in the standard.
And we are basically doing a brute force for the printable characters as user names,
available for user names. But we use this time in Oracle to greatly speed up the brute force.
You see, we begin with measuring the time that it takes for the server to process the user names. We
are only one character long. And for all like 200 possible values for this character, we measure
the time that it takes for the server to process it. And we find the longest time. So we find the
symbol that takes the longest time to process. And it is very likely that this is the correct
symbol, the first symbol of the user name. Because when the mem compare advances to the next symbol,
it takes a little bit more time to process. So this is...
This is why we can use the first guest symbol and advance to the next one. So essentially we do
this in a sequence and we don't brute force the whole key space. We just move from one symbol to
another. And do it a few times in a row just to get better data to measure the response time.
But essentially this works really well on local networks. And you see here, we were
able to guess the right user name. And eventually when we send this user name to authentication
server, we get back the hash, the H mark calculation result. And we just brute force it using
normal methods offline. So this attack can be happening completely offline. So here we use a
word list and we are able to guess the right password. This is the explanation why this works.
So by this moment, we didn't have access to any binaries for the
BMC. We didn't have any access to source code. So this is just a picture that explains to you why
this attack works. You see this mem compare. This is the usual mem compare from libc. It does not
execute in a constant time. This is why it can be used as a timing oracle here. So if you look up
the man page for mem compare, it says don't use it for sensitive data. And this is sensitive data.
So now we have access to the BMC. We guess the user name and the password. So we
only got one user information from the BMC. But we can talk to it. We can issue a request that
requires authentication.
So, sure, as you can see, to put a bit more to the story, we just saw a BMC. We know it's there. We
know there is a known bug from 2006 with the CV from 2013, which we didn't expect to work. But it
works. So interesting. We wanted to log in. It was just pure guess. Like, okay, let's try timing
why not? Like many devices are vulnerable and turns out we have CVE. So, okay, we log
in, one of the devices which VMC managed looks like this is a screenshot of the web interface.
So looks cool. So why are we even speaking more about that? Like, okay, we have access
to VMC, so what? So that's exactly the question. What to do next? And we have a couple of ideas.
We could stop presentation here, but we did not and we did a bit more research. And the
reason why, because at first we wanted to understand a lot of stuff about VMC which we did
not at first. Why this timing protocol at first worked? Because we did not know why this
timing given works. We wanted to find the VMC image, we wanted to reverse this VMC image and
to be able to understand why this timing attack works at first place. And to be able to do that,
we need to have somehow access to the VMC image, yes? We also thought, okay, if the timing
protocol works, maybe there is more bugs, yes? Why not? So we need to have image to be able to
find more bugs there. And also, again, we did not know too much at the time about internal
architecture of the VMC. So we wanted to learn more, like, what is apparently there inside of this
VMC excluding this timing attacks which we found. And also we wanted to analyze, okay, if maybe
there is some other bugs which are good to exploit, what are the difficulties of doing that?
Like, are there any kind of defense or hardening or mitigation techniques? What is the state of
heart of the security of the VMC image? And to be able to do that, we need to have some
how get this VMC image, yes? So how do you get the VMC image? Yes, that's the question. So we
had an idea, like, VMC still has a firmware, yes? So this firmware must be somehow updated. So if
people update the firmware, how many of people, like, delete the updates after they update the
firmware? So we had this idea, maybe some of the host machine will have the, you know, not
deleted updates for the firmware. So let's get now, because we're having the VMC, let's get the
VMC access to the VMC, and we know that the VMC has, like, capabilities of KVM. Let's go from the
VMC to the host and grab the updates for the VMC image, and then we can have ability to analyze
that. So the question, how do we go from the VMC to the host and get the shell? So, of course,
it's not as difficult because we have KVM capabilities, so let's go very old school techniques,
change the bootloader commands and put in it binbash. Of course, sometimes might not that feature
works.
Because you might have secure boot and maybe the commands are assigned, et cetera, but again, we
are in the KVM. So why not to go to the BIOS if there is no password and disable secure boot? We
are in VMC. So there is a lot of opportunities to do so, so we went to the least difficult path,
just changed the command line, binbash, and we end up there. So, yeah, we have shell. And the
disk somehow is not encrypted. Surprise. So we are able now to go and see what is in the host
machine. But then we also realize we are also able to, you know, we are able to, you know,
to blade in the host machine and to mount whatever image we want. So even if the binbash won't work,
you can do a booting attack, disable secure boot and just load our own image if you want to.
But we combine two of these features, apparently. We just put in binbash, half a shell, and we
just mount our tool set to be able to run the commands on the host machine which are not
installed on the specific host. So what commands we wanted to run? So we found the VMC image,
but we don't know how this firmware image looks like, what is inside. So we wrote a new VMC image,
custom tool to be able to pass this image. As you can see here, we found interesting stuff,
like there is essentially model bootloader, model JFFS2, there is model in CRAMFS and one of
them is a root partition as you can see here. And this root partition starts from the
address 2500,000. So what we did, we just very simple stuff. We know exactly how the header
looks like. We know what is inside of this firmware because we just passed that. So we
just do DD, dump all of this byte data and remote into the loopback and now we can able to
browse the BMC and analyze that. So what did we find there? We found at first that BMC includes
the IPMI server binary and libraries. And so what we saw in this interface on the screenshot
which I showed you before, is just a very small subset of all of the APIs which are capabilities
and functionalities which BMC physically can provide.
So it's a very small subset of the feature. And then we started to analyze on the BMC, there's
like a tons of other protocols which are apparently communicating with the BMC and you can poke.
So then we found out that you can just directly send the IPMI messages to the BMC and bypass
all of these web interfaces which they have, which they expose, and you have like a tons
of the functionality. There is tons of the hidden API which is accessible, especially
if you have a username and password, that you can just poke. So as you can imagine, the
attack surface is pretty huge.
So we pull a trigger to find out, let's find out more bugs. And this is one of the bugs
which now Alex will cover as well.
Yeah, we started to poke to this firmware image and found another issue that is identified
by these CVs by MI and NVIDIA. Essentially, we found out that the user database that is
stored and that is implemented in this BMC is managed by Redis database. And what is
more interesting, the passwords,
are encrypted in this database. So it's stored in an encrypted state, but the keys which
are used to encrypt this password are stored right next to the passwords. So it doesn't
make much sense, but the question is, why do they even have this obfuscated passwords
in the database? Why do not hash the passwords instead? So the answer to that is the same
issue with IPMI protocol. They need these plain text passwords to be able to encrypt
the passwords. They need these plain text passwords to be able to authenticate users.
That's why. So we found a really nice API that can be accessed from the host, which is a local
access, considered local, which is not accessible remotely. So this API allows you to do
specific requests to this Redis database. So we analyzed the image and found the correct keys,
the names of the keys that has to be accessed in order to read this password. So we found a lot of
passwords and the user names in the keys, and we implemented the proof of concept that just
accesses this user database through this API and reads the encrypted passwords and then the keys
from the same database and decrypts the passwords. So at this point, we got a full user database
for the BMC, so user names and the passwords in a decrypted form. So we proceeded to analyze
the firmware. We found that the user names and the passwords in a decrypted form. So we
found another issue, which is also interesting, which I will talk about. This is also
identified by these CVEs from IMI and NVIDIA. So there is a server that is actually an IPMI
server that listens on a socket on the BMC site, and services all these requests that come
remotely. And this particular API allows you to read any virtual memory via this server. So this
is within the context of this process, of the server process. So there is special command
that prepares some data, so it has to be issued before you access this API, but also you only
need to have the user name and password, the usual user name and password for the BMC in order
to issue this command. You cannot do it without any authentication. But at this point, we
already have the user name and password, so we can just proceed. As you see here, we have
no checks whatsoever about the region or the size of the data that we want to read. You can
just read anything from this context of this IPMI server. And this is how we found out that,
for example, the server always uses the same base, I mean, it's executed at the same base
every time you boot the BMC, so there is no SLR. And you see here, there are no checks for the
sizes.
And this is the counterpart of this read API. This one writes arbitrary data to this context
of this IPMI server process. So we can use it to write any data or code that we want to this
server process. So that is very useful if you want to build a shell code inside the process.
So at this point, we could use this API to execute any code we want in the context of this
IPMI server process. This is how you...
The actual override happens. You see there is a usual mem copy operation, also without any
checks for the address and the size. And this is the proof of concept for this particular issue.
First, we identify that the server process is actually... Is located at 10,000 hex, so there is
no SLR, but we know the base of this module, so we can just read arbitrary data from it. And we
are interested in reading special errors. So we can just read arbitrary data from it, and we are interested in reading special errors.
We can also see that the array of the configuration array for this features available in this
server, we... Which are controlled by this G-core features array. And in particular, we are
interested in this offset 310... 310 hex. And if we enable the... If we set
a bit at this location, the server now thinks that certain functionality has been enabled.
And this functionality allows you to download arbitrary files from BMC.
So, that is...
By default this is disabled but since we have a write primitive we can now enable it.
And we use this functionality to test and and download ETC Shadow.
Why not? And you see that with dot dot slash dot dot slash string we were able to do it.
So Adam will talk about about this download functionality in detail.
So yeah, first No SLR to recap and the second thing what we wanted to recap so
So we found now read and write capabilities
to entire IPMI process and BMC server,
so essentially we can enable and disable
whatever feature we want.
Of course, why wanna do it?
Because you can still exploit that,
but yeah, we just wanted to find out
what other hidden features we have.
So we identified this CV,
which is assigned by AMI and NVIDIA,
which as Alex mentioned,
the feature itself is nothing wrong with the feature, yes?
You have a feature who allows you to download just a file.
Sometimes you might want to do it, why not?
But the way how the program implemented that,
it's again, old school 90s way
of implementing the one in the file.
So as you can see here,
if you at first enable this feature,
this is the offset,
then this feature is available,
and if it's available,
they construct the path you can see on the bottom here.
So the get file path is static,
but the file path, but file name,
directly comes from the package from the user.
So it's coming from the user,
of course, you just do very old school
dash dash dot, dash dot slash,
dash dot slash,
and that's how we achieve ability
to be able to download an NRB trial file.
So that's what is exploited here in the bottom.
So yeah, it's interesting way.
And even more, when we wanted to put more about this feature,
like how did they implemented that, we also find out,
you can also choose the offset from where you
want to start leading the file.
So if the file is too big, you can just
jump to specific offset.
This is exactly fsig, and it's without any validation,
just come from the user.
So it's not only reading the file,
but also you can choose which portion
of the file precisely you can read.
And another bug which we found is identified
by these CV numbers, by AMI and NVIDIA.
And this bug is exactly in the SNMP protocol,
like functionality for SNMP.
We call it SNMP injection.
So what happens, SNMP is enabled by default,
so we didn't need to use read write primitive
to enable this feature.
And it allows you to change the configuration
of SNMP server.
But whenever you send a new package that you want to reload,
or change the SNMP configuration,
the way how they implemented this logic,
it's kind of unique for the developers, C developers.
So they create a file, new SNMP configuration file from C,
using this path of commands, like SNMP5 and system.
So every line of the configuration
is dynamically created by executing system command.
And that's a unique way of creating configuration file.
So we were surprised, but what is more interesting,
let's see what are the arguments for these system commands.
Yes, and we found out that these read only community variables,
but also LW community, these two variables there,
are essentially directly read from the user,
from the package, yes.
So, and it's enabled by default.
So essentially what we do, you can do just very simple,
again, 90s command injection.
So we just implemented,
this is a proof of concept,
and we didn't know if this bug works or not.
So we combined with the previous exploit
of downloading arbitrary file.
So we just did command injection,
who's supposed to create a new file,
and then we use previous bug to download
if this file is created.
And so here we inject the command echo hi tmp a,
and then first thing what we wanted to check,
how the new configuration for SNMP configuration works.
So because if the command is injected,
this all command and LW command
is supposed to have null attached.
So we download this file,
and you can see there's no arguments for these two.
So likely our command was injected.
And then we download our tmp a file,
using this previous bug, and yes, it exists.
So essentially, you have command injection now.
And the next bug will be covered by Alex.
Oh, before we go there.
So essentially, you don't physically need to write
like very complicated exploit and proof of concept.
The reason why, because we have,
we have from the GitHub APMI tool.
And APMI tool allows you just to be normally executed
as a feature, that's how you run this command,
and then you will be able to exploit
that specific command injection bug.
You just run the tool with this command,
and you have command injection, so yay.
And this bug will be covered by Alex.
Yeah, this is also identified by these TVs,
by my NVIDIA.
This is another shell injection
in another component of the BMC.
So the SNMP
was not accessible, the SNMP config was not accessible
from the web UI that is available to administrator
when it logs in.
But this one is enabled.
So you can set NTP servers that will be used by BMC
to set the time on the device.
So there is also an IPMI counterpart for this API.
So you can call this API to set primary
and secondary NTP servers,
or enable and disable this functionality,
if you want.
So if you enable the functionality,
you can set the, let's say, primary NTP server
with this API.
But the way they parse these addresses
for the servers is interesting.
Yeah, this is where you set this NTP server.
Yeah, so you see here, if this functionality is enabled,
line 42, the code proceeds to just print
the full string that you specify as an address
for the primary server to this buffer,
and later it is saved in the configuration.
And when the code is about to read this configuration,
this is how it is implemented.
They read this string without any checks
to the local buffer on stack,
and then they do printf operation
to build a command line string,
which is supposed to be, you know, a string,
which is supposed to be passed to this NTP date tool.
So NTP date will be executed with the arguments
that basically are user-controlled, or attacker-controlled.
So here you have another kind of shell injection.
There is no validation for this server address.
You can pass any string that will be processed
as a server address, and the NTP date will be executed
with this string attached to the command line.
So this is very easy case
for a shell injection.
And it just executes this command with NTP date.
So what we did, we implemented a proof of concept
that calls this API and sets the server,
primary NTP address of server for this string.
So you see 127.0.0.1, and then echo test,
so we write some data to the just temporary file.
And then we read this file using our, you know,
our download file exploit.
And just to see if it was created or not.
And actually, we see that the file is created.
So when this NTP functionality is restarted,
the new config file is generated,
and this NTP date tool is executed with our string.
And you see that we were able to download TMP test file.
This is how we can now execute arbitrary shell commands
on the BMC side.
Or we can use this to write arbitrary files
on the BMC side.
So we, now we were thinking what can be done else?
How can we attack the host in any interesting way?
So we had full access to BMC.
We can execute arbitrary code.
We can inject shell commands,
read and write any files on BMC side.
So we found this interesting bug
that is not related to BMC anymore.
It is more related to the host.
This is identified by the CVS.
So there is a API on the BMC side
that can be used to perform the update operations
on the host flash, the spy flash of the host.
So we can talk to BMC and ask it to please flash
this piece of data to this location on the host flash.
This is how they implemented it.
So the host spy is just mounted as a block device.
Here.
And they open it as a file and just seek to a location
that we want and write the data.
Or this code just reads the data.
OK, so first we want to be sure that we
are going to overwrite the location
that we want to overwrite.
So we read the data from that location.
We see is this the right place.
And if it's all good, we can write data to it.
Or erase it first and then write.
This is the counterpart that performs the write operation
to this flash.
This is just the usual write call, write API.
So again, there is no validation.
We can overwrite any location on the host flash.
What can we do with this?
We can install persistent implants on the host
by using this functionality.
And this is what we did.
We just read the location 37000 on the host flash.
And we wanted to see some NVRAM structures on this host.
Flash at that location.
And this is what we actually get.
We see that there is NVRAM structures, the NVRAM variable
structures on this data.
So this is the correct place.
This is where the NVRAM starts on this host spy flash.
Then we erase this data.
There is another API on this VMC
that we can use to erase this page of spy flash.
And then we write our updated data.
And then we write our updated data to this spy flash.
And here you see, we write, hello, DEVCOM.
Hello.
We were able to install persistent malware
on this host flash.
Now you cannot just reset the machine to a non-good state.
You have to reflash the firmware first.
So yeah, now I think we broke VMC pretty well.
We have common injection.
We can do spy flash write read.
We can download any file.
We do see that the bugs from 2006 still works.
So why even bother more?
So we thought, OK, let's find some holy grail of the bugs.
We do not allow, does not require some authentication
at all.
And we found one.
And this is identified by this CV number by AMI and NVIDIA.
So we found remote stack overflow
in the pre-authentication phase.
And yeah, knowing the state of the previous bug,
we expected how difficult it will
be to exploit remotely stack overflow
in the pre-authentication phase.
So at first, the bug is here.
There is a telemetry shared library.
Essentially, if you authenticate, you
could have a telemetry enabled, essentially meaning, OK,
let's log in to the VMC, which user name, et cetera.
So yeah, that's a feature.
This feature, the way how they implement it
when the telemetry calls, essentially, it's here.
You can see there is a string and copy.
And as a parameter to string and copy,
it took the local buffer.
Local buffer is a static buffer on the stack.
They get the user name from the package.
And the length of the user name is also coming from the package.
Because why not?
And essentially, directly after the string copy,
there is a function pointer, which
is coming from the shell library.
And it's called immediately after the overflow.
And yeah, so we thought, how difficult
is to be able to exploit the stack overflow in 2023?
We found out.
There is no stack canaries, to our surprise.
So yeah, and then we saw, OK, what about other mitigations?
So the stack is also executable.
And there is no ASLR.
So yeah, but we thought, it will be too easy,
maybe, to exploit just return address.
So there is a function pointer.
So let's try to see where the function pointer is declared.
So this is the local buffer.
It's 16 bytes long for the user name.
So it's not too long.
And apparently, after the buffer, there is no function pointer.
Function pointer is above the buffer.
So we couldn't use this cool technique
to write the function pointer.
So we need to switch to the standard stack overflow
and just write the return address on the stack.
And because of no ASLR executable stack,
it gives us a bit less relaxed time.
And yes, this is exactly in the pre-authentication phase.
So this is a slide which Alex explained at the beginning, when
you just send the first package with the user name.
This user name length just can cause your stack overflow
in the pre-authentication phase.
So yeah, we pull the trigger, and we wrote the exploit.
Why not?
So this is exactly the authentication phase.
As you can see, we send a message
with a very long user name.
And that's a reply.
We're supposed to say you're authenticated or not.
And that's a reply instead.
We just overwrite the messages and the logic on the server.
And we just say, hello, DEF CON.
So yes, that's pre-authentication stack overflow.
Thanks.
So in fact, we found many more bugs.
This is just a few of them to our surprise.
Yes, who would expect that?
And so that's all of the CVs which we found.
There is around 20 of them.
But we only covered, during this talk, these bugs.
So there's so much more to tell you
about the state of the security there and the bugs
which we found.
And just a summary to recap what we saw.
So at first, BMC, of course, is a very critical part
of the security of the entire ecosystem.
And breaking BMC, unfortunately, or fortunately,
depends who you ask, falls into the category of break one
to rule them all.
You just have one component, BMC,
and you have access to all of the host servers.
So it's a nice feature.
And unfortunately, BMC didn't get too much attention
from the security research community,
which is kind of reflected in the current state.
OK.
So the security of it.
What I mean, essentially, we're still
missing a lot of modern mitigation and hardening,
which are expected to have in 2023.
Stack overflows shouldn't be as easily exploitable.
Stack cookies are kind of standard for some decades,
at least.
And yeah, some of the essential vulnerabilities
would still fall into the category,
let's exploit something like in the 90s, yes?
So what we can do with this?
First thing first, probably some kind of recommendation
of secure coding.
We practice supposed to be stabilized
when we develop such kind of firmware, some kind of SDLC
process.
Also, heavy fuzzing could help finding such kind of flow
hanging through it.
So we also ask ourselves, how is it
possible that this pre-authenticated stack
overflow was not found before?
Because a 16 bytes long buffer doesn't
seem like too difficult to overflow.
But apparently, the tool on the user side
limits you how long username you can get.
So the tool just was limiting up to 16 bytes, not the server.
So.
For you, that's why.
Also, switching to memory-safe language
could help and remove such kind of basic memory safety issue.
But there is a catch, which we describe in this talk.
Many of these bugs will not be affected
by the memory-safe language, yes?
The command injection, the downloading of the file,
path traversal, it's a state of the security,
not the state of the language, yes?
And so not everything can be solved by that.
And of course, investing in the security research could help,
and it's recommended to improve overall state of the BMC.
And as an acknowledge, we would like to thank NVIDIA, of course,
especially offensive security research team,
like Jared, Max, and Nikola, because they
did this research with us.
Server folks that allows us to touch
this very expensive hardware, and they were not
scared to give it our hand.
We were scared to touch them, but they don't, so thanks.
And product security team for handling all of this coordination
disclosure, thank you very much.
And everyone.
And of course, Ami.
Ami for allowing us doing this amazing research.
So if you have any question, feel free to shoot us.
We'll try to answer.
I think when a mega-transitized CV, the user
2023 and it's like a spot for BMC which probably
are way over .
So not only by single .
Why is it different than the CS?
How it's connected to something?
SEBASTIAN KONSTANTINOSZEWSKI.
OK, so the answer is because we analyze this BMC first
on our product, so we wanted to have NVIDIA customer
secure as fast as possible.
So we release the CV faster and do some changes by ourselves.
but in the same time we need to go to vendor and do coordination disclosure and it was
taking a bit more time for them to patch as a reference code which could be moved to
everybody who uses BMC. So yeah, there's a time difference between we did the patches and
before the vendor did the patches in the reference code. So yeah, that's the reasons the
difference here.
So basically, you kind of like a patch for your own device, but then you can have their
medical or basically when the CD is issued, it's basically patched for them. Let's say one
vendor released an advisory, they say it should be patched, so maybe it will be additional
tools for the researchers to go through different things. And then we can see if the boxes
are kind of basically fine. How we can suggest to the vendors in such a way that the boxes
really need to be patched and to react to things like blocks and clouds and shortcuts
. It's a good question, just to repeat. So the question is, because there's
time difference between when we found the bug and when NVIDIA fixed the issues and between when
the army fixed the issues and we released the advisor a bit faster to make the protected
customers also earlier, the malicious actor could get our like patches and trying to reverse
engineer to find out if other vendors are also vulnerable. So I don't think it's a good
answer for that. It's a supply chain issue. So there are start-ups who can address the
supply chain and security of the state. And they could help. But, yeah, that's essentially the
industry has some problems which needed reaction which we cannot solve by ourselves and we
need the entire community to work together to make it more secure. And, yes, that's a valid
point and I 100% agree with that statement.
Do you have an idea about how many devices this might affect?
A lot.
Do you have an idea about how many?
No. Let's say no. But, yeah, it's also a valid point. Unfortunately, because BMC is
essentially affecting like a lot of vendors, so, yeah, the impact is high, yes.
.
We try not to touch Redfish. But it's a very good question.
.
Correct. Yes.
.
Correct.
.
Correct. And also requires them to not only patch but also force clients to flush the
firmware.
.
.
Yeah.
The question is how do we interface with the device when we do this timing attack? By
sending UDP packets remotely.
.
.
.
Yeah.
In fact, if you look for the demo, it was not like a single shot. We repeated the same
multiple times and get an average time because latency, et cetera. But it's practical. It
works.
So the question is why they do not run Linux but RTOS, correct? Other way around? Why
they run Linux and not the RTOS? Yeah, I don't know. .
I have no idea. We apparently do not have much visibility of what the vendor does. Apparently,
if you think about the industry of the BMC, there is not as many solutions that you can
choose. You can either use OpenBMC which requires you to essentially set up a team, hire people
who do manage to be like entire process of creating the firmware, the BMC firmware, and
then you can do more influence on the security side or you can go to the vendor and the vendor
just gives you the image and mostly it is out of your hands what you can do there. You
can ask the vendor to do some changes or alignments, but you are pretty limited how much you can
influence that. This was one of the reasons why we apparently pulled the trigger and said,
okay, let's analyze and find out what did we get.
So, the question is, are there any firmware security tools which we recommend? Yes.
Fuzzing. I'm not sure. Alex? Yeah, I don't know, to be honest. But, yes.
For most of these vulnerabilities, is the initial vector, are there more than a few
additional vectors from the BMC management network?
Yes. So, for us, it was, we did slightly opposite, yes, because we saw there is a BMC.
We didn't have any idea how to, you know, be able to get into this BMC. So, our research
was apparently, okay, we know we can poke it. So, let's try to log in. We cannot log
in. Let's try timing attack. And then so on, so on, so on. But, yes, there is many interfaces,
especially to the APMI, but most of them require authentication. There are a lot of them that
require authentication. So, that's why user name finding will be the best, like, entry
point. But, yes. And there is also a local interface from the host. So, host can talk
to BMC through KCS interfaces. Some API are not read. So, the question is, what's the
interface that we can use to access BMC? So, some API that BMC provides, you know,
interfaces are not reachable remotely. So, some of them have to be reached from the host
locally. And this is why we need to have host access. So, we got the shell on the host first.
Yeah. I like this Redis database with the surprise API give me all of your database. It was
through the host. It's a pretty nice feature for attacker, I guess. Okay. So, now we have
to release the PSCs. Unfortunately, we cannot release the PSCs. At least for now. Okay. Thank
you very much again.