Okay, so what are we going to cover today?
We're going to explain what a vehicle diagnostics adapter,
or VDA, is to you.
Then we're going to talk about, you know,
some research we did attacking the VDA.
And then we're going to talk about ways
and approaches we took to defend the VDA.
And then finally, we're going to talk about
some educational things we did with the VDA.
So specifically, we had four challenges
in last year's Car-Convilled CTF
that were based around the VDA.
This is a picture of it here.
We had it in Vegas mode, so...
Yeah, look at that.
Binary modification for Vegas mode, awesome.
And we also did some educational things
at the Cybertruck Challenge we'll talk about.
So Alex and I are on stage today,
but Edward and Sam did a lot of this work as well
as a team effort, so we wanted to acknowledge them.
We also wanted to acknowledge Dr. Jeremy Daly,
who put us on the path to looking at VDAs.
And Friendly Vendor Person, who will not be named today,
but helped give us some of the VDAs
for some of the educational things,
and was a really good partner working with us
to shine light on this part of trucking security.
Cool.
All right, so finally,
what is a vehicle diagnosis adapter?
So your truck has an OBD port in it.
It's a thing that technicians can plug into
to pull diagnostic information off the truck.
I think the VIN number comes through there as well,
so you can pull the VIN number through there,
and also some information from some of the ECUs
or the controllers on the truck.
It also sometimes is used to flash the firmware
of the ECUs within the truck.
So the idea is you pull into a service station,
you can plug in there,
and I guess the way they do it
is there's a little computer, the VDA,
that actually plugs into the truck.
And then there's a,
for some reason it's always a Windows computer
that plugs into the VDA.
So you have like a Windows laptop
with these vendor DLLs that,
there's kind of like a joint protocol,
I think it's called RP-1210,
that everyone used to talk to the VDA,
then the VDA talks to the truck,
and it pulls data back and forth from it.
And you can kind of see here on the right,
this is kind of a diagram of the VDA we looked at.
So on the right, there's the computer interface,
so the USB, Bluetooth, and Wi-Fi,
also is how you connect directly to the VDA.
And then the micropositor connects to the vehicle network,
usually over J1939, unless it's really old.
Okay, so congratulations,
we all now know what a VDA is.
Okay, so now we're gonna talk about attacking the VDA.
So we do a lot of embedded device security around balloons,
so attacking embedded devices,
it's a pretty simple formula.
First, you need to get the thing.
The internet is really good with this.
Pro tip, get more than one of the things,
because you'll probably break one of them in the process.
I think you broke one of these, yeah.
Then you wanna do the hardware reversing.
Figure out, I guess, enumerate all the components
on the board, figure out if it has JTAG, UART,
stuff like that.
Then you wanna get the firmware either from the board
or from some sort of firmware update thing.
Reverse the firmware,
figure out how to update the firmware.
And then if you can modify the firmware
and put the modified firmware back on,
you can change how it works.
And then you win.
So this is pretty complicated,
so when we started working on this,
we decided that it was so hard
that only our interns could do it.
And so that's where Alex came in.
He was an intern at the time, he's a full-time employee now.
So he's gonna walk you through the work he did as an intern,
you know, trying to figure out how this thing works.
Awesome, so the first step that we always go through
when reversing a new device
is grabbing your favorite screwdriver set
and taking it apart.
So that's what we did.
This is a shot of the PCB of one of the VDAs.
And as you can see, the vendor did a great job
making connecting to JTAG and UART super hard
by giving us really awesome adapters to use
and just plug into.
We've identified the flash, the microcontroller,
and then a Wi-Fi chip and a Bluetooth chip on this board.
Now, the more astute of you might be noticing...
the Wi-Fi chip has a Wi-Fi and Bluetooth stack,
but there's also a Bluetooth chip on the board,
which begs the question, why?
We don't have a great answer.
I think these memes probably illustrate
some of our confusion as to why
there were two independent Bluetooth stacks on this device,
but it had both.
So once we took a look at the hardware
and read a couple of data sheets,
we were able to make a super legit power adapter
to plug in the VDA in a way that it's not normally powered.
Yeah, this is what happens when you tell an intern
to power a VDA, but it gets the job done, so that's cool.
And then just grab your favorite female adapter cables
to connect to the UART and not have to do any work with that,
really, because the vendor provided a handy-dandy
interface.
So once we connected to the UART,
unlike most devices that try and obfuscate
and make it really difficult to find out
what the debug options are on a shell,
the vendor just gave us a really easy way of dumping memory.
So, you know, we typed in dump invalid option
and then got every possible permutation of dumping
we could ever want.
So it was pretty simple.
We're just like, hey, can we get the first copy of firmware
on the device?
And then it spits it out handy-dandy over ASCII over UART.
So, you know, I wrote a really complicated extraction algorithm,
which is dump the whole firmware to serial
and then write a Python program to convert that into binary.
So once we actually had the bytes of the firmware
sitting in memory on the device,
what we wanted to do is reverse engineer the file format
that actually puts those bytes on the device
so then we can see whether we can actually change
the bytes that are put on the device.
So as Wyatt mentioned,
this piece of hardware uses a Windows laptop
in order to do the firmware update.
And so I was a little worried
because I hadn't done Windows reversing
that this would be kind of complicated.
But it turns out the firmware update format file
was just put in the files for the application on Windows,
like called firmware.bin.
So that took about two seconds to look at.
And then we had the full firmware update file
for the manufacturer.
This is a colorized GIF of me reversing the firmware update.
It did not take that long.
We're not going to go into the details
of how this firmware format actually works.
But if you spend enough time looking at it,
you will be able to figure it out.
And once we did that,
we knew how the vendor firmware update file
translates to the bytes actually sitting in memory on the device.
So once we'd done all that really complicated reversing work,
we wanted a way to be able to replicate that
so no one would have to go through the same process.
So internally, this is our tool called OFrac.
This is what Red Balloon Security uses internally.
It's also open source. You guys can try it at home.
And it allows us to very easily go through all of the steps
that we normally do to reverse engineer and modify firmware
so we can identify arbitrary binary formats,
unpack them, analyze them, make the changes that we want,
and then repack them up in the format that the device expects.
It's in Python, and we have a really snazzy GUI on the right.
If you guys want to take a look at it in more depth,
we'll be at our booth over there demoing it.
So after I reverse the format,
we can write a script in Python
that specifies the firmware format
and allows us to make the modifications programmatically
and have it spit out a binary
that we can just drop back on the device.
So saving a lot of time for anyone else
to have to do any of the reversing.
So now that we know how to make the firmware file
for the device to take,
we wanted a more fun way of putting it on the device
than just putting that file back in the Windows file system
and clicking through the vendor tool to actually put it on.
So the firmware update happened over USB.
So, you know, computer connected to a thing over USB.
What do you do? You sniff the USB traffic
and store it as a PCAP.
And then we spent a really long time reversing it.
We're going to go through the protocol later
and then wrote our own little Python script
to just do the firmware update
and then got rid of our Windows laptop
because we don't really like Windows.
So to go over the super complicated update procedure
of this device,
you first have to send hex 3
and then the size of the firmware update in hex
as a USB packet and then wait for it.
You just send the entire bytes of the firmware update.
There's no encryption. There's no signature.
It has a CRC check,
but you basically just slop it in a USB packet
and it just takes it up and rewrites the firmware.
For those playing at home, there's bonus points.
This same unauthenticated firmware update works over HTTP.
So if you have Wi-Fi on the device,
you could very easily just connect to the access point
and then do this exact same update format over HTTP.
So now we have a full way of having bytes
that we can modify our computer
and putting it on the target device.
So what do we do with that?
Well, the first thing that we always do
when performing this type of analysis
is testing that are unpacking and repacking
and this whole process works correctly.
So the first thing that we normally do
is just by modifying a string
and seeing whether the device still boots.
So that worked.
And then we wanted to do something more fun,
which we showed you at the beginning,
which was the LED toggle that we hooked into ping.
So you can see the code on the right, pretty complicated.
But again, we used OFRAC,
which is our custom unpacker and packer tool,
in order to put this code into the binary.
And specifically, we used OFRAC
with a tool we wrote called Patchmaker,
which allows us to compile C into bytes
and then link it into the full firmware image.
So it's really easy to iterate and write code
and just plunk it back in the binary
without having to deal with any linking.
So this case was a little bit more difficult than normal.
If you can see on the top left,
the compiler directive, underscore, underscore, reg args.
The calling convention for this toggle LED function
was kind of abnormal.
So the architecture of this was Motorola Coldfire.
And for this specific function,
you needed to pass in the first argument as a register,
which kind of messed with our compiler toolchain.
So we had to use a different toolchain
written by the wonderful Dr. Volkel Barthelme.
Shout out to him, who made the VBCC toolchain.
And then we just had to integrate it into our tool
and compile that code in,
and then, boom, just add in our modification.
Yeah, Dr. Volkel Barthelme. His website's pretty cool.
I would definitely check it out.
VBCC supports a lot of architectures.
It looks like it's from the 90s, but we built it,
and it worked right away.
So just to highlight this, this is kind of how we do
binary modification with OFRAC.
We can use runPingCMD as a function we found in the binary,
and we wanted to overplace it so whenever we pinged,
we would toggle the LED lights.
And so in this case, we were able to call toggleLED
directly in the C code that we're writing.
And then using Patchmaker, we can compile this code
and then link it in so that all the addresses
just work with the target binary,
and then just use OFRAC to copy it back
into the target binary.
So you get pretty readable C,
that's really useful for developing binary modifications.
Awesome. So, so far we showed you a cool demo with LEDs,
like why do we care about this ability
to modify the former of the VDA?
So most VDAs are used to update the engine control unit
in these big trucks.
So the workflow is you have a new engine control unit
firmware from the vendor that needs to be updated.
It gets passed to the VDA,
which then directly uses it
and sends it over to the ECU to flash it.
And so the engine control unit
obviously controls the engine,
very important part of the car.
That's why it's worrying in particular that the VDA
is normally completely unsecured, right?
Like this is a device that technicians use
at the dealership, not really locked up,
not a part of anyone's threat model.
Anyone could sort of walk in
and do this kind of firmware change.
And then in addition, you know,
since we found so many ways to update this firmware,
really not a good thing that this is a critical path
in terms of getting firmware on the engine controller.
Excuse me.
So, you know, as a result, you could, for example,
put an implant on the VDA,
wait for an engine controller update,
add your own modifications to it,
and then unbeknownst to anyone else,
you now have code running on the ECU of that big truck,
which is obviously really not great.
Yeah, so this is kind of where the attack work ended.
We were able to, as you see,
modify the firmware to put it in Vegas mode,
flash the LED lights, which is fun,
but those LED lights are what tell the technician
that the device is on or off, if Wi-Fi is connected,
if data is being transmitted.
So you can use your imagination
to think about what things like that can do.
Alex mentioned, you know, sometimes firmware updates,
these ECUs pass through this device,
some diagnostic data come out,
so it's a really nice place for an attacker to sit.
And this work kind of extends work that Dr.,
that Jeremy Daly and Srikanth Kumar did.
They presented at ESCAR this year
about attacking the Windows application
that communicates with the VDA.
So I'd recommend checking out that work.
They looked at the Windows DLLs
and found that it was pretty easy
to insert shim DLLs into the process
and peek and poke and manipulate packets
on the Windows laptop.
So kind of all the way down to the truck,
there's all these places where you can insert
and manipulate the traffic.
Something interesting just to think about.
Okay, so now we're gonna defend it,
we're gonna fix all the problems and go home, yay.
So we're not actually gonna fix all the problems today,
we're just gonna focus on some of the low-hanging fruit.
So the immediate problem here is,
you might have noticed there's Wi-Fi
and Bluetooth on this VDA.
And I think you can imagine that some people
might not want that Wi-Fi and Bluetooth enabled.
I wouldn't want it enabled.
I think people maintaining military vehicles
might not want to have Wi-Fi access point
on their VDAs they're plugging in, or Bluetooth as well.
But this is kind of a classic situation
in not just automotive, but all embedded devices
where the vendors, either they don't want to
or can't remove the hardware components
and software components that these features
are implemented on.
And so the way you disable the features
is essentially flipping a bit in a config file saying,
don't turn on Wi-Fi, don't turn on Bluetooth.
But as we all know, the code that underlines it
and the hardware is still there.
So an attacker can still modify the firmware
or use an exploit to then turn on these features.
So it's a very ripe attack surface.
So we're going to be talking about VDAs,
but I encourage you to think about,
this is a broader problem.
We have component access features, ABC.
We don't want those features.
We can't remove them because the vendor won't remove them.
So how can we remove them in the binary?
Okay, so what we're going to do today to talk you through
and what we did is like, how can we apply a binary patch
to the VDA firmware that removes all the code
that underpins these features
without having any side effects on the device?
And of course, this is an example of a general problem.
How do we remove features XYZ from component A
so there's no side effects and those features can't work
and the attack surface isn't there for people to use?
So we use a technique called autotomic binary reduction.
So the name is derived from nature, actually.
It's a nice place to look for inspiration.
So autotomy is when animals will shed a part of their body
when they're being attacked
to make it harder for predators to catch them.
So ABR was proposed by our CEO Ong
when he was a graduate student at Columbia.
That's where the work comes from.
So if you want to really get into the math behind it,
you can look at his dissertation.
But to simplify it, there's really just four steps.
First, you identify the high-level feature
that you don't want,
and then you identify the entry point in the code for it.
Then you can use analysis tools
to identify all the code and data
that are exclusively controlful,
dependent on those entry points.
You want to do this very carefully,
so if you're not sure, you'll leave something in.
And then you remove that code and data,
and then you replace the high-level entry point
with a return patch that makes sense
if the program ever calls it.
So if you think about a Bluetooth subsystem,
there's usually some sort of process
that initializes everything.
You can remove the code for that
and insert a return negative one,
or it depends on what is actually in the system,
but return code that will tell the system
this is not up and it's not going to run.
So this graph, I think, simplifies.
Obviously, it's more complex,
but at a high level, this is the idea.
If you have feature A and feature B,
we want to remove feature A.
We look at the control flow graph,
find all the functions that are exclusively controlful,
dependent on feature A.
So function five, we cannot remove,
but functions one and two, three, we can.
And then we insert that return patch there.
So the workflow for this is fairly standard,
and actually, Alex did most of the work already
that you need human input for now in.
So we've understood how the firmware works
and how we can update the firmware.
So next step is just to identify the feature points,
and then all of the actual autotomy work
about identifying the code that needs to be removed
and removing it and packing the firmware,
we can completely automate right now with the work we have.
And what we're doing as a company
is we're trying to move up the stack
to automate this entire process.
So there's minimal human interaction needed into it.
Okay, so identifying the feature entry points,
how do you do that?
Well, this is where you have to like, you know,
roll up your sleeves and do a little reversing.
So on embedded systems like the VDA,
this can be a little complicated,
but sometimes you get lucky.
So in this case, you can use symbols
to find leaked source code online
that you can use to then label the functions
in your project.
This was using ColdFire.
So in this case, we use Gija,
but you can use, you know,
any other disassembly tool as well to do this.
And then you can kind of start labeling
the features inside the binary.
So a new binary comes in,
this whole process can be automated.
You can start with the debug strings within the binary
and then work up to find the functions
and then automate this process removing that code.
So for this like proof of concept,
we generalized five entry points,
the OS tasks that initialize the subsystems.
We also wanted to remove the UR debug commands
because we found them super useful.
And yeah, it should be harder to figure out
how to dump the memory from the device.
I think, yeah.
Then we also targeted some driver initialization code
for these things as well.
And these were the results.
So we use the ABR algorithm that we have ever balloon.
We removed in total 574 functions
and ended up being 144 kilobytes of code.
So that was about 7% of the binary.
And you can kind of visualize it here.
Everything that's dark was unmodified
and the stripes of green here are the areas
where ABR removed the binary code.
So it's kind of cool.
You can see like throughout the binary,
there's huge swaths of functions that are just not needed
and attackers can't use it anymore.
Then we worked with friendly vendor person
to test the image using the functional test suite
that they used.
And they confirmed that performance was consistent
with the original firmware image.
So that was great.
We basically showed we were able to remove this feature
and the regular performance of the device
wasn't modified at all.
Okay, so we didn't fix all the problems.
So as we're reflecting on this, it's kind of Zen.
We were able to use some of the reversing
that attacker would use.
It's the same workflow in the beginning
to identify the features and remove it.
So removing the Bluetooth and Wi-Fi code is good.
We would challenge vendors to reconsider
whether they even need it.
Yeah, I think that's something we can all think about.
Do we need to put this in?
Can we support firmware builds that don't have it in?
Another thing to think about is ideally you wanna
at least not leave the headers on to JTAG and UART.
Ideally disable them as well.
That just makes it a little harder
for people to mess with the device.
For production devices, of course.
So then we get to some of the lower level challenges.
How can we make it so anyone can't modify the firmware?
So some sort of secure firmware update mechanism,
maybe a secure boot mechanism would be useful here.
And then how do we know
that the firmware hasn't been modified?
How do we know that the VDA that's on the shelf
in my technician's office hasn't been tampered with?
There's some sort of runtime protection
and monitoring could help there.
Because then you could see
that the device has been corrupted.
Okay, so we've attacked, we've defended,
and now let's talk about some education we did.
So we repurposed this research
to build two educational things
to kind of take these real world problems
and make it challenges that people can use
to kind of learn about this specific issue.
And this is another area
where the friendly vendor person helped us out
because we only had two of these devices,
but we were able to, at the Cybertruck Challenge,
have, for the last two years, a two hour workshop.
So the Cybertruck Challenge is a pretty cool event
that's held every year where a bunch of students
are paired up with industry experts
in automotive, specifically truck security,
and then a bunch of vendors come with fleets,
and there's a bunch of education that happens,
and then a bunch of assessments
that students do on the devices.
So I encourage you to check it out.
There's also some really good educational resources
on the website as well.
So if you're not super familiar with J1939,
that's a good place to start.
But anyway, in our workshop,
we walked them through this process.
They physically took apart the devices,
dumped the firmware.
Getting UART was a really fun moment
when you get the shell there.
One thing that we've kind of observed
is that familiarity with command line tools
is something that probably we could build on more.
So connecting the UART, getting screen working,
and stuff like that,
it was a really good moment for the students.
And then we walked them through
the bootstring modification,
and a contained environment
where they could toggle the LED lights there.
A little more fun, though.
Last year, as part of the Car-Wrecking Village CTF,
we had a bunch of these VDA boards here,
and four challenges.
So challenge one was a real-world challenge.
We gave them a PCAP,
and we gave them a modified firmware image
and said, flash this image on the device.
And so there, the flag printed in the boot shell
once you're able to flash the device.
Challenge two was a little too difficult.
It was ambitious.
I think we would remove it if we did it again at CTF.
But there, the challenge was to
essentially put the device in Vegas mode.
And then challenge three,
there's a really good write-up online
that kind of walks through the process,
but challenge three involved finding a
super-military-grade encryption scheme on the device.
I think it was just a bunch of XORing data
against the XOR repeating key.
And so the write-up's really cool
because they showed you how they use the ability
to map the firmware to load a Giger project,
find the function.
And I think they actually emulated the function in this
to get the decrypted key there.
And then challenge four was OTA to save the day.
So we had the Wi-Fi access points turned on
and you had to do the firmware update procedure,
but over the air.
Okay, so just to recap.
Yeah, so that's the binary modified image in Vegas mode.
We talked about what VDAs are.
We showed how it's relatively straightforward
to try to modify the firmware on them and attack it.
We can think about what that implies to the industry.
We talked through some ways which binary modifications
can be made to harden the image,
so we don't need to go to the end manufacturers if we want to.
And we talked about how we can use...
I think this is a really good takeaway point.
You've used real-world research and convert it
into challenges to educate both students
and also push people out of their comfort zone.
I think this was the first VDA
that was at the Carr-Akenville CTF.
And yeah, I think we just wanna acknowledge Edward and Sam
who couldn't be here today.
Jeremy and the friendly vendor person
for helping out with this.
So, oh, and lastly...
We're at Red Balloon.
We're hiring now.
So if you wanna learn more about OFRAC for this work,
we can take some questions now,
but we also have a booth over at the Carr-Akenvillage.
And we have two different CTF challenges
that we contributed to the Carr-Akenvillage CTF
this year as well.
So yeah, thanks for your time.