My name is Jong Hyuk Song. I'm the third author of this talk.
Actually, first and second authors are playing CTF now, including me.
So if you want to see first and second authors or me, you can see there.
Anyway, today's talk's title is
Automotive USB Forging, How to Forge USB in Vehicles to Discover the Real-World Vulnerabilities.
In this talk, we propose a way to conduct USB forging on a real car.
We tried USB forging to real cars and found some vulnerabilities.
Today, I'm going to show how we did it.
First, I'm going to introduce my team.
Actually, we are all Red Team members and we are all offensive security researchers at Autocrypt.
Autocrypt is a mobility security company, especially focusing on automotive security.
So we have been conducting penetration tests and forging tests with OEMs and tier companies.
We are actually doing security tests to the real cars with car companies.
So, of course, among the tests, we are doing USB forging.
So I'm going to tell you how we are doing USB forging in the field.
And also, I will explain the limitations of the current method.
And I will describe how we have improved it.
So let's talk about the current USB forging in the automotive industry.
There are three characteristics in USB forging to real cars.
First, it's a black box forging.
OEMs usually don't have a source code of the car because tier suppliers don't provide a source code to the OEMs.
So we also don't have a source code, and so we cannot rebuild the source code for forging.
And in most cases, our target is completed vehicles with a production version of ECUs.
It means that there are no debugger ports, so we cannot monitor inside of the car.
Because of these reasons, most security tests to the cars are usually done in the black box.
So we have to find a way to test well even in the black box testing.
Second, there is no commercial USB forger that can be directly connected to the car with a USB cable.
There are some research about USB forging, but they are not useful forgers for car forging.
In particular, there is no USB forger that can be used in a black box environment.
We were able to test USB on the completed car.
To do that, we needed to connect a test PC to the car by USB.
But there is no commercial USB forger which can connect to the car directly.
Because of the first and second reason, there is a third reason, third characteristic.
To test the attack over USB, automotive OEM began to test the media players in the infotainment system
by inserting USB stick containing malformed media files.
I don't know how all automakers do USB forging, but as far as I know,
many USB forging tests are usually done in this way for the completed car.
So, this slide shows the current USB forging methods, forging procedures that are performed in the field.
First, the pointer is so small, sorry.
First, testers generate the malformed media files by forger in the PC.
Second, testers copy the malformed files to the USB stick.
And third, testers insert the USB stick into the car.
This USB stick actually inserted into the USB port of the head unit issue, infotainment issue.
Fourth, most media players in the car automatically browse the file in the USB stick and play the files.
Fifth, when media player plays the files, the media player could be freezed or issue could be rebooted.
So, testers should keep an eye on the media player while forging to check the failures.
And if all files are played, testers go back to step one and repeat all these steps again.
Most cases, head unit issue.
Head unit.
In this procedure, we usually play hundreds or thousands files at a time.
It depends on the maximum number of files that media player can load at one time.
Of course, the files are abnormal files, malformed files.
The media player does not play most files and just move on to the next file.
So, usually it takes about 10 minutes or 5 minutes to play a few hundred files.
It means that tester has to repeat this process every 10 minutes.
This is quite a burden and annoying task to the testers.
And there is one more important thing.
We are testing the completed car.
It means that testers should do this in the car, sitting on the car seat.
It could be also very uncomfortable to the testers.
Okay, so let me summarize the limitation again.
First, each step requires many manual efforts to testers.
There are many steps in the test procedure.
Testers should generate the files, copy files, and insert sticks.
And every test cycle, testers should repeat that action again and again.
It's very burden to testers.
Second, there is a limited number of files that can be tested at a time.
This is the main reason that testers should repeat these steps several times.
Each media player in the car has the maximum number of files that can be played at one time.
Even if you can put millions of files into a USB stick,
the media player only takes the hundreds or thousands of files and plays only them.
So, when all files on the USB sticks are played,
the tester must generate new files again, and put them into the USB stick again,
and insert it into the car again.
As you know, it's a fuzzing test,
so we need to test as many files as possible to find vulnerabilities.
So, testers should repeat those actions several times to test enough files.
And third, to detect failures by fuzzing,
testers must keep an eye on the car while it's being fuzzed.
So, in this test, testers use a media file fuzzer.
It's not connected to the car.
It just generates malformed files in the PC.
So, the fuzzer cannot detect the failures caused by fuzzing.
Therefore, testers should observe the media player while fuzzing is performed.
As you know, as I said before,
to find vulnerability, testers should test a large number of files.
It means that he stays and watches the media player for a long time in the car.
It's not that good.
Fourth, it's application fuzzing.
This method cannot do kernel fuzzing.
So, it's obvious because it's media player fuzzing.
But testing only media player is not enough to check all USB attacks.
So, in this talk, we suggest a new automotive USB fuzzing method,
which is practically effective.
We try to remove the current limitations that I mentioned just before.
So, what is the requirement of the new fuzzing method?
First, fuzzer should transmit malicious files to the car directly.
Second, fuzzer should monitor the car to detect failures caused by fuzzing.
Third, fuzzer should be able to test the kernel area.
So, therefore, in this talk, I would like to introduce automotive USB fuzzing method
that satisfies these three requirements.
Actually, I will explain later.
Second requirement is difficult to completely satisfy.
In some cases, we cannot satisfy the second requirement yet.
Because it's black box fuzzing, we cannot monitor the inside of the car.
But we try to suggest the best ways to detect failures in black box fuzzing.
So, main contribution of the talk is that we did not develop new fancy fuzzing techniques.
There are already great USB research, and we show how to apply them for fuzzing cars.
And finally, we found some vulnerabilities in several cars.
I'm going to show how to do that.
So, how do we solve the challenges?
First, let's connect fuzzer to the car directly by USB cable.
If fuzzing is possible in this way, fuzzing process becomes very simple.
There are only two steps, connect fuzzer and car, and just start fuzzing.
If it's possible, fuzzer can automatically transmit malformed files to the car,
and fuzzer can monitor the car to detect failures.
It means that testers don't have to move the files and don't have to monitor the car during fuzzing.
So, let's talk about the advantages first.
First, it reduces the workload of testers by removing the test steps.
Because in the previous method, it requires five steps,
and more of the steps should be repeated several times to test enough number of files.
But if fuzzer can transmit the files to the car by itself,
fuzzing process becomes very simple.
Second, in this test, fuzzer can detect your failures,
so testers don't need to monitor all the time.
And third, fuzzer can automatically generate malformed files and send them to the car.
I don't know why it moved to the next page.
Sorry.
Fuzzer automatically generates malformed files and sends them to the vehicle,
so fuzzing can continue without human interaction.
And now we can do corner fuzzing.
So how can we do these things?
Because of these two things, we can do that.
First is Linux USB Gadget.
We need to make the car recognize our device as USB storage.
To do that, we need to use Linux USB Gadget.
Linux USB Gadget makes Linux system appear to be a USB device to a host.
So it's a kind of way to emulate a storage device.
As I know, it's only available in Linux, so we need to prepare Linux device.
So in our test, we use Raspberry Pi.
So it means that we emulate the USB storage using the Raspberry Pi.
So car recognizes a specific folder in the Raspberry Pi as USB storage.
So if we put some files in that folder, the media player will get the files and play them.
Second thing is the USB RAW Gadget.
We also use USB RAW Gadget for corner fuzzing.
It's a Linux corner module that implements a low-level interface.
For the Linux USB Gadget subsystem.
Using the USB RAW Gadget, we can transmit arbitrary USB requests to the USB host.
SysCaller already supports the USB fuzzing using this RAW Gadget.
So we can just use the SysCaller to fuzz USB corner stacks in the car.
So we use these two things to conduct USB fuzzing to the real cars.
So let's summarize.
To USB Gadget mode and RAW Gadget, we need a Linux device.
In this research, we use the Raspberry Pi 4.
And then connect the Raspberry Pi to the car by a USB cable.
To transmit data, you should use a USB cable that supports data transmission.
Some USB cables, you know, only support charging.
That kind of USB cable cannot be used in this fuzzing.
And we set up Raspberry Pi for USB fuzzing.
To do media player fuzzing, we emulate Raspberry Pi as a USB storage using USB Gadget.
And to do corner fuzzing, we use RAW Gadget.
First, I'm going to tell you about how to do media player fuzzing.
This figure shows the overview.
All these processes are automatically done by fuzzer without human interaction.
Except the start command.
First, we need to set up Raspberry Pi as a USB Gadget.
And second, the fuzzer generates malformed media files into the fuzzer.
And third, the fuzzer mounts the folder as a USB storage.
And fourth, if the folder is mounted, the media player automatically recognizes it.
Then media player gets all files in the folder and plays them.
During the files are playing, fuzzer monitors Raspberry Pi's corner log,
which is de-message raw for detecting failures caused by fuzzing.
After all files are played, fuzzer mounts the folder and removes all files in the folder
and go back to the step two and repeat the step.
This is how to do USB fuzzing by directly connecting the fuzzer and the car.
We can do this without the USB stick and without human interaction.
So the whole process can be done without human interaction.
So this is the initial setup for USB Gadget mode.
We do this first step in the previous slide.
Because of time limit, I can't explain details in each line.
But if you want more, you can find the details on the internet.
There are already many good articles, papers, explaining how to make a Raspberry Pi USB storage.
And let's talk about how to generate malformed media files.
To generate the media files, we collect normal media files to use them as seed files.
Then we manipulate the seed files using bit flipping techniques.
We just randomly choose 1% of bits of the seed files and flip the selected bits.
That's all. It's very simple.
Actually, you know, there are already several file fuzzers.
But in our test, in our experience, they are not very useful to the media player fuzzing in the car.
We cannot figure out why. We cannot figure out the exact reason.
But we just guessed that most existing file fuzzers mutate the file signature part.
It means that if file signature is mutated, the media player does not play the file.
Most media players do not play the files which have incorrect file signature.
So when we generate the malformed files, we do not mutate the file signature part.
That is our first tip in the car fuzzing.
For the automotive media players.
Next tip is, if the file content is mutated too much, some media players don't play the media file also.
Even if the media file has correct file signature.
So this is why we just mutate only 1% of the file.
I cannot say that 1% is the magic value for all cars, all cases.
But maybe each car has different media players.
So you should find your values.
And there is one more.
Each automotive media player supports different media file types.
So before you generate the malformed media files,
you should check the supported file types and generate only those files.
Because the media players will only play the files that they support.
So you don't need to generate the files that are not supported by the media player.
Usually media players support many file types.
It's just some example of them.
Now let's talk about how to detect failures.
To detect failures caused by fuzzing, as you can see, our fuzzer monitors the corner log, which is de-message.
But I think you probably noticed that it is not perfect to detect failures.
Because the Raspberry Pi de-message log cannot show the exact state of the target media player or exact state of the car.
It's just the Raspberry Pi's corner log.
But you know, this is black box fuzzing.
The only thing we can use is Raspberry Pi de-message log.
That is currently our best option.
So we have to identify the target state by only using this log.
So let me explain in more detail about how our fuzzer detects failures by the de-message log.
There are two cases.
First, we found out that some issues are rebooted if the media player is crashed.
In that case, USB connection between Raspberry Pi and issue is of course disconnected because issue is rebooted.
After issue is rebooted, Raspberry Pi and issue are connected by USB again.
At that time, USB connection log is created at de-message.
So fuzzer can detect failure.
Fuzzer can know there is something wrong using the de-message log.
And there is a second case.
Some issue is just shut down and not rebooted again.
In this case, there is no USB reconnection log in the de-message because issue is not rebooted.
So fuzzer cannot know the failures.
So the fuzzer just tries to mount the next folder that contains new files.
But the mount will be failed because issue is shut down.
So at that time, fuzzer can detect failures.
Of course, it's not real-time detection, but it's the best option we can do now.
Let's look at these figures.
Left figure shows the fuzzer's log and right figure shows the de-message log.
At this time, we conducted fuzzing using 100 files at each trial.
As I mentioned before, our fuzzer repeats mount and unmount folder several times.
So there will be USB reconnection log as many as the number of mounts.
In this figure, we mount a test folder three times, but there are four USB connection logs in de-message.
And the last log is a little bit different from other logs.
So fuzzer can notice that there is something wrong in the media player or issue.
So in this way, the fuzzer can detect failures.
Actually, of course, there is a limitation of our method.
We are doing black-box fuzzing.
Fuzzer in Raspberry Pi cannot know exactly which file is currently playing in the media player.
Fuzzer generates hundreds or thousands files in the folder and mounts the folder.
Then media player gets all files and plays them.
So when a crash is detected, fuzzer cannot know which file caused the crash.
Fuzzer should find the file causing the crash by replaying the files again one by one.
To solve this limitation, we tried two things.
First, we can just put only one file in the folder and mount the folder and play just only one file.
If we do fuzzing like this, when a crash occurs, we can exactly know which file is causing the crash because there is only one file.
But in this case, mount and unmount is required for the number of files.
Mount and unmount is a very slow operation, but we have to test a large number of files.
So we need to reduce the number of mount and unmount.
So in our experience, playing only one file at once is not a good idea.
So we need to find the appropriate number of files each one cycle.
This is why we generated and played only 100 files at each trial in the previous slide.
We can play more files, but we don't.
Because if there are too many files played at once, it's difficult to find the file that caused the crash.
I cannot say 100 is the ideal number for all fuzzing.
It's very different for each cases.
So you should find the basic number of files for your own test.
And actually, there is one more limitation.
In some cases, some issue is not rebooted and shut down, even if the media player is crashed.
In this case, USB is not disconnected, even if the media player is crashed and stopped.
So fuzzer cannot detect failures.
Test habit monitor car again.
It's one of our limitations and it's one of our future work to solve this one.
We found these cases in GM Volvo cars.
What they have in common is that they use Android Automotive OS.
So we just guessed Android Automotive OS is not good case for our fuzzing.
So until now, I talked about how to fuzz automotive media player by USB.
But to test attacks over USB port, media player fuzzing is not enough.
Let's test automotive kernel by USB.
You know, there is already great research project about fuzzing USB kernel stack.
In this project, this also implements raw gadget for USB kernel fuzzing.
And he also implements USB fuzzer using the raw gadget.
And this fuzzer integrated into Shizkalo.
You know, Shizkalo is the best fuzzer for kernel fuzzing.
So we just use Shizkalo to fuzz USB kernel stack in the car.
But let's think about it.
Shizkalo is the coverage guided fuzzer and it requires kernel build for fuzzing.
But we are doing black box fuzzing now.
We don't have source code of the car.
So it's difficult to use Shizkalo as it is.
So how can you generate the fuzzing inputs for USB kernel stack in the car?
But we can do fuzzing using Shizkalo.
Actually, Shizkalo for USB already found about 300 or more bugs in kernel USB stack.
That bugs are reported to the Shizkalo appspot.com.
In that reports, there are codes to reproduce the bug.
So we can use the reproduction codes for our fuzzing input.
We just collect the reproduction codes and just execute the code by using Shizkalo programs.
Shizkalo provides a way to reproduce the bug with a specific input.
So you can reproduce the bug founded by Shizkalo using those two programs.
I think many of you guys in here already know how to use Shizkalo.
Anyway, this is the example code, example command to reproduce the bug founded by Shizkalo.
In this case, the name of the reproduction code is crash01.log.
And the figure in the bottom shows one of the reproduction codes in the Shizkalo appspot.com.
This figure shows that the bug list reported in the Shizkalo appspot.com.
Actually, there is a lot of bugs on this site except the USB bugs.
So you should filter only the USB bugs.
The link in the bottom will show the only USB bugs.
And there is one more thing.
Actually, all reported bugs don't contain the reproduction code.
So you can only find the reproduction code in the articles if Shiz or Shiz is written in repro field.
So you don't need to collect all the bugs.
So this is the overview of the fuzzing USB kernel stack in the car.
It's very simple.
First, just collect the reproduction code from the Shizkalo appspot.com.
And then just reply the codes.
Of course, we can also mutate the reproduction code.
Mutate the data part of the reproduction code.
But actually, we can find the vulnerability using just the original code.
So in kernel fuzzing, we also monitor the message log to detect failures caused by fuzzing.
All cars we tested were rebooted if kernel crashed.
So fuzzer can detect all failures by using this method.
So now let's talk about testing with real cars.
Actually, this is the most challenging part in the car hacking.
Because we don't have enough test cars.
So even if we have a very good idea, it's very difficult to prove using the real cars.
Luckily, our fuzzing test can be performed without disassembling the car.
And it does not cause serious damage to the car.
So we can test using the rental car.
So we rented cars for a day or just a few hours and tested them.
There were not many different types of cars that we could rent.
But we rented as many as possible and tested them.
These are the summary of the vulnerabilities we found.
We found vulnerabilities from Renaults and Chevrolet and Volkswagen.
And we also found a vulnerability in AGL.
Actually, AGL is tested on the Raspberry Pi.
And actually number 7 vulnerability is not reported.
Because we found number 6 and 7 vulnerability in the same version.
And we just only reported number 6 first.
And Volkswagen patched the vulnerability very quickly.
Before we reported number 7.
And number 7 vulnerability is not working on the patched version.
So we cannot report that.
We should report number 6 and 7 at the same time.
But we made a mistake.
Because we were quite busy at that time.
We just forgot to report number 7.
Anyway, I'm going to show our demo video.
So media player lost the play again.
There is a Raspberry Pi here.
PC just controlled Raspberry Pi.
And it just shut down.
This is the media player fuzzing case.
And this issue is shut down and not rebooted again.
So tester tried to turn off the car and turn on the car again.
Interestingly, car is back working on normally.
But the head unit issue is still not working on.
So we have to hard reset the head unit issue.
Yes, this is the first demo.
So we can crash the head unit issue by the media player crash.
In this test, we tested while driving.
You can see the head unit issue is rebooted.
This is the rebooted case and it's also media player fuzzing.
There is a Raspberry Pi over there.
So you can see that issue is rebooted by crash even if the car is moving.
Third video shows the corner fuzzing demo.
It's also moving.
There is a Raspberry Pi and issue is rebooted even if the car is moving.
The last demo is also corner fuzzing.
Okay, these are our demo video.
So let's conclude my talk.
The current USB fuzzing used in the automotive industry has many limitations.
It's complicated. It requires huge workload to testers.
These limitations can be addressed by directly connecting fuzzer and car via USB cable.
And USB gadget and low gadget makes it possible.
So we can significantly reduce the workload of the testers.
And also we can test the USB corner stack.
And also we can find the real world vulnerabilities.
Okay, it's end of my talk. Thank you.