Thomas, thank you very much for being here.
Looking forward to listening to your talk.
Thank you.
Hello, thanks for staying so late.
We're going to talk about Solana and its JIT compiler
and a couple of lessons we learned from fuzzing it.
First off, I'm Thomas Roth.
I'm a security researcher.
I lead something called Kraken Security Labs, which
is the security research division of Kraken.
Normally, we do things like play
Doom on Bitcoin ATMs, hack hardware wallets,
all that kind of stuff.
But before we get started, there are a couple of people
we need to say thank you to.
So for example, the Solana team,
who are pretty good sports about all of this.
Then Secret Club's Edison deserves a lot of recognition,
as you will see later.
And also, the AFL++ team, who were nice enough to help me
out a couple of times.
Now, before we get into the fuzzing part,
let's quickly talk about Solana.
So Solana is a very well-known and well-known
So Solana self-describes itself as a high-performance
blockchain, which means one of their main goals
is to execute as many transactions per second
as possible.
If you look at, for example, Ethereum,
you normally execute a couple of transactions per second.
But on Solana, their goal is thousands
of transactions per second.
Now, on top of Solana, you can run smart contracts.
And if you've worked with, let's say, Ethereum before,
you probably know that normally those contracts are written
in domain-specific languages.
So for example, if you're working
on Ethereum, it would be Solidity, and so on.
But on Solana, instead, you can write your contracts
in C, C++, and Rust, and so on, which is pretty cool.
And it's pretty cool how they implemented it.
And we are going to take quite a deep look at that.
Solana is a proof-of-stake slash proof-of-history blockchain.
And so no mining involved.
Instead, we have so-called servers
that are called validators that basically achieve
the consensus on the network.
And everyone can become a validator.
It's basically just a beefy server, you know,
like 12 cores, 128 gigs of RAM.
And anyone can basically participate in that.
At any time in the Solana cluster,
one of the validators is elected the leader.
This is just a high-level description.
The details are a bit more involved.
But basically, at each time, we have one leader.
And that leader is basically in control of the order
and the way the transactions are executed.
And so basically, if we submit transactions
to, you know, the Solana network,
they would all arrive at the leader.
And the leader would then basically decide
on which order these get executed in.
Then it would, you know, run the transactions,
execute the smart contracts, and so on.
And then publish first of the order
in which the transactions were executed.
And then finally, a hash of the state of,
let's say, the chain after they got executed.
Then all the other validators basically take the order
that was published.
They execute it.
They read the same transactions again.
And they validate that indeed they get the same hash.
Now, the leader position always rotates through.
And so if we have, for example, a malicious leader,
eventually it will get ejected.
And so, for example, if our leader would publish
a malicious state, or, well, a fake state,
then the validators could say,
hey, we don't agree, and basically eject the leader.
That's just on the chain side.
But what we are really interested in
are the actual smart contracts.
Now, Solana smart contracts are,
can be written in C and C++,
but most that you will actually see are written in Rust.
Rust, I think, is a pretty good choice
because, you know, you get all those nice safety features.
You have memory safety.
You have a lot of other things that you really want to have
when your code is literally dealing with a lot of money.
And so, it's a pretty good choice.
Rust is a compiled language.
And so, as a compiled language, we also need a target.
And in the case of Solana,
they choose eBPF as the target format.
And so, instead of, you know,
directly compiling to, let's say, x64,
and run the smart contracts natively on the validators,
they compile into eBPF.
And so, a smart contract is literally, in this case,
an eBPFL file, just as you would find on Linux or whatever.
And yeah, if you're not familiar with eBPF,
eBPF stands for Extended Berkeley Packet Filter.
Basically, it's used in the Linux system.
It's on the Linux kernel and also, you know, on, like,
DSD for network filtering, for packet filtering.
That's where the name comes from.
And it's a pretty simple RISC machine,
like 64-bit RISC machine, fixed-length instructions.
And in the Linux kernel, we actually have a full eBPF VM.
And so, if you do packet filtering in the Linux kernel,
there's, like, a just-in-time compiler, a program verifier,
and it runs those eBPF programs.
The eBPF machine is pretty simple.
It's just 10 registers and a frame pointer.
The instructions are 64-bit wide,
and the encoding is super simple.
You have one encoding that's just the default encoding
for all instructions, and then you can also optionally
have, like, basically 128-bit wide instruction
where you have a 64-bit immediate value.
We just have your regular instruction classes,
such as load and store and just some basic arithmetic.
It's really not much magic, and as you will see,
we don't really have to understand eBPF too much
to actually find vulnerabilities with it.
And if you work in security, for me,
I mainly know BPF from CVEs, and so I mainly know it
as something that has a lot of vulnerabilities in general.
And so, when I read that Solana uses eBPF,
I got pretty curious how they do it securely, basically.
Now, the eBPF ELF that you generate from your REST code
is basically directly uploaded to the blockchain, so you can,
after programming, you will just get the regular ELF file.
There's no special custom encoding or so,
which is pretty nice because you can use
all the existing tooling for ELF files
on those smart contracts.
You don't need, like, all the domain-specific stuff,
basically.
Now, a couple of more things we need to know
about Solana smart contracts is,
if you've worked with Solidity, so, for example,
if you've ever written an Ethereum smart contract,
you probably know that Ethereum contracts
store their own state.
If you have a global variable in Solidity,
the value of that variable is stored
as part of the contract, basically.
And, for example, here we have this public variable,
just text, which has the state hello,
and if we were to submit a transaction
that calls the function foo,
that state of the contract would get updated.
This is not how Solana works.
Solana contracts are completely stateless,
and they don't have their own storage, basically.
The way you actually interact with data
is that you have separate files, basically,
that are called accounts,
and these accounts basically pay rent to not get deleted.
And so, it's pretty nice because if you, you know,
download a smart contract,
you have all the state you need,
because there is none,
but you still need to figure out
what files on the chain and so on
does the contract actually interact with.
Now, the ELF file itself, it's a shared object,
and shared objects, in the way Solana uses them,
have one big caveat.
You can't have any writable global variables,
simply because, you know,
the shared object gets loaded at once.
If you execute multiple transactions
that run the same contract,
you want to make sure that the state of your in-memory
shared object doesn't change,
and so we don't have any writable global variables.
On the memory side, it's pretty simple.
Basically, the ELF just gets put at a fixed address,
hex 1000.
The stack starts at hex 2000,
heap at 3000,
and finally, we have something called program input
at hex 4000.
The program input is basically just a byte array,
and if you execute a transaction
that calls into the contract,
the byte array is basically the argument
towards the smart contract.
So you don't call functions by name or so.
Instead, you just provide a byte array,
and it gets somehow parsed by our ELF file.
Now, the ELF file, once it's done,
once the contract gets executed,
gets run by the Solana VM,
and if you build a VM for let's say BPF,
you have two choices.
You can either write an interpreter for BPF,
or you write a Just-In-Time compiler.
Now, an interpreter has a lot of advantages
in terms of safety,
because you don't generate native code,
you can very easily sandbox it all,
but it obviously has a performance impact,
and as Solana cares a lot about, you know, the software,
high performance, it's very probable that they use a just-in-time compiler. And just-in-time
compilers historically have been a security issue, let's say. They're pretty difficult
to get right. And that's just because of the concept of them. If you do just-in-time compilation,
you basically take BPF code and you somehow generate native instructions. And if you're
not very careful that, you know, somebody can't generate random instructions that they
can freely choose, then you can very easily get into deep trouble. Because basically the
compiler will just write into memory, then map that memory as executable and run it.
And so if you're not very careful, your JIT code might just exploit your existing, like,
basically the process it's running in. You also need to bring your own memory safety.
So let's say I have BPF code that accesses memory at, I don't know, address zero. I want
to make sure that this memory access doesn't actually happen.
On the, let's say, physical process address zero. And instead you want to catch it and
so on. And this is all a lot of work. And it's fairly difficult to really do this securely.
Because you just have a large list of things to do. And yeah. So this all sounds pretty
interesting for some security research. And so we decided to look deeper into the virtual
machine of Solana. You can find the code online on github.com slash solanalabs slash rbpf.
Contains the full machine.
It's written in Rust, which is great, because we get memory safety and so on. And if we
check the readme, we also can see that indeed they chose to use a JIT compiler that only
at the moment supports 64 bit x86. But just because the virtual machine contains a JIT
compiler doesn't mean that it's actually used in the wild. And so we had to kind of
dig through the code a bit. And eventually just found that we should have just read the
release notes. Because since last year in March, basically the JIT is enabled by the
default. For us this is super interesting, because it means everyone can upload smart
contracts to the chain. If we manage to build a smart contract that can exploit the validators,
we can do a lot of very malicious stuff. And so for us, this was super interesting.
Also in the readme, we can see that Solana rbpf is a fork of rbpf from Quentin Monnet.
And if you see that something is based on something else, it's always good to go to
to the original source and check, like, you know,
issues, to-dos, CVEs, things they might have fixed.
And in the case of RBPF, if we go to the GitHub repository,
there are indeed a couple of interesting things.
And so, for example, in the to-do list,
there's literally a point,
improve safety of JIT compiled programs
with runtime memory checks.
Hmm.
Then you scroll further down, and there's this section,
what about safety?
Safety, and it literally says,
it will crash if your JIT compiled program
tries to perform unauthorized memory accesses.
Hmm.
Now, crash is a very wide range.
Like, crash can mean a lot.
Like, crash could mean, you know, a rust panic,
or crash could mean, oh, you can write into memory,
and it, you know, segfaults.
And also in the caveat section,
there's a whole section called,
the JIT compiler produces an unsafe program.
Memory accesses are not tested at runtime,
used with caution.
Now, we could go and, you know,
mentally check if they added safety checks around this,
or we could do the lazy way,
and we just start fuzzing the VM,
and see what we find.
And so, I'm lazy, so I started fuzzing.
And fuzzing a VM is actually surprisingly simple,
at least in this case.
To fuzz, let's say, the Solana VM,
all we need is some kind of input.
So, for example, a simple eBPF program.
Then we start mutating it semi-randomly,
so we just start flipping bits, adding bytes,
removing bytes, and so on and so forth.
And then we execute it in the Solana VM.
Now, we want to get this all as performant as possible.
And so, the Solana VM, if we get lucky,
we want to execute this multiple thousand times per second.
And so, we want to, you know, try random mutations
as fast as possible on as many machines in parallel,
as possible.
And then, hopefully, we, you know,
eventually crash the machine,
and we collect those crashes.
And if we get very lucky, we can analyze them,
and we maybe find some nice vulnerabilities.
That's the plan, and so let's get started.
Now, first we need some simple eBPF programs.
Now, if you want to fuzz a VM,
you want to be sure that you get a very wide variety
of inputs, you know, you want to cover
as many instructions as possible.
In this case, we're looking at ELF.
ELF in itself is already pretty complex,
and so you want to have a lot of different ELF binaries,
different types, and so on.
And luckily, the RBPF code base contains test cases
for a lot of things.
And so, we can just take these tiny binaries
that are already made to be almost the ideal fuzzing input,
and we can just take them and feed them into our fuzzer,
into the VM.
And now, one of the issues we encountered at this point
is that we wanted to analyze these binaries.
We want to, before we put them into the fuzzer,
we want to look at the instructions
that are generated for them,
we want to look at what did they do,
do we really get the instruction coverage that we require?
Which puts us at the place where we wondered,
how can we actually analyze them?
At this point in time, there was simply
no good reverse engineering tooling
to actually look at the compiled output
of a Solana smart contract.
Now, there were, like, this is something
that gives you a huge tech stamp, but that gives you, you know,
that doesn't really help you,
because Solana contracts tend to be pretty complex
and pretty large, and so you can easily have
a lot of libraries in there,
you can have a lot of dependencies and so on,
and it's very easy for these to get very large,
and you really want to have an interactive disassembler
to work with them.
I'm a big fan of Ghidra, and unfortunately,
Ghidra does not come with native eBPF support,
but Nalin98 built a processor module for BPF,
and so, with that, we can already, you know,
Ghidra understands the machine code,
we can do patching, we can do all kinds of things,
but we can't load Solana binaries yet,
because Solana has this custom memory map,
we have all these custom things that we need for Solana,
and so we decided to write a completely custom loader
based on this eBPF that basically sets up the memory map,
finds the symbols if they are in the ELF file,
and basically allows us to just drag and drop
a Solana contract into Ghidra and analyze it.
So, for example, if we have a simple hello world binary,
we can just drag it into Ghidra,
import it as Solana ELF file, hit okay,
and then simply double click it
and to load it into the code browser,
run the regular analysis as you always do in Ghidra,
then you go grab a coffee, because analysis takes forever,
and then once it's finished, we can just, you know,
navigate through the binary, we get disassembly,
and we even get decompilation of the full contract.
This is pretty nice, because you can really, you know,
dig through a contract, the decompilation will not help us
with analyzing the fuzzing results,
because those, you want to really look at the instructions,
but this actually helps us a lot with something else
that's unrelated to actual virtual machine security,
but that is important for us to be able to
trust public contracts, because verifiable builds
are pretty important.
If you think about, like, a lot of smart contract projects,
you get, like, a link to GitHub, and it's like,
the contract code is there,
but you can't actually easily verify
whether the contract code is identical
to what's stored on chain.
And so, for example, on Ethereum,
if you go to Etherscan and you look at a contract,
they have this thing called contract verification,
where you can just paste in your contract code,
they will compile it with the same Solidity version,
and then they will tell you, yes, this is all verified.
With Rust, it's not as easy.
If you've done, you know, reproducible builds
with Rust before,
it's not very easy, and so, at the moment,
most contracts on Solana, even the very big ones,
are not verified.
But with our Ghidra tooling, we kind of bring
our own verification to the whole thing,
so we can really just go, we can download,
you know, any contract from the chain.
So, for example, using Solana program dump,
you can literally just go and fetch the ELF file
from the chain.
So, if we run file on this, you can see it's really just
a regular 64-bit ELF file, and then we can drag it into Ghidra,
and hopefully, you know, analyze our contract.
Now, obviously, we can't go and manually reverse engineer
the entire contract, because, you know,
it's really, really large and really annoying.
However, what we can do is we can use the Ghidra program
comparison to actually compare our manual compilation
with what we find on chain, and get a very clear picture
if, you know, the code is really identical.
And so, this really helps us, because, you know,
we sometimes need to trust, you know,
contracts with quite a bit of money.
So, with that, we have our simple eBPF program,
we have our analysis tools, and we are basically
almost good to go.
Next, we're gonna first look at the,
at how we execute our simple program in the Solana VM,
because we want those parts running before we start
mutating, so that we can be sure that everything is running.
And it turns out that this was really, really easy.
So, as I said, our BPF is really, really easy.
It's just written in rust, and so we can really
just, you know, write a couple of lines of rust
to load our eBPFL file into memory.
Then we parse the executable.
We JIT compile it, this is just using the RPF API.
You don't have to read the code, you can get the slides
and read it on there.
I just want to show how short it actually is.
And then we verify the executable.
This step will become important later.
We create the virtual machine, and then we, finally,
just go and run it, simple as that.
And so this is just like 20 lines of code
and you have all the base that you need
to start fuzzing the VM.
So with that we have our eBPF program,
we have our VM test set up, almost good to go.
Next we need to start mutating randomly.
Now you could write your own fuzzer
that's maybe even instruction set aware
and so on and so forth, but there are people
who are much better at writing fuzzers,
especially the people at AFL++.
And AFL++ is really nice because it has Rust support,
we don't have to write our own fuzzing infrastructure,
it comes with native dual core and multi-core support
and so on and so forth.
If you are not familiar with AFL++,
it's a fork of Google's AFL, it's a bit faster,
has better instrumentation and it comes with Rust support
and the Rust support is amazing.
It's super simple to use
and it's really, really fast.
Also AFL++ instruments the target binary
and so it's aware that as it goes through the JIT compiler,
which branches has it taken and it tries
to automatically mutate the inputs
to get as much coverage as possible.
And so this allows us to really ensure
that we get a wide range of the JIT compiler actually fast
because as we are fuzzing it will see,
oh I haven't taken that branch yet and so on and so forth.
And it also comes with a nice UI
and it has features to minimize test cases.
So if we find a crash and the crash is,
the backtrace is basically identical to another crash,
it will just ignore it and save you a ton of work.
And what's really nice is that the Rust integration
for AFL++ is really just add a single macro to your code.
Like we just remove the file loading
and we just add the fuzzing macro,
it gives us a byte array with whatever fuzzing input we have
and then we are ready to go.
Now because we are doing Rust,
we also want to add a panic handler
because otherwise any Rust panic will be counted as a crash
which we don't really care about
because we want to get memory corruption,
not you know, Rust panics.
And with that, we are already ready to go.
We are basically ready to fuzz
with like 30 lines of code or so.
Pretty awesome.
And so I was pretty excited about this
because you know, you just let this run for a bit
and hope for the best.
And so you just, you know, started with cargo AFL fuzz.
You provide a directory with inputs
which are just the example BPF programs.
You provide an output directory
with the, where basically the crashes and so on
will be stored.
And then you finally go and launch the binary.
So let's do that.
And now we are fuzzing.
And if you are not familiar with AFL,
it comes with this nice CLI.
It tells you on the top.
Basically the timing of your fuzzer.
It tells you, okay, how long have I been running?
What's the time since I last found a new path in the fuzzer?
Which is pretty important.
It basically tells you, like if this goes very high,
it means it just didn't find a new executable path
in the binary and so you should probably stop fuzzing.
It also tells you, you know, how many crashes has it found,
how many hangs and so on.
And it tells you the execution speed.
And so in this case, we are running at like 3,600.5.
That's a lot of executions or so per second,
which is pretty decent.
If we let this run on like 12 cores for a couple of weeks,
we get a lot, a lot, a lot of results.
And so we have this running for a couple of hours
and then something nice happened.
We found a crash.
And time to party, hopefully.
We'll see.
And we let this run for a couple of weeks
or a couple of days actually at first
and we got a lot of crashes.
Unfortunately, if you look closer at the crashes,
they tell you which signal caused the crash.
Now, does anyone know what signal 08 is on Linux?
Floating point exception.
Doesn't really help us at all.
It just means that we did something numeric wrong.
No real luck.
But however, all between the signal 08
are also three signal 11.
And signal 11 is exactly what you want to find
because signal 11 is a segmentation fault.
And no matter what you do, a virtual machine should, you know,
never cause a segmentation fault.
So even if this is, like, what,
if you see a segmentation fault,
it's most probably a vulnerability.
So I was super excited.
We had three segmentation faults.
Let's start looking at them.
And so, oops, sorry.
I started executing them
and my target binary wouldn't crash.
Basically, I tried it again, again, again, again.
I wouldn't get a segmentation fault.
I eventually wrote a script to simply run this
like 10,000 times and then it would only crash some times.
What's going on?
Well, it turns out that, you know,
if I had read the AFL manual to the end
and not just skipped to the good parts,
they have this indication for something called stability.
And basically, stability tells you
how deterministic your program is.
Basically, if it executes the program
with the same inputs twice
and it sees that different branches are taken,
it reduces stability and tells you, like,
hey, your binary's not deterministic.
Something is sketchy.
Like, for example, you might have an RNG
that is doing weird stuff.
You might have something else going on.
But how could this not be deterministic?
I mean, we are loading into a VM.
We are running the same binary.
What's going on?
Well, it turns out that RBPF has this feature
that introduces random knobs into the code.
And, you know, it's on by default.
And so that causes, each time you run your program,
it causes it to be slightly different in memory.
And so, obviously, sometimes it would crash,
sometimes it wouldn't crash, and so on.
And simply by setting this
to zero, suddenly, our stability goes up to 98.
Yay.
We actually can fuss nicely now.
And indeed, after running this for, I think, 14 days,
we had 59 segmentation falls.
Yay.
Well, this is the fun part.
Because next, you have to do something
that is pretty difficult and pretty annoying.
Namely, you have to triage your crashes.
Now luckily, we don't care about exploitation too much
because, you know, for us, exploitation doesn't matter
because we care about whether we have
or we don't have a vulnerability.
We also don't really care about the severity too much
because, again, like, if the VM crashes,
that's an issue for us, that's a vulnerability.
We don't really submit to bug bounces.
We just want the vulnerability fixed.
And so, for us, triaging is enough to say, okay, we have an out of TJS,
out of bounds read or we have an out of bounds write.
Yeah, we don't have to write a POC.
And even though we have all these limitations
that makes triaging much, much easier,
it's still quite difficult because if you think about it,
you know, you have your eBPF bytecode.
That eBPF bytecode gets JIT compiled by Rust code.
That Rust code then emits native x64 instructions.
Those native instructions then somehow get mapped
into actual process memory.
And we have symbols for the first two parts
and afterwards it's basically just good luck.
If you take one of the crashes and you run it
in let's say GDB, and you know, we just run it
with one of the segmentation faults we found,
and we look at the backtrace,
all you get is question marks.
And that's not nice to reverse engineer
because you basically have
to exactly know what was written at that address,
by which function, and which instruction,
which BPF instruction causes and so on.
It's pretty terrible to be honest.
And the same with LLDB.
LLDB, if we debug with it, we get basically the same.
But it doesn't even give us a backtrace
since we are just stuck somewhere in no man's land.
But at least it tells us kind of what type of access
caused the fault, like whether it's a read or a write
and so on.
And even worse, I didn't notice for quite a bit
that actually on my machine, for whatever reason,
LLDB had ASLR enabled by default.
And so each time I was running it,
it was crashing at a different address.
And if you stare at like hex addresses for 10 hours,
you stop seeing that they are changing.
And I just checked the first byte,
and I was just like, what's going on?
This code was different the last run and so on.
Yeah, 3HG'ing this is not fun, let's just say that.
And so in the end, we still have a problem.
We still wanted to know what's going on.
And so we started to customize the just-in-time compiler
to basically, on one hand, give us the raw x86 binary,
then give us a map of the eBPF instructions that caused it.
And so we basically automatically annotated
the eBPF instructions in Ghidra
to be able to see, okay, what may have caused this.
Spent hours and hours analyzing this.
And found multiple out-of-bound reads
and out-of-bound writes and so on.
And again, we didn't care too much
about exploitability, we just cared that they are there.
So I can't tell you whether this would be
somehow exploitable in the wild on the VM.
But we wanted to at least try on a test chain
to somehow get our validator to crash from the chain side.
And this is again one of these points
where you really should have read the entire code
before you start fuzzing and get too excited
about vulnerabilities.
Because it turns out that the eBPF runtime
contains something called a program verifier.
And that program verifier happens to catch
basically all our crashes.
So we have vulnerabilities in the VM.
But they are not exploitable on chain.
So this lowered kind of the severity for us quite a bit.
But we still had some nice vulnerabilities.
And so let's still, we need to go through disclosure.
We submitted this talk, we did this early in the year,
started disclosure and so on.
And on the first of May we submitted to DEF CON.
On the 11th of May we basically got an email
by the DEF CON review board saying,
hey is this your blog post?
And so it turns out that Edison from Secret Club
basically fuzzed exactly the same VM as we,
using a different fuzzer.
And found obviously, because pretty straightforward,
basically the same things we found.
Some memory corruption vulnerabilities.
And we basically had a full bug collision with Secret Club.
The patch version fixed all our phone vulnerabilities.
And we just told the review board,
hey we didn't know about this, but this is not us.
But it's very similar to what we do.
And yeah, that was kind of a sad day.
Congrats to Secret Club, 200k bounty is pretty nice.
And our fuzzer is basically not useless,
but yeah, it only finds floating point exceptions now.
But yeah, I mean sad, but in the end we still learned a lot.
We still have a lot of takeaways.
First off, just in time compilers are really
a lot of fun to fuzz.
It's pretty cool because you have a lot of different pieces.
They're pretty easy to fuzz,
which was kind of surprising to me
because I always kind of considered it a bit dark magic
because you know, you hear about all these people fuzzing
JavaScript virtual machines and so on.
Turns out with a small JIT it's actually pretty easy.
Triaging it however is pretty difficult and can be
pretty, pretty annoying.
And also performance can really impact security.
Sometimes you have to make a decision on whether you want
performance or security.
In this case, using a just in time compiler,
it's somewhere in the middle.
It's pretty difficult to get it right.
But luckily, like since then, they've added like native
fuzzing infrastructure to the whole VM.
And I think, yeah, it's pretty secure by now.
Also, if you find a vulnerability,
don't get super excited, verify that it's actually
exploitable in the wild.
And unfortunately, bug collisions happen.
And yeah, that's all I have for you today.
I hope you like this.
Thanks a lot.
Cool.
Hope you enjoyed DEF CON and see you next year, hopefully.