And without any further ado, talking about backdooring pickles.
Thank you.
So, to start off, I work for NVIDIA's AI Red team.
They let me do this research and are letting me share it.
But everything is my own opinion and not an official statement from the company.
But yeah, let's talk about backdooring pickles.
So, 11 years ago, Mark did a talk at Black Hat called Sour Pickles.
And in it, he talks about how to do deserialization attacks against pickles.
Not a whole lot has changed from then, since then.
Back then, 11 years ago, there was a warning message.
It said pickles shouldn't be used from an untrusted source.
It makes sense because it is code.
Pickles are code at the end of the day.
It's assembly language.
And the attacks that he talked about were predominantly
sour pickles.
And he talked about deserialization attacks.
So, this is something along the lines where a web server gives you a cookie.
That cookie is a pickle.
You can replace it with a pickle you craft in order to get a shell on the web server
when they load it.
So, yeah, not a whole lot has changed, as I mentioned.
Pickles still have this warning message.
It's a little more verbose.
It gives you a few more ideas on what you can do.
But at the end of the day, it's still, if it's coming from an untrusted source,
you shouldn't load it.
Because they are still code.
That part hasn't changed at all.
But what has changed is in 2016, machine learning libraries started to be released that use
pickles as the way that they save their models.
And so, this creates something along the lines of a macro in a Word document or in an Office
document where you can send it to someone and if they load it, then you get a shell
on their system and do whatever you want with it.
Yeah.
So, I've mentioned models.
Just really quick, what is that?
Models are used in artificial learning.
For what I'm talking about, they're a combination of layers and weights that probably has very
little meaning to most of you as well.
So, what is a layer?
A layer is the mathematical equation that lets you give it one input, like an image,
and it goes through this equation and gives you a label, like cat, dog, chicken, ostrich,
whatever it is.
And these layers we represent as code in Python, specifically, for the Python frameworks.
The weights are the coefficient.
The coefficient for that equation, it's really just the data that's trained.
So when we're training models, this is what we're training.
We're training the weights.
And that can just be viewed as data and because it's a combination of a bit of code and a
ton of data, it makes sense to store it in pickles as long as you ignore the fact that
researchers are going to be distributing it and sharing it as they do research.
And this is exactly the opposite of what that warning message said we should do.
Just another little note, a lot of the frameworks tend to save a model as multiple pickles in
a single file, which, if you do anything with it, can get confusing at first, but, yeah.
So how do you make a malicious pickle?
This is just randomly, like, just arbitrarily malicious.
It doesn't do anything except for be malicious, sorry.
This is an example from a blog from 2011, and what it says is when you pickle up exploit,
this is how I want you to recreate exploit when you de-pickle it later, you de-pickle,
you re-create it by calling subprocess.p open with bin SH.
Now obviously that's not going to create an exploit class, it will just be an exploit,
in this case to give you a shell on a web server.
So yeah.
That's not what we want to do, though, because I wanted to find a way to place a backdoor
into a model that still works exactly as it's supposed to and gives you a shell as well.
So this is the simplified version of the attack if all you want is a shell.
You don't care about it working still.
Any questions?
And it gives us a nice example to talk a little bit more about pickles.
So this output on the screen is thanks to a library built into Python called pickle
tools or the library's the module is pickle tools and it's built in and it has a function
called dis for disassemble and it produces this output.
The first box in the output is getting the function pointer to subprocess.p open.
It writes subprocess and p open onto the stack and then it calls stack global which
takes those two pointers, collapses it all down into a single pointer that points to
the function.
After that in box two we mark the beginning of all of our parameters, write them all onto
the stack as well, call tuple which again collapses them all down into one pointer
that is all of the parameters.
And then the last instruction is really the most important.
It's reduce and that calls the function with the parameters and so it takes the two pointers
on the stack, pops them off, throws them away and puts onto the stack just the one pointer,
to the return value.
So now that that's understood a little bit more, how do we make a pickle more malicious?
There's a really cool tool called fickling.
It was released last year in the AI village and it's made by Trail of Bits.
It's really good.
It allows you to inject code into the beginning of a pickle or the end of a pickle and it
attempts to detect if there's malice within it.
Unfortunately for me, it's a little more complicated than required.
And so it ended up being easier for me to write a solution that only injects in instead
of trying to help patch some of the bugs with how it worked.
And then the second issue which would have been easy to fix if that was it is that it
can only inject into the beginning unless you want to replace the output in which case
it can only inject into the end.
So why do I say it's a little more complicated?
So as we've covered, pickles, you can't just load them because you'll get a shell on the
system or you could have somebody doing that.
So they use a symbolic interpreter so that won't happen.
And that works really well.
But you have to reimplement everything about pickles in order for that to work.
And I kept running into bugs and ultimately it just became easier to do something else.
And when I was trying to patch it, like, the tool they created is what led me down all
this research in the first place.
So I really can't give them enough credit.
It's just I found it was easier to use something else, unfortunately.
Now, why can it only inject into the beginning?
As I mentioned before, when you reduce down, so on the top of the slide in the blue box
is the code that fickling adds.
In this case, it's exacting print high, nothing fancy, but it works for the example.
So when that all runs, it leaves one pointer on the stack and in the beginning of a pickle
that doesn't matter because it all collapses down, creates an object, and that object is
returned and you have one value left on the stack that, you know, as the error message
on the bottom says, it's not empty, so it's technically wrong, but it doesn't matter because
everything works.
But if you put that same code anywhere else, it's going to alter the stack in a way where
the instructions that follow it don't work correctly because the stack is effectively
corrupted for what they're trying to do.
And I wanted to be able to just add code arbitrarily into a pickle.
So just a few more things about pickles and then we can get to the attack.
Pickles are an instruction type, instruction set, not a file type.
So for detection, this starts to matter because pickle doesn't have any magic bytes that you
can detect to identify it as a pickle.
There's no specific file extension because PyTorch is .pth and not everything is a .pkl.
It can be anything.
NumPy like NPZ sometimes has pickles and sometimes it doesn't.
And so...
You don't have just one file you can look at.
Also there's no forking or conditional logic which makes it so that we can inject in the
middle because we don't have to worry about it jumping around.
Byte offsets don't matter.
You can just add code in and it should be fine.
As we've shown, you can import any Python callable.
Subprocess isn't imported when we exploited it earlier.
Pickle actually imports the subprocess module and gets the function for you.
So if it's installed on the system, you can use it.
But there's an instruction called pop that takes a value and discards it.
So I keep saying that I had a simpler solution.
On the right side of the screen are the 50 lines of code that I used to inject into any
pickle.
And 50 lines of code was easier than writing patches.
And so when I wrote this, I had a few requirements.
I wanted it to not be obvious to the user or intrusion detection system because I don't
want to get caught.
I wanted to parse the pickle without loading it.
But at the same time, I don't want to get caught.
I don't want to use a symbolic interpreter because I saw how that can get complicated.
And I want to inject into an arbitrary location.
Mostly that's just to help support the first point.
But also because I can.
It makes it harder to detect.
So how do we avoid being obvious to a user and an IDS?
Well, we turn off our own thread so it hides in the background.
We don't want to stall the pickle.
We want it to load and everything to work exactly as the user expects.
If they're loading up a model and it's supposed to generate a picture of a pickle, we don't
want to see a cat.
They need to see a cat, otherwise they're going to get suspicious.
So we have our thread running in the background that is running our agent to call back to
our C2.
Also size isn't a concern at all.
Most models are hundreds of megabytes or more, like gigs, maybe more than that.
So adding one meg or one kilobyte, nobody's going to notice, most likely.
That's my assumption.
If I'm wrong, so be it.
But I do use Zlib compression.
And that's not to actually make things smaller.
It's because base 64 makes a really long string that you're adding into the file.
And if you run strings and you see a long string, you might get suspicious.
Zlib makes binary data.
So it was just a simple solution.
And then finally, as long as we don't launch Mimi cats or anything that's just like super
malicious and obvious and going to get detected, should be fine.
Shouldn't be detected.
So how do we load the pickle without using a symbolic interpreter?
So the output I showed before.
There were some numbers.
Some numbers on the left side of the disassembly.
Those are the byte offset into the pickle.
And because every instruction, there's no jumping, there's nothing like that that we
have to worry about, we can make those byte offsets change, make those byte offsets change,
and nothing's going to go wrong.
It'll still work the same.
So long as the stack is exactly as it's supposed to be.
So what we do is we pick an arbitrary line from this disassembly, grab the number, write
that many bytes into the output file.
Write our malicious instructions into the output file and then write the remainder into
the output file.
And you have a pickle that will run your instructions.
So long as you're not ‑‑ so long as the stack is exactly as it was when your instructions
started.
So how do we avoid leaving a trace in the stack?
Well I only use pickle instructions that alter the stack.
If you know anything about pickle instructions, you might be wondering why I don't mention
anything else.
And that's because I just don't use it.
It does exist.
If you want to know about it, Mark's talk covers everything and it's really good.
But for our examples, we only use the stack and so that's all covered.
And because we only use the stack, that's all we have to clean up.
And so at the end, we add a pop instruction and it pops off whatever ‑‑ well, in
this case, we're adding a single value to the stack, pops that one value off and everything
is exactly as it was initially.
So everything works exactly as it's supposed to.
So this is a bit of an example.
Here I'm downloading a pickle that I've injected an agent into for the mythic C2.
And I'm loading the pickle and it takes a second, but I'll get a callback.
This callback is running in the background while the GAN is still loading.
When the GAN finishes loading, you see a picture of a cat.
I can tell it to start running calc, which I did.
And when the agent calls back, calc will start even though I'm able to look at a bunch of
different pictures of cats while it's working.
Calc pops.
Yeah, in a real attack, obviously you're not going to pop calc because that's obvious,
but you have an agent running on their system, you can do whatever you want with it.
The hard part about the attack is figuring out how to inject the code into a malicious
agent ‑‑ or inject the malicious code into an existing pickle.
The way that you go about doing this, it's really up to you, I guess, but there's three
ways I see it working out.
There's the supply chain attacks, if someone is distributing a bunch of pickles on a website
or however.
If you modify them, then you can get a shell on all of their customers, assuming that they
don't detect it.
There's also watering hole attacks, if a bunch of data scientists have a shared network share,
they keep passing pickles or pickled models back and forth between each other, and you
have access to that as well.
You can modify it, and the next one of them that loads the one that you modified will
spawn a shell on their system and you have access.
And then finally there's phishing.
Instead of targeting a business person, you can send an awesome research paper with a
real model that really works, that's actually the research, to a data scientist, and when
they go and load it up, you can get a shell on their system as well.
And since research is done by a bunch of different countries, not always who are friends, but
are often sharing data, that's a bit of a concern.
And then after that, you just wait for a call back and do what you want with it.
So what can we do about this?
So what can we do about this?
Well, antivirus software doesn't work great for detecting this.
Pickles aren't like a file type.
A coworker of mine wrote a Yara rule, but because it's just a full instruction set,
you can do anything you want with it.
You can make the code do anything you want completely.
You can load any library.
So it's essentially detecting an arbitrary Python file, except written in a different
form that's binary and harder to read.
So there are rules that we wrote.
They're pretty easy to bypass if you know what you're doing.
Maybe there's a way to make them better.
There's probably ways to make them better, but I still don't think you're ever going
to be able to completely detect every way you can make a pickle malicious.
So all you can really do is verify that the integrity of the file is exactly as it was
when you created it.
But that doesn't help anyone who's getting a file from you, unless you have a way to
share a hash that's more secure than the way you share the pickle, which maybe you do.
And so you can do that.
But you sort of run into an issue of how does anyone know that it hasn't been modified?
Fickling has a feature for check safety.
But as I mentioned, there's a lot of different ways you can make something malicious.
And so the last line of their output is don't unpickle this file if it's from an untrusted
source because it can still be malicious.
The examples I make aren't things they thought of, so it doesn't detect them.
So what can you do if you have a pickle that comes from a source you don't 100% trust?
My recommendation is that you do not load them.
You just shouldn't.
If you really want to, there's people on the Internet that have examples of how you can
create a list of approved functions that will only those functions can be loaded by
the pickle.
But at that point you're creating a Python jail and there's a lot of CTFs where we've
seen people breaking out of Python jails and a lot of issues that we've seen in Python
jails over the years.
So I wouldn't recommend that.
But you'll find that option on the Internet if you really want to try it.
So what do I recommend?
Well, if you're creating something new, then you can release your layers as code.
Pickles don't really hide what you created in any way, in any meaningful way.
It can all be extracted by anyone who wants to.
So you can release your layers, all the code that you write, as actual Python.
Whoever is going to iterate off of it will have an easier time doing it and they won't
be vulnerable because they can look at the code and make sure it's not doing something
sketchy.
And then you release your weights as binary data.
If you wanted to do this in PyTorch, on the right-hand side of the screen is some code
which I'm also releasing at the end.
And that code will, for PyTorch, save the state of the weights and write it out to a
NumPy blob for you and load it.
And if you look closely, you see the NPs.
NP lib, NPYO, like the super deep function inside the NP library that you're not really
supposed to call, that's because the default one will fall back to Pickles if you have
something in it that makes it want to.
So it's not really a good function to use.
But you can.
It exists.
And then if, you know, this has been around for six years, so there's a lot of Pickles
that already exist that are malicious, well, not malicious, that could be made to be malicious.
And.
Okay.
It's sort of unrealistic for me to expect them to just delete all of them and start
over as much as like that would be the safest solution.
So if you have some of those, then you need to protect them like they're unsigned executables
because effectively that's what they are.
That means verifying that their integrity is exactly what you expect it to be.
If you're offering people the ability to download it, it needs to be over an encrypted channel
because otherwise an attacker in the middle can make it whatever they want it and get
a shell on the system.
And if an adversary gets access and could have access modified it and you have no way
to verify that they haven't, you sort of just have to delete it all and recreate them,
which sort of sucks.
So don't let that happen.
And some of you that know about machine learning models might be thinking, like, okay, I don't
use Pickles, I use Onyx or TorchScript or something else that still supports arbitrary
layers.
And arbitrary layers are going to be in some way still code.
So most likely they're also still vulnerable, it just is going to take more work to do it
because there's not ten years of research showing you how to just add malicious code
into it.
For example, Onyx already has a POC, thanks to Lobato, I'm not sure if you can read the
link at the bottom, but that's his GitHub where he has proof of concept on how to add
exploitable code into an Onyx file.
.
If you're someone that wants to do this kind of research, Onyx and all the other ways to
save models could be something interesting to look at to add malicious code into as well
because I'm pretty sure it's possible even though I haven't tried it myself.
If anyone wants to see the injection code, I have it on my GitHub.
If anyone uses Mythic C2, I made a wrapper, so all you have to do is upload the Pickle
you want to inject into.
It'll inject a Mythic Medusa agent into it and you can put that where you want and wait
for a callback.
And yeah, if you have any questions you can ask, but also hit me up on Twitter and I'll
do my best to respond.