Here's our second talk of the day. It's Eric Gallican.
He's a... well, you see his Who Am I page, but he's also a PhD student at Brandeis and
an all-around awesome guy. He does way too much stuff and is very enthusiastic and involved in
the community. So I'm just going to let him talk about... on his intro talk about...
for machine learning people.
Hi, everybody. Welcome to my talk. This is Baby's First 100 MLSat Words.
My name is Eric Gallican. I'm an artificial intelligence researcher at Rapid7. I specialize
in breaking machine learning stuff and applying machine learning to detecting bad things.
I also do research at the Montreal AI Ethics Institute on applying DevOps principle to
machine learning ethics. And last year, I was here at DEF CON in person, not here in my house
at DEF CON, presenting at the Cloud Village on a piece of malware that I wrote. Malware,
not actually malware, just in case the legal team is listening. And I presented that piece of proof
of concept malware called Sassy Boy. But mostly I do boring math stuff. And if you can't see my
speaker video for some reason, I put a picture of my face on the right so that you can see it. Or
you can see it twice. You can see that my hair hasn't changed basically at all in years. So
before we get started, let's just answer the question, what is MLSat? Because I get asked
to that a lot by practitioners in security and in machine learning, honestly. And mostly it's
an argument. And it's an argument between three groups of people, as it usually is. And the first
group of people is people who think it's only applications of machine learning and artificial
intelligence to security problems. So that's automated malware detection. That's next generation
SIMs. That's automated red teaming and that sort of stuff. And then there are people who think that
it's just securing artificial intelligence in machine learning systems. And then there are
those of us who are correct who think it's both. It's kind of any intersection of artificial
intelligence, right? That broad category of machine learning slash AI and security.
And since this is my talk, and since I am the Mao Zedong of MLSat, I get to decide that it's both.
And if anybody argues with you or you hear anybody having this totally asinine argument, you can just
tell them that I said it's both. And then the argument can be over and everybody can do more
productive things with their lives. So we'll start with deep fakes, right? So D is for deep fakes.
And this man on the right, if I didn't have a speaker video, I probably could have just put
that picture up and said it was me. And everybody would have believed it because he's got kind of
he looks like the kind of guy, right? And I'm allowed to say that because he's not real.
This is not a real person. This is completely an AI-generated image.
And so, you know, to give deep fakes a real definition, they are a convincing synthetic
image, video, or audio recording which purports to be real. So, you know, I'm pretty convinced
by this picture. If I saw it in the wild, right, if somebody put it as like their Twitter avatar
and was like, yeah, I am a software engineer at Google, I could buy that. I could buy that
based on this image, right? You know? We have our stereotypes, right? The thin-rimmed glasses,
the scruff. And it's super convincing. It's super convincing. And I think that's really,
when we think about impact, that's what it is. It's that they're convincing to users. So, we also
see deep fakes in the text generation space where it purports to be a real email. Or we see deep
fake writing which mimics the style of famous authors. We see deep fake audio recordings which
sound like the person who they are trained to sound like. Now, there's a piece of commercial
software out there called Liarbird which does this really well. And so, you know, they're very
convincing to users. And as a result, you know, they get used widely and they're easy to make,
right? That's the other thing, is they don't take as much time as finding and crafting an image or
carefully learning a style or mimicking a voice, right? You just kind of train the model and push
them out. So, they're easy to make and they're convincing to users. Which means that they can
have real impact. But that impact is mostly societal, right? So, there was a famous, you know,
semi-viral video of Jordan Peele doing a deep fake where it was an image of President Obama
speaking. And you can look it up. It's really easy to find. And so, like, the risk there is really
that broad swaths of people will believe that a doctored or deep faked video is real. And so,
there's something like President Donald J. Trump, you know, declaring war on North Korea
on a hot mic or suggesting that he might invade North Korea on a hot mic. That could be really
convincing to people. And so, that's the real risk of deep fakes, is that they are a powerful
disinformation tool which is powered by artificial intelligence. And so, you know,
our mitigation against deep fakes is kind of, eh. Like, there's very little we can do about deep
fakes. They're difficult to detect. They're very convincing to users. They're easy to make. And so,
really, it's a broader social problem that we need to tackle of having people be inherently
less trustful of information sources and really vet where they get their information from,
which is a hard problem. But this isn't an enterprise security threat. This is a social
threat. So, moving back to thinking about deployed models, we'll look at adversarial examples. And
this gif over here on the right is from a pretty well-known example that was really,
really hyped up. And it was the turtle rifle paper. Because the Google computer vision API
is what they attacked. And they 3D printed a turtle. And this is pretty obviously a turtle
to anybody watching who isn't powered by the Google Vision API. And if you are an artificial
intelligence powered by the Google Vision API, I'm so sorry for the rest of this talk. Because
you're not gonna like it. That, you know, it's an input to the classifier that's specifically
crafted to force misclassification. And so, this object as a turtle, just a little, like,
toy turtle, right? Being misclassified as a rifle, the risk there is if you have a CCTV system
that's powered by Google Vision API, which seems reasonable to license, that's looking for
potential security threats. And, you know, you don't input all of that, like, racist physiognomy
that we already see in a lot of those systems. But instead just say, you know what? Let's just
look for weapons. Well, this could falsely trigger those systems. And so, there's a risk there, too.
Correspondingly, if you 3D printed a gun, right? That got misclassified as a turtle,
that would also be a threat, right? So, the impact, really, is that it causes misclassification.
And so, that real world security threat and 3D printed examples are really rare,
and they're really difficult to do. And they don't tend to work in all situations,
and any researcher in computer vision can explain to you why it's not a real part of the threat
model. And that's fine. But our threat is that it causes misclassification. As security practitioners,
we try to look for ways to use technologies to detect bad things. And so, if you integrate
artificial intelligence into your malware detector, for example, an attacker can bypass detection
using adversarial examples. And the thing with adversarial examples is when we think about how
to mitigate them, one of the only truly effective ways to do it is through what's called
adversarial training, where you basically show it a bunch of adversarial examples and tell it,
like, no, classify these correctly. And that's really time consuming, and it's expensive,
and it's hard. And so, not a lot of people do it. And it requires you to take your model out
of production and put it back into training, which, again, has a business impact. So, one of
the other ways to sort of mitigate it is through what's called ensembling, where you don't just
take one canonical model. You take several models and sort of average their output or take a weighted
average of their output or whatever. And so, that's another way to do it. And that's kind of
because it's a lot harder to come up with an adversarial example that works across a variety
of classifiers, especially if they're not trained on the same data, if they don't have the same
architecture. And it's still possible to do, but it sort of raises that barrier to entry. And if we
think about it in terms of, like, cryptography, that's really what we're trying to do, right?
Is we know that there is hypothetically always an attack, right? You can brute force the key space.
That's a hypothetically feasible attack against AES, right? But it's good enough, and it takes
long enough to brute force that key space that, you know, it's fine. It does the job, right? And
that's sort of the same thing here, is you just have to raise that barrier to entry to make it
not worth it for an attacker. So, the next thing we want to talk about
in terms of threats to deployed models are backdoors. And so, backdoors in a machine
learning context are manipulations of trained model weights that result in a specific outcome
each time. So, a common way to do this in neural networks is bias poisoning, where you take one
class and you make the bias for that one class in just the output layer really, really, really big.
So, you're only changing one weight in the whole network. So, it doesn't change a lot, right?
But by virtue of making that weight really big, you'll always get the same
classification. And if it's a binary classifier, right, say it's, again,
our malware detector, because people love to make neural network powered malware detectors
for whatever reason. And I've made one. So, I'm allowed to criticize.
People love to do it. And that's a really easy way to just be like, nope, everything's benign all
the time. Always. 100% of the time. So, you know, the impact is that it causes misclassification.
And I say but not really, because really what it does is remove a class or remove all other
classes from the classifier. So, it's not really misclassification. It's just making
it a one class classifier. And our mitigation is just don't let attackers get access to your model.
And by your model, I mean the trained model weights. And I say this kind of flippantly,
because if an attacker has the ability to manipulate your model weights,
where you're hosting your model, they can do way worse things than backdoor your model.
So, it's not... it's like a local privilege escalation that requires you to have admin
to begin with. It's not really a threat. So, again, attacks against deployed models.
There's model theft. And I want to talk about model theft and give a lot of credit to Will
Pierce, formerly of Silent Break Security, who I stole this graphic from his fantastic DerbyCon
presentation last year. Was given the first CVE for machine learning, which is super dope.
And so, model theft is the process of creating a copycat model by querying a trained model.
Right? So, essentially, what we're doing is we're hitting a model over and over and over again.
And using the outputs of that trained model, we train our own model to give us an approximation
of that model. So, you know, if you have a box and you put in a 1 and you get out a 2,
and then you have a second box, you want to put in the same input and get the same output.
And it doesn't really matter what happens inside of that box as much as it matters that whatever
input you give it, you get the same output. And that's what happens with model theft.
So, our impact here is kind of twofold. And so, the potential for adversarial attacks in model
theft goes way up. Because somebody has a model they can specifically query against to validate
whether or not it gets misclassified without interacting with your model. So, it gives them
sort of a private development environment for these adversarial attacks. And that's what Will
did in the proof putting. Is he created this copycat model so that he could test phishing
emails to bypass the proof point email security appliance. And it was incredibly clever. Super
dope. I recommend watching his talk. But after you watch all the other talks at the AI Village.
Maybe, like, after DEF CON, watch it. It's a really good talk.
The other impact, though, is the loss of intellectual property.
So, one of the things that I think kind of got overlooked in how cool the ability to bypass
security appliances is, is that if you hire a bunch of data scientists and collect a bunch of
data and clean a bunch of data and spend years creating and building and deploying a model that
really differentiates you from the competition, and some unscrupulous company in not Canada,
you know, steals your model, approximates your model, and just shoves it into their product
and says, yeah, we do the same thing. Well, it costs them a lot less money to just copy
your model. It costs them almost nothing in the grand scheme of things to copy your model.
And so, there's no moat, so to speak. So, it really hurts your ability to create a differentiated
product. So, this is a real vulnerability from that standpoint. And we'll get one layer deeper
in our next slide. So, when we think about mitigations for model theft, right,
there's two. And the first one is limiting queries to the model. And this one kind of
feels bad. Because you don't want to limit queries to your model too much, since most
queries to your model are going to be legitimate. For the most part, you're expecting these inputs
and giving outputs, and you created this model, and you're hosting this model, because it's
supposed to be useful to someone. And for most people, because of just the corporatization
of information security and the corporatization of artificial intelligence, it's probably the
people who pay your paycheck who want this. And so, inherently, you want to have really,
but you want to balance that really high uptime and the fact that there may be a single endpoint
that's making a lot of legitimate queries to your model with the fact that there may be an attacker
who's trying to use your model for evil. Trying to steal your model.
And so, the other thing we can do is limiting information returned from the model. And we saw
the efficacy of this with SQL injection, where, like, back in the olden days,
you know, a million years ago, because it's 2020, and so time doesn't matter anymore,
we would return these detailed error messages for SQL queries, and it made it really, really easy
for attackers to find the information they were looking for in SQL databases, especially SQL
databases that didn't do a great job of input validation. And so, you know, once we started
returning basically there was an error as the error message, it became much harder, right? And
it's still feasible to do, but it became much harder. And it's similar here. If all that was
returned was a blocked or not blocked, that binary signal versus a detailed header,
then it would be much more difficult to create a copycat model. And so, you know, that is a
real mitigation. And when we think about model theft, kind of the next step is inversion. Model
inversion. And what model inversion is, is recovering training data from a trained model.
So, there was a paper by Nick Carlini that came out two years ago, I think, in 2018,
about how neural networks unintentionally memorize specific training examples.
There have been papers on model inversion, both traditional and using generative adversarial
networks that have shown that it's pretty effective. And so, the impact here is the
loss of data, right? You're losing training data. And maybe that doesn't matter a ton.
So, like at Rapid7, we have our open dataset, right? And that's free and open for anyone to
use for noncommercial purposes, and we make it available for, you know, other stuff, right?
And so, like, if our model was trained on open data, and somebody inverted our model,
we probably wouldn't care that much, because, like, that data's out there, right? But if you
are training a model for, like, a large medical device manufacturer, and it's trained on sensitive
medical information, and that information gets recovered, well, that's a lot worse.
If you're a data scientist at Equifax or some other credit rating bureau,
and you have a bunch of sensitive financial data that you train a model on,
that's pretty terrifying, right? The penalties for losing everybody's data at Equifax could be huge.
You could have to pay a bunch of people's credit monitoring service for a while,
and that would be bad for your business. So, when we think about mitigations,
really, the first one is to protect access to the model. So, if they have direct access to
the model, it's much easier to do inversion. It's much, much easier to do inversion.
There are different techniques you can use if you have direct access to the model.
And then the other mitigations for model inversion are essentially don't let people
steal your model. Because then it's easier to use it as sort of the discriminator in the GAN.
The architecture is a little more complicated than a traditional generative adversarial network,
but the idea is pretty similar. You basically show it an example and say, does this look familiar?
And it goes yes or no. And if it says yes, then you may have inverted the information.
So, our next threat is poisoning. Data poisoning. And data poisoning is when malicious users
inject bad training data into a model to corrupt it. So, usually, this happens in online learning
where the model is continuously trained on the input to the model. And that is definitely a threat
that needs to be considered. The other option is that they can, you know, get access to wherever
you store your data and just inject bad examples. So, our impact is misclassification. It sort of
shifts our decision boundary to deliberately misclassify samples. So, you know, letting spam
through because now your threshold for what is spam is so high because you've just been doing
online learning, right? And so, when we mitigate it, right, we want to go back and think about
protecting access to our data. So, we want data integrity, right? So, having data versioning
is really important. Making sure that your training data has a version, has an associated
hash, has a label. That's super important. And then our next is we don't want to just allow
unvalidated, untrusted input to our model. That's something we see a ton is people who just
accept whatever input with no validation on the front end. It's just whatever gets input to the
API gets pushed into the model. And that can really cause problems when you're retraining
if you just take, you know, those inputs that you get and sort of store them off for training later.
So, you know, we're coming up on the end of this talk. And so, we want to talk about,
like, what can I do, right? And this is sort of a choose your own adventure story. It depends
on your role. So, if you're a hacker, and by hacker, I mean this in the colloquial sense,
a black hat, or a red teamer, right? Black hats are already doing some of this stuff.
So, I really mean, you know, if you're a red teamer, if you're catching up to the elite
APT black hat hackers, right? Take advantage of the lack of defenses on machine learning systems.
Microsoft put out a report earlier this year saying that in their survey of large companies
and government organizations out of, I think it was 28, only three had any meaningful defenses
on their machine learning systems. So, if you're a red teamer, like, check it out.
If you're a vulnerability discovery person, like, check it out. You can definitely find stuff.
And that's because nobody knows what you're doing, and nobody's looking for you, right?
Nobody is looking for black hat hackers, right? APTs, whoever.
And if you are, let me know. Yell at me in the discord, and I'll be like, oh damn, cool.
If you're a defender, if you're a blue teamer, right? If you're a sock monkey, if you're
a researcher, if you're at DEF CON, you're probably one of these two categories.
Test your machine learning systems as if they're part of your infrastructure.
So, what we see a lot is that machine learning engineers and data scientists develop these models,
and they take these models in, like, a .py file, and they hand it off to engineering,
and engineering goes, I don't know what this is, and they slap a Django API in front of it,
put it in a Docker container, and deploy it in Kubernetes as an API, and then when InfoSec
looks at it, they go, I don't know what this is. It's magic. You put in JSON, and you get
an output. It's magic, and we don't mess with it. And that sucks. Don't do that. Don't do that.
Test your systems. Test the machine learning systems. Work with your data scientists. Work
with your ops people, because this is part of your attack surface. This is part of your attack
surface, right? And don't let people hype you up over AI-generated phishing emails.
When GPT-2 came out, people went nuts and were like, this is the end of detecting phishing.
They're going to be too convincing. And, like, just don't. Okay? It's the same thing that we've
been saying for 20 years. Patch your systems. Look for bad stuff, and patch your systems.
That's it. There's nothing new under the sun. It's the exact same thing we've been saying for
20 years. Just keep looking for bad stuff, and patch your systems. And if anybody gives you
pushback on patching your systems, like, have them talk to me, and I'll yell at them for you.
Because you have to patch your systems. If you're watching this talk and you're a data
scientist or a machine learning engineer, first of all, thank you for coming to DEF CON.
DEF CON is dope. And, like, even though it's free and remote this year, come back next year,
because it's cool. Conduct threat modeling on your models. So, before you put a model into
deployment, work with your infosec team, work with ops, and sort of ask, like, what could go
wrong? Right? What are the risks of an adversarial attack on this? What are the risks associated with
this particular model? And what is the attack surface of this model?
You know, and also when you're deploying those models, work with infosec and ops to balance
uptime with the risk of model theft. These two kind of go hand in hand, is, like, you're going
to deploy these models. Somebody may try to steal them. Is there a risk if somebody steals your
model? Like, does it matter? And if it doesn't matter, like, think about would it be okay to
publish the source code and data for this model? And if your answer to that is no, then you really
need to make sure that infosec and operations are aware at deployment time. And finally, don't
hype people up over text generation models. Like, GPT-2 and GPT-3 are super cool from an NLP
standpoint, natural language processing, but they're not going to change the security landscape.
Stop scaring people, please. I'm literally begging you, don't hype people up over text
generation models. Here are some references. These are some of the things I talked about.
And thank you so much for attending my talk. I'll be in the Discord most of the weekend,
so if you have any questions, feel free to drop me a line. Thank you so much. Have a great weekend.
Enjoy your con.