Oh, okay. Hi, welcome to our next talk. This one's by Gavin Kondyk, also known as GTKondyk on
Twitter. He's a pre-circuited researcher, consultant, and has been super involved in
the AI village community for a long time. He's an awesome guy, so I'll let him take it away.
Hello, and welcome to Hacking with Skynet, How AI is Empowering Adversaries.
My name is Gavin. I go by the pseudonym GTKondyk. I am a security researcher
and senior security consultant. I'm very passionate about network security,
both the attack and defense side. And through that passion, I run a project called NetSecExplained,
which is a blog and YouTube channel where I explain intermediate and advanced level concepts
in an easy-to-understand way. So, machine learning, let's get everybody on the same page.
What is machine learning? Well, first, we have this big umbrella term of AI. And parts of AI
are machine learning. Parts of AI are unrelated. So, we have rule-based artificial intelligence.
We have classical image recognition. But inside of AI, we have this small subset called machine
learning. And machine learning specifically allows us to take statistical analysis and
draw inferences from data and perform data discovery techniques. And then out of machine
learning, we also have deep learning, which is specifically deep neural networks. And this can
be convolutional neural networks, which are used primarily for image recognition, or recurrent
neural networks, which are used for cyclical pattern recognition, such as a heartbeat monitor.
So, the burning question is, how is AI empowering adversaries? Or how can it empower adversaries
moving forward? So, I see the age of AI today, or the age of machine learning today,
we're at the very beginning of what it can do for the security industry on both the attack and the
defense side. Now, if you remember back in the 80s and 90s, antivirus, intrusion detection,
vulnerability scanners, these were all simple scripts that were written. And then over time,
they grew into what are now multi-billion dollar industries. And so, I think that we're at the very
beginning of what AI and machine learning can do for the security community. So, the way that I'm
going to bring about this talk is really in two major parts. So, the first is to talk about
offensive AI tools. Why do we want these? What's out there already? What are we seeing in the wild?
How do we build them? And then adversarial machine learning, which is attacking the machine
learning algorithms themselves. So, let's start with offensive AI tools.
So, the first question is, why would we even want these? Well, offensive AI tools and AI tools in
general will allow us to operate at a much faster speed and a much larger scale than we're currently
able to do. And this is going to allow us to automate a lot of manual tasks. Now, we don't
need to think about it as automating everything. But if there's something that requires or manages
or deals with a lot of data, and these are things that we can apply a certain amount of fuzzy logic
to, then this is a perfect candidate for automating with some form of AI or machine learning.
And then, of course, this allows us to dynamically and intelligently explore our target attack
surface and uncover hidden blind spots that come about some of the biases that we intrinsically have.
And so, being able to statistically analyze some of this data or perform some machine learning
operations, this will allow us to kind of uncover some of those blind spots
and look for things that we don't already know. And then, how realistic is this?
Well, back in 2016, DARPA kind of took on a large undertaking in their CyberGram challenge.
In here, what they did was they required participants to develop some form of artificial
intelligence that will perform a series of attack chain tasks. So, in this case, it was going to be
vulnerability discovery, exploitation, patching, and then data theft. So, in the form of data
theft, they have capture the flag. And then, as part of their major grand challenge, they had
an attack, defend, capture the flag event that was air gapped. And you can see that in the picture
kind of off to the side, where there's no human interaction whatsoever for the entire event.
And everything had to be automated. So, this was a big reveal on a world stage that let us see
what the power of artificial intelligence is, especially when it comes to a number of these
tasks. And then, where are we seeing applications of AI already? So, the current proof of concepts
we can see for social engineering. We can see in defense detection and evasion. There are some
researchers that are identifying intrusion detection systems and sandbox detection systems
so that attacks can be stealthier and break out of various systems and bypass existing
defense measures. Evaluating data leaks. So, large data sets such as password dumps,
subdomains, things of that nature, websites, applications, configuration files. Automated
network exploitation. So, this will be going from standard user to domain admin, automating that
process, as well as normal internal and external network pen tests. And then, software exploitation.
And so, we'll see this with AFL, which is a software fuzzer that will utilize its own custom
compiler, put in certain key points in the software itself, and then it allows the system to fuzz and
track where through the application user input is being provided and how that can reach potentially
vulnerable functions. So, let's start with social engineering. Social engineering, there's a couple
things that are in the works right now that will allow us to improve our social engineering
capabilities and improve attackers' abilities to exploit corporations and systems. So, in this case,
we have an NVIDIA project called StyleGAN. StyleGAN, what it does is generate human faces.
So, all four of the images up at the top you see are actually generated by StyleGAN. And then, just
to kind of show you that there's no tricks up my sleeve, on the bottom row, we have StyleGAN
generated cats. And you can see here in the bottom right corner, this abomination that kind of looks
like a cat, but is really messed up and difficult to see, difficult to really recognize for a human.
And this is kind of the imperfect model generating a cat-like image. But the level of detail that
these can go into, if you look at the cat over here on the left, just kind of sitting on the
carpet, you can see the individual hair, the whiskers, the texture of the carpet underneath.
And so, these are really difficult to tell whether or not they are legitimate faces or cats, or if
these are machine learning generated. One of the telltale signs, if you look here in the top right
corner, this gentleman, it seems like there's a picket fence behind him, but it's a little warped
in the back. You also notice that all of the images have a blurred background. And if you
look really carefully, all of the eyes point directly at you. So, it doesn't matter what
direction they're faced, the eyes are pointing straight. So, there's already some technologies
and some research being performed to kind of identify what is a legitimate human face and
what is a generated human face. Now, why is this important? How can we use this for social
engineering? Well, for several types of social engineering campaigns, what's pretty common to do
is go to the 10th page of a Google image search, grab somebody's stock photo or profile photo,
and then you can paste that up as your own LinkedIn profile photo and say, okay, I am now
Becky. I'm not a 32-year-old male. My name is Becky. I'm 27. I'm starting my career and I want
to ask somebody, hey, as a fellow woman, how did you get started? And then I can use that to build
rapport, have my pretext, and perform social engineering tasks to try and exploit employees,
maybe in the HR department, sales, wherever. And so, instead of grabbing that image and throwing
it up as a pretext where an analyst can go in, grab that same image, do a reverse Google image
search, see that it's a stock photo, call me out, realize that I'm fake, what I can do is use
something like StyleGAN to generate my own face. So, why steal a person when I can just make a
whole new person? But there needs to be a little bit more legitimacy to my pretext. So, this is
where GPT-2 and Grover come into play. GPT-2 and Grover, they operate very similarly. These are
automated generation of long-form content. So, these can be used to generate emails,
blog posts, articles, anything of that nature, with very minimal effort from the attacker or
the adversary in that respect. And so, unfortunately, the website that was used to generate
these has been taken offline, but the GPT-2 model is still something that you can download, put on
your own systems, and generate long-form content. So, if I was developing a pretext and I can use
this to write phishing emails that could get past a spam filter, or that look distinctly unique,
so that I'm not sending the same email to multiple people within an environment,
or I can use this to write blog posts and long-form content, and because any sort of
Google image search or... what is that called? Plagiarism detection, it will see that I am the
only source of truth, therefore I must be a real person, therefore my website must be legitimate.
And so, this will kind of add to the pretext. And then on top of that, long-form content Google
really enjoys, and so that can give us some Google ranking to add a little bit more credibility to our
initial pretext. So, this is one of the scarier ones that I see as being at the very beginning
of the research, and I'm curious to see where this will be in the next several years. We have
Lyrebird and Tacotron. Lyrebird and Tacotron, these are text-to-speech that are based on human voices
and use those human voices to generate conversation bits. And so, we can use this,
I foresee, in the phishing campaigns where we're actually calling people or leaving voicemails.
So, it's a lot less automated sounding, a little less robocall,
and then you just add a little bit of compression and it sounds like you just have a really bad
connection. So, I have a couple audio clips. I want you to hear them. So, these first two,
I have the original voice and then the second one is the synthetic voice based off that original
voice. So, let's go ahead and take a listen. In another moment, down went Alice after it.
Never once did she consider how in the world she was to get out again.
Awesome. And then the synthetic voice built off of that sample. My voice might be generated by
a computer, but I think it sounds pretty human. I don't know exactly how they made it, but I'm
really impressed. So, what I'm imagining is that as these models get better, what we can do is
take interviews of CEOs and VPs of large corporations, grab their audio, isolate it,
and then put it into a model like this so that we can have our own version of synthetic text to
speech using their voice. So, we can call throughout the company and impersonate these
individuals in order to cause some sort of social engineering action. Now, this last one is really
interesting. This is Tacotron. This is a Google project. And I have the text over here on the
right, but what I want you to do while you're listening to this is pay attention to the breaths.
As humans, we tend to have filler words like um and uh, and then in between sentences and
commas, we usually have a bit of a pause or a breath. And so, the Tacotron model actually
picked that up. This is fully synthetic voice. Just take a listen. Only the photograph really
showed how much time had passed. Ten years ago, there had been lots of pictures of what looked
like a large pink beach ball wearing different colored bonnets. But Dudley Dursley was no longer
a baby, and now the photograph showed a large blond boy riding his first bicycle
on a carousel at the fair playing a computer game with his father being hugged and kissed
by his mother by his mother. That's pretty scary. So, these projects are still underway. Liarbird
is actually advertising itself as a transcription or reverse transcription service, which is just
text-to-speech. In their original pay model that they had pushed out, currently they're in closed
beta, but in their original pay model that they pushed out, you can transcribe three hours of
audio pretty cheaply. And then if you're so inclined to pay, you can transcribe obviously
more. But if a malicious user or somebody who's maliciously inclined, they can just grab those
voice samples, throw them into Liarbird or Tacotron, or just use the voice samples that
are already in there and generate their own adversarial examples.
So, next we get into sandbox detection. And so, this is the defense detection and evasion
part of the system. So, we have these researchers over at Silent Brick Security, and what they did
was decide that with many different types of sandbox, what you can do is take the process list
as data itself. And so, based on the process list, the number of processes,
the number of users on a system, you can generate and decide whether or not you're in a sandbox.
And then you can take it a step further and decide what kind of sandbox you're in,
whether you're in FireEye or Cuckoo sandbox. So, what the researchers did was kind of classify,
they took a number of basic Windows systems and they took a number of sandboxes and just kind of
identified how many processes there are, how many users there are. Let's take a look at what malware
can identify. And so, when we have a low process count, you see over here in column A, we have 33.
In column F, we have 34. And then we have a high user count, such as, again, A and F, we have 4
and 4. And the process count per user, 8.25, 8.5. It gives us a very high probability that it is a
sandbox. And so, we can see that with our host total score. And then based off that total score,
we can set a threshold. In this case, it looks like they set the host average score as their
threshold. So, anything below the average score is considered a sandbox. Anything above that
is considered a legitimate system. And so, they can have malware that completely shuts down or
erases itself if it identifies that it's in a sandbox, or it can use targeted evasion techniques
per sandbox. So, if I'm in FireEye, these are the things that I'll do to get past it. If I'm in
Cuckoo, these are the things that I'll do to get past it. So, it's a really interesting article,
highly recommend taking a look at that. And then for automated network exploitation,
we have systems like DeepExploit and Goethon. Goethon and DeepExploit, they both operate very
similarly. The way that these work is they will go out, they will perform an nmap scan based on
the results of that nmap scan, if it sees common services that it knows are exploitable or things
that it would like to kind of take to that next level. For example, port 80 or 443, it's web
traffic. So, it's going to go ahead and run Scrappy to go and crawl that system. I'm sure they could
add other tools such as Go Buster, which will go out and enumerate a lot of the files or directories
or URLs that are on that system. Try and fingerprint the server. You want to see if it's running
Nginx, Apache, PHP, Java, things of that nature. And then it performs additional content exploration
and its own Google hacking, which is kind of cool. So, that will go out, do a little bit of research,
see if there's any sort of exploitable scripts, things of that nature. And it takes all of this
data and it processes it using some statistical techniques and then it uses Metasploit as a
targeted exploitation engine and it goes and tries to exploit those things. So, if it sees
certain versions of JBoss or Jenkins, it will go and try and target that with maybe
password spray or any sort of Metasploit module. Tomcat, same thing. So, this is kind of cool
as a proof of concept of automating the entire process of network discovery all the way up to
exploitation and in some cases post-exploitation. One of my favorite while I was researching this
is Deathstar. And this is my favorite just kind of in the way that it is assembled. So, Deathstar,
it's built on PowerShell Empire. And the way that it works is it starts, it kind of covers its
baseline and it is designed to go from standard user all the way up to domain administrator.
And so, it charts and maps attack paths all the way to that location. And so, it starts at its
baseline and it says, am I already a domain administrator? If so, cancel out. And we see
that's the path off to the left. If I have domain administrator credentials, all right, cool. Log
in. And that charts off the path to the right. But if it's neither of those, then it goes straight
down the middle and it says, okay, let's spawn a new agent. Let's go ahead and identify where is
the domain administrators and where is the domain controllers. And so, it runs over here to the
right. We can see get net domain controller, get net group member, invoke user hunter. And so,
this is going to enumerate the domain controllers, enumerate the domain admins, and enumerate the
users that it can on the systems that it can. And you can see it goes through this entire tree
kind of off here to the bottom left. It identifies and says, hey, is this a vulnerable,
known vulnerable version of Windows, for example, Windows 7? If yes, let's run Mimikatz,
try and do Mimikatz, secret stump, things of that nature, collect credentials, and then we can kind
of start back over at the top. Let's go ahead and enumerate the running processes. Do we see any
processes that are already being ran by other users that we don't have in our inventory? If so,
let's add those users to our inventory and try and do some lateral movement.
And so, in this case, it'll use psinject, and then it'll go ahead and spawn a new instance.
And so, it goes through this iterative process all the way until it reaches domain admin credentials.
And so, if you've ever used a system like Bloodhound, where you'll generate an attack
graph and say, okay, this user is part of this group, which has these members, which this member
is also part of domain admins. And so, this is the attack path that you want to go in order to
become domain admin. And so, Deathstar kind of takes that idea a step further and actually tries
to navigate that attack path in an automated fashion. So, that's all I have for the machine
learning tools that already exist, some of the proof of concepts. There's a lot more information
that I wanted to cover that I wouldn't be able to cover in such a short amount of time. So,
at the end of this presentation, I have a link to my slide deck, which has a lot more detail
and a lot more tools and a lot more techniques in it. But in the meantime, I kind of want to
answer the question of how do I go about creating my own machine learning tools
or AI tools? What should I look for? How do I automate that? What is that process? And so,
honestly, my advice would be to identify somewhere that either utilizes a lot of data
or identify somewhere that uses a... it's a common process, but it uses some sort of fuzzy logic,
right? Something that's kind of hard to script, hard to write a signature for,
but with AI and machine learning, you can use some of that fuzzy logic to generate some heuristics
and it can perform based off of that. So, in the case of some of these, what we can do is say,
all right, NMAP scans, everybody does NMAP scans, and then you have to manually look at them,
and then based off of that, you have to follow up and go over here. Instead, like I showed with
Goython, is you can just use the NMAP scan and then automate the process of, okay, let's enumerate
and fingerprint these services, and then let's see if any of these services are already exploitable
using Metasploit, or let's go with next steps. I identified a web service, let's run web analysis
techniques. So, Go Buster, Crawler, any sort of things like that. So, that's kind of the thing
that you want to keep an eye on if you want to develop your own machine learning tools. And
there's more that are coming out all of the time, and it's really interesting and a really fast-paced
area, and it's just ripe with research. So, go ahead and play around with some of that.
So, the next bit that I want to get into is adversarial machine learning,
and this is attacking the machine learning algorithms themselves. So, adversarial machine
learning, there's really three ways to go about attacking a machine learning model.
So, the first way we have is model evasion. This is to take what is clearly a malicious example
and get the model to consider it to be benign or normal or standard to bypass that filter.
So, this would be taking a spam message and getting past a spam filter, or this would be
taking a piece of malware and getting it past antivirus or anti-malware solutions.
Model poisoning is where we add our own data into the training data set. Now, machine learning
models specifically are... they're done learning as soon as they're done being trained. To get
around this, there are some models that are called online systems, and so what they'll do is they'll
take data that has been classified by itself already. So, say I send it a piece of data,
like a piece of malware, it looks at that, says, okay, bam, that's malware,
now that goes into my training set, and I learn from that malware. So, data poisoning allows us
to slowly inject certain pieces of data so that we push that threshold a little off to the side
into what is more acceptable, and then we target where that threshold has been pushed.
And we'll see an example of that in a bit. Data leakage. Data leakage is actually
a combination of two smaller bits. So, one is stealing the model itself. So, for example,
say you're an IoT engineering company and you develop an IoT system that utilizes some form
of machine learning. What I can do is buy that device and then reverse engineer it,
and now I have your model, and so I can use that model or train another model based off of that
instead of having to go through the hundreds of thousands of dollars that you went through
to develop that in the first place. And so, this is where the model itself is easy to be grabbed.
There's also data leakage, which is I can identify what data you use to train the model.
And so, this is really damaging, especially for healthcare industry or
HIPAA-protected information. So, if I can identify any sort of clients or
patients that you have, any sort of their PII, then that's really damaging.
So, when it comes to model testing, there's really two approaches. You have a white box where you
know what the model is, you know how the hyperparameters are set, you know everything
that's involved in the building of this, and then you target that model, much like you would do in
software where you already have all of the code involved. And then you have a black box, which is
just the model itself. You aren't able to peek into the way that the model operates, but what you
can do is treat it like an oracle. So, if you've ever done any sort of cryptography, you know,
exercises, you have an oracle, you ask yes-no questions, or things of that nature, and then
you can generate what is called a surrogate model. So, based off of the yes-no questions that I asked
the black box model, I generated my own model, and then based off of that model, I can perform attacks
and that those attacks should be able to impact the original black box itself.
Now, the way that this operates is under a principle called attack transferability.
This is a documented phenomenon, and the way that attack transferability works is,
it doesn't matter what model or the underlying architecture that these models are designed with,
an attack on one model can also be an attack on another model. So, if I have a deep learning
model over here, and I find adversarial examples that I can use to attack this model,
then here is a random forest, a logistic regression, SVM, or anything else, any other
machine learning algorithm or architecture, that those same adversarial examples will be able to
impact that. And then, of course, the attack surface. So, where are the machine learning
models themselves vulnerable? So, machine learning, as it works, is you have a physical object,
and then you take that physical object and you try to digitize it. So, say you have a picture
of a panda or a car, you take a photograph of it, now it's digitized, and you can make
information off of that. You take that digitized information, you run your pre-processing as you
would per the normal machine learning process, and then you put that into the machine learning
model as part of its input features, it presents some output, usually a probability or a number
of some sort, and then a decision is made based off of that action. So, for example, if I am 67%
confident that this is malware, then I will probably flag it as malware. If I'm only 42%
confident, then I'll just let it continue to operate. So, where can we attack things? Well,
one, we can actually impact the physical object itself. So, down below, I have the network
construction detection system as kind of an example that follows this generic machine learning model
path. So, we have the attack traffic. We can actually, as attackers, modify the attack traffic
in such a way that as it goes through the pipeline of the machine learning pipeline,
it will modify the behaviors at different points. So, I can slow down my attack traffic,
I can use my attack traffic for multiple hosts so that it's a little bit harder to detect.
And then that attack traffic is then digitized. The digitization process, we don't really have
a lot of impact over as attackers. It's not realistic for us to be able to affect TCP dump.
There's really famous examples. For example, you may have come across this if you ever looked at
adversarial machine learning, a picture of a panda, and you modify some pixels,
and now it thinks it's a gibbon or a bird or some sort. But if we have the ability to modify pixels,
why don't we just give it a picture of a bird at that point? And then more to the point,
if we can modify pictures, if you have something like a self-driving car, it has a camera,
how do we interface with that camera in an adversarial perspective? We would already have
to have access to the system, which at that point we can just shut it down or do whatever.
It's physical access. So, we're not actually able to interface with the digitization of the object,
but what we can do is change the way that the attack traffic is modified.
We can change the input features, the input vectors, the machine learning model, especially
with the model poisoning. We can change the way that the data is represented.
So, in the case of the network intrusion detection system, we have the packet metadata,
and then it spits out an attack probability, and then off of that it makes a decision.
And in between the machine learning model and the decision-making process, you got to remember
this is just software, and you can approach it like any other piece of software that you're
pen testing or trying to hack. So, you have the machine learning model in it, and then
all wrapped around it is a nice piece of software that probably has some of its own vulnerabilities
involved. So, model invasion. Model evasion, not invasion, sorry.
So, we're going to try and hide in the blind spots. Now, we have this massive theoretical
space. This is all of the possible pieces of malware that could possibly exist, but we can't
collect all of that, realistically speaking. Antivirus companies can't collect all of it,
and so what they have is a subset, and this is considered the training space. So, this is what
they are realistically able to collect. So, in their training space, they perform their training,
and then off of a small piece of that that's already clean labeled data, they perform their
testing to make sure that the model operates properly. Well, part of that testing space,
kind of right there off the sliver, this is the adversarial space. This is where we live.
This is where we are able to generate adversarial examples that target that model specifically.
And so, this is kind of what we want to play around with and hide in the blind spots,
so to speak. So, a couple really good examples is, in this case, multinomial
Naive Bayesian spam filters, or just any of the early spam filters that used some rudimentary
forms of machine learning, how they would operate is they would look at keywords. So,
if you had something like, buy Viagra now, the probability that the word Viagra shows up in a
spam email is much higher than the probability of the word Viagra showing up in a regular email.
And so, based off of these keywords, it does its own statistical analysis and says,
I am 99% sure this is spam. Well, the interesting thing about emails is that
emails are usually represented as HTML pages or web pages. And so, as malicious actors,
what we can do is actually add in a comment. So, in this case, we can add in the entire Wikipedia
page article for a horse. And so, we have a bunch of bad words, but we have even more good words.
What this lets us do is kind of skip past the spam filter by showing, hey, we have a lot more
good words than bad words, so we must be good, right? It says, yeah, absolutely. I am 99% sure
you are not spam. And the thing is that the way that this is presented to users is as an HTML
web page. And so, the comment itself is not actually rendered for the users. So, the users
still see the original spam email as though it was unmodified. And so, this is kind of a real
world example on some of the spam filters. A really good example of this is by Skylight Cyber.
The researchers over there were actually able to bypass the silence machine learning antivirus
in this article, Silence, I Kill You. Really good article. Highly recommend you take a look at it.
We can see some familiar malware samples here. So, we have CoinMiner, Emotet, Zeus,
and we see the original scores that they were actually able to pull out of the silence antivirus,
negative 826, negative 923, negative 997. So, these are really bad, really bad, flagged immediately.
So, what the researchers over at Skylight Cyber did was they took the silence antivirus,
reverse engineered it, and they were able to pull out the scores based on some of the
syscalls and other key features that silence looked at for malware.
One of the curious things is that they identified certain things that were whitelisted. Now,
why would silence whitelist certain things, even though the standard machine learning process would
actually identify it as potentially malware? Well, the reason why is because when you think
about malware, think about how malware operates, right? So, maybe it asks for access to your webcam
or your microphone, or it tries to resize or open up new windows. Maybe it makes calls out to the
internet. Well, what's another thing that does that? Chrome, Firefox, Internet Explorer. They
operate very similar to some modern malware. In this case, what they found was a video game.
I'm not sure which video game it is. My money is on Fortnite or League of Legends.
But the video games that they found were certain whitelisted keywords. So, what they did was took
these clearly bad malware and they just added these keywords to the end of the malware. So,
it didn't change the way that the malware operated, but it changed the way that silence looked at it
and said, okay, you have a bunch of bad boy points, but you have even more good boy points.
So, we're going to give you some good boy scores. So, in this case, CoinMiner had 884.
My favorite is looking at Zeus, negative 997, one of the worst in this table,
all the way up to positive 997, which is one of the best in this table.
And so, it shows that model evasion techniques are very useful in a number of different ways.
So, how do we defend against this? Well, the first and probably one of the better
is adversarial training. Adversarial training kind of comes from the idea of chaos engineering.
Netflix is a really big proponent of chaos engineering, where you already know something
bad is going to happen, so you might as well account for those bad things to happen. So,
in this case, you train it with adversarial examples. You generate your own adversarial
examples and you train the model based off of that to kind of harden and make it a little bit
more robust. Again, if it's not a robust model, it's not a good model. Chaos engineering as a
whole, the way that it operates is you already know something is going to go bad. So, you have
Netflix, for example. Netflix, they know that their systems, they need to have that uptime.
And so, they know that it's possible one of their systems goes down. They want to make sure
their redundancy operates properly. So, you purposefully go into production and you break
things. And then you develop processes and policies around knowing things are going to
break. So, in some cases, you'll have the CIO walk into a data center and just start unplugging
servers. So, adversarial training is a really good way to kind of counteract this. Another
slightly more technical way is called defensive distillation. Defensive distillation is a
smoothing algorithm that kind of smooths the curve. So, instead of having, okay, malware,
malware, malware, malware, nope, it's okay. You can't have sharp edges like that. So, what it
does is it smooths it. So, instead of going straight down, it just kind of goes in a downward
trend. And so, it's a little bit harder, a little bit more robust way of analyzing and dealing with
the adversarial examples. And then the last example would be monotonic classification.
Which means that the graph grows in one direction. So, we see here in the image off to the right,
the non-monotonic example, we see that it's getting worse and worse and worse as far as a
keyword. So, it goes worse and worse and worse across that threshold. Now, it's considered bad,
it's spam, it's malware, it's whatever. We're ready to throw it out, but we're still analyzing
it. And so, it's crossed that threshold, but wait, no, it's getting better and better and better.
We just hit that pad of keywords that are really positive. And so, we went from good to bad,
all the way back up to good. Monotonic classification means that we don't count the
good words at all, we say, we measure your badness. How many bad words do you have?
And so, in this case, on the bottom example, we have the monotonic classification, which says,
okay, we have some bad words, we have some bad words, up, we all went all the way up to the top,
and then we hit all of the padded good words. And it doesn't matter, you're still bad,
Jesus is not going to save you, you're done. And so, monotonic classification is a really
powerful technique to deal with these model evasion techniques.
Next, we have model poisoning. Model poisoning almost exclusively works on online learning
systems. So, as adversaries, we need to find a way to get adversarial examples into the training
space. And so, this is where systems will grab data that it's already classified a certain way,
and add it into the training space. And so, we're just trying to push that barrier
past a certain threshold where we can actually operate safely as adversaries without it detecting
us as such. So, some real world examples. Tay AI, if you're not already familiar,
I highly recommend checking it out. Tay AI was a Microsoft Twitter chat bot, and it was designed
to learn from real conversations by real people. Well, 4chan, of course, gets a hold of this, and
they decide, how much does it learn? How much can we teach it? And about 16 hours after coming online,
it was taken offline permanently, because within that time, 4chan was able to get it to say,
Hitler did nothing wrong, and racial slurs. If you want more information on Tay AI, and I highly
recommend checking this out, it's called The People's Chatbot by the Internet Historian on
YouTube. Really funny guy, really great story, and it's just incredible to think of how many
data scientists and developers and people behind this project put time in, put effort, put work,
and it was shut down by a bunch of kids on 4chan. The other one is Jacobian Map Saliency
Attack, and this kind of goes into the pixel manipulation. So, what you do is you grab a
couple pixels, and you modify them, or you tweak them a little bit, or you tweak the data a little
bit, doesn't necessarily need to be image data, and you pass that on as a certain class.
And so, as the model is learning from this data, learning from this class, it starts identifying
these tweaks, and then you can implement those tweaks in an adversarial example. So, in this
case, we have a 70 kilometer per hour sign, we add in these little tweaks, and it latches onto those
tweaks and says, okay, you're a 30 kilometer per hour sign. And so, imagine the kind of chaos that
that can create. Again, these tweaks are difficult to implement in real world examples. This is also
difficult to pull off simply because it relies on an online learning system, something that learns
from the data that you've already presented it. But these are things that researchers have been
able to identify and actually use to attack machine learning systems.
So, how do we defend against poisoning attacks? Basically, it comes down to, you can still use a
lot of the data, but what you want to do is add your own little smoothing filter. So, in this case,
have longer periods of time between retraining. So, what that will do is kind of narrow the window,
not narrow the window, it will expand the window that an attacker would need to generate adversarial
examples for. And then they would also have to compete for votes against legitimate traffic.
So, it's kind of like, if you look at Bitcoin, you need 51% of the vote in order to kind of
take over cryptocurrency networks. So, this is kind of their own example of doing this.
You can also analyze longer periods of data. Again, it broadens the window that you're looking
at. And so, the adversaries, the attackers are going to have to generate more traffic,
more examples in a longer period of time, which is difficult to do. And then you just want to
generally minimize the impact of adversarial training examples. So, whether you have a human
in loop that's manually looking at some of these examples, or maybe you're taking a small sample
of the stuff that you're seeing and only allowing that to affect the model very...
a little bit. So, it's just kind of a general rule of thumb. It's just minimize the impact,
come to the understanding that people will try and poison the well, account for that.
And then, of course, we have data leakage. So, data leakage, this is usually when,
if we're trying to steal data, not steal the model, but steal data itself, this is usually
when the models are trained to be a little too good. So, this image down below, you see aerial
photographs, generated map, and then an aerial reconstruction. And so, articles at the time,
this is a project by Google, articles at the time were kind of grandstanding and saying,
oh my god, the Google AI found a way to cheat and generate aerial photographs.
It was able to embed its own data. From the machine learning perspective or data science
perspective, this is actually called overfitting. It's a very common problem. It's nothing
spectacular. It's just, you didn't train your model right. And so, especially with overfit
models, you have the training data in the model itself, or the model is able to generate that
training data. And so, as I was saying earlier, with things like credit card information, PCI
protected information, personally identifiable information, public health, or not public,
sorry, personal health information, or HIPAA protected data, things of that nature,
it can be very damaging for a machine learning model to leak real training data. And so,
it's important to kind of evaluate and make sure that the models are not overfit, are not underfit,
just hidden just right, or that the data is being preprocessed in such a way that it's
a one directional. So, in some cases, what they'll do is generate a hash of certain features. For
example, instead of putting in a credit card number, they generate a hash of the credit
card number. And so, it's not reversible in any meaningful way. And then, of course, model theft,
model theft by competitors. There's a technique called federated learning, which is you encrypt
the machine learning model itself so that the model has to be used in that encrypted form. You
can't steal the model, you can't decrypt it. It's a one way kind of thing, or a one way street in
that scenario. And so, this will allow like IoT devices to be deployed with machine learning
models. And it will prevent competitors or attackers, malicious actors from reversing it,
stealing that model and building their own product off of it.
And then, one last thing that I kind of want to cover is adversarial stickers. So,
you've probably seen adversarial stickers. They look like melted crayons, in my opinion.
But adversarial stickers, they have a bit of a problem. This is considered a brittle attack,
and it's not very repeatable. And the reason why is because it doesn't have that attack transfer
ability that I brought up earlier. So, in this case, kind of off to the bottom right,
the stop sign, this is one of the first articles I was talking about adversarial stickers, and they
were trying to modify a stop sign in such a way that the machine learning model didn't recognize
it as a stop sign anymore, recognized it as something else. And so, this could be really
damaging for self-driving vehicles. One of the problems is that this paper ran into some
repeatability problems. And the reason why is because this attack is so brittle, they were
able to convince their model, but they were not able to convince, or people who were trying to
replicate this were not able to convince other models. So, it's not very useful. You can't just
slap an adversarial sticker onto a sign and hope that you're driving a self-driving vehicle off
into a wall or something of that nature. And then off here to the left is kind of the same idea with
facial recognition. So, this researcher here said, okay, I'm going to set some constraints. I only
want the pixels or the image around my eyes in the shape of glasses to be modified so that I look like
this actress. Honestly, I don't remember who this is. I apologize. But I'm a man. I want to look like
this actress per a facial recognition model. And so, he was successfully able to do that.
The problem is that these glasses are designed for his face and the rest of the features of his face.
And so, if I or many of you put on these glasses, it'll either look like somebody else or it won't
recognize it as an adversarial example and it'll just recognize us as people, the same people,
just wearing stupid glasses. So, attack transferability, adversarial stickers don't
really have it. It's not a really good attack. It's very brittle. And I just kind of wanted to
bring that up. For more information on this, I highly recommend DEF CON 2018 AI Village. Sven
has a really good 30-minute talk on adversarial stickers and kind of the challenge with them.
And then, for the facial recognition specifically, I know that Rich Horang in AI Village 2019 has a
really good talk talking about facial recognition and trying to bypass that.
So, to recap, how has AI empowered attackers? Well, it allows us to operate at a speed and scale
like never before. We can operate at machine speed. And then, what we are seeing is the very
beginning of what AI can bring to the offensive security space. So, we've already seen some
adversarial attacks in the wild. I anticipate us seeing more as more researchers are coming out.
AI and machine learning is kind of the hot new buzzword that everybody's latching onto. So,
CEOs and decision makers are grabbing the AI-powered or machine learning-powered
detection systems, defense systems. And it's kind of become a bit of a buzzword
where you say that in a lot of common companies, especially in places like DEF CON, and they tell
you to go talk to the Bitcoin folks and blockchain and all that fun stuff. There's no silver bullet,
but we are seeing the very beginning of what is possible. And I anticipate, and I'm really hoping
in the next several years, that we see a much larger impact of this AI versus AI kind of warfare
that should be coming about. But anyways, I appreciate it. Thank you for your time. Thanks
for sticking around this long. You can go ahead and email me if you have any questions. I'm also
going to be in the chat. And then, for the slides, I have that are much more detailed are on
Slideshare, and you can find them at slideshare.net slash GTKalandai. So that is my presentation. I
hope you learned something, and thank you for sticking around. Bye!