start	end	text
720	8780	Oh, okay. Hi, welcome to our next talk. This one's by Gavin Kondyk, also known as GTKondyk on
8780	16040	Twitter. He's a pre-circuited researcher, consultant, and has been super involved in
16040	24060	the AI village community for a long time. He's an awesome guy, so I'll let him take it away.
27770	34470	Hello, and welcome to Hacking with Skynet, How AI is Empowering Adversaries.
34750	39850	My name is Gavin. I go by the pseudonym GTKondyk. I am a security researcher
39850	44410	and senior security consultant. I'm very passionate about network security,
44410	49570	both the attack and defense side. And through that passion, I run a project called NetSecExplained,
49570	55110	which is a blog and YouTube channel where I explain intermediate and advanced level concepts
55110	62870	in an easy-to-understand way. So, machine learning, let's get everybody on the same page.
63210	70410	What is machine learning? Well, first, we have this big umbrella term of AI. And parts of AI
70410	76350	are machine learning. Parts of AI are unrelated. So, we have rule-based artificial intelligence.
76350	82750	We have classical image recognition. But inside of AI, we have this small subset called machine
82750	88910	learning. And machine learning specifically allows us to take statistical analysis and
88910	94710	draw inferences from data and perform data discovery techniques. And then out of machine
94710	100570	learning, we also have deep learning, which is specifically deep neural networks. And this can
100570	105750	be convolutional neural networks, which are used primarily for image recognition, or recurrent
105750	111210	neural networks, which are used for cyclical pattern recognition, such as a heartbeat monitor.
113610	122110	So, the burning question is, how is AI empowering adversaries? Or how can it empower adversaries
122110	129110	moving forward? So, I see the age of AI today, or the age of machine learning today,
129110	133270	we're at the very beginning of what it can do for the security industry on both the attack and the
133270	140350	defense side. Now, if you remember back in the 80s and 90s, antivirus, intrusion detection,
140350	144570	vulnerability scanners, these were all simple scripts that were written. And then over time,
144570	150030	they grew into what are now multi-billion dollar industries. And so, I think that we're at the very
150030	157550	beginning of what AI and machine learning can do for the security community. So, the way that I'm
157550	162930	going to bring about this talk is really in two major parts. So, the first is to talk about
162930	169450	offensive AI tools. Why do we want these? What's out there already? What are we seeing in the wild?
169450	174390	How do we build them? And then adversarial machine learning, which is attacking the machine
174390	181490	learning algorithms themselves. So, let's start with offensive AI tools.
184370	190850	So, the first question is, why would we even want these? Well, offensive AI tools and AI tools in
190850	197450	general will allow us to operate at a much faster speed and a much larger scale than we're currently
197450	202510	able to do. And this is going to allow us to automate a lot of manual tasks. Now, we don't
202510	208370	need to think about it as automating everything. But if there's something that requires or manages
208370	214570	or deals with a lot of data, and these are things that we can apply a certain amount of fuzzy logic
214570	221210	to, then this is a perfect candidate for automating with some form of AI or machine learning.
221370	225690	And then, of course, this allows us to dynamically and intelligently explore our target attack
225690	232330	surface and uncover hidden blind spots that come about some of the biases that we intrinsically have.
232610	237050	And so, being able to statistically analyze some of this data or perform some machine learning
237050	240710	operations, this will allow us to kind of uncover some of those blind spots
240710	249710	and look for things that we don't already know. And then, how realistic is this?
249710	257930	Well, back in 2016, DARPA kind of took on a large undertaking in their CyberGram challenge.
258050	263250	In here, what they did was they required participants to develop some form of artificial
263250	270710	intelligence that will perform a series of attack chain tasks. So, in this case, it was going to be
270710	277050	vulnerability discovery, exploitation, patching, and then data theft. So, in the form of data
277050	282250	theft, they have capture the flag. And then, as part of their major grand challenge, they had
283730	288710	an attack, defend, capture the flag event that was air gapped. And you can see that in the picture
288710	296230	kind of off to the side, where there's no human interaction whatsoever for the entire event.
296230	302950	And everything had to be automated. So, this was a big reveal on a world stage that let us see
302950	307790	what the power of artificial intelligence is, especially when it comes to a number of these
307790	316170	tasks. And then, where are we seeing applications of AI already? So, the current proof of concepts
316170	324930	we can see for social engineering. We can see in defense detection and evasion. There are some
324930	329250	researchers that are identifying intrusion detection systems and sandbox detection systems
329250	336350	so that attacks can be stealthier and break out of various systems and bypass existing
336350	342790	defense measures. Evaluating data leaks. So, large data sets such as password dumps,
342790	351010	subdomains, things of that nature, websites, applications, configuration files. Automated
351010	356790	network exploitation. So, this will be going from standard user to domain admin, automating that
356790	364680	process, as well as normal internal and external network pen tests. And then, software exploitation.
365630	372170	And so, we'll see this with AFL, which is a software fuzzer that will utilize its own custom
372170	380390	compiler, put in certain key points in the software itself, and then it allows the system to fuzz and
380390	386830	track where through the application user input is being provided and how that can reach potentially
386830	392790	vulnerable functions. So, let's start with social engineering. Social engineering, there's a couple
392790	398970	things that are in the works right now that will allow us to improve our social engineering
398970	406290	capabilities and improve attackers' abilities to exploit corporations and systems. So, in this case,
406290	412070	we have an NVIDIA project called StyleGAN. StyleGAN, what it does is generate human faces.
412710	418970	So, all four of the images up at the top you see are actually generated by StyleGAN. And then, just
418970	423610	to kind of show you that there's no tricks up my sleeve, on the bottom row, we have StyleGAN
423610	430950	generated cats. And you can see here in the bottom right corner, this abomination that kind of looks
430950	439410	like a cat, but is really messed up and difficult to see, difficult to really recognize for a human.
439410	445530	And this is kind of the imperfect model generating a cat-like image. But the level of detail that
445530	448790	these can go into, if you look at the cat over here on the left, just kind of sitting on the
448790	454350	carpet, you can see the individual hair, the whiskers, the texture of the carpet underneath.
454950	461810	And so, these are really difficult to tell whether or not they are legitimate faces or cats, or if
461810	466450	these are machine learning generated. One of the telltale signs, if you look here in the top right
466450	471250	corner, this gentleman, it seems like there's a picket fence behind him, but it's a little warped
471250	477370	in the back. You also notice that all of the images have a blurred background. And if you
477370	482850	look really carefully, all of the eyes point directly at you. So, it doesn't matter what
482850	489070	direction they're faced, the eyes are pointing straight. So, there's already some technologies
489530	493910	and some research being performed to kind of identify what is a legitimate human face and
493910	499410	what is a generated human face. Now, why is this important? How can we use this for social
499410	505950	engineering? Well, for several types of social engineering campaigns, what's pretty common to do
505950	514250	is go to the 10th page of a Google image search, grab somebody's stock photo or profile photo,
514250	518950	and then you can paste that up as your own LinkedIn profile photo and say, okay, I am now
518950	526370	Becky. I'm not a 32-year-old male. My name is Becky. I'm 27. I'm starting my career and I want
526370	531950	to ask somebody, hey, as a fellow woman, how did you get started? And then I can use that to build
531950	537830	rapport, have my pretext, and perform social engineering tasks to try and exploit employees,
537830	547090	maybe in the HR department, sales, wherever. And so, instead of grabbing that image and throwing
547090	552490	it up as a pretext where an analyst can go in, grab that same image, do a reverse Google image
552490	558610	search, see that it's a stock photo, call me out, realize that I'm fake, what I can do is use
558610	564570	something like StyleGAN to generate my own face. So, why steal a person when I can just make a
564570	571950	whole new person? But there needs to be a little bit more legitimacy to my pretext. So, this is
571950	578410	where GPT-2 and Grover come into play. GPT-2 and Grover, they operate very similarly. These are
578410	584370	automated generation of long-form content. So, these can be used to generate emails,
584370	591550	blog posts, articles, anything of that nature, with very minimal effort from the attacker or
591550	598110	the adversary in that respect. And so, unfortunately, the website that was used to generate
598110	604610	these has been taken offline, but the GPT-2 model is still something that you can download, put on
604610	611410	your own systems, and generate long-form content. So, if I was developing a pretext and I can use
611410	619690	this to write phishing emails that could get past a spam filter, or that look distinctly unique,
619690	623750	so that I'm not sending the same email to multiple people within an environment,
624370	629910	or I can use this to write blog posts and long-form content, and because any sort of
629910	639010	Google image search or... what is that called? Plagiarism detection, it will see that I am the
639010	644210	only source of truth, therefore I must be a real person, therefore my website must be legitimate.
644950	650790	And so, this will kind of add to the pretext. And then on top of that, long-form content Google
650790	656370	really enjoys, and so that can give us some Google ranking to add a little bit more credibility to our
656370	664130	initial pretext. So, this is one of the scarier ones that I see as being at the very beginning
664130	669710	of the research, and I'm curious to see where this will be in the next several years. We have
669710	677450	Lyrebird and Tacotron. Lyrebird and Tacotron, these are text-to-speech that are based on human voices
678130	684970	and use those human voices to generate conversation bits. And so, we can use this,
684970	690790	I foresee, in the phishing campaigns where we're actually calling people or leaving voicemails.
690790	694950	So, it's a lot less automated sounding, a little less robocall,
694950	698730	and then you just add a little bit of compression and it sounds like you just have a really bad
698730	704090	connection. So, I have a couple audio clips. I want you to hear them. So, these first two,
704090	709670	I have the original voice and then the second one is the synthetic voice based off that original
709670	717150	voice. So, let's go ahead and take a listen. In another moment, down went Alice after it.
717150	721330	Never once did she consider how in the world she was to get out again.
723310	729090	Awesome. And then the synthetic voice built off of that sample. My voice might be generated by
729250	734290	a computer, but I think it sounds pretty human. I don't know exactly how they made it, but I'm
734290	740110	really impressed. So, what I'm imagining is that as these models get better, what we can do is
740110	746590	take interviews of CEOs and VPs of large corporations, grab their audio, isolate it,
746590	751930	and then put it into a model like this so that we can have our own version of synthetic text to
751930	759890	speech using their voice. So, we can call throughout the company and impersonate these
759890	766010	individuals in order to cause some sort of social engineering action. Now, this last one is really
766010	771250	interesting. This is Tacotron. This is a Google project. And I have the text over here on the
771250	775630	right, but what I want you to do while you're listening to this is pay attention to the breaths.
775630	781330	As humans, we tend to have filler words like um and uh, and then in between sentences and
781330	785670	commas, we usually have a bit of a pause or a breath. And so, the Tacotron model actually
785670	791490	picked that up. This is fully synthetic voice. Just take a listen. Only the photograph really
791490	796350	showed how much time had passed. Ten years ago, there had been lots of pictures of what looked
796350	800890	like a large pink beach ball wearing different colored bonnets. But Dudley Dursley was no longer
801050	805190	a baby, and now the photograph showed a large blond boy riding his first bicycle
805750	810390	on a carousel at the fair playing a computer game with his father being hugged and kissed
810390	820930	by his mother by his mother. That's pretty scary. So, these projects are still underway. Liarbird
820930	828090	is actually advertising itself as a transcription or reverse transcription service, which is just
828090	833450	text-to-speech. In their original pay model that they had pushed out, currently they're in closed
833450	839610	beta, but in their original pay model that they pushed out, you can transcribe three hours of
839610	845650	audio pretty cheaply. And then if you're so inclined to pay, you can transcribe obviously
845650	851130	more. But if a malicious user or somebody who's maliciously inclined, they can just grab those
851130	856290	voice samples, throw them into Liarbird or Tacotron, or just use the voice samples that
856290	861690	are already in there and generate their own adversarial examples.
864720	870480	So, next we get into sandbox detection. And so, this is the defense detection and evasion
871100	876020	part of the system. So, we have these researchers over at Silent Brick Security, and what they did
876020	882640	was decide that with many different types of sandbox, what you can do is take the process list
882640	888220	as data itself. And so, based on the process list, the number of processes,
888220	892740	the number of users on a system, you can generate and decide whether or not you're in a sandbox.
892740	897140	And then you can take it a step further and decide what kind of sandbox you're in,
897140	906120	whether you're in FireEye or Cuckoo sandbox. So, what the researchers did was kind of classify,
906120	911940	they took a number of basic Windows systems and they took a number of sandboxes and just kind of
911940	916600	identified how many processes there are, how many users there are. Let's take a look at what malware
916600	922720	can identify. And so, when we have a low process count, you see over here in column A, we have 33.
922720	929500	In column F, we have 34. And then we have a high user count, such as, again, A and F, we have 4
929500	938320	and 4. And the process count per user, 8.25, 8.5. It gives us a very high probability that it is a
938320	944220	sandbox. And so, we can see that with our host total score. And then based off that total score,
944220	950780	we can set a threshold. In this case, it looks like they set the host average score as their
950780	955280	threshold. So, anything below the average score is considered a sandbox. Anything above that
955280	961160	is considered a legitimate system. And so, they can have malware that completely shuts down or
961160	967160	erases itself if it identifies that it's in a sandbox, or it can use targeted evasion techniques
967160	972880	per sandbox. So, if I'm in FireEye, these are the things that I'll do to get past it. If I'm in
972880	978100	Cuckoo, these are the things that I'll do to get past it. So, it's a really interesting article,
978100	985260	highly recommend taking a look at that. And then for automated network exploitation,
985260	990300	we have systems like DeepExploit and Goethon. Goethon and DeepExploit, they both operate very
990300	996020	similarly. The way that these work is they will go out, they will perform an nmap scan based on
996020	1002100	the results of that nmap scan, if it sees common services that it knows are exploitable or things
1002100	1007400	that it would like to kind of take to that next level. For example, port 80 or 443, it's web
1007400	1012380	traffic. So, it's going to go ahead and run Scrappy to go and crawl that system. I'm sure they could
1012380	1018840	add other tools such as Go Buster, which will go out and enumerate a lot of the files or directories
1018840	1023400	or URLs that are on that system. Try and fingerprint the server. You want to see if it's running
1023400	1032120	Nginx, Apache, PHP, Java, things of that nature. And then it performs additional content exploration
1032120	1037740	and its own Google hacking, which is kind of cool. So, that will go out, do a little bit of research,
1037740	1042920	see if there's any sort of exploitable scripts, things of that nature. And it takes all of this
1042920	1051040	data and it processes it using some statistical techniques and then it uses Metasploit as a
1051040	1056260	targeted exploitation engine and it goes and tries to exploit those things. So, if it sees
1056260	1061120	certain versions of JBoss or Jenkins, it will go and try and target that with maybe
1061680	1069340	password spray or any sort of Metasploit module. Tomcat, same thing. So, this is kind of cool
1069340	1075060	as a proof of concept of automating the entire process of network discovery all the way up to
1075060	1082620	exploitation and in some cases post-exploitation. One of my favorite while I was researching this
1082620	1088940	is Deathstar. And this is my favorite just kind of in the way that it is assembled. So, Deathstar,
1088940	1094160	it's built on PowerShell Empire. And the way that it works is it starts, it kind of covers its
1094160	1100560	baseline and it is designed to go from standard user all the way up to domain administrator.
1100560	1106140	And so, it charts and maps attack paths all the way to that location. And so, it starts at its
1106140	1110780	baseline and it says, am I already a domain administrator? If so, cancel out. And we see
1110780	1116460	that's the path off to the left. If I have domain administrator credentials, all right, cool. Log
1116460	1121420	in. And that charts off the path to the right. But if it's neither of those, then it goes straight
1121420	1127040	down the middle and it says, okay, let's spawn a new agent. Let's go ahead and identify where is
1127040	1132660	the domain administrators and where is the domain controllers. And so, it runs over here to the
1132660	1138160	right. We can see get net domain controller, get net group member, invoke user hunter. And so,
1138160	1144660	this is going to enumerate the domain controllers, enumerate the domain admins, and enumerate the
1144660	1149440	users that it can on the systems that it can. And you can see it goes through this entire tree
1149440	1153660	kind of off here to the bottom left. It identifies and says, hey, is this a vulnerable,
1153660	1158560	known vulnerable version of Windows, for example, Windows 7? If yes, let's run Mimikatz,
1158560	1164040	try and do Mimikatz, secret stump, things of that nature, collect credentials, and then we can kind
1164040	1169420	of start back over at the top. Let's go ahead and enumerate the running processes. Do we see any
1169420	1175440	processes that are already being ran by other users that we don't have in our inventory? If so,
1175440	1179120	let's add those users to our inventory and try and do some lateral movement.
1179580	1184380	And so, in this case, it'll use psinject, and then it'll go ahead and spawn a new instance.
1184380	1190540	And so, it goes through this iterative process all the way until it reaches domain admin credentials.
1191300	1197260	And so, if you've ever used a system like Bloodhound, where you'll generate an attack
1197260	1202640	graph and say, okay, this user is part of this group, which has these members, which this member
1202640	1207360	is also part of domain admins. And so, this is the attack path that you want to go in order to
1207360	1213500	become domain admin. And so, Deathstar kind of takes that idea a step further and actually tries
1213500	1222720	to navigate that attack path in an automated fashion. So, that's all I have for the machine
1222720	1227060	learning tools that already exist, some of the proof of concepts. There's a lot more information
1227060	1231760	that I wanted to cover that I wouldn't be able to cover in such a short amount of time. So,
1231760	1237940	at the end of this presentation, I have a link to my slide deck, which has a lot more detail
1237940	1244280	and a lot more tools and a lot more techniques in it. But in the meantime, I kind of want to
1244280	1249460	answer the question of how do I go about creating my own machine learning tools
1250360	1258180	or AI tools? What should I look for? How do I automate that? What is that process? And so,
1258180	1264100	honestly, my advice would be to identify somewhere that either utilizes a lot of data
1264660	1270900	or identify somewhere that uses a... it's a common process, but it uses some sort of fuzzy logic,
1270900	1275060	right? Something that's kind of hard to script, hard to write a signature for,
1275060	1280400	but with AI and machine learning, you can use some of that fuzzy logic to generate some heuristics
1280400	1285660	and it can perform based off of that. So, in the case of some of these, what we can do is say,
1285660	1292060	all right, NMAP scans, everybody does NMAP scans, and then you have to manually look at them,
1292060	1297180	and then based off of that, you have to follow up and go over here. Instead, like I showed with
1297180	1303720	Goython, is you can just use the NMAP scan and then automate the process of, okay, let's enumerate
1303720	1307540	and fingerprint these services, and then let's see if any of these services are already exploitable
1307540	1314240	using Metasploit, or let's go with next steps. I identified a web service, let's run web analysis
1314240	1321820	techniques. So, Go Buster, Crawler, any sort of things like that. So, that's kind of the thing
1321820	1326300	that you want to keep an eye on if you want to develop your own machine learning tools. And
1326300	1331920	there's more that are coming out all of the time, and it's really interesting and a really fast-paced
1332440	1337280	area, and it's just ripe with research. So, go ahead and play around with some of that.
1339200	1343260	So, the next bit that I want to get into is adversarial machine learning,
1343260	1348360	and this is attacking the machine learning algorithms themselves. So, adversarial machine
1348360	1353900	learning, there's really three ways to go about attacking a machine learning model.
1354160	1359780	So, the first way we have is model evasion. This is to take what is clearly a malicious example
1360300	1366500	and get the model to consider it to be benign or normal or standard to bypass that filter.
1366500	1370700	So, this would be taking a spam message and getting past a spam filter, or this would be
1370700	1376780	taking a piece of malware and getting it past antivirus or anti-malware solutions.
1377420	1385160	Model poisoning is where we add our own data into the training data set. Now, machine learning
1385160	1393680	models specifically are... they're done learning as soon as they're done being trained. To get
1393680	1398420	around this, there are some models that are called online systems, and so what they'll do is they'll
1398980	1404860	take data that has been classified by itself already. So, say I send it a piece of data,
1404860	1408680	like a piece of malware, it looks at that, says, okay, bam, that's malware,
1408680	1414820	now that goes into my training set, and I learn from that malware. So, data poisoning allows us
1414820	1422580	to slowly inject certain pieces of data so that we push that threshold a little off to the side
1422580	1429240	into what is more acceptable, and then we target where that threshold has been pushed.
1429680	1435160	And we'll see an example of that in a bit. Data leakage. Data leakage is actually
1436360	1443560	a combination of two smaller bits. So, one is stealing the model itself. So, for example,
1443560	1448820	say you're an IoT engineering company and you develop an IoT system that utilizes some form
1448820	1455000	of machine learning. What I can do is buy that device and then reverse engineer it,
1455000	1460300	and now I have your model, and so I can use that model or train another model based off of that
1460300	1463900	instead of having to go through the hundreds of thousands of dollars that you went through
1463900	1470660	to develop that in the first place. And so, this is where the model itself is easy to be grabbed.
1470660	1479440	There's also data leakage, which is I can identify what data you use to train the model.
1479760	1483240	And so, this is really damaging, especially for healthcare industry or
1485320	1490860	HIPAA-protected information. So, if I can identify any sort of clients or
1492120	1497280	patients that you have, any sort of their PII, then that's really damaging.
1499760	1505080	So, when it comes to model testing, there's really two approaches. You have a white box where you
1505080	1508140	know what the model is, you know how the hyperparameters are set, you know everything
1508140	1513280	that's involved in the building of this, and then you target that model, much like you would do in
1513280	1517820	software where you already have all of the code involved. And then you have a black box, which is
1517820	1524000	just the model itself. You aren't able to peek into the way that the model operates, but what you
1524000	1527260	can do is treat it like an oracle. So, if you've ever done any sort of cryptography, you know,
1527260	1534280	exercises, you have an oracle, you ask yes-no questions, or things of that nature, and then
1534280	1540020	you can generate what is called a surrogate model. So, based off of the yes-no questions that I asked
1540020	1545760	the black box model, I generated my own model, and then based off of that model, I can perform attacks
1546620	1552020	and that those attacks should be able to impact the original black box itself.
1552660	1557780	Now, the way that this operates is under a principle called attack transferability.
1557900	1563160	This is a documented phenomenon, and the way that attack transferability works is,
1563160	1568460	it doesn't matter what model or the underlying architecture that these models are designed with,
1568460	1573600	an attack on one model can also be an attack on another model. So, if I have a deep learning
1573600	1578300	model over here, and I find adversarial examples that I can use to attack this model,
1578300	1583740	then here is a random forest, a logistic regression, SVM, or anything else, any other
1583740	1590220	machine learning algorithm or architecture, that those same adversarial examples will be able to
1590220	1597100	impact that. And then, of course, the attack surface. So, where are the machine learning
1597100	1603980	models themselves vulnerable? So, machine learning, as it works, is you have a physical object,
1603980	1608120	and then you take that physical object and you try to digitize it. So, say you have a picture
1608120	1612840	of a panda or a car, you take a photograph of it, now it's digitized, and you can make
1612840	1618200	information off of that. You take that digitized information, you run your pre-processing as you
1618200	1622540	would per the normal machine learning process, and then you put that into the machine learning
1622540	1629460	model as part of its input features, it presents some output, usually a probability or a number
1629460	1636180	of some sort, and then a decision is made based off of that action. So, for example, if I am 67%
1636180	1641320	confident that this is malware, then I will probably flag it as malware. If I'm only 42%
1641320	1647200	confident, then I'll just let it continue to operate. So, where can we attack things? Well,
1647200	1653620	one, we can actually impact the physical object itself. So, down below, I have the network
1653620	1659020	construction detection system as kind of an example that follows this generic machine learning model
1659020	1664160	path. So, we have the attack traffic. We can actually, as attackers, modify the attack traffic
1664160	1669380	in such a way that as it goes through the pipeline of the machine learning pipeline,
1670040	1678040	it will modify the behaviors at different points. So, I can slow down my attack traffic,
1678040	1683000	I can use my attack traffic for multiple hosts so that it's a little bit harder to detect.
1683680	1689560	And then that attack traffic is then digitized. The digitization process, we don't really have
1689660	1697080	a lot of impact over as attackers. It's not realistic for us to be able to affect TCP dump.
1699220	1703880	There's really famous examples. For example, you may have come across this if you ever looked at
1703880	1708880	adversarial machine learning, a picture of a panda, and you modify some pixels,
1708880	1715260	and now it thinks it's a gibbon or a bird or some sort. But if we have the ability to modify pixels,
1715260	1722120	why don't we just give it a picture of a bird at that point? And then more to the point,
1722120	1726480	if we can modify pictures, if you have something like a self-driving car, it has a camera,
1726480	1731200	how do we interface with that camera in an adversarial perspective? We would already have
1731200	1734860	to have access to the system, which at that point we can just shut it down or do whatever.
1734860	1742820	It's physical access. So, we're not actually able to interface with the digitization of the object,
1742820	1748200	but what we can do is change the way that the attack traffic is modified.
1748200	1751620	We can change the input features, the input vectors, the machine learning model, especially
1751620	1756920	with the model poisoning. We can change the way that the data is represented.
1758460	1762620	So, in the case of the network intrusion detection system, we have the packet metadata,
1762620	1768520	and then it spits out an attack probability, and then off of that it makes a decision.
1768840	1772340	And in between the machine learning model and the decision-making process, you got to remember
1772340	1777540	this is just software, and you can approach it like any other piece of software that you're
1777540	1781440	pen testing or trying to hack. So, you have the machine learning model in it, and then
1781440	1785840	all wrapped around it is a nice piece of software that probably has some of its own vulnerabilities
1786360	1794840	involved. So, model invasion. Model evasion, not invasion, sorry.
1795680	1799600	So, we're going to try and hide in the blind spots. Now, we have this massive theoretical
1799600	1805660	space. This is all of the possible pieces of malware that could possibly exist, but we can't
1805660	1811080	collect all of that, realistically speaking. Antivirus companies can't collect all of it,
1811080	1815520	and so what they have is a subset, and this is considered the training space. So, this is what
1815520	1821140	they are realistically able to collect. So, in their training space, they perform their training,
1821140	1825960	and then off of a small piece of that that's already clean labeled data, they perform their
1825960	1832040	testing to make sure that the model operates properly. Well, part of that testing space,
1832040	1835820	kind of right there off the sliver, this is the adversarial space. This is where we live.
1835820	1842020	This is where we are able to generate adversarial examples that target that model specifically.
1842440	1846760	And so, this is kind of what we want to play around with and hide in the blind spots,
1846760	1852240	so to speak. So, a couple really good examples is, in this case, multinomial
1852240	1857860	Naive Bayesian spam filters, or just any of the early spam filters that used some rudimentary
1857860	1862360	forms of machine learning, how they would operate is they would look at keywords. So,
1862360	1869920	if you had something like, buy Viagra now, the probability that the word Viagra shows up in a
1869920	1875160	spam email is much higher than the probability of the word Viagra showing up in a regular email.
1875160	1879200	And so, based off of these keywords, it does its own statistical analysis and says,
1879200	1885680	I am 99% sure this is spam. Well, the interesting thing about emails is that
1885680	1894440	emails are usually represented as HTML pages or web pages. And so, as malicious actors,
1894440	1899700	what we can do is actually add in a comment. So, in this case, we can add in the entire Wikipedia
1899700	1905200	page article for a horse. And so, we have a bunch of bad words, but we have even more good words.
1905200	1911060	What this lets us do is kind of skip past the spam filter by showing, hey, we have a lot more
1911060	1916960	good words than bad words, so we must be good, right? It says, yeah, absolutely. I am 99% sure
1916960	1923280	you are not spam. And the thing is that the way that this is presented to users is as an HTML
1923280	1928520	web page. And so, the comment itself is not actually rendered for the users. So, the users
1928520	1936620	still see the original spam email as though it was unmodified. And so, this is kind of a real
1936620	1943680	world example on some of the spam filters. A really good example of this is by Skylight Cyber.
1943680	1949200	The researchers over there were actually able to bypass the silence machine learning antivirus
1950960	1956620	in this article, Silence, I Kill You. Really good article. Highly recommend you take a look at it.
1956620	1962060	We can see some familiar malware samples here. So, we have CoinMiner, Emotet, Zeus,
1962880	1968980	and we see the original scores that they were actually able to pull out of the silence antivirus,
1968980	1977920	negative 826, negative 923, negative 997. So, these are really bad, really bad, flagged immediately.
1978980	1984920	So, what the researchers over at Skylight Cyber did was they took the silence antivirus,
1984920	1989920	reverse engineered it, and they were able to pull out the scores based on some of the
1989920	1994740	syscalls and other key features that silence looked at for malware.
1995940	2002120	One of the curious things is that they identified certain things that were whitelisted. Now,
2002120	2007820	why would silence whitelist certain things, even though the standard machine learning process would
2007820	2012220	actually identify it as potentially malware? Well, the reason why is because when you think
2012220	2017720	about malware, think about how malware operates, right? So, maybe it asks for access to your webcam
2017720	2025360	or your microphone, or it tries to resize or open up new windows. Maybe it makes calls out to the
2025360	2033280	internet. Well, what's another thing that does that? Chrome, Firefox, Internet Explorer. They
2033280	2039360	operate very similar to some modern malware. In this case, what they found was a video game.
2039820	2044640	I'm not sure which video game it is. My money is on Fortnite or League of Legends.
2045240	2050520	But the video games that they found were certain whitelisted keywords. So, what they did was took
2050520	2057660	these clearly bad malware and they just added these keywords to the end of the malware. So,
2057660	2063340	it didn't change the way that the malware operated, but it changed the way that silence looked at it
2063340	2067440	and said, okay, you have a bunch of bad boy points, but you have even more good boy points.
2067440	2073020	So, we're going to give you some good boy scores. So, in this case, CoinMiner had 884.
2073120	2078460	My favorite is looking at Zeus, negative 997, one of the worst in this table,
2078460	2081960	all the way up to positive 997, which is one of the best in this table.
2082680	2090140	And so, it shows that model evasion techniques are very useful in a number of different ways.
2092100	2095680	So, how do we defend against this? Well, the first and probably one of the better
2095680	2102220	is adversarial training. Adversarial training kind of comes from the idea of chaos engineering.
2102220	2106600	Netflix is a really big proponent of chaos engineering, where you already know something
2106600	2111500	bad is going to happen, so you might as well account for those bad things to happen. So,
2111500	2115360	in this case, you train it with adversarial examples. You generate your own adversarial
2115360	2120120	examples and you train the model based off of that to kind of harden and make it a little bit
2120120	2127060	more robust. Again, if it's not a robust model, it's not a good model. Chaos engineering as a
2127060	2131820	whole, the way that it operates is you already know something is going to go bad. So, you have
2131820	2137260	Netflix, for example. Netflix, they know that their systems, they need to have that uptime.
2137260	2140280	And so, they know that it's possible one of their systems goes down. They want to make sure
2140280	2145540	their redundancy operates properly. So, you purposefully go into production and you break
2145540	2153360	things. And then you develop processes and policies around knowing things are going to
2153360	2158000	break. So, in some cases, you'll have the CIO walk into a data center and just start unplugging
2158000	2165180	servers. So, adversarial training is a really good way to kind of counteract this. Another
2165180	2171840	slightly more technical way is called defensive distillation. Defensive distillation is a
2171840	2176520	smoothing algorithm that kind of smooths the curve. So, instead of having, okay, malware,
2176520	2181900	malware, malware, malware, nope, it's okay. You can't have sharp edges like that. So, what it
2181900	2187220	does is it smooths it. So, instead of going straight down, it just kind of goes in a downward
2187220	2194680	trend. And so, it's a little bit harder, a little bit more robust way of analyzing and dealing with
2194680	2201400	the adversarial examples. And then the last example would be monotonic classification.
2201400	2206100	Which means that the graph grows in one direction. So, we see here in the image off to the right,
2206100	2213560	the non-monotonic example, we see that it's getting worse and worse and worse as far as a
2213560	2217620	keyword. So, it goes worse and worse and worse across that threshold. Now, it's considered bad,
2217980	2222700	it's spam, it's malware, it's whatever. We're ready to throw it out, but we're still analyzing
2222700	2226680	it. And so, it's crossed that threshold, but wait, no, it's getting better and better and better.
2226680	2232780	We just hit that pad of keywords that are really positive. And so, we went from good to bad,
2232780	2238080	all the way back up to good. Monotonic classification means that we don't count the
2238080	2243280	good words at all, we say, we measure your badness. How many bad words do you have?
2243560	2247680	And so, in this case, on the bottom example, we have the monotonic classification, which says,
2247680	2251420	okay, we have some bad words, we have some bad words, up, we all went all the way up to the top,
2251420	2256620	and then we hit all of the padded good words. And it doesn't matter, you're still bad,
2256620	2261780	Jesus is not going to save you, you're done. And so, monotonic classification is a really
2261780	2266500	powerful technique to deal with these model evasion techniques.
2268480	2276600	Next, we have model poisoning. Model poisoning almost exclusively works on online learning
2276600	2281940	systems. So, as adversaries, we need to find a way to get adversarial examples into the training
2281940	2290240	space. And so, this is where systems will grab data that it's already classified a certain way,
2290240	2293760	and add it into the training space. And so, we're just trying to push that barrier
2295040	2300080	past a certain threshold where we can actually operate safely as adversaries without it detecting
2300080	2309380	us as such. So, some real world examples. Tay AI, if you're not already familiar,
2309380	2317460	I highly recommend checking it out. Tay AI was a Microsoft Twitter chat bot, and it was designed
2317460	2326000	to learn from real conversations by real people. Well, 4chan, of course, gets a hold of this, and
2326000	2334580	they decide, how much does it learn? How much can we teach it? And about 16 hours after coming online,
2334580	2341280	it was taken offline permanently, because within that time, 4chan was able to get it to say,
2341280	2348500	Hitler did nothing wrong, and racial slurs. If you want more information on Tay AI, and I highly
2348500	2353660	recommend checking this out, it's called The People's Chatbot by the Internet Historian on
2353660	2360400	YouTube. Really funny guy, really great story, and it's just incredible to think of how many
2360400	2367240	data scientists and developers and people behind this project put time in, put effort, put work,
2367240	2375880	and it was shut down by a bunch of kids on 4chan. The other one is Jacobian Map Saliency
2375880	2383440	Attack, and this kind of goes into the pixel manipulation. So, what you do is you grab a
2383440	2388540	couple pixels, and you modify them, or you tweak them a little bit, or you tweak the data a little
2388540	2394240	bit, doesn't necessarily need to be image data, and you pass that on as a certain class.
2394880	2398880	And so, as the model is learning from this data, learning from this class, it starts identifying
2398880	2405720	these tweaks, and then you can implement those tweaks in an adversarial example. So, in this
2405720	2412180	case, we have a 70 kilometer per hour sign, we add in these little tweaks, and it latches onto those
2412180	2422120	tweaks and says, okay, you're a 30 kilometer per hour sign. And so, imagine the kind of chaos that
2422120	2430280	that can create. Again, these tweaks are difficult to implement in real world examples. This is also
2430280	2436980	difficult to pull off simply because it relies on an online learning system, something that learns
2436980	2441680	from the data that you've already presented it. But these are things that researchers have been
2441680	2445600	able to identify and actually use to attack machine learning systems.
2447440	2453540	So, how do we defend against poisoning attacks? Basically, it comes down to, you can still use a
2453540	2460040	lot of the data, but what you want to do is add your own little smoothing filter. So, in this case,
2460040	2466620	have longer periods of time between retraining. So, what that will do is kind of narrow the window,
2466620	2473600	not narrow the window, it will expand the window that an attacker would need to generate adversarial
2473600	2479690	examples for. And then they would also have to compete for votes against legitimate traffic.
2480240	2487280	So, it's kind of like, if you look at Bitcoin, you need 51% of the vote in order to kind of
2487280	2496060	take over cryptocurrency networks. So, this is kind of their own example of doing this.
2496060	2501180	You can also analyze longer periods of data. Again, it broadens the window that you're looking
2501180	2505060	at. And so, the adversaries, the attackers are going to have to generate more traffic,
2505060	2510680	more examples in a longer period of time, which is difficult to do. And then you just want to
2510680	2514640	generally minimize the impact of adversarial training examples. So, whether you have a human
2514640	2520960	in loop that's manually looking at some of these examples, or maybe you're taking a small sample
2520960	2526240	of the stuff that you're seeing and only allowing that to affect the model very...
2528420	2535060	a little bit. So, it's just kind of a general rule of thumb. It's just minimize the impact,
2535060	2541900	come to the understanding that people will try and poison the well, account for that.
2543460	2549420	And then, of course, we have data leakage. So, data leakage, this is usually when,
2549420	2554160	if we're trying to steal data, not steal the model, but steal data itself, this is usually
2554160	2560440	when the models are trained to be a little too good. So, this image down below, you see aerial
2560440	2565320	photographs, generated map, and then an aerial reconstruction. And so, articles at the time,
2565320	2570300	this is a project by Google, articles at the time were kind of grandstanding and saying,
2570300	2576900	oh my god, the Google AI found a way to cheat and generate aerial photographs.
2577460	2582640	It was able to embed its own data. From the machine learning perspective or data science
2582640	2586360	perspective, this is actually called overfitting. It's a very common problem. It's nothing
2586360	2594140	spectacular. It's just, you didn't train your model right. And so, especially with overfit
2594140	2598040	models, you have the training data in the model itself, or the model is able to generate that
2598040	2603440	training data. And so, as I was saying earlier, with things like credit card information, PCI
2604000	2608480	protected information, personally identifiable information, public health, or not public,
2608480	2614160	sorry, personal health information, or HIPAA protected data, things of that nature,
2614160	2621680	it can be very damaging for a machine learning model to leak real training data. And so,
2621680	2628500	it's important to kind of evaluate and make sure that the models are not overfit, are not underfit,
2628500	2633700	just hidden just right, or that the data is being preprocessed in such a way that it's
2633780	2639760	a one directional. So, in some cases, what they'll do is generate a hash of certain features. For
2639760	2643800	example, instead of putting in a credit card number, they generate a hash of the credit
2643800	2652140	card number. And so, it's not reversible in any meaningful way. And then, of course, model theft,
2652140	2658340	model theft by competitors. There's a technique called federated learning, which is you encrypt
2658340	2666180	the machine learning model itself so that the model has to be used in that encrypted form. You
2666180	2672620	can't steal the model, you can't decrypt it. It's a one way kind of thing, or a one way street in
2672620	2678680	that scenario. And so, this will allow like IoT devices to be deployed with machine learning
2678680	2683920	models. And it will prevent competitors or attackers, malicious actors from reversing it,
2683920	2686820	stealing that model and building their own product off of it.
2689420	2693760	And then, one last thing that I kind of want to cover is adversarial stickers. So,
2693760	2698720	you've probably seen adversarial stickers. They look like melted crayons, in my opinion.
2700980	2705700	But adversarial stickers, they have a bit of a problem. This is considered a brittle attack,
2705700	2711680	and it's not very repeatable. And the reason why is because it doesn't have that attack transfer
2711680	2719980	ability that I brought up earlier. So, in this case, kind of off to the bottom right,
2719980	2724440	the stop sign, this is one of the first articles I was talking about adversarial stickers, and they
2724440	2731960	were trying to modify a stop sign in such a way that the machine learning model didn't recognize
2731960	2736360	it as a stop sign anymore, recognized it as something else. And so, this could be really
2736360	2742560	damaging for self-driving vehicles. One of the problems is that this paper ran into some
2742560	2748400	repeatability problems. And the reason why is because this attack is so brittle, they were
2748400	2754080	able to convince their model, but they were not able to convince, or people who were trying to
2754080	2760980	replicate this were not able to convince other models. So, it's not very useful. You can't just
2760980	2769440	slap an adversarial sticker onto a sign and hope that you're driving a self-driving vehicle off
2769440	2775480	into a wall or something of that nature. And then off here to the left is kind of the same idea with
2775480	2782360	facial recognition. So, this researcher here said, okay, I'm going to set some constraints. I only
2782360	2788300	want the pixels or the image around my eyes in the shape of glasses to be modified so that I look like
2788300	2795860	this actress. Honestly, I don't remember who this is. I apologize. But I'm a man. I want to look like
2795860	2801560	this actress per a facial recognition model. And so, he was successfully able to do that.
2801760	2809380	The problem is that these glasses are designed for his face and the rest of the features of his face.
2809420	2814440	And so, if I or many of you put on these glasses, it'll either look like somebody else or it won't
2814440	2821740	recognize it as an adversarial example and it'll just recognize us as people, the same people,
2821740	2826620	just wearing stupid glasses. So, attack transferability, adversarial stickers don't
2826620	2831800	really have it. It's not a really good attack. It's very brittle. And I just kind of wanted to
2831800	2840140	bring that up. For more information on this, I highly recommend DEF CON 2018 AI Village. Sven
2840140	2845940	has a really good 30-minute talk on adversarial stickers and kind of the challenge with them.
2845940	2853420	And then, for the facial recognition specifically, I know that Rich Horang in AI Village 2019 has a
2853420	2859580	really good talk talking about facial recognition and trying to bypass that.
2859580	2867180	So, to recap, how has AI empowered attackers? Well, it allows us to operate at a speed and scale
2867180	2873460	like never before. We can operate at machine speed. And then, what we are seeing is the very
2873460	2878540	beginning of what AI can bring to the offensive security space. So, we've already seen some
2878540	2884940	adversarial attacks in the wild. I anticipate us seeing more as more researchers are coming out.
2885120	2890180	AI and machine learning is kind of the hot new buzzword that everybody's latching onto. So,
2890180	2895480	CEOs and decision makers are grabbing the AI-powered or machine learning-powered
2895480	2901000	detection systems, defense systems. And it's kind of become a bit of a buzzword
2902180	2907260	where you say that in a lot of common companies, especially in places like DEF CON, and they tell
2907260	2911980	you to go talk to the Bitcoin folks and blockchain and all that fun stuff. There's no silver bullet,
2911980	2917880	but we are seeing the very beginning of what is possible. And I anticipate, and I'm really hoping
2917880	2925820	in the next several years, that we see a much larger impact of this AI versus AI kind of warfare
2927480	2933720	that should be coming about. But anyways, I appreciate it. Thank you for your time. Thanks
2933720	2938160	for sticking around this long. You can go ahead and email me if you have any questions. I'm also
2938160	2944300	going to be in the chat. And then, for the slides, I have that are much more detailed are on
2944300	2950820	Slideshare, and you can find them at slideshare.net slash GTKalandai. So that is my presentation. I
2950820	2956100	hope you learned something, and thank you for sticking around. Bye!