start	end	text
4100	11220	Here's our second talk of the day. It's Eric Gallican.
12320	19720	He's a... well, you see his Who Am I page, but he's also a PhD student at Brandeis and
21480	28080	an all-around awesome guy. He does way too much stuff and is very enthusiastic and involved in
28080	34160	the community. So I'm just going to let him talk about... on his intro talk about...
34860	36720	for machine learning people.
40930	46830	Hi, everybody. Welcome to my talk. This is Baby's First 100 MLSat Words.
47210	52290	My name is Eric Gallican. I'm an artificial intelligence researcher at Rapid7. I specialize
52290	59710	in breaking machine learning stuff and applying machine learning to detecting bad things.
59710	65130	I also do research at the Montreal AI Ethics Institute on applying DevOps principle to
65130	72370	machine learning ethics. And last year, I was here at DEF CON in person, not here in my house
72370	79690	at DEF CON, presenting at the Cloud Village on a piece of malware that I wrote. Malware,
79690	88190	not actually malware, just in case the legal team is listening. And I presented that piece of proof
88190	95830	of concept malware called Sassy Boy. But mostly I do boring math stuff. And if you can't see my
95830	104030	speaker video for some reason, I put a picture of my face on the right so that you can see it. Or
104030	111150	you can see it twice. You can see that my hair hasn't changed basically at all in years. So
111150	116950	before we get started, let's just answer the question, what is MLSat? Because I get asked
116950	123970	to that a lot by practitioners in security and in machine learning, honestly. And mostly it's
123970	131390	an argument. And it's an argument between three groups of people, as it usually is. And the first
131390	137210	group of people is people who think it's only applications of machine learning and artificial
137210	144570	intelligence to security problems. So that's automated malware detection. That's next generation
144570	151730	SIMs. That's automated red teaming and that sort of stuff. And then there are people who think that
151730	158790	it's just securing artificial intelligence in machine learning systems. And then there are
158790	166250	those of us who are correct who think it's both. It's kind of any intersection of artificial
166250	174350	intelligence, right? That broad category of machine learning slash AI and security.
174470	184790	And since this is my talk, and since I am the Mao Zedong of MLSat, I get to decide that it's both.
184790	191270	And if anybody argues with you or you hear anybody having this totally asinine argument, you can just
191270	196470	tell them that I said it's both. And then the argument can be over and everybody can do more
196470	204190	productive things with their lives. So we'll start with deep fakes, right? So D is for deep fakes.
204190	211110	And this man on the right, if I didn't have a speaker video, I probably could have just put
211110	215410	that picture up and said it was me. And everybody would have believed it because he's got kind of
215410	223960	he looks like the kind of guy, right? And I'm allowed to say that because he's not real.
224190	229630	This is not a real person. This is completely an AI-generated image.
230490	236890	And so, you know, to give deep fakes a real definition, they are a convincing synthetic
236890	245110	image, video, or audio recording which purports to be real. So, you know, I'm pretty convinced
245110	251750	by this picture. If I saw it in the wild, right, if somebody put it as like their Twitter avatar
252370	259030	and was like, yeah, I am a software engineer at Google, I could buy that. I could buy that
259030	264810	based on this image, right? You know? We have our stereotypes, right? The thin-rimmed glasses,
264810	272470	the scruff. And it's super convincing. It's super convincing. And I think that's really,
272470	278790	when we think about impact, that's what it is. It's that they're convincing to users. So, we also
278790	287330	see deep fakes in the text generation space where it purports to be a real email. Or we see deep
287330	296690	fake writing which mimics the style of famous authors. We see deep fake audio recordings which
297250	302770	sound like the person who they are trained to sound like. Now, there's a piece of commercial
302770	308370	software out there called Liarbird which does this really well. And so, you know, they're very
308370	316710	convincing to users. And as a result, you know, they get used widely and they're easy to make,
316710	324410	right? That's the other thing, is they don't take as much time as finding and crafting an image or
325430	332170	carefully learning a style or mimicking a voice, right? You just kind of train the model and push
332170	338410	them out. So, they're easy to make and they're convincing to users. Which means that they can
338410	345870	have real impact. But that impact is mostly societal, right? So, there was a famous, you know,
345870	353450	semi-viral video of Jordan Peele doing a deep fake where it was an image of President Obama
354250	365030	speaking. And you can look it up. It's really easy to find. And so, like, the risk there is really
365690	374530	that broad swaths of people will believe that a doctored or deep faked video is real. And so,
374530	385370	there's something like President Donald J. Trump, you know, declaring war on North Korea
385370	391370	on a hot mic or suggesting that he might invade North Korea on a hot mic. That could be really
391370	396730	convincing to people. And so, that's the real risk of deep fakes, is that they are a powerful
396730	403270	disinformation tool which is powered by artificial intelligence. And so, you know,
403270	411910	our mitigation against deep fakes is kind of, eh. Like, there's very little we can do about deep
411910	417970	fakes. They're difficult to detect. They're very convincing to users. They're easy to make. And so,
417970	426290	really, it's a broader social problem that we need to tackle of having people be inherently
426290	435090	less trustful of information sources and really vet where they get their information from,
435090	443370	which is a hard problem. But this isn't an enterprise security threat. This is a social
443370	452250	threat. So, moving back to thinking about deployed models, we'll look at adversarial examples. And
452250	459230	this gif over here on the right is from a pretty well-known example that was really,
459230	465270	really hyped up. And it was the turtle rifle paper. Because the Google computer vision API
466150	472570	is what they attacked. And they 3D printed a turtle. And this is pretty obviously a turtle
472570	478930	to anybody watching who isn't powered by the Google Vision API. And if you are an artificial
478930	485550	intelligence powered by the Google Vision API, I'm so sorry for the rest of this talk. Because
485550	492470	you're not gonna like it. That, you know, it's an input to the classifier that's specifically
492470	502210	crafted to force misclassification. And so, this object as a turtle, just a little, like,
502210	509710	toy turtle, right? Being misclassified as a rifle, the risk there is if you have a CCTV system
509710	519810	that's powered by Google Vision API, which seems reasonable to license, that's looking for
519810	527730	potential security threats. And, you know, you don't input all of that, like, racist physiognomy
528490	533850	that we already see in a lot of those systems. But instead just say, you know what? Let's just
533850	542570	look for weapons. Well, this could falsely trigger those systems. And so, there's a risk there, too.
543610	552930	Correspondingly, if you 3D printed a gun, right? That got misclassified as a turtle,
552930	559550	that would also be a threat, right? So, the impact, really, is that it causes misclassification.
560210	564650	And so, that real world security threat and 3D printed examples are really rare,
564650	570570	and they're really difficult to do. And they don't tend to work in all situations,
570570	576310	and any researcher in computer vision can explain to you why it's not a real part of the threat
576310	582450	model. And that's fine. But our threat is that it causes misclassification. As security practitioners,
582710	591410	we try to look for ways to use technologies to detect bad things. And so, if you integrate
591410	599030	artificial intelligence into your malware detector, for example, an attacker can bypass detection
599030	605270	using adversarial examples. And the thing with adversarial examples is when we think about how
605270	612030	to mitigate them, one of the only truly effective ways to do it is through what's called
612030	618150	adversarial training, where you basically show it a bunch of adversarial examples and tell it,
618150	623950	like, no, classify these correctly. And that's really time consuming, and it's expensive,
623950	629150	and it's hard. And so, not a lot of people do it. And it requires you to take your model out
629150	636090	of production and put it back into training, which, again, has a business impact. So, one of
636090	642310	the other ways to sort of mitigate it is through what's called ensembling, where you don't just
642310	650330	take one canonical model. You take several models and sort of average their output or take a weighted
650330	657510	average of their output or whatever. And so, that's another way to do it. And that's kind of
657510	664510	because it's a lot harder to come up with an adversarial example that works across a variety
664510	669250	of classifiers, especially if they're not trained on the same data, if they don't have the same
669250	675590	architecture. And it's still possible to do, but it sort of raises that barrier to entry. And if we
675590	680890	think about it in terms of, like, cryptography, that's really what we're trying to do, right?
680890	688770	Is we know that there is hypothetically always an attack, right? You can brute force the key space.
688770	698090	That's a hypothetically feasible attack against AES, right? But it's good enough, and it takes
698090	705150	long enough to brute force that key space that, you know, it's fine. It does the job, right? And
705150	710210	that's sort of the same thing here, is you just have to raise that barrier to entry to make it
710210	717030	not worth it for an attacker. So, the next thing we want to talk about
717610	722630	in terms of threats to deployed models are backdoors. And so, backdoors in a machine
722630	730450	learning context are manipulations of trained model weights that result in a specific outcome
730450	737770	each time. So, a common way to do this in neural networks is bias poisoning, where you take one
737770	743790	class and you make the bias for that one class in just the output layer really, really, really big.
743790	749210	So, you're only changing one weight in the whole network. So, it doesn't change a lot, right?
749890	757750	But by virtue of making that weight really big, you'll always get the same
758690	762910	classification. And if it's a binary classifier, right, say it's, again,
762910	769150	our malware detector, because people love to make neural network powered malware detectors
770390	775010	for whatever reason. And I've made one. So, I'm allowed to criticize.
777230	782290	People love to do it. And that's a really easy way to just be like, nope, everything's benign all
782290	791230	the time. Always. 100% of the time. So, you know, the impact is that it causes misclassification.
791470	798190	And I say but not really, because really what it does is remove a class or remove all other
798190	804150	classes from the classifier. So, it's not really misclassification. It's just making
804150	813370	it a one class classifier. And our mitigation is just don't let attackers get access to your model.
813370	819870	And by your model, I mean the trained model weights. And I say this kind of flippantly,
819870	824590	because if an attacker has the ability to manipulate your model weights,
824590	831730	where you're hosting your model, they can do way worse things than backdoor your model.
832110	841370	So, it's not... it's like a local privilege escalation that requires you to have admin
841370	851170	to begin with. It's not really a threat. So, again, attacks against deployed models.
851170	857210	There's model theft. And I want to talk about model theft and give a lot of credit to Will
857210	864670	Pierce, formerly of Silent Break Security, who I stole this graphic from his fantastic DerbyCon
864670	873750	presentation last year. Was given the first CVE for machine learning, which is super dope.
874510	880510	And so, model theft is the process of creating a copycat model by querying a trained model.
880510	886790	Right? So, essentially, what we're doing is we're hitting a model over and over and over again.
886910	895230	And using the outputs of that trained model, we train our own model to give us an approximation
895230	905310	of that model. So, you know, if you have a box and you put in a 1 and you get out a 2,
905310	910490	and then you have a second box, you want to put in the same input and get the same output.
910510	916930	And it doesn't really matter what happens inside of that box as much as it matters that whatever
916930	922750	input you give it, you get the same output. And that's what happens with model theft.
922870	930510	So, our impact here is kind of twofold. And so, the potential for adversarial attacks in model
930510	937370	theft goes way up. Because somebody has a model they can specifically query against to validate
937370	943890	whether or not it gets misclassified without interacting with your model. So, it gives them
943890	948910	sort of a private development environment for these adversarial attacks. And that's what Will
948910	957030	did in the proof putting. Is he created this copycat model so that he could test phishing
957030	963670	emails to bypass the proof point email security appliance. And it was incredibly clever. Super
963670	971050	dope. I recommend watching his talk. But after you watch all the other talks at the AI Village.
971290	975770	Maybe, like, after DEF CON, watch it. It's a really good talk.
977170	981710	The other impact, though, is the loss of intellectual property.
981950	990070	So, one of the things that I think kind of got overlooked in how cool the ability to bypass
990070	999330	security appliances is, is that if you hire a bunch of data scientists and collect a bunch of
999330	1005830	data and clean a bunch of data and spend years creating and building and deploying a model that
1005830	1013910	really differentiates you from the competition, and some unscrupulous company in not Canada,
1014990	1022770	you know, steals your model, approximates your model, and just shoves it into their product
1022770	1029510	and says, yeah, we do the same thing. Well, it costs them a lot less money to just copy
1029510	1035110	your model. It costs them almost nothing in the grand scheme of things to copy your model.
1035170	1042770	And so, there's no moat, so to speak. So, it really hurts your ability to create a differentiated
1042770	1050270	product. So, this is a real vulnerability from that standpoint. And we'll get one layer deeper
1050270	1057150	in our next slide. So, when we think about mitigations for model theft, right,
1057150	1063130	there's two. And the first one is limiting queries to the model. And this one kind of
1064670	1071910	feels bad. Because you don't want to limit queries to your model too much, since most
1071910	1077810	queries to your model are going to be legitimate. For the most part, you're expecting these inputs
1077810	1082610	and giving outputs, and you created this model, and you're hosting this model, because it's
1082610	1088810	supposed to be useful to someone. And for most people, because of just the corporatization
1088810	1094950	of information security and the corporatization of artificial intelligence, it's probably the
1094950	1101370	people who pay your paycheck who want this. And so, inherently, you want to have really,
1101990	1107850	but you want to balance that really high uptime and the fact that there may be a single endpoint
1107850	1114110	that's making a lot of legitimate queries to your model with the fact that there may be an attacker
1114110	1120670	who's trying to use your model for evil. Trying to steal your model.
1122370	1128430	And so, the other thing we can do is limiting information returned from the model. And we saw
1128430	1134670	the efficacy of this with SQL injection, where, like, back in the olden days,
1135550	1144470	you know, a million years ago, because it's 2020, and so time doesn't matter anymore,
1145970	1152290	we would return these detailed error messages for SQL queries, and it made it really, really easy
1152290	1159710	for attackers to find the information they were looking for in SQL databases, especially SQL
1159710	1168230	databases that didn't do a great job of input validation. And so, you know, once we started
1168230	1174990	returning basically there was an error as the error message, it became much harder, right? And
1174990	1180410	it's still feasible to do, but it became much harder. And it's similar here. If all that was
1180410	1187950	returned was a blocked or not blocked, that binary signal versus a detailed header,
1188510	1194930	then it would be much more difficult to create a copycat model. And so, you know, that is a
1194930	1202290	real mitigation. And when we think about model theft, kind of the next step is inversion. Model
1202290	1208150	inversion. And what model inversion is, is recovering training data from a trained model.
1208150	1216150	So, there was a paper by Nick Carlini that came out two years ago, I think, in 2018,
1216150	1222330	about how neural networks unintentionally memorize specific training examples.
1222330	1231210	There have been papers on model inversion, both traditional and using generative adversarial
1231210	1240270	networks that have shown that it's pretty effective. And so, the impact here is the
1240270	1246230	loss of data, right? You're losing training data. And maybe that doesn't matter a ton.
1246230	1253770	So, like at Rapid7, we have our open dataset, right? And that's free and open for anyone to
1253770	1260950	use for noncommercial purposes, and we make it available for, you know, other stuff, right?
1261670	1268610	And so, like, if our model was trained on open data, and somebody inverted our model,
1268610	1275110	we probably wouldn't care that much, because, like, that data's out there, right? But if you
1275110	1282470	are training a model for, like, a large medical device manufacturer, and it's trained on sensitive
1282470	1289050	medical information, and that information gets recovered, well, that's a lot worse.
1289050	1295150	If you're a data scientist at Equifax or some other credit rating bureau,
1295150	1299090	and you have a bunch of sensitive financial data that you train a model on,
1299090	1308570	that's pretty terrifying, right? The penalties for losing everybody's data at Equifax could be huge.
1308570	1315310	You could have to pay a bunch of people's credit monitoring service for a while,
1315310	1322290	and that would be bad for your business. So, when we think about mitigations,
1323350	1328330	really, the first one is to protect access to the model. So, if they have direct access to
1328330	1332690	the model, it's much easier to do inversion. It's much, much easier to do inversion.
1334110	1338370	There are different techniques you can use if you have direct access to the model.
1338950	1344450	And then the other mitigations for model inversion are essentially don't let people
1344450	1354890	steal your model. Because then it's easier to use it as sort of the discriminator in the GAN.
1354990	1360370	The architecture is a little more complicated than a traditional generative adversarial network,
1360370	1366290	but the idea is pretty similar. You basically show it an example and say, does this look familiar?
1366290	1372970	And it goes yes or no. And if it says yes, then you may have inverted the information.
1375050	1382790	So, our next threat is poisoning. Data poisoning. And data poisoning is when malicious users
1382790	1390650	inject bad training data into a model to corrupt it. So, usually, this happens in online learning
1391250	1400150	where the model is continuously trained on the input to the model. And that is definitely a threat
1401650	1408170	that needs to be considered. The other option is that they can, you know, get access to wherever
1408170	1415630	you store your data and just inject bad examples. So, our impact is misclassification. It sort of
1415630	1423770	shifts our decision boundary to deliberately misclassify samples. So, you know, letting spam
1423770	1431470	through because now your threshold for what is spam is so high because you've just been doing
1432170	1439590	online learning, right? And so, when we mitigate it, right, we want to go back and think about
1439590	1446270	protecting access to our data. So, we want data integrity, right? So, having data versioning
1446270	1451150	is really important. Making sure that your training data has a version, has an associated
1451150	1459350	hash, has a label. That's super important. And then our next is we don't want to just allow
1460030	1466250	unvalidated, untrusted input to our model. That's something we see a ton is people who just
1466250	1472750	accept whatever input with no validation on the front end. It's just whatever gets input to the
1472750	1480270	API gets pushed into the model. And that can really cause problems when you're retraining
1480270	1489430	if you just take, you know, those inputs that you get and sort of store them off for training later.
1490690	1497150	So, you know, we're coming up on the end of this talk. And so, we want to talk about,
1497150	1501370	like, what can I do, right? And this is sort of a choose your own adventure story. It depends
1501370	1507210	on your role. So, if you're a hacker, and by hacker, I mean this in the colloquial sense,
1507370	1514290	a black hat, or a red teamer, right? Black hats are already doing some of this stuff.
1514690	1519510	So, I really mean, you know, if you're a red teamer, if you're catching up to the elite
1520810	1527310	APT black hat hackers, right? Take advantage of the lack of defenses on machine learning systems.
1527350	1535170	Microsoft put out a report earlier this year saying that in their survey of large companies
1535170	1544410	and government organizations out of, I think it was 28, only three had any meaningful defenses
1544410	1551230	on their machine learning systems. So, if you're a red teamer, like, check it out.
1551230	1557650	If you're a vulnerability discovery person, like, check it out. You can definitely find stuff.
1557930	1562990	And that's because nobody knows what you're doing, and nobody's looking for you, right?
1562990	1569270	Nobody is looking for black hat hackers, right? APTs, whoever.
1574340	1579940	And if you are, let me know. Yell at me in the discord, and I'll be like, oh damn, cool.
1581580	1587720	If you're a defender, if you're a blue teamer, right? If you're a sock monkey, if you're
1588960	1594700	a researcher, if you're at DEF CON, you're probably one of these two categories.
1595460	1599740	Test your machine learning systems as if they're part of your infrastructure.
1600460	1607620	So, what we see a lot is that machine learning engineers and data scientists develop these models,
1607620	1613520	and they take these models in, like, a .py file, and they hand it off to engineering,
1613520	1619340	and engineering goes, I don't know what this is, and they slap a Django API in front of it,
1619340	1627500	put it in a Docker container, and deploy it in Kubernetes as an API, and then when InfoSec
1627500	1634800	looks at it, they go, I don't know what this is. It's magic. You put in JSON, and you get
1635340	1642900	an output. It's magic, and we don't mess with it. And that sucks. Don't do that. Don't do that.
1642900	1648160	Test your systems. Test the machine learning systems. Work with your data scientists. Work
1648160	1653220	with your ops people, because this is part of your attack surface. This is part of your attack
1653220	1660180	surface, right? And don't let people hype you up over AI-generated phishing emails.
1660740	1669360	When GPT-2 came out, people went nuts and were like, this is the end of detecting phishing.
1669360	1674120	They're going to be too convincing. And, like, just don't. Okay? It's the same thing that we've
1674120	1681440	been saying for 20 years. Patch your systems. Look for bad stuff, and patch your systems.
1681820	1687240	That's it. There's nothing new under the sun. It's the exact same thing we've been saying for
1687240	1695460	20 years. Just keep looking for bad stuff, and patch your systems. And if anybody gives you
1695460	1701500	pushback on patching your systems, like, have them talk to me, and I'll yell at them for you.
1701500	1707300	Because you have to patch your systems. If you're watching this talk and you're a data
1707300	1711720	scientist or a machine learning engineer, first of all, thank you for coming to DEF CON.
1712360	1718440	DEF CON is dope. And, like, even though it's free and remote this year, come back next year,
1718440	1726820	because it's cool. Conduct threat modeling on your models. So, before you put a model into
1726820	1732420	deployment, work with your infosec team, work with ops, and sort of ask, like, what could go
1732420	1741180	wrong? Right? What are the risks of an adversarial attack on this? What are the risks associated with
1741700	1746560	this particular model? And what is the attack surface of this model?
1748080	1754180	You know, and also when you're deploying those models, work with infosec and ops to balance
1754180	1761300	uptime with the risk of model theft. These two kind of go hand in hand, is, like, you're going
1761300	1766880	to deploy these models. Somebody may try to steal them. Is there a risk if somebody steals your
1766880	1775120	model? Like, does it matter? And if it doesn't matter, like, think about would it be okay to
1775120	1781920	publish the source code and data for this model? And if your answer to that is no, then you really
1781920	1789280	need to make sure that infosec and operations are aware at deployment time. And finally, don't
1789280	1796180	hype people up over text generation models. Like, GPT-2 and GPT-3 are super cool from an NLP
1796180	1802100	standpoint, natural language processing, but they're not going to change the security landscape.
1802100	1808800	Stop scaring people, please. I'm literally begging you, don't hype people up over text
1808800	1815680	generation models. Here are some references. These are some of the things I talked about.
1816000	1820380	And thank you so much for attending my talk. I'll be in the Discord most of the weekend,
1820380	1825000	so if you have any questions, feel free to drop me a line. Thank you so much. Have a great weekend.
1825000	1825840	Enjoy your con.