WEBVTT 00:04.100 --> 00:06.640 Here's our second talk of the day. 00:07.620 --> 00:11.220 It's Eric Gallican. 00:12.320 --> 00:24.020 He's a... well, you see his Who Am I page, but he's also a PhD student at Brandeis and an all-around awesome guy. 00:24.180 --> 00:29.300 He does way too much stuff and is very enthusiastic and involved in the community. 00:30.120 --> 00:34.160 So I'm just going to let him talk about... on his intro talk about... 00:34.860 --> 00:36.720 for machine learning people. 00:40.930 --> 00:42.210 Hi, everybody. 00:42.210 --> 00:43.270 Welcome to my talk. 00:43.270 --> 00:46.830 This is Baby's First 100 MLSat Words. 00:47.210 --> 00:48.730 My name is Eric Gallican. 00:48.730 --> 00:51.670 I'm an artificial intelligence researcher at Rapid7. 00:51.670 --> 00:59.710 I specialize in breaking machine learning stuff and applying machine learning to detecting bad things. 00:59.710 --> 01:06.990 I also do research at the Montreal AI Ethics Institute on applying DevOps principle to machine learning ethics. 01:07.290 --> 01:18.730 And last year, I was here at DEF CON in person, not here in my house at DEF CON, presenting at the Cloud Village on a piece of malware that I wrote. 01:18.730 --> 01:25.210 Malware, not actually malware, just in case the legal team is listening. 01:25.830 --> 01:31.290 And I presented that piece of proof of concept malware called Sassy Boy. 01:31.870 --> 01:34.150 But mostly I do boring math stuff. 01:34.450 --> 01:43.810 And if you can't see my speaker video for some reason, I put a picture of my face on the right so that you can see it. 01:43.810 --> 01:45.210 Or you can see it twice. 01:45.210 --> 01:50.550 You can see that my hair hasn't changed basically at all in years. 01:50.890 --> 01:55.670 So before we get started, let's just answer the question, what is MLSat? 01:55.690 --> 02:03.090 Because I get asked to that a lot by practitioners in security and in machine learning, honestly. 02:03.250 --> 02:04.850 And mostly it's an argument. 02:05.150 --> 02:10.890 And it's an argument between three groups of people, as it usually is. 02:10.890 --> 02:19.670 And the first group of people is people who think it's only applications of machine learning and artificial intelligence to security problems. 02:19.670 --> 02:22.230 So that's automated malware detection. 02:22.230 --> 02:25.830 That's next generation SIMs. 02:25.830 --> 02:29.770 That's automated red teaming and that sort of stuff. 02:30.490 --> 02:37.030 And then there are people who think that it's just securing artificial intelligence in machine learning systems. 02:37.690 --> 02:42.850 And then there are those of us who are correct who think it's both. 02:42.850 --> 02:48.770 It's kind of any intersection of artificial intelligence, right? 02:48.770 --> 02:54.350 That broad category of machine learning slash AI and security. 02:54.470 --> 03:04.790 And since this is my talk, and since I am the Mao Zedong of MLSat, I get to decide that it's both. 03:04.790 --> 03:13.530 And if anybody argues with you or you hear anybody having this totally asinine argument, you can just tell them that I said it's both. 03:13.610 --> 03:18.310 And then the argument can be over and everybody can do more productive things with their lives. 03:18.650 --> 03:22.010 So we'll start with deep fakes, right? 03:22.010 --> 03:24.190 So D is for deep fakes. 03:24.190 --> 03:32.890 And this man on the right, if I didn't have a speaker video, I probably could have just put that picture up and said it was me. 03:32.890 --> 03:39.790 And everybody would have believed it because he's got kind of he looks like the kind of guy, right? 03:40.790 --> 03:43.960 And I'm allowed to say that because he's not real. 03:44.190 --> 03:46.310 This is not a real person. 03:46.310 --> 03:49.630 This is completely an AI-generated image. 03:50.490 --> 04:02.110 And so, you know, to give deep fakes a real definition, they are a convincing synthetic image, video, or audio recording which purports to be real. 04:02.110 --> 04:06.450 So, you know, I'm pretty convinced by this picture. 04:06.710 --> 04:18.370 If I saw it in the wild, right, if somebody put it as like their Twitter avatar and was like, yeah, I am a software engineer at Google, I could buy that. 04:18.370 --> 04:20.630 I could buy that based on this image, right? 04:20.950 --> 04:21.470 You know? 04:21.470 --> 04:23.070 We have our stereotypes, right? 04:23.070 --> 04:26.030 The thin-rimmed glasses, the scruff. 04:26.510 --> 04:29.450 And it's super convincing. 04:29.450 --> 04:30.710 It's super convincing. 04:30.710 --> 04:36.050 And I think that's really, when we think about impact, that's what it is. 04:36.050 --> 04:38.330 It's that they're convincing to users. 04:38.330 --> 04:45.270 So, we also see deep fakes in the text generation space where it purports to be a real email. 04:45.270 --> 04:53.990 Or we see deep fake writing which mimics the style of famous authors. 04:53.990 --> 05:01.670 We see deep fake audio recordings which sound like the person who they are trained to sound like. 05:01.670 --> 05:06.710 Now, there's a piece of commercial software out there called Liarbird which does this really well. 05:06.710 --> 05:10.230 And so, you know, they're very convincing to users. 05:10.630 --> 05:17.630 And as a result, you know, they get used widely and they're easy to make, right? 05:17.630 --> 05:29.810 That's the other thing, is they don't take as much time as finding and crafting an image or carefully learning a style or mimicking a voice, right? 05:29.810 --> 05:32.670 You just kind of train the model and push them out. 05:32.670 --> 05:35.050 So, they're easy to make and they're convincing to users. 05:35.050 --> 05:39.690 Which means that they can have real impact. 05:39.690 --> 05:43.510 But that impact is mostly societal, right? 05:43.510 --> 05:55.370 So, there was a famous, you know, semi-viral video of Jordan Peele doing a deep fake where it was an image of President Obama speaking. 05:55.890 --> 05:57.270 And you can look it up. 05:57.270 --> 05:59.150 It's really easy to find. 06:00.850 --> 06:13.990 And so, like, the risk there is really that broad swaths of people will believe that a doctored or deep faked video is real. 06:13.990 --> 06:19.810 And so, there's something like President Donald J. 06:19.810 --> 06:30.530 Trump, you know, declaring war on North Korea on a hot mic or suggesting that he might invade North Korea on a hot mic. 06:30.530 --> 06:32.450 That could be really convincing to people. 06:32.450 --> 06:41.970 And so, that's the real risk of deep fakes, is that they are a powerful disinformation tool which is powered by artificial intelligence. 06:42.330 --> 06:48.030 And so, you know, our mitigation against deep fakes is kind of, eh. 06:48.510 --> 06:52.670 Like, there's very little we can do about deep fakes. 06:52.670 --> 06:54.270 They're difficult to detect. 06:54.270 --> 06:56.090 They're very convincing to users. 06:56.090 --> 06:57.470 They're easy to make. 06:57.470 --> 07:17.390 And so, really, it's a broader social problem that we need to tackle of having people be inherently less trustful of information sources and really vet where they get their information from, which is a hard problem. 07:17.390 --> 07:21.390 But this isn't an enterprise security threat. 07:21.390 --> 07:25.030 This is a social threat. 07:25.030 --> 07:32.150 So, moving back to thinking about deployed models, we'll look at adversarial examples. 07:32.150 --> 07:40.170 And this gif over here on the right is from a pretty well-known example that was really, really hyped up. 07:40.170 --> 07:42.430 And it was the turtle rifle paper. 07:42.430 --> 07:48.270 Because the Google computer vision API is what they attacked. 07:48.270 --> 07:51.110 And they 3D printed a turtle. 07:51.110 --> 07:57.550 And this is pretty obviously a turtle to anybody watching who isn't powered by the Google Vision API. 07:57.550 --> 08:05.290 And if you are an artificial intelligence powered by the Google Vision API, I'm so sorry for the rest of this talk. 08:05.290 --> 08:07.150 Because you're not gonna like it. 08:08.670 --> 08:15.990 That, you know, it's an input to the classifier that's specifically crafted to force misclassification. 08:16.330 --> 08:24.130 And so, this object as a turtle, just a little, like, toy turtle, right? 08:24.430 --> 08:42.310 Being misclassified as a rifle, the risk there is if you have a CCTV system that's powered by Google Vision API, which seems reasonable to license, that's looking for potential security threats. 08:42.350 --> 08:52.030 And, you know, you don't input all of that, like, racist physiognomy that we already see in a lot of those systems. 08:52.030 --> 08:53.450 But instead just say, you know what? 08:53.450 --> 08:55.610 Let's just look for weapons. 08:56.170 --> 09:00.370 Well, this could falsely trigger those systems. 09:00.490 --> 09:02.570 And so, there's a risk there, too. 09:03.610 --> 09:08.090 Correspondingly, if you 3D printed a gun, right? 09:08.730 --> 09:15.910 That got misclassified as a turtle, that would also be a threat, right? 09:15.910 --> 09:19.550 So, the impact, really, is that it causes misclassification. 09:20.210 --> 09:26.730 And so, that real world security threat and 3D printed examples are really rare, and they're really difficult to do. 09:27.730 --> 09:36.830 And they don't tend to work in all situations, and any researcher in computer vision can explain to you why it's not a real part of the threat model. 09:36.830 --> 09:37.950 And that's fine. 09:38.230 --> 09:40.710 But our threat is that it causes misclassification. 09:40.710 --> 09:50.030 As security practitioners, we try to look for ways to use technologies to detect bad things. 09:50.030 --> 10:01.710 And so, if you integrate artificial intelligence into your malware detector, for example, an attacker can bypass detection using adversarial examples. 10:02.430 --> 10:18.390 And the thing with adversarial examples is when we think about how to mitigate them, one of the only truly effective ways to do it is through what's called adversarial training, where you basically show it a bunch of adversarial examples and tell it, like, 10:18.390 --> 10:20.590 no, classify these correctly. 10:21.430 --> 10:25.170 And that's really time consuming, and it's expensive, and it's hard. 10:25.550 --> 10:27.450 And so, not a lot of people do it. 10:27.450 --> 10:34.210 And it requires you to take your model out of production and put it back into training, which, again, has a business impact. 10:35.330 --> 10:44.970 So, one of the other ways to sort of mitigate it is through what's called ensembling, where you don't just take one canonical model. 10:45.270 --> 10:52.530 You take several models and sort of average their output or take a weighted average of their output or whatever. 10:53.870 --> 10:56.190 And so, that's another way to do it. 10:56.270 --> 11:10.350 And that's kind of because it's a lot harder to come up with an adversarial example that works across a variety of classifiers, especially if they're not trained on the same data, if they don't have the same architecture. 11:10.570 --> 11:15.250 And it's still possible to do, but it sort of raises that barrier to entry. 11:15.310 --> 11:20.890 And if we think about it in terms of, like, cryptography, that's really what we're trying to do, right? 11:20.890 --> 11:26.790 Is we know that there is hypothetically always an attack, right? 11:26.790 --> 11:28.770 You can brute force the key space. 11:28.770 --> 11:35.130 That's a hypothetically feasible attack against AES, right? 11:36.050 --> 11:43.950 But it's good enough, and it takes long enough to brute force that key space that, you know, it's fine. 11:43.950 --> 11:45.010 It does the job, right? 11:45.010 --> 11:52.950 And that's sort of the same thing here, is you just have to raise that barrier to entry to make it not worth it for an attacker. 11:54.850 --> 12:01.170 So, the next thing we want to talk about in terms of threats to deployed models are backdoors. 12:01.370 --> 12:11.530 And so, backdoors in a machine learning context are manipulations of trained model weights that result in a specific outcome each time. 12:11.530 --> 12:23.790 So, a common way to do this in neural networks is bias poisoning, where you take one class and you make the bias for that one class in just the output layer really, really, really big. 12:23.790 --> 12:26.990 So, you're only changing one weight in the whole network. 12:26.990 --> 12:29.210 So, it doesn't change a lot, right? 12:29.890 --> 12:39.810 But by virtue of making that weight really big, you'll always get the same classification. 12:39.810 --> 12:51.890 And if it's a binary classifier, right, say it's, again, our malware detector, because people love to make neural network powered malware detectors for whatever reason. 12:51.890 --> 12:53.130 And I've made one. 12:53.130 --> 12:55.010 So, I'm allowed to criticize. 12:57.230 --> 12:58.790 People love to do it. 12:58.790 --> 13:03.230 And that's a really easy way to just be like, nope, everything's benign all the time. 13:03.310 --> 13:04.070 Always. 13:04.070 --> 13:05.510 100% of the time. 13:06.290 --> 13:11.230 So, you know, the impact is that it causes misclassification. 13:11.470 --> 13:21.730 And I say but not really, because really what it does is remove a class or remove all other classes from the classifier. 13:21.730 --> 13:23.610 So, it's not really misclassification. 13:23.610 --> 13:26.390 It's just making it a one class classifier. 13:27.750 --> 13:33.370 And our mitigation is just don't let attackers get access to your model. 13:33.370 --> 13:36.550 And by your model, I mean the trained model weights. 13:37.730 --> 13:51.730 And I say this kind of flippantly, because if an attacker has the ability to manipulate your model weights, where you're hosting your model, they can do way worse things than backdoor your model. 13:52.110 --> 14:02.890 So, it's not... it's like a local privilege escalation that requires you to have admin to begin with. 14:02.890 --> 14:05.710 It's not really a threat. 14:07.290 --> 14:11.170 So, again, attacks against deployed models. 14:11.170 --> 14:12.750 There's model theft. 14:12.750 --> 14:26.790 And I want to talk about model theft and give a lot of credit to Will Pierce, formerly of Silent Break Security, who I stole this graphic from his fantastic DerbyCon presentation last year. 14:27.810 --> 14:33.750 Was given the first CVE for machine learning, which is super dope. 14:34.510 --> 14:40.510 And so, model theft is the process of creating a copycat model by querying a trained model. 14:40.510 --> 14:41.710 Right? 14:41.710 --> 14:46.790 So, essentially, what we're doing is we're hitting a model over and over and over again. 14:46.910 --> 14:56.850 And using the outputs of that trained model, we train our own model to give us an approximation of that model. 14:56.850 --> 15:10.490 So, you know, if you have a box and you put in a 1 and you get out a 2, and then you have a second box, you want to put in the same input and get the same output. 15:10.510 --> 15:20.110 And it doesn't really matter what happens inside of that box as much as it matters that whatever input you give it, you get the same output. 15:20.290 --> 15:22.750 And that's what happens with model theft. 15:22.870 --> 15:26.790 So, our impact here is kind of twofold. 15:26.830 --> 15:32.050 And so, the potential for adversarial attacks in model theft goes way up. 15:32.050 --> 15:43.090 Because somebody has a model they can specifically query against to validate whether or not it gets misclassified without interacting with your model. 15:43.390 --> 15:48.210 So, it gives them sort of a private development environment for these adversarial attacks. 15:48.210 --> 15:51.130 And that's what Will did in the proof putting. 15:51.330 --> 16:01.090 Is he created this copycat model so that he could test phishing emails to bypass the proof point email security appliance. 16:01.090 --> 16:03.410 And it was incredibly clever. 16:03.410 --> 16:04.310 Super dope. 16:04.310 --> 16:06.910 I recommend watching his talk. 16:07.370 --> 16:11.050 But after you watch all the other talks at the AI Village. 16:11.290 --> 16:14.170 Maybe, like, after DEF CON, watch it. 16:14.170 --> 16:15.770 It's a really good talk. 16:17.170 --> 16:21.710 The other impact, though, is the loss of intellectual property. 16:21.950 --> 16:49.230 So, one of the things that I think kind of got overlooked in how cool the ability to bypass security appliances is, is that if you hire a bunch of data scientists and collect a bunch of data and clean a bunch of data and spend years creating and building and deploying a model that really differentiates you from the competition, 16:49.230 --> 17:06.330 and some unscrupulous company in not Canada, you know, steals your model, approximates your model, and just shoves it into their product and says, yeah, we do the same thing. 17:06.830 --> 17:10.590 Well, it costs them a lot less money to just copy your model. 17:10.670 --> 17:15.110 It costs them almost nothing in the grand scheme of things to copy your model. 17:15.170 --> 17:19.330 And so, there's no moat, so to speak. 17:19.410 --> 17:23.730 So, it really hurts your ability to create a differentiated product. 17:24.530 --> 17:27.930 So, this is a real vulnerability from that standpoint. 17:28.410 --> 17:33.050 And we'll get one layer deeper in our next slide. 17:33.350 --> 17:39.030 So, when we think about mitigations for model theft, right, there's two. 17:39.430 --> 17:42.070 And the first one is limiting queries to the model. 17:42.070 --> 17:46.350 And this one kind of feels bad. 17:46.770 --> 17:54.310 Because you don't want to limit queries to your model too much, since most queries to your model are going to be legitimate. 17:54.670 --> 18:04.850 For the most part, you're expecting these inputs and giving outputs, and you created this model, and you're hosting this model, because it's supposed to be useful to someone. 18:05.010 --> 18:18.290 And for most people, because of just the corporatization of information security and the corporatization of artificial intelligence, it's probably the people who pay your paycheck who want this. 18:18.310 --> 18:38.650 And so, inherently, you want to have really, but you want to balance that really high uptime and the fact that there may be a single endpoint that's making a lot of legitimate queries to your model with the fact that there may be an attacker who's trying to use your model for evil. 18:38.650 --> 18:40.670 Trying to steal your model. 18:42.370 --> 18:47.730 And so, the other thing we can do is limiting information returned from the model. 18:47.730 --> 19:11.910 And we saw the efficacy of this with SQL injection, where, like, back in the olden days, you know, a million years ago, because it's 2020, and so time doesn't matter anymore, we would return these detailed error messages for SQL queries, and it made it really, 19:11.910 --> 19:24.590 really easy for attackers to find the information they were looking for in SQL databases, especially SQL databases that didn't do a great job of input validation. 19:25.490 --> 19:34.910 And so, you know, once we started returning basically there was an error as the error message, it became much harder, right? 19:34.910 --> 19:38.030 And it's still feasible to do, but it became much harder. 19:38.210 --> 19:39.470 And it's similar here. 19:39.470 --> 19:52.890 If all that was returned was a blocked or not blocked, that binary signal versus a detailed header, then it would be much more difficult to create a copycat model. 19:52.890 --> 19:56.230 And so, you know, that is a real mitigation. 19:56.470 --> 20:01.670 And when we think about model theft, kind of the next step is inversion. 20:01.670 --> 20:02.970 Model inversion . 20:03.270 --> 20:08.150 And what model inversion is, is recovering training data from a trained model. 20:08.150 --> 20:22.330 So, there was a paper by Nick Carlini that came out two years ago, I think, in 2018, about how neural networks unintentionally memorize specific training examples. 20:22.330 --> 20:35.650 There have been papers on model inversion, both traditional and using generative adversarial networks that have shown that it's pretty effective. 20:37.150 --> 20:41.710 And so, the impact here is the loss of data, right? 20:41.710 --> 20:43.830 You're losing training data. 20:43.830 --> 20:46.230 And maybe that doesn't matter a ton. 20:46.230 --> 20:50.470 So, like at Rapid7, we have our open dataset, right? 20:50.470 --> 21:00.950 And that's free and open for anyone to use for noncommercial purposes, and we make it available for, you know, other stuff, right? 21:01.670 --> 21:13.990 And so, like, if our model was trained on open data, and somebody inverted our model, we probably wouldn't care that much, because, like, that data's out there, right? 21:14.170 --> 21:29.050 But if you are training a model for, like, a large medical device manufacturer, and it's trained on sensitive medical information, and that information gets recovered, well, that's a lot worse. 21:29.050 --> 21:42.430 If you're a data scientist at Equifax or some other credit rating bureau, and you have a bunch of sensitive financial data that you train a model on, that's pretty terrifying, right? 21:42.430 --> 21:48.570 The penalties for losing everybody's data at Equifax could be huge. 21:48.570 --> 21:58.050 You could have to pay a bunch of people's credit monitoring service for a while, and that would be bad for your business. 21:59.690 --> 22:06.490 So, when we think about mitigations, really, the first one is to protect access to the model. 22:06.490 --> 22:10.770 So, if they have direct access to the model, it's much easier to do inversion. 22:10.770 --> 22:12.690 It's much, much easier to do inversion. 22:14.110 --> 22:18.370 There are different techniques you can use if you have direct access to the model. 22:18.950 --> 22:26.170 And then the other mitigations for model inversion are essentially don't let people steal your model. 22:26.630 --> 22:34.890 Because then it's easier to use it as sort of the discriminator in the GAN. 22:34.990 --> 22:42.330 The architecture is a little more complicated than a traditional generative adversarial network, but the idea is pretty similar. 22:42.710 --> 22:46.290 You basically show it an example and say, does this look familiar? 22:46.290 --> 22:48.430 And it goes yes or no. 22:48.570 --> 22:52.970 And if it says yes, then you may have inverted the information. 22:55.050 --> 22:58.310 So, our next threat is poisoning. 22:58.310 --> 22:59.710 Data poisoning. 22:59.930 --> 23:07.070 And data poisoning is when malicious users inject bad training data into a model to corrupt it. 23:07.070 --> 23:16.350 So, usually, this happens in online learning where the model is continuously trained on the input to the model. 23:16.390 --> 23:24.190 And that is definitely a threat that needs to be considered. 23:24.190 --> 23:31.530 The other option is that they can, you know, get access to wherever you store your data and just inject bad examples. 23:31.650 --> 23:34.450 So, our impact is misclassification. 23:35.070 --> 23:41.430 It sort of shifts our decision boundary to deliberately misclassify samples. 23:41.610 --> 23:54.530 So, you know, letting spam through because now your threshold for what is spam is so high because you've just been doing online learning, right? 23:55.370 --> 24:01.670 And so, when we mitigate it, right, we want to go back and think about protecting access to our data. 24:01.670 --> 24:04.330 So, we want data integrity, right? 24:04.330 --> 24:07.570 So, having data versioning is really important. 24:07.570 --> 24:13.250 Making sure that your training data has a version, has an associated hash, has a label. 24:13.350 --> 24:15.290 That's super important. 24:15.870 --> 24:22.830 And then our next is we don't want to just allow unvalidated, untrusted input to our model. 24:22.830 --> 24:30.690 That's something we see a ton is people who just accept whatever input with no validation on the front end. 24:30.690 --> 24:35.370 It's just whatever gets input to the API gets pushed into the model. 24:36.130 --> 24:49.430 And that can really cause problems when you're retraining if you just take, you know, those inputs that you get and sort of store them off for training later. 24:50.690 --> 24:54.650 So, you know, we're coming up on the end of this talk. 24:54.650 --> 24:58.710 And so, we want to talk about, like, what can I do, right? 24:58.710 --> 25:01.050 And this is sort of a choose your own adventure story. 25:01.050 --> 25:02.410 It depends on your role. 25:02.830 --> 25:10.850 So, if you're a hacker, and by hacker, I mean this in the colloquial sense, a black hat, or a red teamer, right? 25:10.850 --> 25:14.290 Black hats are already doing some of this stuff. 25:14.690 --> 25:23.410 So, I really mean, you know, if you're a red teamer, if you're catching up to the elite APT black hat hackers, right? 25:23.410 --> 25:27.310 Take advantage of the lack of defenses on machine learning systems. 25:27.350 --> 25:46.550 Microsoft put out a report earlier this year saying that in their survey of large companies and government organizations out of, I think it was 28, only three had any meaningful defenses on their machine learning systems. 25:47.790 --> 25:51.230 So, if you're a red teamer, like, check it out. 25:51.230 --> 25:55.570 If you're a vulnerability discovery person, like, check it out. 25:55.570 --> 25:57.650 You can definitely find stuff. 25:57.930 --> 26:02.990 And that's because nobody knows what you're doing, and nobody's looking for you, right? 26:02.990 --> 26:07.730 Nobody is looking for black hat hackers, right? 26:07.770 --> 26:09.270 APTs, whoever. 26:14.340 --> 26:16.140 And if you are, let me know. 26:16.140 --> 26:19.940 Yell at me in the discord, and I'll be like, oh damn, cool. 26:21.580 --> 26:24.700 If you're a defender, if you're a blue teamer, right? 26:24.700 --> 26:34.700 If you're a sock monkey, if you're a researcher, if you're at DEF CON, you're probably one of these two categories. 26:35.460 --> 26:39.740 Test your machine learning systems as if they're part of your infrastructure. 26:40.460 --> 26:59.340 So, what we see a lot is that machine learning engineers and data scientists develop these models, and they take these models in, like, a .py file, and they hand it off to engineering, and engineering goes, I don't know what this is, and they slap a Django API in front of it, 26:59.340 --> 27:11.720 put it in a Docker container, and deploy it in Kubernetes as an API, and then when InfoSec looks at it, they go, I don't know what this is. 27:11.720 --> 27:12.660 It's magic. 27:12.660 --> 27:16.400 You put in JSON, and you get an output. 27:16.400 --> 27:19.040 It's magic, and we don't mess with it. 27:19.280 --> 27:20.660 And that sucks. 27:20.660 --> 27:21.760 Don't do that. 27:21.760 --> 27:22.900 Don't do that. 27:22.900 --> 27:24.040 Test your systems. 27:24.040 --> 27:25.860 Test the machine learning systems. 27:26.380 --> 27:27.980 Work with your data scientists. 27:27.980 --> 27:32.540 Work with your ops people, because this is part of your attack surface. 27:32.540 --> 27:34.420 This is part of your attack surface, right? 27:34.520 --> 27:40.180 And don't let people hype you up over AI-generated phishing emails. 27:40.740 --> 27:49.360 When GPT-2 came out, people went nuts and were like, this is the end of detecting phishing. 27:49.360 --> 27:51.000 They're going to be too convincing. 27:51.300 --> 27:52.640 And, like, just don't. 27:52.640 --> 27:53.000 Okay? 27:53.000 --> 27:56.400 It's the same thing that we've been saying for 20 years. 27:57.120 --> 27:58.760 Patch your systems. 27:58.780 --> 28:01.440 Look for bad stuff, and patch your systems. 28:01.820 --> 28:03.020 That's it. 28:03.320 --> 28:05.300 There's nothing new under the sun. 28:05.300 --> 28:08.460 It's the exact same thing we've been saying for 20 years. 28:08.780 --> 28:13.160 Just keep looking for bad stuff, and patch your systems. 28:13.340 --> 28:21.500 And if anybody gives you pushback on patching your systems, like, have them talk to me, and I'll yell at them for you. 28:21.500 --> 28:23.860 Because you have to patch your systems. 28:25.240 --> 28:31.720 If you're watching this talk and you're a data scientist or a machine learning engineer, first of all, thank you for coming to DEF CON. 28:32.360 --> 28:34.140 DEF CON is dope. 28:34.460 --> 28:39.840 And, like, even though it's free and remote this year, come back next year, because it's cool. 28:42.280 --> 28:44.860 Conduct threat modeling on your models. 28:45.040 --> 28:53.660 So, before you put a model into deployment, work with your infosec team, work with ops, and sort of ask, like, what could go wrong? 28:53.840 --> 28:54.340 Right? 28:54.340 --> 28:58.660 What are the risks of an adversarial attack on this? 28:58.660 --> 29:03.440 What are the risks associated with this particular model? 29:03.440 --> 29:06.560 And what is the attack surface of this model? 29:08.080 --> 29:17.140 You know, and also when you're deploying those models, work with infosec and ops to balance uptime with the risk of model theft. 29:17.700 --> 29:22.880 These two kind of go hand in hand, is, like, you're going to deploy these models. 29:23.380 --> 29:25.040 Somebody may try to steal them. 29:25.040 --> 29:27.520 Is there a risk if somebody steals your model? 29:27.520 --> 29:28.820 Like, does it matter? 29:29.640 --> 29:38.640 And if it doesn't matter, like, think about would it be okay to publish the source code and data for this model? 29:38.640 --> 29:47.580 And if your answer to that is no, then you really need to make sure that infosec and operations are aware at deployment time. 29:47.940 --> 29:51.780 And finally, don't hype people up over text generation models. 29:51.780 --> 30:02.100 Like, GPT-2 and GPT-3 are super cool from an NLP standpoint, natural language processing, but they're not going to change the security landscape. 30:02.100 --> 30:04.380 Stop scaring people, please. 30:04.420 --> 30:10.120 I'm literally begging you, don't hype people up over text generation models. 30:10.860 --> 30:12.220 Here are some references. 30:12.860 --> 30:15.680 These are some of the things I talked about. 30:16.000 --> 30:18.040 And thank you so much for attending my talk. 30:18.080 --> 30:23.240 I'll be in the Discord most of the weekend, so if you have any questions, feel free to drop me a line. 30:23.300 --> 30:24.160 Thank you so much. 30:24.160 --> 30:25.000 Have a great weekend. 30:25.000 --> 30:25.840 Enjoy your con.