1
00:00:04,100 --> 00:00:06,640
Here's our second talk of the day.

2
00:00:07,620 --> 00:00:11,220
It's Eric Gallican.

3
00:00:12,320 --> 00:00:24,020
He's a... well, you see his Who Am I page, but he's also a PhD student at Brandeis and an all-around awesome guy.

4
00:00:24,180 --> 00:00:29,300
He does way too much stuff and is very enthusiastic and involved in the community.

5
00:00:30,120 --> 00:00:34,160
So I'm just going to let him talk about... on his intro talk about...

6
00:00:34,860 --> 00:00:36,720
for machine learning people.

7
00:00:40,930 --> 00:00:42,210
Hi, everybody.

8
00:00:42,210 --> 00:00:43,270
Welcome to my talk.

9
00:00:43,270 --> 00:00:46,830
This is Baby's First 100 MLSat Words.

10
00:00:47,210 --> 00:00:48,730
My name is Eric Gallican.

11
00:00:48,730 --> 00:00:51,670
I'm an artificial intelligence researcher at Rapid7.

12
00:00:51,670 --> 00:00:59,710
I specialize in breaking machine learning stuff and applying machine learning to detecting bad things.

13
00:00:59,710 --> 00:01:06,990
I also do research at the Montreal AI Ethics Institute on applying DevOps principle to machine learning ethics.

14
00:01:07,290 --> 00:01:18,730
And last year, I was here at DEF CON in person, not here in my house at DEF CON, presenting at the Cloud Village on a piece of malware that I wrote.

15
00:01:18,730 --> 00:01:25,210
Malware, not actually malware, just in case the legal team is listening.

16
00:01:25,830 --> 00:01:31,290
And I presented that piece of proof of concept malware called Sassy Boy.

17
00:01:31,870 --> 00:01:34,150
But mostly I do boring math stuff.

18
00:01:34,450 --> 00:01:43,810
And if you can't see my speaker video for some reason, I put a picture of my face on the right so that you can see it.

19
00:01:43,810 --> 00:01:45,210
Or you can see it twice.

20
00:01:45,210 --> 00:01:50,550
You can see that my hair hasn't changed basically at all in years.

21
00:01:50,890 --> 00:01:55,670
So before we get started, let's just answer the question, what is MLSat?

22
00:01:55,690 --> 00:02:03,090
Because I get asked to that a lot by practitioners in security and in machine learning, honestly.

23
00:02:03,250 --> 00:02:04,850
And mostly it's an argument.

24
00:02:05,150 --> 00:02:10,890
And it's an argument between three groups of people, as it usually is.

25
00:02:10,890 --> 00:02:19,670
And the first group of people is people who think it's only applications of machine learning and artificial intelligence to security problems.

26
00:02:19,670 --> 00:02:22,230
So that's automated malware detection.

27
00:02:22,230 --> 00:02:25,830
That's next generation SIMs.

28
00:02:25,830 --> 00:02:29,770
That's automated red teaming and that sort of stuff.

29
00:02:30,490 --> 00:02:37,030
And then there are people who think that it's just securing artificial intelligence in machine learning systems.

30
00:02:37,690 --> 00:02:42,850
And then there are those of us who are correct who think it's both.

31
00:02:42,850 --> 00:02:48,770
It's kind of any intersection of artificial intelligence, right?

32
00:02:48,770 --> 00:02:54,350
That broad category of machine learning slash AI and security.

33
00:02:54,470 --> 00:03:04,790
And since this is my talk, and since I am the Mao Zedong of MLSat, I get to decide that it's both.

34
00:03:04,790 --> 00:03:13,530
And if anybody argues with you or you hear anybody having this totally asinine argument, you can just tell them that I said it's both.

35
00:03:13,610 --> 00:03:18,310
And then the argument can be over and everybody can do more productive things with their lives.

36
00:03:18,650 --> 00:03:22,010
So we'll start with deep fakes, right?

37
00:03:22,010 --> 00:03:24,190
So D is for deep fakes.

38
00:03:24,190 --> 00:03:32,890
And this man on the right, if I didn't have a speaker video, I probably could have just put that picture up and said it was me.

39
00:03:32,890 --> 00:03:39,790
And everybody would have believed it because he's got kind of he looks like the kind of guy, right?

40
00:03:40,790 --> 00:03:43,960
And I'm allowed to say that because he's not real.

41
00:03:44,190 --> 00:03:46,310
This is not a real person.

42
00:03:46,310 --> 00:03:49,630
This is completely an AI-generated image.

43
00:03:50,490 --> 00:04:02,110
And so, you know, to give deep fakes a real definition, they are a convincing synthetic image, video, or audio recording which purports to be real.

44
00:04:02,110 --> 00:04:06,450
So, you know, I'm pretty convinced by this picture.

45
00:04:06,710 --> 00:04:18,370
If I saw it in the wild, right, if somebody put it as like their Twitter avatar and was like, yeah, I am a software engineer at Google, I could buy that.

46
00:04:18,370 --> 00:04:20,630
I could buy that based on this image, right?

47
00:04:20,950 --> 00:04:21,470
You know?

48
00:04:21,470 --> 00:04:23,070
We have our stereotypes, right?

49
00:04:23,070 --> 00:04:26,030
The thin-rimmed glasses, the scruff.

50
00:04:26,510 --> 00:04:29,450
And it's super convincing.

51
00:04:29,450 --> 00:04:30,710
It's super convincing.

52
00:04:30,710 --> 00:04:36,050
And I think that's really, when we think about impact, that's what it is.

53
00:04:36,050 --> 00:04:38,330
It's that they're convincing to users.

54
00:04:38,330 --> 00:04:45,270
So, we also see deep fakes in the text generation space where it purports to be a real email.

55
00:04:45,270 --> 00:04:53,990
Or we see deep fake writing which mimics the style of famous authors.

56
00:04:53,990 --> 00:05:01,670
We see deep fake audio recordings which sound like the person who they are trained to sound like.

57
00:05:01,670 --> 00:05:06,710
Now, there's a piece of commercial software out there called Liarbird which does this really well.

58
00:05:06,710 --> 00:05:10,230
And so, you know, they're very convincing to users.

59
00:05:10,630 --> 00:05:17,630
And as a result, you know, they get used widely and they're easy to make, right?

60
00:05:17,630 --> 00:05:29,810
That's the other thing, is they don't take as much time as finding and crafting an image or carefully learning a style or mimicking a voice, right?

61
00:05:29,810 --> 00:05:32,670
You just kind of train the model and push them out.

62
00:05:32,670 --> 00:05:35,050
So, they're easy to make and they're convincing to users.

63
00:05:35,050 --> 00:05:39,690
Which means that they can have real impact.

64
00:05:39,690 --> 00:05:43,510
But that impact is mostly societal, right?

65
00:05:43,510 --> 00:05:55,370
So, there was a famous, you know, semi-viral video of Jordan Peele doing a deep fake where it was an image of President Obama speaking.

66
00:05:55,890 --> 00:05:57,270
And you can look it up.

67
00:05:57,270 --> 00:05:59,150
It's really easy to find.

68
00:06:00,850 --> 00:06:13,990
And so, like, the risk there is really that broad swaths of people will believe that a doctored or deep faked video is real.

69
00:06:13,990 --> 00:06:19,810
And so, there's something like President Donald J.

70
00:06:19,810 --> 00:06:30,530
Trump, you know, declaring war on North Korea on a hot mic or suggesting that he might invade North Korea on a hot mic.

71
00:06:30,530 --> 00:06:32,450
That could be really convincing to people.

72
00:06:32,450 --> 00:06:41,970
And so, that's the real risk of deep fakes, is that they are a powerful disinformation tool which is powered by artificial intelligence.

73
00:06:42,330 --> 00:06:48,030
And so, you know, our mitigation against deep fakes is kind of, eh.

74
00:06:48,510 --> 00:06:52,670
Like, there's very little we can do about deep fakes.

75
00:06:52,670 --> 00:06:54,270
They're difficult to detect.

76
00:06:54,270 --> 00:06:56,090
They're very convincing to users.

77
00:06:56,090 --> 00:06:57,470
They're easy to make.

78
00:06:57,470 --> 00:07:17,390
And so, really, it's a broader social problem that we need to tackle of having people be inherently less trustful of information sources and really vet where they get their information from, which is a hard problem.

79
00:07:17,390 --> 00:07:21,390
But this isn't an enterprise security threat.

80
00:07:21,390 --> 00:07:25,030
This is a social threat.

81
00:07:25,030 --> 00:07:32,150
So, moving back to thinking about deployed models, we'll look at adversarial examples.

82
00:07:32,150 --> 00:07:40,170
And this gif over here on the right is from a pretty well-known example that was really, really hyped up.

83
00:07:40,170 --> 00:07:42,430
And it was the turtle rifle paper.

84
00:07:42,430 --> 00:07:48,270
Because the Google computer vision API is what they attacked.

85
00:07:48,270 --> 00:07:51,110
And they 3D printed a turtle.

86
00:07:51,110 --> 00:07:57,550
And this is pretty obviously a turtle to anybody watching who isn't powered by the Google Vision API.

87
00:07:57,550 --> 00:08:05,290
And if you are an artificial intelligence powered by the Google Vision API, I'm so sorry for the rest of this talk.

88
00:08:05,290 --> 00:08:07,150
Because you're not gonna like it.

89
00:08:08,670 --> 00:08:15,990
That, you know, it's an input to the classifier that's specifically crafted to force misclassification.

90
00:08:16,330 --> 00:08:24,130
And so, this object as a turtle, just a little, like, toy turtle, right?

91
00:08:24,430 --> 00:08:42,310
Being misclassified as a rifle, the risk there is if you have a CCTV system that's powered by Google Vision API, which seems reasonable to license, that's looking for potential security threats.

92
00:08:42,350 --> 00:08:52,030
And, you know, you don't input all of that, like, racist physiognomy that we already see in a lot of those systems.

93
00:08:52,030 --> 00:08:53,450
But instead just say, you know what?

94
00:08:53,450 --> 00:08:55,610
Let's just look for weapons.

95
00:08:56,170 --> 00:09:00,370
Well, this could falsely trigger those systems.

96
00:09:00,490 --> 00:09:02,570
And so, there's a risk there, too.

97
00:09:03,610 --> 00:09:08,090
Correspondingly, if you 3D printed a gun, right?

98
00:09:08,730 --> 00:09:15,910
That got misclassified as a turtle, that would also be a threat, right?

99
00:09:15,910 --> 00:09:19,550
So, the impact, really, is that it causes misclassification.

100
00:09:20,210 --> 00:09:26,730
And so, that real world security threat and 3D printed examples are really rare, and they're really difficult to do.

101
00:09:27,730 --> 00:09:36,830
And they don't tend to work in all situations, and any researcher in computer vision can explain to you why it's not a real part of the threat model.

102
00:09:36,830 --> 00:09:37,950
And that's fine.

103
00:09:38,230 --> 00:09:40,710
But our threat is that it causes misclassification.

104
00:09:40,710 --> 00:09:50,030
As security practitioners, we try to look for ways to use technologies to detect bad things.

105
00:09:50,030 --> 00:10:01,710
And so, if you integrate artificial intelligence into your malware detector, for example, an attacker can bypass detection using adversarial examples.

106
00:10:02,430 --> 00:10:18,390
And the thing with adversarial examples is when we think about how to mitigate them, one of the only truly effective ways to do it is through what's called adversarial training, where you basically show it a bunch of adversarial examples and tell it, like,

107
00:10:18,390 --> 00:10:20,590
no, classify these correctly.

108
00:10:21,430 --> 00:10:25,170
And that's really time consuming, and it's expensive, and it's hard.

109
00:10:25,550 --> 00:10:27,450
And so, not a lot of people do it.

110
00:10:27,450 --> 00:10:34,210
And it requires you to take your model out of production and put it back into training, which, again, has a business impact.

111
00:10:35,330 --> 00:10:44,970
So, one of the other ways to sort of mitigate it is through what's called ensembling, where you don't just take one canonical model.

112
00:10:45,270 --> 00:10:52,530
You take several models and sort of average their output or take a weighted average of their output or whatever.

113
00:10:53,870 --> 00:10:56,190
And so, that's another way to do it.

114
00:10:56,270 --> 00:11:10,350
And that's kind of because it's a lot harder to come up with an adversarial example that works across a variety of classifiers, especially if they're not trained on the same data, if they don't have the same architecture.

115
00:11:10,570 --> 00:11:15,250
And it's still possible to do, but it sort of raises that barrier to entry.

116
00:11:15,310 --> 00:11:20,890
And if we think about it in terms of, like, cryptography, that's really what we're trying to do, right?

117
00:11:20,890 --> 00:11:26,790
Is we know that there is hypothetically always an attack, right?

118
00:11:26,790 --> 00:11:28,770
You can brute force the key space.

119
00:11:28,770 --> 00:11:35,130
That's a hypothetically feasible attack against AES, right?

120
00:11:36,050 --> 00:11:43,950
But it's good enough, and it takes long enough to brute force that key space that, you know, it's fine.

121
00:11:43,950 --> 00:11:45,010
It does the job, right?

122
00:11:45,010 --> 00:11:52,950
And that's sort of the same thing here, is you just have to raise that barrier to entry to make it not worth it for an attacker.

123
00:11:54,850 --> 00:12:01,170
So, the next thing we want to talk about in terms of threats to deployed models are backdoors.

124
00:12:01,370 --> 00:12:11,530
And so, backdoors in a machine learning context are manipulations of trained model weights that result in a specific outcome each time.

125
00:12:11,530 --> 00:12:23,790
So, a common way to do this in neural networks is bias poisoning, where you take one class and you make the bias for that one class in just the output layer really, really, really big.

126
00:12:23,790 --> 00:12:26,990
So, you're only changing one weight in the whole network.

127
00:12:26,990 --> 00:12:29,210
So, it doesn't change a lot, right?

128
00:12:29,890 --> 00:12:39,810
But by virtue of making that weight really big, you'll always get the same classification.

129
00:12:39,810 --> 00:12:51,890
And if it's a binary classifier, right, say it's, again, our malware detector, because people love to make neural network powered malware detectors for whatever reason.

130
00:12:51,890 --> 00:12:53,130
And I've made one.

131
00:12:53,130 --> 00:12:55,010
So, I'm allowed to criticize.

132
00:12:57,230 --> 00:12:58,790
People love to do it.

133
00:12:58,790 --> 00:13:03,230
And that's a really easy way to just be like, nope, everything's benign all the time.

134
00:13:03,310 --> 00:13:04,070
Always.

135
00:13:04,070 --> 00:13:05,510
100% of the time.

136
00:13:06,290 --> 00:13:11,230
So, you know, the impact is that it causes misclassification.

137
00:13:11,470 --> 00:13:21,730
And I say but not really, because really what it does is remove a class or remove all other classes from the classifier.

138
00:13:21,730 --> 00:13:23,610
So, it's not really misclassification.

139
00:13:23,610 --> 00:13:26,390
It's just making it a one class classifier.

140
00:13:27,750 --> 00:13:33,370
And our mitigation is just don't let attackers get access to your model.

141
00:13:33,370 --> 00:13:36,550
And by your model, I mean the trained model weights.

142
00:13:37,730 --> 00:13:51,730
And I say this kind of flippantly, because if an attacker has the ability to manipulate your model weights, where you're hosting your model, they can do way worse things than backdoor your model.

143
00:13:52,110 --> 00:14:02,890
So, it's not... it's like a local privilege escalation that requires you to have admin to begin with.

144
00:14:02,890 --> 00:14:05,710
It's not really a threat.

145
00:14:07,290 --> 00:14:11,170
So, again, attacks against deployed models.

146
00:14:11,170 --> 00:14:12,750
There's model theft.

147
00:14:12,750 --> 00:14:26,790
And I want to talk about model theft and give a lot of credit to Will Pierce, formerly of Silent Break Security, who I stole this graphic from his fantastic DerbyCon presentation last year.

148
00:14:27,810 --> 00:14:33,750
Was given the first CVE for machine learning, which is super dope.

149
00:14:34,510 --> 00:14:40,510
And so, model theft is the process of creating a copycat model by querying a trained model.

150
00:14:40,510 --> 00:14:41,710
Right?

151
00:14:41,710 --> 00:14:46,790
So, essentially, what we're doing is we're hitting a model over and over and over again.

152
00:14:46,910 --> 00:14:56,850
And using the outputs of that trained model, we train our own model to give us an approximation of that model.

153
00:14:56,850 --> 00:15:10,490
So, you know, if you have a box and you put in a 1 and you get out a 2, and then you have a second box, you want to put in the same input and get the same output.

154
00:15:10,510 --> 00:15:20,110
And it doesn't really matter what happens inside of that box as much as it matters that whatever input you give it, you get the same output.

155
00:15:20,290 --> 00:15:22,750
And that's what happens with model theft.

156
00:15:22,870 --> 00:15:26,790
So, our impact here is kind of twofold.

157
00:15:26,830 --> 00:15:32,050
And so, the potential for adversarial attacks in model theft goes way up.

158
00:15:32,050 --> 00:15:43,090
Because somebody has a model they can specifically query against to validate whether or not it gets misclassified without interacting with your model.

159
00:15:43,390 --> 00:15:48,210
So, it gives them sort of a private development environment for these adversarial attacks.

160
00:15:48,210 --> 00:15:51,130
And that's what Will did in the proof putting.

161
00:15:51,330 --> 00:16:01,090
Is he created this copycat model so that he could test phishing emails to bypass the proof point email security appliance.

162
00:16:01,090 --> 00:16:03,410
And it was incredibly clever.

163
00:16:03,410 --> 00:16:04,310
Super dope.

164
00:16:04,310 --> 00:16:06,910
I recommend watching his talk.

165
00:16:07,370 --> 00:16:11,050
But after you watch all the other talks at the AI Village.

166
00:16:11,290 --> 00:16:14,170
Maybe, like, after DEF CON, watch it.

167
00:16:14,170 --> 00:16:15,770
It's a really good talk.

168
00:16:17,170 --> 00:16:21,710
The other impact, though, is the loss of intellectual property.

169
00:16:21,950 --> 00:16:49,230
So, one of the things that I think kind of got overlooked in how cool the ability to bypass security appliances is, is that if you hire a bunch of data scientists and collect a bunch of data and clean a bunch of data and spend years creating and building and deploying a model that really differentiates you from the competition,

170
00:16:49,230 --> 00:17:06,330
and some unscrupulous company in not Canada, you know, steals your model, approximates your model, and just shoves it into their product and says, yeah, we do the same thing.

171
00:17:06,830 --> 00:17:10,590
Well, it costs them a lot less money to just copy your model.

172
00:17:10,670 --> 00:17:15,110
It costs them almost nothing in the grand scheme of things to copy your model.

173
00:17:15,170 --> 00:17:19,330
And so, there's no moat, so to speak.

174
00:17:19,410 --> 00:17:23,730
So, it really hurts your ability to create a differentiated product.

175
00:17:24,530 --> 00:17:27,930
So, this is a real vulnerability from that standpoint.

176
00:17:28,410 --> 00:17:33,050
And we'll get one layer deeper in our next slide.

177
00:17:33,350 --> 00:17:39,030
So, when we think about mitigations for model theft, right, there's two.

178
00:17:39,430 --> 00:17:42,070
And the first one is limiting queries to the model.

179
00:17:42,070 --> 00:17:46,350
And this one kind of feels bad.

180
00:17:46,770 --> 00:17:54,310
Because you don't want to limit queries to your model too much, since most queries to your model are going to be legitimate.

181
00:17:54,670 --> 00:18:04,850
For the most part, you're expecting these inputs and giving outputs, and you created this model, and you're hosting this model, because it's supposed to be useful to someone.

182
00:18:05,010 --> 00:18:18,290
And for most people, because of just the corporatization of information security and the corporatization of artificial intelligence, it's probably the people who pay your paycheck who want this.

183
00:18:18,310 --> 00:18:38,650
And so, inherently, you want to have really, but you want to balance that really high uptime and the fact that there may be a single endpoint that's making a lot of legitimate queries to your model with the fact that there may be an attacker who's trying to use your model for evil.

184
00:18:38,650 --> 00:18:40,670
Trying to steal your model.

185
00:18:42,370 --> 00:18:47,730
And so, the other thing we can do is limiting information returned from the model.

186
00:18:47,730 --> 00:19:11,910
And we saw the efficacy of this with SQL injection, where, like, back in the olden days, you know, a million years ago, because it's 2020, and so time doesn't matter anymore, we would return these detailed error messages for SQL queries, and it made it really,

187
00:19:11,910 --> 00:19:24,590
really easy for attackers to find the information they were looking for in SQL databases, especially SQL databases that didn't do a great job of input validation.

188
00:19:25,490 --> 00:19:34,910
And so, you know, once we started returning basically there was an error as the error message, it became much harder, right?

189
00:19:34,910 --> 00:19:38,030
And it's still feasible to do, but it became much harder.

190
00:19:38,210 --> 00:19:39,470
And it's similar here.

191
00:19:39,470 --> 00:19:52,890
If all that was returned was a blocked or not blocked, that binary signal versus a detailed header, then it would be much more difficult to create a copycat model.

192
00:19:52,890 --> 00:19:56,230
And so, you know, that is a real mitigation.

193
00:19:56,470 --> 00:20:01,670
And when we think about model theft, kind of the next step is inversion.

194
00:20:01,670 --> 00:20:02,970
Model inversion .

195
00:20:03,270 --> 00:20:08,150
And what model inversion is, is recovering training data from a trained model.

196
00:20:08,150 --> 00:20:22,330
So, there was a paper by Nick Carlini that came out two years ago, I think, in 2018, about how neural networks unintentionally memorize specific training examples.

197
00:20:22,330 --> 00:20:35,650
There have been papers on model inversion, both traditional and using generative adversarial networks that have shown that it's pretty effective.

198
00:20:37,150 --> 00:20:41,710
And so, the impact here is the loss of data, right?

199
00:20:41,710 --> 00:20:43,830
You're losing training data.

200
00:20:43,830 --> 00:20:46,230
And maybe that doesn't matter a ton.

201
00:20:46,230 --> 00:20:50,470
So, like at Rapid7, we have our open dataset, right?

202
00:20:50,470 --> 00:21:00,950
And that's free and open for anyone to use for noncommercial purposes, and we make it available for, you know, other stuff, right?

203
00:21:01,670 --> 00:21:13,990
And so, like, if our model was trained on open data, and somebody inverted our model, we probably wouldn't care that much, because, like, that data's out there, right?

204
00:21:14,170 --> 00:21:29,050
But if you are training a model for, like, a large medical device manufacturer, and it's trained on sensitive medical information, and that information gets recovered, well, that's a lot worse.

205
00:21:29,050 --> 00:21:42,430
If you're a data scientist at Equifax or some other credit rating bureau, and you have a bunch of sensitive financial data that you train a model on, that's pretty terrifying, right?

206
00:21:42,430 --> 00:21:48,570
The penalties for losing everybody's data at Equifax could be huge.

207
00:21:48,570 --> 00:21:58,050
You could have to pay a bunch of people's credit monitoring service for a while, and that would be bad for your business.

208
00:21:59,690 --> 00:22:06,490
So, when we think about mitigations, really, the first one is to protect access to the model.

209
00:22:06,490 --> 00:22:10,770
So, if they have direct access to the model, it's much easier to do inversion.

210
00:22:10,770 --> 00:22:12,690
It's much, much easier to do inversion.

211
00:22:14,110 --> 00:22:18,370
There are different techniques you can use if you have direct access to the model.

212
00:22:18,950 --> 00:22:26,170
And then the other mitigations for model inversion are essentially don't let people steal your model.

213
00:22:26,630 --> 00:22:34,890
Because then it's easier to use it as sort of the discriminator in the GAN.

214
00:22:34,990 --> 00:22:42,330
The architecture is a little more complicated than a traditional generative adversarial network, but the idea is pretty similar.

215
00:22:42,710 --> 00:22:46,290
You basically show it an example and say, does this look familiar?

216
00:22:46,290 --> 00:22:48,430
And it goes yes or no.

217
00:22:48,570 --> 00:22:52,970
And if it says yes, then you may have inverted the information.

218
00:22:55,050 --> 00:22:58,310
So, our next threat is poisoning.

219
00:22:58,310 --> 00:22:59,710
Data poisoning.

220
00:22:59,930 --> 00:23:07,070
And data poisoning is when malicious users inject bad training data into a model to corrupt it.

221
00:23:07,070 --> 00:23:16,350
So, usually, this happens in online learning where the model is continuously trained on the input to the model.

222
00:23:16,390 --> 00:23:24,190
And that is definitely a threat that needs to be considered.

223
00:23:24,190 --> 00:23:31,530
The other option is that they can, you know, get access to wherever you store your data and just inject bad examples.

224
00:23:31,650 --> 00:23:34,450
So, our impact is misclassification.

225
00:23:35,070 --> 00:23:41,430
It sort of shifts our decision boundary to deliberately misclassify samples.

226
00:23:41,610 --> 00:23:54,530
So, you know, letting spam through because now your threshold for what is spam is so high because you've just been doing online learning, right?

227
00:23:55,370 --> 00:24:01,670
And so, when we mitigate it, right, we want to go back and think about protecting access to our data.

228
00:24:01,670 --> 00:24:04,330
So, we want data integrity, right?

229
00:24:04,330 --> 00:24:07,570
So, having data versioning is really important.

230
00:24:07,570 --> 00:24:13,250
Making sure that your training data has a version, has an associated hash, has a label.

231
00:24:13,350 --> 00:24:15,290
That's super important.

232
00:24:15,870 --> 00:24:22,830
And then our next is we don't want to just allow unvalidated, untrusted input to our model.

233
00:24:22,830 --> 00:24:30,690
That's something we see a ton is people who just accept whatever input with no validation on the front end.

234
00:24:30,690 --> 00:24:35,370
It's just whatever gets input to the API gets pushed into the model.

235
00:24:36,130 --> 00:24:49,430
And that can really cause problems when you're retraining if you just take, you know, those inputs that you get and sort of store them off for training later.

236
00:24:50,690 --> 00:24:54,650
So, you know, we're coming up on the end of this talk.

237
00:24:54,650 --> 00:24:58,710
And so, we want to talk about, like, what can I do, right?

238
00:24:58,710 --> 00:25:01,050
And this is sort of a choose your own adventure story.

239
00:25:01,050 --> 00:25:02,410
It depends on your role.

240
00:25:02,830 --> 00:25:10,850
So, if you're a hacker, and by hacker, I mean this in the colloquial sense, a black hat, or a red teamer, right?

241
00:25:10,850 --> 00:25:14,290
Black hats are already doing some of this stuff.

242
00:25:14,690 --> 00:25:23,410
So, I really mean, you know, if you're a red teamer, if you're catching up to the elite APT black hat hackers, right?

243
00:25:23,410 --> 00:25:27,310
Take advantage of the lack of defenses on machine learning systems.

244
00:25:27,350 --> 00:25:46,550
Microsoft put out a report earlier this year saying that in their survey of large companies and government organizations out of, I think it was 28, only three had any meaningful defenses on their machine learning systems.

245
00:25:47,790 --> 00:25:51,230
So, if you're a red teamer, like, check it out.

246
00:25:51,230 --> 00:25:55,570
If you're a vulnerability discovery person, like, check it out.

247
00:25:55,570 --> 00:25:57,650
You can definitely find stuff.

248
00:25:57,930 --> 00:26:02,990
And that's because nobody knows what you're doing, and nobody's looking for you, right?

249
00:26:02,990 --> 00:26:07,730
Nobody is looking for black hat hackers, right?

250
00:26:07,770 --> 00:26:09,270
APTs, whoever.

251
00:26:14,340 --> 00:26:16,140
And if you are, let me know.

252
00:26:16,140 --> 00:26:19,940
Yell at me in the discord, and I'll be like, oh damn, cool.

253
00:26:21,580 --> 00:26:24,700
If you're a defender, if you're a blue teamer, right?

254
00:26:24,700 --> 00:26:34,700
If you're a sock monkey, if you're a researcher, if you're at DEF CON, you're probably one of these two categories.

255
00:26:35,460 --> 00:26:39,740
Test your machine learning systems as if they're part of your infrastructure.

256
00:26:40,460 --> 00:26:59,340
So, what we see a lot is that machine learning engineers and data scientists develop these models, and they take these models in, like, a .py file, and they hand it off to engineering, and engineering goes, I don't know what this is, and they slap a Django API in front of it,

257
00:26:59,340 --> 00:27:11,720
put it in a Docker container, and deploy it in Kubernetes as an API, and then when InfoSec looks at it, they go, I don't know what this is.

258
00:27:11,720 --> 00:27:12,660
It's magic.

259
00:27:12,660 --> 00:27:16,400
You put in JSON, and you get an output.

260
00:27:16,400 --> 00:27:19,040
It's magic, and we don't mess with it.

261
00:27:19,280 --> 00:27:20,660
And that sucks.

262
00:27:20,660 --> 00:27:21,760
Don't do that.

263
00:27:21,760 --> 00:27:22,900
Don't do that.

264
00:27:22,900 --> 00:27:24,040
Test your systems.

265
00:27:24,040 --> 00:27:25,860
Test the machine learning systems.

266
00:27:26,380 --> 00:27:27,980
Work with your data scientists.

267
00:27:27,980 --> 00:27:32,540
Work with your ops people, because this is part of your attack surface.

268
00:27:32,540 --> 00:27:34,420
This is part of your attack surface, right?

269
00:27:34,520 --> 00:27:40,180
And don't let people hype you up over AI-generated phishing emails.

270
00:27:40,740 --> 00:27:49,360
When GPT-2 came out, people went nuts and were like, this is the end of detecting phishing.

271
00:27:49,360 --> 00:27:51,000
They're going to be too convincing.

272
00:27:51,300 --> 00:27:52,640
And, like, just don't.

273
00:27:52,640 --> 00:27:53,000
Okay?

274
00:27:53,000 --> 00:27:56,400
It's the same thing that we've been saying for 20 years.

275
00:27:57,120 --> 00:27:58,760
Patch your systems.

276
00:27:58,780 --> 00:28:01,440
Look for bad stuff, and patch your systems.

277
00:28:01,820 --> 00:28:03,020
That's it.

278
00:28:03,320 --> 00:28:05,300
There's nothing new under the sun.

279
00:28:05,300 --> 00:28:08,460
It's the exact same thing we've been saying for 20 years.

280
00:28:08,780 --> 00:28:13,160
Just keep looking for bad stuff, and patch your systems.

281
00:28:13,340 --> 00:28:21,500
And if anybody gives you pushback on patching your systems, like, have them talk to me, and I'll yell at them for you.

282
00:28:21,500 --> 00:28:23,860
Because you have to patch your systems.

283
00:28:25,240 --> 00:28:31,720
If you're watching this talk and you're a data scientist or a machine learning engineer, first of all, thank you for coming to DEF CON.

284
00:28:32,360 --> 00:28:34,140
DEF CON is dope.

285
00:28:34,460 --> 00:28:39,840
And, like, even though it's free and remote this year, come back next year, because it's cool.

286
00:28:42,280 --> 00:28:44,860
Conduct threat modeling on your models.

287
00:28:45,040 --> 00:28:53,660
So, before you put a model into deployment, work with your infosec team, work with ops, and sort of ask, like, what could go wrong?

288
00:28:53,840 --> 00:28:54,340
Right?

289
00:28:54,340 --> 00:28:58,660
What are the risks of an adversarial attack on this?

290
00:28:58,660 --> 00:29:03,440
What are the risks associated with this particular model?

291
00:29:03,440 --> 00:29:06,560
And what is the attack surface of this model?

292
00:29:08,080 --> 00:29:17,140
You know, and also when you're deploying those models, work with infosec and ops to balance uptime with the risk of model theft.

293
00:29:17,700 --> 00:29:22,880
These two kind of go hand in hand, is, like, you're going to deploy these models.

294
00:29:23,380 --> 00:29:25,040
Somebody may try to steal them.

295
00:29:25,040 --> 00:29:27,520
Is there a risk if somebody steals your model?

296
00:29:27,520 --> 00:29:28,820
Like, does it matter?

297
00:29:29,640 --> 00:29:38,640
And if it doesn't matter, like, think about would it be okay to publish the source code and data for this model?

298
00:29:38,640 --> 00:29:47,580
And if your answer to that is no, then you really need to make sure that infosec and operations are aware at deployment time.

299
00:29:47,940 --> 00:29:51,780
And finally, don't hype people up over text generation models.

300
00:29:51,780 --> 00:30:02,100
Like, GPT-2 and GPT-3 are super cool from an NLP standpoint, natural language processing, but they're not going to change the security landscape.

301
00:30:02,100 --> 00:30:04,380
Stop scaring people, please.

302
00:30:04,420 --> 00:30:10,120
I'm literally begging you, don't hype people up over text generation models.

303
00:30:10,860 --> 00:30:12,220
Here are some references.

304
00:30:12,860 --> 00:30:15,680
These are some of the things I talked about.

305
00:30:16,000 --> 00:30:18,040
And thank you so much for attending my talk.

306
00:30:18,080 --> 00:30:23,240
I'll be in the Discord most of the weekend, so if you have any questions, feel free to drop me a line.

307
00:30:23,300 --> 00:30:24,160
Thank you so much.

308
00:30:24,160 --> 00:30:25,000
Have a great weekend.

309
00:30:25,000 --> 00:30:25,840
Enjoy your con.