1
00:00:00,820 --> 00:00:03,180
Okay, say something.

2
00:00:05,280 --> 00:00:09,420
Nope, the audio is definitely broken again.

3
00:00:10,220 --> 00:00:11,680
Now say something.

4
00:00:19,120 --> 00:00:20,920
Time with the audio issues.

5
00:00:20,920 --> 00:00:22,100
There we go.

6
00:00:22,180 --> 00:00:24,040
The audio seems to be working now.

7
00:00:24,040 --> 00:00:25,140
All right, awesome.

8
00:00:26,280 --> 00:00:42,700
Yeah, I almost feel like we got to have... whoever is the poor soul stuck with jockeying this in the future almost needs to have two laptops, one with Twitch and everything going and one with just OBS and Zoom, yeah.

9
00:00:43,880 --> 00:00:45,260
Yeah, that's...

10
00:00:45,260 --> 00:00:49,840
Nothing can confuse OBS's tiny little brain.

11
00:00:50,060 --> 00:00:59,860
But then the problem is when you forget to mute that thing and then this thing and then there's one of your tabs is doing the wrong noise and you're like, oh, this is wrong.

12
00:01:00,220 --> 00:01:00,920
Yep.

13
00:01:00,920 --> 00:01:02,720
Sven, you're being heroic.

14
00:01:02,760 --> 00:01:03,300
It's just...

15
00:01:04,020 --> 00:01:05,960
Yep, we are live, by the way.

16
00:01:05,960 --> 00:01:09,880
The internet is listening to us talk, so enjoying the banter.

17
00:01:09,920 --> 00:01:11,240
You're still being heroic.

18
00:01:11,420 --> 00:01:12,200
Yes.

19
00:01:12,420 --> 00:01:23,120
Yeah, so for those of you watching, Sven has been the main guy troubleshooting all of the audio issues and fighting with the stream for basically all of DEF CON.

20
00:01:23,120 --> 00:01:29,860
And it's a testament to his hard work that things have gone as smoothly as they have.

21
00:01:29,860 --> 00:01:35,060
So you've made it a lot easier that things were pre-recorded.

22
00:01:35,060 --> 00:01:42,840
Can you imagine trying to hunt people down, hunt speakers down, and then their sharing is not quite right and all that?

23
00:01:43,400 --> 00:01:45,420
Yeah, no, I mean...

24
00:01:45,420 --> 00:01:47,880
I mean, this year, at least they're not passed out in the bar.

25
00:01:49,660 --> 00:01:52,260
How many speakers have been passed out in the bar?

26
00:01:53,360 --> 00:01:57,720
None of ours that I know of, but we've definitely hooked a few of them out from the nearest bar.

27
00:01:58,240 --> 00:01:59,600
Yeah, well, you know.

28
00:02:04,250 --> 00:02:08,810
Are you expecting anyone else or we want to get this show on the road?

29
00:02:09,130 --> 00:02:16,930
Yeah, let me... well, let me pull up this thing for myself.

30
00:02:18,090 --> 00:02:24,970
I'm going to try a new layout as soon as I can find the thing.

31
00:02:25,210 --> 00:02:31,210
So while I'm looking for it, Rich, do you want to introduce the paper to everyone?

32
00:02:33,470 --> 00:02:34,670
Oh, sorry.

33
00:02:34,670 --> 00:02:36,270
So we're going to do a couple of things.

34
00:02:36,290 --> 00:02:37,430
Hi, Barton.

35
00:02:37,510 --> 00:02:40,810
So we're going to start with just the four of us talking about it for a bit.

36
00:02:40,810 --> 00:02:42,410
And then I'm going to open up the...

37
00:02:43,770 --> 00:02:48,710
and join the voice chat in the Defcon Discord.

38
00:02:48,710 --> 00:02:52,310
And so you guys will be able to talk on stream and talk with us and everything.

39
00:02:53,870 --> 00:02:56,950
So for the first, like, 20 minutes, it's going to be just us.

40
00:02:56,950 --> 00:03:02,070
And then we'll bring everyone we can in and have a bit more of a Wild West thing.

41
00:03:02,070 --> 00:03:03,790
We're going to try out how that works.

42
00:03:04,430 --> 00:03:07,530
And the other thing is I'm just going to play with the layouts a little bit.

43
00:03:07,750 --> 00:03:10,750
So that's the structure of it.

44
00:03:10,750 --> 00:03:13,670
We've also got a journal club coming up on this Wednesday.

45
00:03:14,370 --> 00:03:17,770
We will put the link to that paper in the Defcon Discord.

46
00:03:17,770 --> 00:03:21,190
And then if you want to participate, you should join our Discord.

47
00:03:22,810 --> 00:03:24,370
Link in the Twitter bio.

48
00:03:24,810 --> 00:03:26,410
Yeah, it's in the Twitter bio.

49
00:03:26,410 --> 00:03:28,690
And you can just DM one of us.

50
00:03:28,690 --> 00:03:32,730
And we'll send you an invite to the AI Village Discord.

51
00:03:32,730 --> 00:03:36,050
Please don't use the AI Village Discord while Defcon is going on.

52
00:03:36,050 --> 00:03:41,770
Please be respectful of the hard work they have put into building their Discord and getting it running.

53
00:03:41,970 --> 00:03:45,330
And only use it, like, tomorrow or Tuesday.

54
00:03:46,730 --> 00:03:48,910
So take it away, Rich.

55
00:03:49,490 --> 00:03:50,530
Okay.

56
00:03:50,590 --> 00:03:51,070
Cool.

57
00:03:51,070 --> 00:03:54,690
So this is actually a slightly older paper.

58
00:03:54,690 --> 00:03:56,950
It's from 2017.

59
00:03:58,010 --> 00:04:06,890
It is called Summoning Demons, the Pursuit of Exploitable Bugs in Machine Learning, which is probably one of the coolest paper titles I've come across in a while.

60
00:04:07,310 --> 00:04:12,310
And it's by Rock Stevens and Tudor Dimitrescu's group.

61
00:04:12,310 --> 00:04:22,050
So the basic idea of the paper, it's kind of an interesting spin on the adversarial ML papers that I think started coming out about the same time.

62
00:04:22,050 --> 00:04:32,130
And in this, rather than actually, like, using the ML model itself to sort of guide inputs, they're looking for more...

63
00:04:32,770 --> 00:04:37,930
they're more looking for flaws in how the different components of the system hook up.

64
00:04:37,930 --> 00:04:54,370
And to do that, they actually crack out an old classic American fuzzy lump, the fuzzer, and they re-instrument a lot of it so that weird actions from the ML components of the system register as crashes to AFL.

65
00:04:54,570 --> 00:05:02,530
And so that lets them essentially do sort of like a black box adversarial attack, if you want to think of it that way, against the ML system.

66
00:05:02,570 --> 00:05:13,950
And what they find is that in addition to sort of the usual stuff you can get, where you have, like, inputs that look like faces that don't register as faces, because they've had some pixels perturbed, they also run into, like, image format parsing errors,

67
00:05:13,950 --> 00:05:21,730
where a component of the ML system won't parse, for example, a face correctly, completely fails to find the face.

68
00:05:21,730 --> 00:05:25,830
But if you open it up in other programs, it actually looks good.

69
00:05:25,990 --> 00:05:39,330
So I think this is kind of in the vein of the same sort of things that Ariel Herbert-Boss talks about a lot, where it's not just attacking the ML system itself that's important.

70
00:05:39,330 --> 00:05:48,830
You've got all of these other components in the pipeline, from ingesting the actual image all the way through feature extraction, pre-processing, and so on.

71
00:05:48,830 --> 00:05:51,650
And each of those is actually a point that you can attack.

72
00:05:51,650 --> 00:06:04,510
And so in this paper, they're basically using fuzzing as a way to find some of those, you know, broken assumptions or breakdowns in the link from one stage of processing to the next.

73
00:06:05,350 --> 00:06:08,590
And I thought it was a super clever paper with a really good name.

74
00:06:08,590 --> 00:06:11,250
And they dig a couple of CVs out of it, too.

75
00:06:11,250 --> 00:06:23,510
So super cool application of sort of more classic security methodology to what, from my perspective, is kind of an adversarial ML problem.

76
00:06:26,210 --> 00:06:30,530
So that's my, I guess, like two-minute summary of the paper.

77
00:06:35,440 --> 00:06:38,140
So I guess maybe to kick off conversation,

78
00:06:41,340 --> 00:06:55,180
so does this, do you feel like, does anyone have an opinion, strong opinion one way or the other, if this falls into the same sort of general field of adversarial ML as sort of like gradient-based attacks?

79
00:06:55,180 --> 00:07:14,520
Like I'm trying to think of, the name escapes me off the top of my head, like fast gradient sign method, or like these Jacobian-based methods, or saliency-based methods, or maybe even like, you could almost argue it's like a genetic algorithm-based attack against a particular ML pipeline.

80
00:07:14,580 --> 00:07:18,420
Well, to be honest, I thought that adversarial ML covered all this already.

81
00:07:18,420 --> 00:07:20,540
I was a bit surprised that it was new.

82
00:07:21,800 --> 00:07:36,440
Yeah, it's genetic algorithm-based because they're using FuzzyLock, which is genetic algorithm-based, which is also interesting because there were network extensions to genetic algorithms that might be more appropriate if you're searching spaces.

83
00:07:36,980 --> 00:07:40,100
So there's some interesting stuff on the front that could be done.

84
00:07:40,100 --> 00:07:41,980
I kept wanting to say, what happened next?

85
00:07:50,890 --> 00:08:10,710
Likewise, in the context of healthcare, practical realities of securing systems, it seems like a lot more emphasis right now is placed on more theoretical attacks, where it's more so once you have uninhibited access to being able to query this API that's being explored.

86
00:08:10,710 --> 00:08:23,450
So it's nice to see a paper like this that's a bit more in an all-systems set of things, because at least from healthcare IT, it seems like it's a more likely encounter that you'll find in the industry.

87
00:08:24,670 --> 00:08:33,090
Yeah, one thing that I did really like was their focus on implementations rather than just theoretical properties, right?

88
00:08:33,090 --> 00:08:42,710
Because in some sense, something like the FastGradientSign method would apply to almost any of these models.

89
00:08:42,710 --> 00:08:48,170
But here, they're sort of like, okay, this model was instantiated in this way and is doing these things.

90
00:08:48,170 --> 00:08:52,850
And now how can we attack sort of that specific version of the model?

91
00:08:53,450 --> 00:09:02,890
Because it's a lot less academic in some ways, and a lot of the adversarial ML research seems to be very academic still.

92
00:09:03,150 --> 00:09:06,190
This was like down and dirty in how do we attack this thing.

93
00:09:06,310 --> 00:09:09,150
It's kind of in my vein of stuff.

94
00:09:09,270 --> 00:09:13,810
Just look at the system, work out how to break the system, and take it to pieces.

95
00:09:14,050 --> 00:09:25,070
I loved the way that they hacked that little hack on FuzzyLop, where they went, okay, we're going to take out the things that actually broke, and then we're going to induce breaks for the things we care about.

96
00:09:25,490 --> 00:09:27,430
Yeah, changing the target.

97
00:09:27,430 --> 00:09:28,570
Yeah, changing the target.

98
00:09:29,290 --> 00:09:31,250
It was just beautifully done.

99
00:09:31,650 --> 00:09:45,510
I thought it would be useful for defense as well, because we're also looking now at things like Strafe's work, so Brittany Prostakoff's work on attacking with robots.

100
00:09:45,990 --> 00:09:50,290
What could we do to screw up their vision systems?

101
00:09:51,210 --> 00:09:53,710
So how could we mess that up?

102
00:09:53,710 --> 00:09:55,850
There's some beautiful adversarial there too.

103
00:09:56,910 --> 00:09:58,870
They can't see, they can't move.

104
00:09:59,190 --> 00:09:59,750
Yeah.

105
00:10:00,470 --> 00:10:03,650
Yeah, so actually, I thought table one in this was...

106
00:10:04,490 --> 00:10:06,130
I don't know, I'm surprised.

107
00:10:06,830 --> 00:10:08,270
Flips to table one.

108
00:10:08,270 --> 00:10:09,010
Flips to table one, yeah.

109
00:10:09,010 --> 00:10:10,510
Waves to table one.

110
00:10:11,170 --> 00:10:13,190
It's, I think it's on page four.

111
00:10:13,390 --> 00:10:14,310
Yeah, yeah.

112
00:10:14,310 --> 00:10:15,270
Yeah, page four.

113
00:10:15,330 --> 00:10:20,330
Yeah, they actually like, I feel like, again, this is very much like Ariel's jam, right?

114
00:10:20,330 --> 00:10:23,650
They go over sort of the attack surface and the kinds of things you can do to it.

115
00:10:23,650 --> 00:10:28,130
And it reminds me a little bit of the contest there was...

116
00:10:28,130 --> 00:10:35,670
So I think it was a late entry to the contest last year that was announced at the AI Village where they were trying to do ML evasions.

117
00:10:35,830 --> 00:10:47,650
And one guy actually found essentially a bunch of ways to just completely trash the feature extraction so that the ML prediction just wouldn't run on the samples he uploaded.

118
00:10:48,550 --> 00:10:54,310
And so, yeah, he was a little upset that that didn't count as a win, which I think is justifiable, right?

119
00:10:54,310 --> 00:10:57,290
Because if you can just like stop the analysis cold, right?

120
00:10:57,470 --> 00:10:59,530
That's almost as good as evading it.

121
00:10:59,530 --> 00:11:00,230
But yeah.

122
00:11:01,170 --> 00:11:05,610
It depends what you're trying to do if you kind of sneak into a system and it's a bunch of systems, right?

123
00:11:05,610 --> 00:11:07,830
We just want to, you know, the usual thing.

124
00:11:07,830 --> 00:11:10,930
Get in, do your thing, get out without no one noticing.

125
00:11:12,090 --> 00:11:14,850
And we see enough of that in real life of the stuff.

126
00:11:14,850 --> 00:11:17,350
It's like, oh crap, we didn't realize that was a problem.

127
00:11:18,070 --> 00:11:19,770
Before someone else adds them.

128
00:11:21,050 --> 00:11:23,650
There's also another thing to that.

129
00:11:23,770 --> 00:11:33,350
Like if you wanted to attack the Ember model with the Ember dataset, it only looks at your headers and you could just change other parts.

130
00:11:33,350 --> 00:11:36,570
Like don't even think about it in terms of adversarial ML.

131
00:11:36,570 --> 00:11:48,190
Just copy like a notepad.exe's header and just put your own entry point function in and start launching weird stuff from there.

132
00:11:48,190 --> 00:11:51,870
And that's a different way to bypass the thing that is also not...

133
00:11:52,810 --> 00:11:54,670
Academics also not aware of.

134
00:11:56,330 --> 00:11:58,110
Yeah, so that's sort of like...

135
00:11:58,110 --> 00:12:03,010
I guess that's maybe like a...

136
00:12:03,010 --> 00:12:04,910
You can classify that if you're gonna...

137
00:12:04,910 --> 00:12:05,910
I'm gonna screw this up.

138
00:12:05,910 --> 00:12:10,110
So if I'm getting the terminology wrong, security people, please like yell at me.

139
00:12:10,110 --> 00:12:20,570
But it's almost sort of like a logic attack on the feature extraction rather than what they're doing here, which seems to be more going for crashes, right?

140
00:12:20,570 --> 00:12:31,830
If you know how the feature extraction process works, then you can sort of write your way around it deliberately rather than just throwing shit at it until something blows up.

141
00:12:32,930 --> 00:12:35,490
Some of it's blowing it up, some of it's sneaking in the back.

142
00:12:35,670 --> 00:12:39,630
Some of it's like sneaking in the back conventionally depending on your system.

143
00:12:40,770 --> 00:12:46,710
The other thing that interests me is that only some of these became CVEs.

144
00:12:46,710 --> 00:12:53,830
That the response to the people who are responsible for the algorithms for some of this was like, won't fix and who cares?

145
00:12:54,330 --> 00:12:57,990
And that's part of our journey, I guess.

146
00:12:58,370 --> 00:13:09,630
It is just like every other security thing is trying to get people to accept that yes, you may have to drop a little bit of responsiveness in your system.

147
00:13:10,110 --> 00:13:13,410
But you'll get it more secure and it matters.

148
00:13:14,570 --> 00:13:26,670
Yeah, I'm kind of reminded a little bit of some of the argument that went around for the proof point CVE, where people were arguing whether or not that actually counted as like an information leak CVE.

149
00:13:26,670 --> 00:13:36,510
Because so if people aren't familiar, the short version is the people who did it, Will Pierce and someone whose name I'm forgetting, I'm sorry.

150
00:13:37,040 --> 00:13:45,030
They found a way to essentially like send an email through proof points, spam detection service that would then bounce back to them.

151
00:13:45,030 --> 00:13:49,650
And in the header of the email, it had a score for how spammy it looked.

152
00:13:50,110 --> 00:13:53,750
And so that essentially was the information leak.

153
00:13:53,750 --> 00:13:59,270
The fact that they could get that score back and that was enough for them to rebuild essentially a proxy model for it.

154
00:13:59,750 --> 00:14:12,790
And yeah, over on in various like AI village discussion channels, we had very strong debates over whether or not that actually should have counted as like an information leak CVE.

155
00:14:12,830 --> 00:14:17,890
Because, you know, it's in a lot of cases, it's useful contextual information.

156
00:14:17,890 --> 00:14:22,950
But when you look at it in an ML context, you're actually enabling like a model stealing attack.

157
00:14:23,590 --> 00:14:30,110
Which again, it's one of these, we're kind of in new territory as far as the security of these ML based systems goes.

158
00:14:30,130 --> 00:14:47,630
And a lot of people that are more familiar with conventional security, maybe don't quite get what these various information leaks or attack surfaces could do once you wrap it in like how ML based systems work.

159
00:14:47,790 --> 00:14:56,750
And I think another thing we get out of this is by doing this sort of attack, we start showing where the things that break up.

160
00:14:56,750 --> 00:15:01,590
And one of the things I'd love about the fact that MLSec exists is it makes machine learning better.

161
00:15:02,530 --> 00:15:05,650
Just by dint of the spotlight we throw on it.

162
00:15:05,650 --> 00:15:09,370
You chuck hackers at it, they take it to pieces, they show you where it's broken.

163
00:15:09,990 --> 00:15:14,290
And hell, we spend most of our life trying to get our damn systems to work.

164
00:15:14,450 --> 00:15:16,210
But I'm going to give a little bit of history.

165
00:15:16,210 --> 00:15:17,170
I'm old.

166
00:15:17,510 --> 00:15:20,590
Jump in for just a second.

167
00:15:20,630 --> 00:15:22,470
I just saw in the Discord chat.

168
00:15:22,470 --> 00:15:27,170
So it was Will Pierce and Nick Landers who got the Proofpoint CVE.

169
00:15:27,170 --> 00:15:30,010
So I'm very sorry that I forgot your name, Nick.

170
00:15:30,010 --> 00:15:31,410
Okay, sorry, carry on.

171
00:15:31,590 --> 00:15:36,870
I was just going to say about sort of some things, the findings, that it's double float precision thing.

172
00:15:37,250 --> 00:15:44,710
So way back before we had, I think we had Python, but we sure as hell didn't have scikit-learn and friends.

173
00:15:44,710 --> 00:15:46,590
We had matplotlib.

174
00:15:46,650 --> 00:15:48,270
I'm not... matmap.

175
00:15:48,390 --> 00:15:51,670
We use MATLAB for a hell of a lot of machine learning stuff.

176
00:15:51,910 --> 00:15:54,770
And there was a neural network kit in the MATLAB.

177
00:15:55,010 --> 00:15:59,170
And that neural network kit suffered horrendously from underflow errors.

178
00:15:59,690 --> 00:16:02,090
You could break it really, really, really easily.

179
00:16:02,090 --> 00:16:03,950
So Aston University built its own.

180
00:16:04,510 --> 00:16:08,170
They built a separate kit because they couldn't get the first one fixed.

181
00:16:08,810 --> 00:16:11,910
I mean, this sort of stuff, like the float to double errors.

182
00:16:12,890 --> 00:16:25,950
I mean, crossing my fingers that actually all of those hidden faults that are probably still going on with neural network stuff and deep learning, we're not even noticing because we just trust the systems.

183
00:16:25,950 --> 00:16:26,890
Yeah.

184
00:16:26,890 --> 00:16:29,670
I mean, even in Python, right?

185
00:16:29,670 --> 00:16:43,070
Like under the hood, a ton of these neural network libraries, and PyTorch, at least, sklearn as well, use pickle, which is just broken kind of by default, right?

186
00:16:43,070 --> 00:16:46,310
It's a stack machine that you can write arbitrary code for.

187
00:16:46,570 --> 00:16:53,090
And yeah, people still share wait files that are essentially pickles.

188
00:16:53,090 --> 00:16:56,850
And so you're downloading this off the internet and running code.

189
00:16:56,850 --> 00:16:57,690
And I keep threatening.

190
00:16:57,690 --> 00:17:11,470
One of these days, I'm actually going to write a pickle poisoner that just replaces calls to the relu function with some sort of stochastic function so that half the time it works and half the time it gives you garbage.

191
00:17:12,030 --> 00:17:14,670
Comment from Discord.

192
00:17:14,670 --> 00:17:16,510
Yeah , numpy as well uses pickles.

193
00:17:16,510 --> 00:17:17,810
So it's everywhere.

194
00:17:17,810 --> 00:17:18,890
Keras, yeah.

195
00:17:18,890 --> 00:17:22,710
Is that what the numpy save file is?

196
00:17:24,330 --> 00:17:26,050
I thought that was secure.

197
00:17:26,350 --> 00:17:27,570
I'm not sure.

198
00:17:27,570 --> 00:17:29,130
I know, I'm pretty sure it has one...

199
00:17:30,570 --> 00:17:31,970
One pickle thing in there.

200
00:17:32,030 --> 00:17:33,110
One or two.

201
00:17:33,110 --> 00:17:37,890
It's got one or two formats that are not.

202
00:17:37,890 --> 00:17:39,390
Apparently, there's a couple that...

203
00:17:39,390 --> 00:17:44,210
The scikit-learn form that these guys found, numpy was part of that.

204
00:17:44,210 --> 00:17:45,490
I'm just flicking to the page.

205
00:17:45,490 --> 00:17:47,990
Page seven at the bottom, if you're reading along with us.

206
00:17:49,210 --> 00:17:56,650
One interesting thing is that they weren't able to induce misclassification or from the precision loss problem.

207
00:17:56,650 --> 00:18:00,610
So they tried different values of epsilon, and basically said that it's possible in theory.

208
00:18:00,610 --> 00:18:09,330
But it's interesting to consider how would you model that epsilon based on the syntax to induce this misclassification.

209
00:18:15,120 --> 00:18:16,840
How much of this is induced by...

210
00:18:18,440 --> 00:18:25,560
Basically, all the machine learning code is written by academics and over-enthusiastic academics.

211
00:18:25,560 --> 00:18:26,200
Guilty.

212
00:18:27,860 --> 00:18:29,540
Who aren't experienced in...

213
00:18:30,820 --> 00:18:35,180
It's incredibly hard to write numerically efficient code.

214
00:18:35,180 --> 00:18:43,440
You don't have time to really learn how to multiply matrices incredibly efficiently and make sure you open files correctly.

215
00:18:43,960 --> 00:18:52,260
So we've chosen a lot of lazy paths to formatting and opening things versus secure paths.

216
00:18:53,720 --> 00:18:56,260
And also, you've got tiny numbers and people don't...

217
00:18:56,260 --> 00:19:00,640
I mean, you go wading on in, build your stuff, and don't think about how those numbers interact.

218
00:19:00,640 --> 00:19:03,260
You just assume the system is going to magically deal with it.

219
00:19:03,900 --> 00:19:04,220
Yeah.

220
00:19:04,220 --> 00:19:12,200
And I mean, there's also the fact that writing secure code is very much its own skill set, right?

221
00:19:14,440 --> 00:19:21,040
Some people are writing research code because they don't expect it to be used in Angular, they don't expect it to be used in a production context.

222
00:19:21,040 --> 00:19:26,160
Some people are writing numerically efficient code because that's their jam and God bless them.

223
00:19:26,900 --> 00:19:38,120
And then you've got sort of security people who are probably gazing at a lot of this with sort of a feeling of vague horror that they're actually going into production.

224
00:19:39,180 --> 00:19:55,260
Yeah, I know even in my day job, right, a lot of what we've been doing recently is sort of going back and rethinking how all of this stuff is put together so that we can feel confident in actually pushing out products containing it.

225
00:19:56,120 --> 00:20:04,060
There's a middle ground where you can enable basically more flexible code, but also execute it in a way that perhaps minimizes the blast radius.

226
00:20:04,140 --> 00:20:10,420
I don't mean put it in a container, obviously, but it's something more to the point of running code in secure enclaves.

227
00:20:10,420 --> 00:20:19,980
For instance, the work with silo type of workflows or do you see anything where you try to mitigate the impacts of this code?

228
00:20:20,560 --> 00:20:22,060
And could we have code checkers?

229
00:20:22,060 --> 00:20:26,100
I mean, literally just some way, some sort of test data sets.

230
00:20:26,180 --> 00:20:27,740
But we know this stuff is bad.

231
00:20:27,740 --> 00:20:29,360
Run it through your system.

232
00:20:30,020 --> 00:20:30,560
Yeah.

233
00:20:31,660 --> 00:20:43,120
Yeah, I mean, I had a great amount of fun, again, at my day job, actually like taking our feature abstraction code and fuzzing it just to see what would happen with it.

234
00:20:43,120 --> 00:20:51,580
Fortunately, it turned out that I spent more time than I care to think of beating on it, and it turned out fine.

235
00:20:51,580 --> 00:20:54,840
But yeah, if it hadn't, what would we have done then, right?

236
00:20:55,660 --> 00:21:06,700
And I think the idea of building test vectors of just really bizarre, but technically conformant files for this sort of thing is an awesome idea.

237
00:21:07,820 --> 00:21:18,940
But extending that to the actual data science part, how often have you issued a PR about some data science thing and sent it to someone and they're gone?

238
00:21:18,940 --> 00:21:20,540
It looks good to me.

239
00:21:22,360 --> 00:21:32,160
But there's some stuff that you can check easily, but a lot of the code that goes into BERT is incredibly complicated.

240
00:21:32,520 --> 00:21:34,180
Very hard to check.

241
00:21:34,280 --> 00:21:40,480
When you make a small change, you could have broken things fundamentally because it's software.

242
00:21:41,720 --> 00:21:53,380
How do you write a unit test for I broke BERT when you input this one slight thing and now it causes this whole class of things to be false positives, false negatives?

243
00:21:55,680 --> 00:21:58,140
Like, how do you unit test machine learning?

244
00:21:58,580 --> 00:22:01,740
There's a... I mean, you can do...

245
00:22:01,740 --> 00:22:04,600
I mean, essentially, you do test vectors, right?

246
00:22:04,800 --> 00:22:07,980
You say, you know, if I load...

247
00:22:08,660 --> 00:22:11,340
You can consider the weights are obviously part of the model.

248
00:22:11,340 --> 00:22:12,520
You say, I've got this set of weights.

249
00:22:12,520 --> 00:22:23,580
If I load in, you know, this set of features, I should get this exact set of outputs up to numerical precision, which, again, numerical precision is kind of its own nightmare.

250
00:22:25,020 --> 00:22:29,660
Yeah, but then if you change the weights, right, then kind of all bets are off.

251
00:22:29,660 --> 00:22:44,040
And that's one of the painful things about debugging ML, even when you're not under adversarial conditions, is just, I retrained the model, do I still trust it?

252
00:22:44,180 --> 00:22:48,440
I mean, there's a difference between I messed up my models, which is like, we do this all the time.

253
00:22:48,440 --> 00:22:55,360
And somebody is actively trying to attack my models or may actively try to attack my models using known vulnerabilities.

254
00:22:55,860 --> 00:22:59,480
Because sure as hell, half the systems out there haven't patched any of this.

255
00:23:00,060 --> 00:23:06,420
How many machine learning developers even know unit testing or even know good software practices?

256
00:23:06,420 --> 00:23:08,580
Like, a lot of us came from academia.

257
00:23:08,580 --> 00:23:11,300
I have a PhD in math, not software engineering.

258
00:23:11,740 --> 00:23:17,520
So I have a philosophical question I want to pose to the panel from Discord.

259
00:23:18,080 --> 00:23:21,580
And Yaga asks, what is a terror and...

260
00:23:21,580 --> 00:23:23,560
Sorry, what's a terror?

261
00:23:23,560 --> 00:23:25,420
This is terror right here.

262
00:23:25,420 --> 00:23:28,100
What is an error and what's an attack?

263
00:23:28,100 --> 00:23:32,980
And like, is there any meaningful distinction between the two?

264
00:23:32,980 --> 00:23:35,100
I guess I would follow up.

265
00:23:36,080 --> 00:23:43,200
I think if you're looking at adversarial images, I mean, the fact you have like the little image patches to screw up your system is an attack.

266
00:23:43,280 --> 00:23:44,440
It's not an error.

267
00:23:44,700 --> 00:23:48,620
You just don't randomly get sort of extra patches on a stop sign.

268
00:23:49,280 --> 00:23:51,960
No matter how much undergrowth you got on it.

269
00:23:52,140 --> 00:23:54,280
I think the difference is intention.

270
00:23:55,540 --> 00:23:57,840
Yeah, I mean, that's the same thing in security, right?

271
00:23:57,840 --> 00:24:02,660
Bugs can be... I mean, exploits are essentially software bugs, right?

272
00:24:02,980 --> 00:24:05,300
Maybe that's being too general, but if you...

273
00:24:06,140 --> 00:24:15,060
You know, a subset of software bugs or exploits and you use them to make the system do things you don't want it, it shouldn't be doing under normal circumstances.

274
00:24:16,400 --> 00:24:18,360
Or wasn't intended to be doing.

275
00:24:18,360 --> 00:24:18,880
Right.

276
00:24:18,880 --> 00:24:20,380
And sometimes they're useful.

277
00:24:21,740 --> 00:24:35,640
Would you consider a well-informed person building an attack against a machine learning system where they understand the machine learning system, they understand the feature extract and all that stuff, and they've built a false positive.

278
00:24:36,460 --> 00:24:41,180
Is that an adversarial attack against the machine learning or is that like an exploit?

279
00:24:42,120 --> 00:24:44,700
Like, where would you classify the mistake?

280
00:24:44,700 --> 00:24:50,020
Is it like a software mistake or a exploit or what?

281
00:24:51,580 --> 00:24:54,240
I think that's the tricky bit about ML models, right?

282
00:24:54,240 --> 00:24:57,180
I mean, they're inherently statistical.

283
00:24:59,620 --> 00:25:03,140
So it kind of gets back to testing after you retrain it, right?

284
00:25:03,140 --> 00:25:18,320
So if you have a single example and it turns out to be a false positive, and maybe that's one of the bridges that I feel like as a data scientist, I have to cross a lot when I'm talking to security specialists.

285
00:25:18,320 --> 00:25:25,380
If I have a model and it FPs on a single file, that doesn't call into question the efficacy of the model, right?

286
00:25:25,380 --> 00:25:28,560
You have to look at the statistics of what the model does, right?

287
00:25:28,560 --> 00:25:41,560
And if you say, well, yeah, it got this FP and that was a pretty boneheaded FP, but for 99% detection, you get one FP in 10,000 negative samples.

288
00:25:41,720 --> 00:25:43,380
That's a pretty good model.

289
00:25:43,520 --> 00:25:55,360
And so being able to sort of drive that switch from, hey, here's the good, bad to, hey, you have to think of this sort of heuristically and statistically.

290
00:25:55,980 --> 00:26:01,140
Is a big part of just like communicating as a data scientist, I think.

291
00:26:02,360 --> 00:26:03,940
Go ahead, Sarah.

292
00:26:03,940 --> 00:26:09,840
Yeah, if you've got somebody actively attacking, then maybe some pre-processing is enough to cut down the attacks.

293
00:26:09,840 --> 00:26:11,540
Yeah, yeah, for sure.

294
00:26:12,020 --> 00:26:15,860
All of that goes out the window when you've got someone maliciously trying to...

295
00:26:17,280 --> 00:26:34,980
Yeah, but so my question is, so if you have like a fancy ML from an academic paper bypass of your model, versus someone trying things, what would be...

296
00:26:37,100 --> 00:26:43,360
When is it like a machine learning mistake or versus a mistake in your featurizer and stuff?

297
00:26:44,040 --> 00:26:51,640
If you look at the decision boundary, basically for something and seeing how the perturbations relate to that?

298
00:26:51,900 --> 00:27:12,640
Yeah, I kind of feel like maybe to pull it back to this paper a little bit, if it gets through your feature extraction process in one shape, in one piece, then maybe it's a modeling issue.

299
00:27:12,640 --> 00:27:24,680
And if it crashes your feature extraction process or it gives you something that you wouldn't expect from your feature extraction process, then it's a featurization bug, right?

300
00:27:25,140 --> 00:27:30,740
If you feed garbage into a model, you can't blame the model for producing garbage outputs.

301
00:27:31,660 --> 00:27:33,020
How would you basically...

302
00:27:33,020 --> 00:27:36,360
What are some methods for quantifying unexpectedness in this scenario?

303
00:27:36,360 --> 00:27:40,660
Is it more like error counts or latencies in response?

304
00:27:40,660 --> 00:27:46,040
What would you pay attention to if you were trying to understand if something was misbehaving?

305
00:27:47,960 --> 00:27:49,720
You know what, when you see it.

306
00:27:49,820 --> 00:27:52,240
Yeah, like if you take a look in the paper, right?

307
00:27:52,240 --> 00:27:53,520
They've got an example.

308
00:27:53,520 --> 00:27:54,780
I want to find it.

309
00:27:55,440 --> 00:27:56,620
Figure two, right?

310
00:27:56,620 --> 00:27:59,740
So they show an example of two different images.

311
00:27:59,840 --> 00:28:07,320
One is an image of sort of the very top of a person's head and then the rest of it is gray.

312
00:28:07,580 --> 00:28:11,040
And one is a picture of a person's head.

313
00:28:11,040 --> 00:28:11,660
Yes, there you go.

314
00:28:11,660 --> 00:28:12,340
Thank you.

315
00:28:13,480 --> 00:28:15,340
Page six, if you're following along.

316
00:28:15,340 --> 00:28:26,740
Right, so OpenCV has a bug that these guys found by their guided fuzzing technique, which means that OpenCV can't properly load the file and then you run it...

317
00:28:26,740 --> 00:28:30,300
Obviously, you run it through feature extraction and it doesn't get anywhere.

318
00:28:30,300 --> 00:28:34,980
So it's like all these different individual components that you got to worry about, right?

319
00:28:36,300 --> 00:28:38,940
And I think on that topic, there was a...

320
00:28:38,940 --> 00:28:41,500
From Hacker Factor in Discord, he's got a question.

321
00:28:41,500 --> 00:28:46,740
Are you making a distinction between the ML model versus software that drives the ML system?

322
00:28:46,880 --> 00:28:49,020
He says fundamental error versus algorithmic.

323
00:28:49,020 --> 00:28:51,240
And I think here, right?

324
00:28:51,240 --> 00:28:53,900
This is what this paper is going after, right?

325
00:28:53,900 --> 00:29:00,600
It's saying the software that drives the ML system, the implementation of the system also has bugs and they find them here.

326
00:29:00,600 --> 00:29:04,840
And , you know, if you go to table two, right?

327
00:29:04,840 --> 00:29:09,020
They've got in OpenCV, they've got like two heap corruptions.

328
00:29:09,020 --> 00:29:11,180
They've got a weird rendering bug that they found.

329
00:29:11,180 --> 00:29:13,980
Page six, thank you, yeah.

330
00:29:14,880 --> 00:29:16,700
Right, so all these different...

331
00:29:16,700 --> 00:29:21,760
Like they literally have like heap corruption bugs that they're discovering with these funky inputs.

332
00:29:22,380 --> 00:29:24,440
And that has nothing to do with the model.

333
00:29:24,440 --> 00:29:28,220
That's you're attacking the plumbing around the model, which is...

334
00:29:28,220 --> 00:29:31,660
But you can use that plumbing to go and tweak the model.

335
00:29:32,480 --> 00:29:33,700
Right, exactly, yeah.

336
00:29:33,700 --> 00:29:36,740
So that's what the third line down does.

337
00:29:36,740 --> 00:29:39,320
That's the figure one, figure two?

338
00:29:39,320 --> 00:29:40,680
I forget, figure two.

339
00:29:40,680 --> 00:29:51,720
Yeah, so you break the rendering and then the rendering goes to the feature extraction and the feature extraction can't get anything because it's just got a bunch of gray pixels to look at.

340
00:29:51,720 --> 00:29:53,160
And then the model, right?

341
00:29:53,160 --> 00:29:54,040
Garbage in, garbage out.

342
00:29:54,040 --> 00:29:56,780
The model can't do anything because it doesn't have good features.

343
00:29:57,860 --> 00:30:01,260
So I'm going to try to open this up to Discord.

344
00:30:01,740 --> 00:30:09,140
If you are in the DevCon Discord, go to airvillage-general-voice.

345
00:30:09,760 --> 00:30:14,960
We'll try unmute you and get you so you can talk.

346
00:30:16,180 --> 00:30:22,080
If you're human plus, that's the correct way to do it, but we have a workaround.

347
00:30:22,480 --> 00:30:27,220
So head over to airvillage-general-voice to ask any questions.

348
00:30:27,360 --> 00:30:32,960
If you can't, we'll still be paying attention to the text chat in airvillage-journal-text.

349
00:30:35,900 --> 00:30:37,940
The moment of truth.

350
00:30:39,940 --> 00:30:42,260
I'm watching journal club text.

351
00:30:42,360 --> 00:30:43,980
Yeah, me too.

352
00:30:45,220 --> 00:30:48,080
Audio, managing audio is...

353
00:30:48,080 --> 00:30:50,280
I'm so glad I don't have to deal with it.

354
00:30:57,170 --> 00:31:05,690
I wish I had seen this paper before I started, like, really trying to solve adversarial machine learning as a security problem.

355
00:31:05,890 --> 00:31:07,340
Because there's a lot...

356
00:31:07,860 --> 00:31:13,540
this seems a lot easier, like, to attack for a lot of attack people.

357
00:31:14,240 --> 00:31:22,400
I think a lot of attackers have more knowledge on how to break a parser than how to break a neural network.

358
00:31:23,880 --> 00:31:32,640
So this, to me, feels like more of a realistic threat model of, like, your pipeline was... they broke your pipeline.

359
00:31:32,640 --> 00:31:34,400
You did something wrong in AWS.

360
00:31:34,400 --> 00:31:38,940
You did something wrong with your container setup.

361
00:31:38,940 --> 00:31:42,200
And it's parsing your images incorrectly now.

362
00:31:44,140 --> 00:31:52,100
Yeah, I mean, that's tons of the bugs that you find in sort of real systems, right, as parsing bugs.

363
00:31:52,100 --> 00:32:07,380
And yeah, a lot of the research right now focuses less on that and more on sort of these theoretical properties of machine learning models, which, I mean, I don't want to dig on those.

364
00:32:07,380 --> 00:32:08,200
They're fascinating, right?

365
00:32:08,200 --> 00:32:09,400
I love these papers.

366
00:32:09,400 --> 00:32:13,320
But they're also, again, keep coming back to this paper.

367
00:32:13,320 --> 00:32:20,420
This paper makes the really good point that these sort of theoretical models that have these nice properties are embedded in real world systems.

368
00:32:20,420 --> 00:32:26,680
And we know that real world systems are always kind of full of bugs that can be attacked and exploited.

369
00:32:27,380 --> 00:32:32,040
And so it gives you sort of multiple entries into the problem, right?

370
00:32:32,040 --> 00:32:42,360
On the one hand, you can exploit the preprocessor, and maybe that gives you some sort of heap corruption or some sort of like direct, you know, denial of service or even code execution.

371
00:32:42,500 --> 00:32:48,720
And on the other hand, you can break the feature extraction enough so that everything downstream is getting garbage data.

372
00:32:50,160 --> 00:32:51,600
Or mildly corrupted.

373
00:32:51,600 --> 00:32:57,480
I mean, I say again, with like the sneaking in, you want to do low and slows on some of this.

374
00:32:59,000 --> 00:33:02,520
Again, I am really like the idea of using this as defense.

375
00:33:02,520 --> 00:33:09,120
I mean, there's a whole bunch of applications where the bad guys are using machine learning based systems.

376
00:33:09,520 --> 00:33:12,020
I like the idea of breaking their systems.

377
00:33:12,160 --> 00:33:14,060
Yeah, as I would.

378
00:33:14,060 --> 00:33:15,200
Great hats on.

379
00:33:16,780 --> 00:33:26,170
Yeah, just like yesterday's paper with the Forbes masks.

380
00:33:27,390 --> 00:33:28,290
Mm-hmm.

381
00:33:28,690 --> 00:33:31,730
Yeah, so Will has a comment.

382
00:33:31,730 --> 00:33:34,350
He says, I'm surprised a paper is the vehicle for this knowledge.

383
00:33:34,350 --> 00:33:36,710
There's a whole security industry that has advice for this stuff.

384
00:33:36,710 --> 00:33:47,670
And I think, yeah, this is part of what we're trying to do, right, with AI Village is draw a link between security and machine learning, right?

385
00:33:47,670 --> 00:33:56,690
So we have a lot of like wicked smart security people who know about all about these implementation bugs.

386
00:33:56,690 --> 00:34:06,350
And then we've got a bunch of like really smart academics and machine learning people who know about like sort of these, you know, feature based attacks or weird properties of ML systems.

387
00:34:06,350 --> 00:34:15,270
And getting sort of that knowledge transfer going on so that we can find out where those intersect and how those two, you know, sort of sets of attacks inform each other.

388
00:34:15,270 --> 00:34:20,270
I think that's like a really rich area for future development.

389
00:34:20,270 --> 00:34:37,650
And it kind of makes me sad a little bit that this paper came out in 2017 and got comparatively little attention when it really is going right to the heart of a lot of really big questions in this, you know, in ML slash security.

390
00:34:37,650 --> 00:34:46,090
So in practice, you see a lot of data science teams disconnected from both like infrastructure application network security sort of teams.

391
00:34:46,090 --> 00:34:56,070
And so like the whole MLOps sort of formation of teams that can interface between data scientists and implementation has been emerging.

392
00:34:56,070 --> 00:35:02,250
But I'm curious what the panel thinks as far as what's the appropriate like role, a person, a cross-functional team?

393
00:35:02,250 --> 00:35:07,170
Do we want to have security people sitting with data scientists as these things are being developed?

394
00:35:08,270 --> 00:35:08,730
And...

395
00:35:08,730 --> 00:35:16,730
Yeah, I mean, weirdly enough, I used to be one of the people going into large companies working out where to put the data scientists.

396
00:35:16,750 --> 00:35:22,770
And there was always this argument between you want to embed the data scientists out into the rest of the team.

397
00:35:23,310 --> 00:35:26,550
And have them work with because you're informing the rest of the team everywhere.

398
00:35:27,710 --> 00:35:29,870
But then you've got data scientists out on their own.

399
00:35:29,870 --> 00:35:35,490
And we're kind of, you know, you see in this paper that we get ignored quite a bit when we're on our own.

400
00:35:35,670 --> 00:35:41,290
Or having these unicorn pens full of data scientists who talk to each other, but not really to anybody else.

401
00:35:41,810 --> 00:35:46,790
And there's this sense of you put people out, but you make sure they're still part of TRIPE.

402
00:35:46,790 --> 00:35:51,570
So you literally build TRIPE or whatever the good word now is for that.

403
00:35:51,570 --> 00:35:53,650
It used to be called TRIPE, but cross...

404
00:35:53,650 --> 00:35:54,510
Village?

405
00:35:54,610 --> 00:35:55,850
Cross, yeah, village.

406
00:35:56,450 --> 00:36:04,330
So you build villages for the data scientists to keep connected to each other, but they need to be out in the rest of the teams.

407
00:36:04,770 --> 00:36:08,270
Because otherwise, you know, our work is everywhere.

408
00:36:08,290 --> 00:36:09,630
We should be everywhere too.

409
00:36:10,350 --> 00:36:11,970
That's strongly our belief though.

410
00:36:12,510 --> 00:36:15,690
Yeah, I mean, I think it's, but it's got to be sort of a two-way street, right?

411
00:36:15,690 --> 00:36:32,210
On the one hand, yeah, there is kind of, for data scientists and sort of ML practitioners, I think that, yeah, we do sort of tend to stay in our own little bubble and don't think about these sort of externalities.

412
00:36:32,210 --> 00:36:38,190
I don't know what the right word for it is, but like these other ways, you know, it's not just going to be like fast, radiant, sign-based attacks.

413
00:36:38,190 --> 00:36:41,110
It's going to be like, no, someone broke the parsing.

414
00:36:41,110 --> 00:36:59,450
But then at the same time, and this is kind of going back to maybe a little bit of a hype panel yesterday, having people that aren't data scientists understand what the models can and can't do and what kind of behavior to expect from them so that they don't freak out when they see something weird happen and be like,

415
00:36:59,450 --> 00:37:01,710
oh, you know, whatever.

416
00:37:01,710 --> 00:37:04,430
We flagged this DLL as being malware when it's not.

417
00:37:04,430 --> 00:37:06,530
We're clearly under attack, right?

418
00:37:06,530 --> 00:37:08,250
And you're just like, it's statistics.

419
00:37:08,250 --> 00:37:09,730
Sometimes it makes mistakes.

420
00:37:10,590 --> 00:37:13,470
Well, just explaining it's not magic.

421
00:37:13,630 --> 00:37:15,330
It's hard, but it's not magic.

422
00:37:15,330 --> 00:37:16,030
Yeah, it's hard.

423
00:37:16,030 --> 00:37:16,570
It's useful.

424
00:37:16,570 --> 00:37:19,210
It does cool stuff, but it's a tool.

425
00:37:26,530 --> 00:37:34,590
Jager's talking about implementing a security development lifecycle and a secure software development lifecycle into model creation and upkeep.

426
00:37:34,750 --> 00:37:36,770
I'm going to make a data scientist cry.

427
00:37:38,570 --> 00:37:39,410
Yeah.

428
00:37:40,030 --> 00:37:50,730
I had issues with just being like, cool, we need to just write a few unit tests to make sure that the functions all kind of do what they're supposed to do.

429
00:37:51,390 --> 00:38:05,430
There's a bit of pushback from our more academically minded folk of just a little bit of basics like that, which may have solved some of these open CV issues.

430
00:38:09,550 --> 00:38:25,770
Interesting that if you basically treat the lifecycle of a data project as more of an internal product for a company and treat it as a product which serves a purpose, provides value, and then is integrated with other ways of developing software products,

431
00:38:25,770 --> 00:38:34,950
then data science and machine learning, they just become tools in the engineering toolbox, but then everything else falls within methods of continuous integration and testing and delivery.

432
00:38:34,950 --> 00:38:38,810
So then perhaps the risk, rather than the unicorn, depends on data scientists.

433
00:38:38,810 --> 00:38:39,370
Yeah.

434
00:38:39,770 --> 00:38:40,690
That's what we're doing.

435
00:38:40,690 --> 00:38:52,690
We're doing things like hypothesis-based development, which is like the next thing up from behavior-based development, which is next thing up for test-driven development.

436
00:38:52,910 --> 00:39:00,650
So you very much work out what the universe you'd like to see is, go do the maths on it, build the systems in it.

437
00:39:00,970 --> 00:39:03,090
So there are ways of working.

438
00:39:05,090 --> 00:39:06,350
Yeah, so maybe looping...

439
00:39:06,990 --> 00:39:07,850
Sorry, go ahead, Sven.

440
00:39:07,850 --> 00:39:08,790
Go ahead.

441
00:39:09,690 --> 00:39:18,410
Yeah, so I was going to loop back to the paper one more time and talk about the disclosure experience that they mentioned towards the end of the article.

442
00:39:18,830 --> 00:39:28,570
So basically, three of the vulnerabilities got new CVEs because they enabled arbitrary code execution or denial-of-service attacks, right?

443
00:39:28,570 --> 00:39:31,770
So those are both very firmly within the wheelhouse of what people...

444
00:39:31,770 --> 00:39:32,550
Page seven.

445
00:39:32,550 --> 00:39:33,490
Page seven, thank you.

446
00:39:34,270 --> 00:39:40,930
Very much within the wheelhouse of what security people think of as, oh yeah, that's a bug, that's an issue, that's a security gap that we need to fix.

447
00:39:40,930 --> 00:39:47,350
But there were a couple of other ones that they found where they could impact or manipulate the prediction.

448
00:39:47,350 --> 00:40:00,930
And then they call out in particular the mouthier memory corruption they found, where basically they can rewrite the feature vector, which allows you to basically make the model give you any output you want, essentially.

449
00:40:01,150 --> 00:40:06,150
Doesn't have to have anything to do with what the actual file that you were analyzing was.

450
00:40:06,250 --> 00:40:15,730
And those specifically are the ones that didn't get a CVE, didn't get called out, and mostly got labeled as won't fix.

451
00:40:15,730 --> 00:40:17,030
Or as won't fix.

452
00:40:17,030 --> 00:40:23,650
So that's sort of like, we were kind of bashing the data scientists earlier saying, oh, we got to think of this as part of a security system.

453
00:40:23,650 --> 00:40:25,770
But then it goes back the other way, right?

454
00:40:25,770 --> 00:40:32,570
We need to be able to communicate to people that as data scientists, as machine learning experts, look, this is bad, right?

455
00:40:32,570 --> 00:40:50,710
And how do we convince people that something that allows arbitrary misclassification or arbitrary predictions to be produced is as serious as, or maybe not as serious, but has some degree of severity, just like a remote code execution or a denial of service.

456
00:40:50,710 --> 00:40:54,190
I like the example that Eric brought up in his talk of a turtle rifle.

457
00:40:54,370 --> 00:40:57,930
Like that is a very effective way of communicating potential problems.

458
00:40:58,430 --> 00:41:08,510
Whereas for those who haven't seen it, like a turtle 3D printed object could be applied to a turtle to make it misclassified as a rifle.

459
00:41:08,570 --> 00:41:16,210
You can imagine like systems that track security and stuff to do certain events.

460
00:41:17,230 --> 00:41:19,290
I mean, it's like any other ethics discussion.

461
00:41:19,290 --> 00:41:27,390
We have to talk about consequences and then track it back and just keep pushing and pushing examples, real examples if we've got them.

462
00:41:28,830 --> 00:41:30,270
It's hard.

463
00:41:31,450 --> 00:41:40,990
Yeah, I mean, I guess like that's, you have a joke, but we're not really a joke, that like airline regulations are written in blood, right?

464
00:41:40,990 --> 00:41:47,710
And so when you've got all of these safety requirements, it's because a lot of people got hurt before someone was like, we ought to do something about that.

465
00:41:47,750 --> 00:41:58,390
And it would be nice to think that we could find a way to not have like ML disclosure requirements written in blood, right?

466
00:41:58,390 --> 00:42:13,290
Like maybe, you know, we're at a place now where we could maybe like jump ahead of the game a little bit and get an understanding of this out there so that we don't have to see people hurt before people are like, yeah, we should maybe take that seriously.

467
00:42:13,290 --> 00:42:17,010
I actually have a historical example for that too, for being old.

468
00:42:17,010 --> 00:42:23,900
So I ran one of the unmanned air vehicle safety teams back in the day before UAVs were sexy.

469
00:42:25,230 --> 00:42:34,810
And we didn't have the privilege, I guess, that the rest of the airspace industry had.

470
00:42:34,810 --> 00:42:38,090
So aerospace regulations are literally written in blood.

471
00:42:38,090 --> 00:42:44,690
So an aircraft would crash into something else or hit the ground and you would change the regulations.

472
00:42:44,710 --> 00:42:46,990
And that's literally how the regulations are written.

473
00:42:46,990 --> 00:42:48,210
Same with fire safety.

474
00:42:48,550 --> 00:42:51,270
Fire happens, change the regulations.

475
00:42:53,690 --> 00:43:05,570
And we were told explicitly and understood explicitly that we couldn't just put UAVs into manned airspace, have them crash into stuff and then rewrite the rules.

476
00:43:06,050 --> 00:43:13,090
We literally had to think all the scenarios and write the rules for how to do it safely.

477
00:43:13,610 --> 00:43:20,050
And it's whole space safety, not just individual aircraft safety before we were allowed to fly in the same spaces.

478
00:43:21,330 --> 00:43:23,330
So there's some precedent.

479
00:43:23,350 --> 00:43:28,490
I mean, there's maybe some looking in those spaces to see how it was done well.

480
00:43:28,830 --> 00:43:30,330
That's a good way to look.

481
00:43:31,670 --> 00:43:33,770
But a lot of it's just getting the will.

482
00:43:34,970 --> 00:43:38,270
Now, nothing in focus gets your attention like a whole bunch of airliners about to crash.

483
00:43:38,290 --> 00:43:43,770
But yeah, we could maybe get there before that point.

484
00:43:46,650 --> 00:43:51,950
So one more question from Discord, which is good because that's where I wanted to go next.

485
00:43:52,430 --> 00:43:53,750
So Yaga has a question.

486
00:43:53,750 --> 00:43:56,230
How do we force open source software to fix itself?

487
00:43:56,230 --> 00:43:57,210
We don't fix it ourselves.

488
00:43:57,210 --> 00:43:58,530
That's why it's open source.

489
00:43:58,570 --> 00:44:07,390
And the authors actually make a comment in the future work where they say that it's unclear who should be responsible for fixing them as well.

490
00:44:07,390 --> 00:44:15,170
So when they found a bug in the Malheur feature processing, that was because Malheur relied on libarchive.

491
00:44:15,610 --> 00:44:18,370
And the bug was actually in libarchive.

492
00:44:18,370 --> 00:44:24,090
And again, this gets back to this notion of like, you have all these different components that feed into the system.

493
00:44:25,130 --> 00:44:29,150
So even though the bug was in libarchive, it affected Malheur.

494
00:44:29,150 --> 00:44:33,090
So where do you put the responsibility for fixing it?

495
00:44:33,090 --> 00:44:46,210
And how do you convince, you know, like whoever's maintaining libarchive, look, this is serious enough that, you know, yes, I don't think it doesn't affect you directly, but it does affect this other system and it could be a critical impact.

496
00:44:46,210 --> 00:44:50,590
Like, how do we, how can we navigate that sort of uncertainty?

497
00:44:52,170 --> 00:44:59,150
How does Linux and Ubuntu and other open source software ecosystems do it?

498
00:45:02,130 --> 00:45:03,330
Do they have liability?

499
00:45:03,870 --> 00:45:12,710
I mean, I know for a lot of systems, the end users, especially if they're big end users, just get involved with the open source communities and work in there.

500
00:45:12,710 --> 00:45:22,730
But if you're using even like Python and your physical system crashes, who's responsible?

501
00:45:26,030 --> 00:45:26,570
Yeah.

502
00:45:26,570 --> 00:45:34,990
I mean, a lot of licensing argument, a lot of licensing agreements basically say like, yeah, you agree that you're using this at your own risk too.

503
00:45:35,030 --> 00:45:42,290
So, you know, basically everyone's going around not me, which, you know, it's fair, right?

504
00:45:42,290 --> 00:45:46,510
And people have, like, they have, they can't assume liability for everything that they do.

505
00:45:46,510 --> 00:45:52,630
But, you know, they can't agree to fix every single thing.

506
00:45:52,910 --> 00:45:55,110
But at some point, right, there's a balance, right?

507
00:45:55,110 --> 00:46:01,510
If something's widely adopted, it's widely used, and there's like a bug three steps up the chain that's causing it to behave weirdly.

508
00:46:01,910 --> 00:46:07,170
It feels like somebody ought to, there ought to be like some impetus to fix it somehow.

509
00:46:07,170 --> 00:46:20,830
The low tail of companies like Fortune 500 is more bothered to use these open source libraries than in theory, you know, this is AI, so you have a lot of, like, boost in productivity, like, and there's a lot of value that's being derived, and especially if they've started to use it for critical applications or decisions.

510
00:46:20,890 --> 00:46:33,430
So I think the onus and responsibility also needs to fall on the end users of, like, widely adopted packages, and then somehow making, making it very clear to decision makers in those companies that, hey, if you're using this to predict insurance rates or something and it fails,

511
00:46:33,430 --> 00:46:37,830
that's what you could potentially face, and then somehow channeling that into the projects themselves.

512
00:46:38,010 --> 00:46:44,270
Are you proposing that large companies that make a lot of money using open source tools should actually give back to the open source community?

513
00:46:48,330 --> 00:46:50,730
Smells like communism to me.

514
00:46:51,850 --> 00:46:55,550
It's kind of terrifying that we all just laughed when you said that.

515
00:46:57,390 --> 00:46:58,270
Yep.

516
00:46:58,330 --> 00:47:02,070
Sorry, I got to get my, got to get my commentary in once in a while.

517
00:47:03,110 --> 00:47:11,390
These are complex systems, and, like, just communicating the impact of some small, like, overflow bug to somebody who only cares about, like, the bottom line is a problem.

518
00:47:11,390 --> 00:47:16,630
Maybe there needs to be, like, awesome ML failures, GitHub repository, or something like that, where you can find it.

519
00:47:17,070 --> 00:47:18,810
We haven't had this.

520
00:47:18,950 --> 00:47:21,990
Sounds like, actually, a pretty good weekend project for someone.

521
00:47:21,990 --> 00:47:29,810
Yeah, I mean, like, you know, when we started up the ethics stuff years ago, it just, just takes some people determined to make this thing a thing, a visible thing.

522
00:47:29,950 --> 00:47:30,550
Yeah.

523
00:47:30,610 --> 00:47:31,630
Hey, community.

524
00:47:32,030 --> 00:47:33,550
Anyone want to take this one on?

525
00:47:35,130 --> 00:47:39,510
So, how many, like, solid examples of ML failure do we have?

526
00:47:41,470 --> 00:47:42,130
If we want to...

527
00:47:42,130 --> 00:47:43,230
I mean, a lot of it is.

528
00:47:43,310 --> 00:47:44,710
Compile those, yeah.

529
00:47:44,710 --> 00:47:46,410
How much do we actually know?

530
00:47:46,670 --> 00:48:01,170
Yeah, that's, I want to say, Andrew Davis over in different Discord had a pretty good comment on it, which is that a lot of time when ML fails, it kind of fails silently, right?

531
00:48:01,170 --> 00:48:12,990
It just, you don't flag something, and it comes back to a lot of the earlier discussion about is it a bug, or is it, like, normal, just sort of, like, statistical failure, right?

532
00:48:12,990 --> 00:48:20,770
A 1% failure rate means that if it's not failing 1% of the time, you should be vaguely surprised, right?

533
00:48:20,830 --> 00:48:29,570
So, yeah, like, I think it's possible that there's a lot of ML failures that we just kind of assume are in the statistical noise.

534
00:48:29,570 --> 00:48:36,550
A lot of them are probably kept quiet because, you know, there's potentially, like, PR risks or PR damage, right?

535
00:48:36,550 --> 00:48:42,370
Or, you know, like, like, actual, like, gross, you know, liability attached to it.

536
00:48:43,150 --> 00:48:50,730
Yeah, I mean, it's an open question how much is actually sort of flying beneath the radar out there.

537
00:48:50,790 --> 00:49:04,410
I mentioned a rental company saying, or a background check company who uses ML going, so we didn't give a lot of people their approved, you know, their credit check approval because of a mistake.

538
00:49:04,410 --> 00:49:08,870
And we want to publicly apologize for that when it's an ML mistake.

539
00:49:08,870 --> 00:49:14,830
And they can just be like, well, the model just was, yeah, sweep it under the rug, fix it quietly, never mention it.

540
00:49:17,390 --> 00:49:19,650
Like, that would be an ethical nightmare.

541
00:49:21,050 --> 00:49:27,590
Yeah, I mean, there's lots of startups that have proposed to do things.

542
00:49:27,590 --> 00:49:37,750
There was one, what was it, about two, three years ago, oh, we'll use ML to pre-screen your babysitters by digging through their social media profiles and stuff.

543
00:49:38,050 --> 00:49:49,090
And I have my own opinions on how likely that was to succeed versus how likely it was to simply just recommend white people.

544
00:49:49,090 --> 00:49:57,230
But how do you even measure or quantify failures in that case?

545
00:49:57,230 --> 00:50:08,010
Because it's all going to be completely internal and all you can do is essentially try and get a proxy model to be like, aha, here are the features, right?

546
00:50:08,010 --> 00:50:12,610
I've submitted a thousand different profiles and here are the features that seem to be triggering, right?

547
00:50:12,610 --> 00:50:16,410
And you have to affirmatively dig for it, right?

548
00:50:16,410 --> 00:50:20,170
Those failures aren't going to be obvious unless you're actually doing something like that.

549
00:50:21,170 --> 00:50:31,330
A recent streak of sort of facial recognition failures that lead to characteristics that elicits a visceral gut reaction like, yes, this is wrong.

550
00:50:31,730 --> 00:50:39,450
So I think in some cases, just seeing the outcomes, like Claire said, helps understand whether it's a failure or not.

551
00:50:39,670 --> 00:50:43,130
This gets back to the bias question.

552
00:50:43,130 --> 00:50:49,070
Like, how do you tell if something's biased or just like for this one instance, it was wrong?

553
00:50:49,070 --> 00:50:49,890
Oh, man.

554
00:50:51,530 --> 00:50:52,890
Yeah, it kind of works.

555
00:50:52,890 --> 00:50:54,710
We're going to have to clean that one out, Tommy.

556
00:50:54,770 --> 00:50:55,170
Yeah .

557
00:50:55,170 --> 00:50:58,210
In images is, oh, God.

558
00:50:58,910 --> 00:51:01,690
How much longer do we have for this conversation?

559
00:51:02,530 --> 00:51:04,550
Yeah, we can solve it in eight minutes.

560
00:51:04,550 --> 00:51:05,070
We've got eight minutes.

561
00:51:05,070 --> 00:51:06,130
Yeah, yeah, no problem.

562
00:51:06,170 --> 00:51:13,690
Yeah, I mean, the problem in a lot of the bias stuff, right, is like the terms aren't even defined sometimes.

563
00:51:13,690 --> 00:51:22,030
So like the example I keep coming back to is like recidivism and like predicting re-arrest rates, right?

564
00:51:22,030 --> 00:51:26,670
And someone might point out, hey, look, this is biased, right?

565
00:51:26,670 --> 00:51:30,790
It's clearly predicting re-arrest rates that are much higher for this group, right?

566
00:51:30,790 --> 00:51:34,090
And one argument would be, well, that's clearly wrong, right?

567
00:51:35,150 --> 00:51:40,130
It's biased because it's trained on biased data.

568
00:51:40,130 --> 00:51:44,270
And the counter argument would be like, look, it's reproducing the data, right?

569
00:51:44,270 --> 00:51:46,090
We don't live in a perfect society.

570
00:51:46,090 --> 00:51:47,570
The data is what it is.

571
00:51:47,570 --> 00:51:52,530
And within that box, the ML has done as much as you can ask it to do.

572
00:51:53,290 --> 00:51:59,090
And so it always boils down to these normative questions.

573
00:51:59,090 --> 00:52:12,070
And this is my own totally personal opinion that talking about unbiased systems, unbiased ML is almost inherently impossible, right?

574
00:52:12,070 --> 00:52:24,690
It's a question of how is it producing the kinds of outcomes that you want and how is the ML system moving power around, right?

575
00:52:24,690 --> 00:52:25,650
Who is it empowering?

576
00:52:25,650 --> 00:52:27,080
Who is it disenfranchising?

577
00:52:27,890 --> 00:52:31,470
And who is it helping and who is it hurting?

578
00:52:31,470 --> 00:52:51,730
And chasing after some vague notion of fairness, a fair system or an unbiased system is kind of like a distraction from more fundamental questions of who's being helped, who's being hurt, what's being reinforced or whose concerns are being downplayed.

579
00:52:52,010 --> 00:52:57,450
And who's accountable, I guess, also because of the reference recent Twitter story between Timothy Kimbrough and Yann LeCun.

580
00:52:57,450 --> 00:53:02,210
Basically, is it the ML engineer or is it actually the data scientist or is it throughout the entire chain?

581
00:53:02,410 --> 00:53:02,950
Yep.

582
00:53:04,630 --> 00:53:05,590
Yeah, and I mean...

583
00:53:06,370 --> 00:53:06,930
Sorry, go ahead.

584
00:53:06,930 --> 00:53:07,830
I'm talking too much.

585
00:53:07,830 --> 00:53:14,270
I'm seeing over in the chat that we already have awesome ML failures in the AI village as a repo.

586
00:53:14,270 --> 00:53:19,030
And Stellar Athena seems to be getting in there already.

587
00:53:19,710 --> 00:53:20,690
Outstanding.

588
00:53:20,690 --> 00:53:25,710
Suggestions for implementation failures, ethical failures, HCI failures and security failures.

589
00:53:26,010 --> 00:53:27,730
You got that Stellar?

590
00:53:28,130 --> 00:53:28,930
Yeah.

591
00:53:28,930 --> 00:53:29,790
Yeah, I think...

592
00:53:30,250 --> 00:53:32,170
Do we have a category for...

593
00:53:32,170 --> 00:53:36,570
Because we were playing around with this cat does not exist a couple days ago.

594
00:53:36,570 --> 00:53:38,270
I think we maybe need a category.

595
00:53:38,270 --> 00:53:40,210
This ML should not exist.

596
00:53:40,370 --> 00:53:41,810
Yeah, nightmare fuel.

597
00:53:42,810 --> 00:53:45,090
Yeah, well, I mean, I'm thinking of things like...

598
00:53:45,090 --> 00:53:45,590
Yeah , you're right.

599
00:53:45,590 --> 00:53:49,070
There are a whole bunch of machine learning systems that just should not be on this planet.

600
00:53:49,190 --> 00:53:49,730
Yeah.

601
00:53:49,790 --> 00:53:54,650
And anyone asked to contribute to it should have been like, perhaps not.

602
00:53:56,370 --> 00:53:58,810
Yeah, as far as the responsibility...

603
00:53:58,810 --> 00:54:01,790
And we would never have had Facebook if anyone had that call.

604
00:54:01,910 --> 00:54:02,590
Anyway.

605
00:54:02,590 --> 00:54:03,810
I mean, hindsight.

606
00:54:03,810 --> 00:54:11,870
But you can look at really short-term stuff like even just like genderify, right?

607
00:54:11,870 --> 00:54:18,130
It was should have been pretty obvious to everyone involved that this was just like a losing proposition from the start.

608
00:54:18,930 --> 00:54:22,350
Yeah, and the recidivism through face images stuff from China.

609
00:54:22,350 --> 00:54:23,070
Oh , God.

610
00:54:23,070 --> 00:54:23,390
Yeah.

611
00:54:23,390 --> 00:54:24,470
Okay, so...

612
00:54:24,470 --> 00:54:26,070
Way to replicate somebody's bias.

613
00:54:26,070 --> 00:54:31,190
So we do have a workshop coming up in a couple of minutes.

614
00:54:31,190 --> 00:54:34,550
So I think I want to kind of like put a pin in this.

615
00:54:34,550 --> 00:54:42,550
I think if everyone's enjoying this conversation, having an awesome ethics in AI panel, which is coming up in a couple of hours.

616
00:54:42,550 --> 00:54:44,850
I'm not sure someone fill me in.

617
00:54:45,030 --> 00:54:46,510
Two o'clock, two o'clock.

618
00:54:46,590 --> 00:54:47,530
Thank you.

619
00:54:47,530 --> 00:54:52,750
So everyone should absolutely turn, tune in for that if they're enjoying this discussion.

620
00:54:53,210 --> 00:55:04,150
But yeah, I guess I want to just like go around just to wrap this up and see if we have any like last minute, like big idea takeaways from this paper that people thought were really, really cool.

621
00:55:04,990 --> 00:55:06,630
I love the way they did it.

622
00:55:06,650 --> 00:55:11,250
The idea of attacking the system and looking at taking it apart is just kind of my thing.

623
00:55:11,250 --> 00:55:17,510
And I like the idea of using it to attack systems that shouldn't exist.

624
00:55:18,310 --> 00:55:19,250
Yep.

625
00:55:19,430 --> 00:55:20,390
Sven?

626
00:55:21,050 --> 00:55:24,670
I really like the idea of attacking the feature extractors.

627
00:55:24,670 --> 00:55:34,030
And I think we should also add like bypassing the feature extractors by just doing something you know the ML system is going to miss.

628
00:55:35,110 --> 00:55:40,010
Because you can understand it well to this whole thing.

629
00:55:41,030 --> 00:55:41,610
Cool.

630
00:55:41,610 --> 00:55:42,250
Martin?

631
00:55:43,310 --> 00:55:45,970
Yeah, that also really enjoyed the paper and discussion.

632
00:55:46,190 --> 00:55:53,010
I think to me what was clear is that it can occur at many levels of the basically stack.

633
00:55:53,010 --> 00:55:57,390
So just instrumentation and stuff become very, very important to monitoring what happens.

634
00:55:57,390 --> 00:56:07,390
And I would really like to see follow on work that applies this to TensorFlow, PyTorch, basically the learning libraries and seeing how you can play with intermediate representations and things like that too.

635
00:56:08,410 --> 00:56:09,630
Yeah, for sure.

636
00:56:09,630 --> 00:56:20,770
Yeah, the reminder that like these ML things, ML projects don't exist in some abstract theoretical space and are implemented in real code and real systems that come with their own flaws.

637
00:56:21,190 --> 00:56:28,250
Yeah, like you know it but being reminded of it sort of viscerally by, look, we have CVEs is always a great thing.

638
00:56:28,530 --> 00:56:29,970
So cool.

639
00:56:30,050 --> 00:56:36,150
So I think this is we probably want to leave a few minutes for people to fight with the audio as we transition over to the workshop.

640
00:56:36,170 --> 00:56:40,230
So I think we want to call that a panel or a discussion.

641
00:56:40,230 --> 00:56:41,050
So awesome.

642
00:56:41,050 --> 00:56:41,890
Thanks, everyone.

643
00:56:41,890 --> 00:56:42,330
Thanks, Rich.

644
00:56:42,330 --> 00:56:43,170
One second.

645
00:56:43,450 --> 00:56:47,890
Before we jump off, we have one on Wednesday at 5 p.m.

646
00:56:47,890 --> 00:56:49,030
Pacific time.

647
00:56:50,510 --> 00:56:55,310
And we have the, I'm guessing, I think we should go with your paper, Rich.

648
00:56:56,310 --> 00:56:57,510
The slide deck on...

649
00:56:58,770 --> 00:57:07,470
Yeah, the anomaly detection outside the closed world on using machine learning for network confusion detection by Robin Subban and Vern Paxson.

650
00:57:08,050 --> 00:57:13,710
So we'll post that in the Defcon Discord and also be posted to our Discord and Twitter.

651
00:57:13,710 --> 00:57:17,930
So if you want to participate, you should find your way over to our Discord.

652
00:57:17,930 --> 00:57:24,530
We can't post a link to that in the Defcon Discord, but the links are places and you can just DM us for an invite.

653
00:57:24,770 --> 00:57:29,610
And we'll be discussing that for probably an hour and a half on Wednesday.

654
00:57:29,610 --> 00:57:33,650
And we might be able to get one of the authors in, even though this is a 10-year-old paper.

655
00:57:36,050 --> 00:57:37,750
How far we have come.

656
00:57:39,130 --> 00:57:40,010
All right.

657
00:57:40,010 --> 00:57:40,730
Thanks, everyone.

658
00:57:40,730 --> 00:57:41,390
Catch you later.