1
00:00:01,000 --> 00:00:03,040
Hello?

2
00:00:13,060 --> 00:00:16,060
Hey, how are you doing?

3
00:00:19,100 --> 00:00:21,140
Sorry?

4
00:00:30,640 --> 00:00:35,540
I can stream this thing easily to Twitch chat.

5
00:00:41,040 --> 00:00:42,280
Okay.

6
00:00:42,280 --> 00:00:44,360
I don't care if we do this or Zoom.

7
00:00:44,960 --> 00:00:46,080
Whichever.

8
00:00:46,080 --> 00:00:47,300
Whichever works.

9
00:00:52,790 --> 00:00:59,610
It's funny, I was talking about the Fox paper with a co-worker of mine on Friday.

10
00:00:59,870 --> 00:01:03,270
We were wondering about reverse engineering it.

11
00:01:03,270 --> 00:01:05,250
Is the author currently on?

12
00:01:05,790 --> 00:01:06,510
No.

13
00:01:06,510 --> 00:01:07,310
Do you know the author?

14
00:01:07,310 --> 00:01:09,010
Do you want to ping him quickly?

15
00:01:09,770 --> 00:01:10,370
No.

16
00:01:10,690 --> 00:01:15,790
I mean, I think I've met him once.

17
00:01:15,790 --> 00:01:20,470
I did my undergrad at Chicago, but I don't know any of them.

18
00:01:20,970 --> 00:01:27,270
By the way, we're also live on Twitch with MyUglyMug as the only visual reference.

19
00:01:27,790 --> 00:01:28,430
Gotcha.

20
00:01:28,430 --> 00:01:29,510
Hello, Twitch.

21
00:01:30,350 --> 00:01:35,510
So if you want to participate in this journal club, the link is in both...

22
00:01:36,270 --> 00:01:39,010
This is the experimental side of the village.

23
00:01:39,010 --> 00:01:41,450
So the link is in Twitch chat.

24
00:01:41,450 --> 00:01:43,910
The link is also in the Discord.

25
00:01:43,910 --> 00:01:55,490
If you are interested in using Adverse Fields examples productively and want to talk about that, join the AI village general voice and just ask away.

26
00:01:55,490 --> 00:02:01,090
It would be nice if you've read the paper, but don't worry about that.

27
00:02:01,090 --> 00:02:14,950
If you want to talk about this principle, like, you know, adversarial examples for good sort of thing, just come.

28
00:02:15,470 --> 00:02:16,990
We'd love to hear from you.

29
00:02:26,160 --> 00:02:30,700
So there are multiple ways for you to get on here.

30
00:02:34,420 --> 00:02:35,880
Hey, Rich.

31
00:02:39,460 --> 00:02:40,520
Hey.

32
00:02:40,940 --> 00:02:44,680
For too long.

33
00:02:44,900 --> 00:02:46,300
Yeah, welcome.

34
00:02:55,510 --> 00:02:57,110
Where are they?

35
00:02:58,630 --> 00:03:06,970
So I'm in the process of moving and I discovered my COVID panic beans.

36
00:03:09,140 --> 00:03:15,620
You know, when the whole thing was starting, I bought a bunch of beans because I like making bean curries.

37
00:03:15,620 --> 00:03:20,160
And I was worried that beans would be a little bit hard to find.

38
00:03:22,840 --> 00:03:28,020
And as I was moving, I found my giant bag of COVID panic beans.

39
00:03:30,980 --> 00:03:34,700
Isn't that like exactly what we were told to not do?

40
00:03:34,700 --> 00:03:37,540
I mean, I only...

41
00:03:38,700 --> 00:03:40,620
I use about two cups of that.

42
00:03:40,620 --> 00:03:42,700
So that would be like...

43
00:03:42,700 --> 00:03:51,240
If I was eating bean curries two, three times a week, that's about three weeks of beans.

44
00:03:52,200 --> 00:03:55,000
Some people bought like six months of beans.

45
00:03:58,290 --> 00:04:07,430
There was a while we actually had to get beans, flour and yeast essentially off of basically like the black market.

46
00:04:07,430 --> 00:04:17,970
It was a local company that was buying it in bulk and then like repackaging it and reselling it under the table.

47
00:04:19,830 --> 00:04:26,190
Nothing like people exploiting a pandemic to make a quick buck.

48
00:04:26,190 --> 00:04:28,650
That's true American ingenuity right there.

49
00:04:28,650 --> 00:04:30,750
Yeah, it's capitalism.

50
00:04:33,550 --> 00:04:36,950
This is the part where we say, yay, capitalism, right?

51
00:04:40,330 --> 00:04:41,890
Capitalism is the problem.

52
00:04:42,370 --> 00:04:45,590
Sometimes I get my speaking points confused.

53
00:04:47,690 --> 00:04:48,770
Yeah.

54
00:04:50,150 --> 00:04:53,590
Let's see how... well...

55
00:04:59,130 --> 00:04:59,850
How

56
00:05:08,700 --> 00:05:09,420
many...

57
00:05:09,420 --> 00:05:13,460
All right.

58
00:05:13,460 --> 00:05:26,560
So for everybody on Twitch that is new to Drone and Phone, basically what this is, is we run this once a week on Wednesdays usually at five o'clock Pacific Standard Time.

59
00:05:26,560 --> 00:05:28,560
Or 8 p.m.

60
00:05:28,640 --> 00:05:30,460
East Coast Time.

61
00:05:32,100 --> 00:05:35,060
And we have a paper that we're discussing.

62
00:05:35,060 --> 00:05:37,900
We usually get way off topic very quickly.

63
00:05:41,000 --> 00:05:47,800
And oftentimes we have the authors on and we basically talk about the paper and the surrounding topics.

64
00:05:53,060 --> 00:05:56,940
And it's a really interesting... they usually have really interesting discussions.

65
00:05:56,940 --> 00:06:03,580
There's some like motifs because of our... from our perspectives, like threat models.

66
00:06:03,580 --> 00:06:09,520
Whenever we have an academic paper with a threat model in it, we usually rag on it for a bit.

67
00:06:09,820 --> 00:06:12,000
Unless the threat model is really good.

68
00:06:14,520 --> 00:06:23,180
Because if you've never actually worked in security, building a threat model from a purely academic perspective is hard.

69
00:06:23,180 --> 00:06:27,840
And they do kind of... there's a lot of people who mess that up.

70
00:06:27,920 --> 00:06:30,560
But they at least try and try to figure it out.

71
00:06:30,560 --> 00:06:36,680
And when we have authors on, we'll talk about that all.

72
00:06:40,200 --> 00:06:44,980
And if you want to join us today, it's free, easy, open.

73
00:06:45,000 --> 00:06:49,340
You can come jump on the Discord voice chat.

74
00:06:49,340 --> 00:06:52,380
So we have AI Village General Voice.

75
00:06:52,620 --> 00:07:01,720
If you've got any questions about using adversarial examples to defend against facial recognition systems, please come jump on.

76
00:07:03,220 --> 00:07:07,580
Stella, since you've got like... you're invested in this paper a bit.

77
00:07:07,580 --> 00:07:12,440
Do you want to describe what it goes over and give us an overview of what's going on?

78
00:07:12,860 --> 00:07:13,820
Okay, sure.

79
00:07:13,820 --> 00:07:16,800
So is the author going to join us?

80
00:07:16,880 --> 00:07:19,680
Or does it look like no right now?

81
00:07:19,680 --> 00:07:22,220
No, it doesn't look like the author is going to join us.

82
00:07:23,720 --> 00:07:27,180
Someone had a note that they were still trying to get a hold of him.

83
00:07:27,180 --> 00:07:29,640
So that's the last update I heard.

84
00:07:29,640 --> 00:07:31,740
I don't think we're going to get him, unfortunately.

85
00:07:31,940 --> 00:07:34,460
Okay, gotcha.

86
00:07:34,600 --> 00:07:40,360
All right, so the paper we're discussing is called Fox Protecting Privacy Against Unauthorized Deep Learning Models.

87
00:07:40,360 --> 00:07:43,140
I hope I'm in the right talk, right?

88
00:07:43,140 --> 00:07:44,400
Yeah, that's the one.

89
00:07:45,640 --> 00:07:55,240
So the background concept of this paper is that there are a lot of ways that people upload photos to the internet.

90
00:07:55,240 --> 00:08:05,220
And companies like Facebook, like Google, process your photo through a machine learning algorithm and use it to identify you.

91
00:08:05,320 --> 00:08:09,100
And they can use this to identify you in future photos.

92
00:08:09,180 --> 00:08:16,960
A couple years ago, Facebook rolled out an automatic tagging process, where when you upload a new photo, it tags your friends in it automatically.

93
00:08:17,140 --> 00:08:19,300
And this creeped out a lot of people.

94
00:08:19,380 --> 00:08:23,480
They were surprised that Facebook was able to immediately identify who they were.

95
00:08:23,480 --> 00:08:41,360
And so the conceit of this paper is to give people a defense, a way to opt out of this kind of analysis without necessarily opting out of existing on the internet as a person.

96
00:08:41,360 --> 00:09:09,360
Another important connection that the authors draw specifically is Clearview AI, which is a now infamous company that did analytics drawing on what was technically speaking public data to draw all sorts of conclusions about people, where the users were not particularly comfortable with the amount of information that Clearview AI was able to obtain about them.

97
00:09:09,360 --> 00:09:15,380
And this is kind of a repeating motif in public use of technology.

98
00:09:15,380 --> 00:09:18,200
Nobody reads Terms of Service.

99
00:09:18,200 --> 00:09:23,100
Nobody has any idea what data agreements they're accepting and stuff like that.

100
00:09:23,340 --> 00:09:27,460
And even if you did, Facebook has a monopoly on being Facebook.

101
00:09:27,460 --> 00:09:29,060
Twitter has a monopoly on being Twitter.

102
00:09:29,060 --> 00:09:38,180
You either go along with whatever they demand of you, or you just don't exist on Twitter.

103
00:09:38,180 --> 00:09:46,860
And it's not like there's a Twitter 2 that you can go on to and have the same sort of experience, just not with the people whose data policies you don't like.

104
00:09:48,240 --> 00:10:03,400
So an idea that's been incubating over the past year or so in the adversarial AI community is, hey, defeating algorithms like this is exactly what adversarial AIs are supposed to do.

105
00:10:04,160 --> 00:10:18,880
Adversarial AI, or adversarial examples, I should say, is supposed to take photos, transform them in a way that they look normal to humans, but they can't be processed by AI algorithms correctly.

106
00:10:18,880 --> 00:10:30,000
So instead of using these things to fool... these things are usually conceived in the literature as a negative, as a bad thing.

107
00:10:30,000 --> 00:10:33,360
They're used to fool AI algorithms and do harm.

108
00:10:33,940 --> 00:10:37,980
But in terms of data privacy, you can view them as a very positive thing.

109
00:10:37,980 --> 00:10:47,820
You can take an image of yourself and transform it so that AIs that look at it don't think that you're a person, or think that you're a person who you're not.

110
00:10:47,820 --> 00:10:54,560
And then when you upload it, that photo that's been transformed onto the internet, it looks like you to humans.

111
00:10:54,740 --> 00:10:59,040
That's kind of the whole point, but it doesn't look like you to AI algorithms.

112
00:10:59,540 --> 00:11:06,120
And so that's the core idea at the center of this paper, and that's ultimately what they try to achieve.

113
00:11:06,400 --> 00:11:12,160
And what they do achieve with their approach.

114
00:11:12,460 --> 00:11:15,120
In the paper, they refer to it as a cloak.

115
00:11:33,760 --> 00:11:36,140
Yeah, that sounds about right.

116
00:12:30,420 --> 00:12:37,200
To cloak you, but they're going to cloak you in a way that makes you look like an image from this reference data set.

117
00:12:37,720 --> 00:12:42,380
So they're going to make you look like another person or look like another object.

118
00:12:42,900 --> 00:12:53,440
And the reason that they give for this in the paper is that they find that this helps it get past screening algorithms.

119
00:12:53,440 --> 00:12:58,860
That try to notice and reject adversarial examples.

120
00:12:59,580 --> 00:13:06,340
Typically, when you use adversarial examples, you're inserting a lot of noise, and it's not anything in particular.

121
00:13:06,420 --> 00:13:09,540
It just happens to make you look different to the AI.

122
00:13:09,840 --> 00:13:16,040
But what they're doing here is trying to inject some kind of pattern thing that makes me look like Gwyneth Paltrow.

123
00:13:16,040 --> 00:13:25,880
And then once it's made me look like Gwyneth Paltrow, the idea is that the algorithms are supposed to detect when this is going on.

124
00:13:25,880 --> 00:13:32,340
Aren't going to notice because they're going to say this looks like a non-adversarial photo of Gwyneth Paltrow.

125
00:13:34,340 --> 00:13:50,020
I'm kind of salty about this because I spent a huge amount of time last year before the talk I did on facial recognition trying to come up with a single universal adversarial vector that would make one model think everyone was John Malkovich.

126
00:13:50,860 --> 00:13:52,540
I'm super salty about this.

127
00:13:52,540 --> 00:13:56,160
I've literally written two-thirds of a paper on this exact topic.

128
00:13:56,340 --> 00:13:59,400
Yeah, definitely an idea that's in the air.

129
00:13:59,400 --> 00:14:03,100
No, yeah, this has been in the air for a long time.

130
00:14:03,760 --> 00:14:13,220
There's a, from an ethics point of view, because a lot of the framework for this is ethically oriented, you know, giving users control over their data.

131
00:14:13,220 --> 00:14:22,430
There's a really good paper from either earlier this year or last year called The Ethics of Adversarial... The Politics of Adversarial Examples.

132
00:14:23,620 --> 00:14:30,470
That kind of discusses the fact that the AI literature like almost universally treats adversarial examples as a bad thing.

133
00:14:30,880 --> 00:14:37,860
And how that's not really fair and discusses different ways that one could use them to achieve good.

134
00:14:37,860 --> 00:14:44,100
Though it's a it's a ethics policy oriented paper and it doesn't do any of them.

135
00:14:44,160 --> 00:14:48,860
So yeah, this is definitely an idea that's been around the block a couple times.

136
00:14:53,060 --> 00:14:54,240
For sure.

137
00:14:54,280 --> 00:14:58,720
I know a lot of people who have had thoughts along these lines.

138
00:14:59,660 --> 00:15:05,020
Yeah, I want to say there was a talk at, there was actually a talk at The Village last year.

139
00:15:05,020 --> 00:15:08,100
I'm drawing a complete blank on what they called it.

140
00:15:08,220 --> 00:15:15,460
I think they were specifically targeting Facebook's model and they were doing something a bit different.

141
00:15:15,460 --> 00:15:20,920
They were basically breaking the, they were basically breaking like the facial detection step.

142
00:15:20,920 --> 00:15:32,880
So, you know, skipping the, so the point where the system would actually like put the bounding box and clip out the face tile around the face so that like nothing looked like a face at all in it.

143
00:15:32,880 --> 00:15:39,000
And so it wouldn't even go to the next step of like finding the vector for the face and then doing the embedding and then trying to match it.

144
00:15:39,680 --> 00:15:48,840
Yeah, because a lot of a lot of these algorithms use some kind of bounding boxes or try to try to identify, you know, images are complex.

145
00:15:48,840 --> 00:15:53,580
So you kind of have to first pick out where there's something that you're going to look at before you do that.

146
00:15:53,580 --> 00:16:02,200
I don't think I've ever read that paper, but I assume you're talking about like disrupting its ability to find bounding boxes or what faces are.

147
00:16:02,200 --> 00:16:02,980
Yeah.

148
00:16:02,980 --> 00:16:08,380
So most, most of these facial recognition systems go through like a three step process, right?

149
00:16:08,380 --> 00:16:14,020
You, you first, you draw bounding boxes around the face and you get a lot of the time.

150
00:16:14,020 --> 00:16:18,720
Most of them, I think you still, still use like landmark based approaches.

151
00:16:18,900 --> 00:16:23,220
Then you use the landmarks to sort of like rotate and realign the face.

152
00:16:23,220 --> 00:16:27,480
So it's sort of like dead on and in a standard orientation for those faces.

153
00:16:27,480 --> 00:16:35,460
And then you put it through the actual facial recognition model, which will take the face and embed it into a vector space.

154
00:16:35,460 --> 00:16:38,260
And then you find like nearest matches within the vector space.

155
00:16:38,260 --> 00:16:40,360
And that's the actual recognition step.

156
00:16:40,420 --> 00:16:42,280
So this is what these guys are going after.

157
00:16:42,280 --> 00:16:44,220
They're saying, no, take this face.

158
00:16:44,380 --> 00:16:49,360
We'll let you find the face, but now we're going to make, you know, we're going to make Stella look like Gwyneth Paltrow.

159
00:16:49,360 --> 00:16:55,360
And what the guys at AI Village last year did was they said, we're not going to even let you find the face, right?

160
00:16:55,360 --> 00:17:00,180
It's going to look like, you know, like a landscape with no faces in it or something like that.

161
00:17:00,180 --> 00:17:01,540
But it was the same idea, right?

162
00:17:01,540 --> 00:17:07,860
It's trying to find like the minimal, like least disturbing perturbation you can use.

163
00:17:08,420 --> 00:17:13,680
That would still like, you know, so your picture looks nice and you can share it on the internet.

164
00:17:13,680 --> 00:17:20,080
But, you know, either you can't find any faces in it or you do find faces, but they're tagged as the wrong people.

165
00:17:20,080 --> 00:17:23,220
Yeah, that is very similar.

166
00:17:23,220 --> 00:17:26,900
And I think you just hit on something that's important to call out specifically.

167
00:17:26,960 --> 00:17:31,560
You know, there are a lot of different types of data that might benefit from this sort of masking.

168
00:17:31,560 --> 00:17:35,820
Here, they're using faces and they're doing that for a very important reason.

169
00:17:35,960 --> 00:17:46,200
Adversarial examples for faces seem to be... their performance is not better than it is on other kinds of data.

170
00:17:46,200 --> 00:17:51,450
But for facial analytics, you can make very difficult to perceive to human ones.

171
00:17:51,820 --> 00:18:01,340
So an adversarial example for images and especially for facial analytics looks to a human like you haven't edited the image at all.

172
00:18:01,340 --> 00:18:08,940
When you talk about like audio adversarial examples, like Carlini has a kind of the first major paper on that.

173
00:18:08,940 --> 00:18:12,640
You can hear in the background that there's a hum.

174
00:18:13,240 --> 00:18:17,500
And as the strength of the adversarial example increases, the hum increases.

175
00:18:17,500 --> 00:18:28,540
And I've played around with these and, you know, even at the loudest, the kind of noise in the background is, you can still clearly hear what's going on.

176
00:18:29,240 --> 00:18:35,140
But it is definitely perceptively modified to a human.

177
00:18:35,140 --> 00:18:46,540
And so, you know, when you're talking about these being imperceptible changes, because they're talking about, you know, editing your own data and then uploading it to Facebook as if it was an original photo.

178
00:18:46,540 --> 00:18:47,980
That's something that's really important here.

179
00:18:47,980 --> 00:18:50,280
You don't want people to have to...

180
00:18:50,280 --> 00:19:02,460
The ideal is for people to be able to upload things that to their, you know, their followers or their friends or their, I don't know, whatever you want from your favorite social media platform.

181
00:19:02,460 --> 00:19:09,040
You want it to be as close to the original content as possible to your desired audience.

182
00:19:09,040 --> 00:19:19,760
Let's say to refer to the people you want to be listening to this, despite being very different for your, you know, your undesired audience of Facebook's analytics algorithms.

183
00:19:39,200 --> 00:19:40,480
Correct.

184
00:19:40,680 --> 00:19:50,640
And it doesn't help much with the, however, 10-15 years that social media has been a thing where it already has tons of pictures of you.

185
00:19:50,640 --> 00:19:51,020
So there's...

186
00:19:53,800 --> 00:20:02,020
Yeah, I, as much as I love this line of research from a practical point of view, it really does feel like...

187
00:20:04,180 --> 00:20:15,800
Like, it's like the work is technically great, but in terms of making it, A, accessible to a wide range of people, right?

188
00:20:18,120 --> 00:20:18,640
Nobody...

189
00:20:18,640 --> 00:20:23,100
So, like, a lot of people, like, just don't care about this sort of privacy thing, right?

190
00:20:23,100 --> 00:20:29,220
And so they'll update photos of, upload photos of themselves or of you and kind of not give a shit.

191
00:20:29,220 --> 00:20:43,120
And then, you know, and then we've also got the issue that there are tons of places where you just don't have any control over the photo before it goes up, right?

192
00:20:43,120 --> 00:20:51,660
So we're, you know, if you're getting like an ID photo taken of yourself, right, that goes into the system as it goes into the system.

193
00:20:51,660 --> 00:20:56,920
And we know that that's like part of what was being collected by systems like Clearview, right?

194
00:20:56,920 --> 00:21:01,220
They're getting like, you know, state ID cards and passport photos and stuff like that.

195
00:21:01,220 --> 00:21:03,920
And those are all getting like dragged into this dragnet.

196
00:21:03,920 --> 00:21:12,980
And it doesn't take very many photos of a single person to get a good registration of them for a facial recognition system.

197
00:21:12,980 --> 00:21:15,840
I mean, in a lot of cases, you can get away with just one photo.

198
00:21:15,840 --> 00:21:36,120
So I understand that you can't go back and change the pre-existing photos, but couldn't you give the application permission to contact friends who have posted photos of you to ask them to use the application to blur you from them?

199
00:21:36,880 --> 00:21:39,940
By the application, what are you referring to?

200
00:21:39,940 --> 00:21:41,980
Because obviously Facebook isn't going to give you that option.

201
00:21:41,980 --> 00:21:44,500
Do you mean like the program you download to use this?

202
00:21:44,500 --> 00:21:51,760
Yeah, the one you were talking about blurring the image so Facebook can't identify you.

203
00:22:06,570 --> 00:22:10,670
So couldn't you send a message to your friends to use that?

204
00:22:11,170 --> 00:22:15,730
I mean, you could, but it's going to be another hoop for them to jump through.

205
00:22:15,730 --> 00:22:17,110
I mean, people are lazy, right?

206
00:22:17,110 --> 00:22:22,730
This is one of the biggest problems with doing any sort of security is people are lazy.

207
00:22:22,730 --> 00:22:24,690
They don't want to jump through extra hoops.

208
00:22:24,690 --> 00:22:37,070
I understand that people are lazy, but it's easier if you can just invite someone to use a product than having to find the link and send it to them.

209
00:22:37,070 --> 00:22:42,050
They're far more likely to use it if you just send them an invite that's really easy to use.

210
00:22:42,050 --> 00:22:44,590
Yes, that's definitely true.

211
00:22:45,090 --> 00:23:03,370
I believe that the authors are currently working on a... one of the authors has a note on their website that they're currently developing like a download an exe plug and play pre-trained model for people to use that will be like compatible with Windows OS and stuff like that.

212
00:23:03,370 --> 00:23:11,790
So I believe the best answer is that they are aware of distribution problems and accessibility problems.

213
00:23:11,890 --> 00:23:14,330
And that's something that they're actively working on.

214
00:23:14,790 --> 00:23:21,590
I don't know if invite links is something that they're building into their program or not.

215
00:23:21,590 --> 00:23:27,490
It's gonna be very interesting to see kind of what Sean, Emily, and the rest include.

216
00:23:28,590 --> 00:23:32,470
But I don't know if that's something they're developing, but I think you're right.

217
00:23:32,470 --> 00:23:45,830
Another question I have, if you don't mind me asking, is does this cause problems for the visually impaired for reading from like alt text?

218
00:24:16,610 --> 00:24:26,110
I don't think that this is something that these authors... I mean, it's a valid question.

219
00:24:26,110 --> 00:24:28,330
It's definitely something that should be considered.

220
00:24:28,330 --> 00:24:33,550
I don't think this is quite what these authors were focusing on in this paper, though.

221
00:24:33,550 --> 00:24:35,990
I understand their focus.

222
00:24:35,990 --> 00:24:41,950
But it's absolutely, yeah, no, like usability and accessibility concerns are major things that definitely...

223
00:24:41,950 --> 00:24:55,970
My thing is, there's currently the interpretation of the ADA has been changed by judges.

224
00:24:55,970 --> 00:25:03,490
And now if you can't read the image, whoever created that as the problem can be sued.

225
00:25:09,600 --> 00:25:21,660
It would be... I think it would be on the person who's making it impossible for Facebook to do their job of making it readable.

226
00:25:23,440 --> 00:25:27,620
The ADA doesn't apply to private individuals, though.

227
00:25:27,620 --> 00:25:28,840
There's no obligation...

228
00:25:29,780 --> 00:25:30,840
It didn't apply...

229
00:25:30,840 --> 00:25:45,580
If I, as myself, not as a company or as a non-profit or something, but if I just as myself upload a home video to YouTube, there's no requirement that that video be ADA compliant in any way.

230
00:25:45,580 --> 00:25:52,760
That was true until last year when that changed.

231
00:25:52,920 --> 00:25:54,200
Really?

232
00:25:54,460 --> 00:25:55,740
Yes.

233
00:25:56,300 --> 00:25:58,560
Do you know the court case or something?

234
00:25:58,560 --> 00:25:59,420
I'm very curious.

235
00:25:59,420 --> 00:26:00,960
Yeah, I can bring it up.

236
00:26:01,440 --> 00:26:03,240
Do we have a text chat?

237
00:26:03,240 --> 00:26:04,920
I can put it in a text chat.

238
00:26:04,920 --> 00:26:07,800
Yeah, put it in the AIG Journal Club text.

239
00:26:11,200 --> 00:26:14,380
Yeah, this is a topic I'm very interested in.

240
00:26:14,380 --> 00:26:15,620
I have disabilities myself.

241
00:26:15,620 --> 00:26:17,260
I care a lot about accessibility.

242
00:26:19,820 --> 00:26:22,860
Honestly, that wasn't something I thought about in this context.

243
00:26:22,860 --> 00:26:28,340
And I'm right now on Facebook trying to find out what the alt text for a photo of me is.

244
00:26:28,340 --> 00:26:30,760
I'm having trouble doing so.

245
00:26:31,460 --> 00:26:43,160
If anyone knows, like, you know, I don't know if the alt text says this is a photo of two people or if this is a photo of Stella and Madeline or a photo of Stella and her girlfriend Madeline.

246
00:26:45,580 --> 00:26:48,420
Or just like a user uploaded photo.

247
00:26:48,420 --> 00:27:01,300
But that is definitely a very interesting question as to how it affects people who rely on augmented computer devices.

248
00:27:03,420 --> 00:27:12,540
Yeah, I mean, I think it's like the first step in any of these things is, okay, can we solve the immediate problem that we're worried about?

249
00:27:12,540 --> 00:27:22,940
Can I actually successfully convince Facebook to reliably retag photos of me as photos of John Malkovich?

250
00:27:23,600 --> 00:27:33,880
If you can't do that, then worrying about the accessibility stuff, right, it's kind of like the cart before the horse.

251
00:27:33,880 --> 00:27:42,680
But yeah, before you roll this into a widespread thing, I mean, it's definitely worth doing a careful look at accessibility issues.

252
00:27:42,680 --> 00:27:51,660
And I think there's also other things that probably are worth considering, like, if you deploy this tool so that it can make everyone look like, you know, John Malkovich, whatever.

253
00:27:51,660 --> 00:28:02,800
I'm using John Malkovich as a silly example, but has John Malkovich been consulted about this, about whether he wants to be tagged in a million, you know, photos on Facebook all of a sudden?

254
00:28:02,800 --> 00:28:12,720
I don't know if, you know, like, again, John Malkovich is a made up example, but if you're going to do a targeted attack, like this paper is proposing, who's the target?

255
00:28:13,280 --> 00:28:17,640
And, like, do they really want to be the target?

256
00:28:17,640 --> 00:28:21,140
So it's interesting, because the paper kind of talks around that.

257
00:28:21,140 --> 00:28:36,400
The paper, like, specifically draws the distinction between private individuals, which is a phrase that uses to refer to the people they imagine using their photos, and celebrities and public individuals, who are the people they imagine being in their database.

258
00:28:36,400 --> 00:28:41,000
And the reason I use Gwyneth Paltrow as an example is because this is an example used in the paper.

259
00:28:41,580 --> 00:28:52,740
Her image is in the paper several times, and it shows, I'm belaying on his name, but the actor who plays Derek Shepard in Grey's Anatomy is another example in the paper.

260
00:28:52,740 --> 00:28:58,980
And so it shows you transforming Derek Shepard into Gwyneth Paltrow, and vice versa.

261
00:28:59,960 --> 00:29:05,520
But the idea that these famous people are also private individuals, I think, is really important.

262
00:29:05,520 --> 00:29:15,360
It is something that the paper kind of had a swing and a miss on, I think, because they certainly didn't ignore it, but I think you're right to question the way that they approached it.

263
00:29:16,200 --> 00:29:19,180
There is a way to do both.

264
00:29:19,180 --> 00:29:21,380
Can I jump in really quick?

265
00:29:21,380 --> 00:29:31,040
We've got someone on the stream who is asking... Tech Critter wants to jump in with a question.

266
00:29:31,060 --> 00:29:36,600
Tech Critter, if you're following on the stream, type it in the stream, otherwise type it in Discord.

267
00:29:36,700 --> 00:29:43,140
I want to make sure that we get other voices in here because we've got, I think it's been like three people talking most of the time.

268
00:29:44,220 --> 00:29:45,100
Definitely.

269
00:29:56,840 --> 00:30:00,980
Okay, or journalclub-text if you want to just drop a note in.

270
00:30:09,630 --> 00:30:14,610
Yeah, but we've got a couple other people in the Discord voice channel.

271
00:30:14,610 --> 00:30:18,010
I don't know if they want to chime in with any comments or questions.

272
00:30:55,680 --> 00:31:00,420
A slot is Robles versus Domino's Pizza.

273
00:31:03,710 --> 00:31:10,510
Yeah, so I just, if anyone's following on Discord, it just got posted in the aiv-offtopic-text channel.

274
00:31:10,510 --> 00:31:14,450
I'm going to post a newer one there.

275
00:31:14,450 --> 00:31:22,730
This is the newest one and there's been following cases after this was rolled.

276
00:31:23,370 --> 00:31:35,950
So the Domino's Pizza case says that the ADA applies to online venues such as websites, which was not necessarily obviously true before.

277
00:31:40,010 --> 00:31:49,090
This actually reminds me, there was definitely something that the University of California ran into trouble where they had to take down a bunch of their videos.

278
00:31:49,090 --> 00:31:50,230
I assume this must be related.

279
00:31:50,230 --> 00:31:58,490
A bunch of their online education videos that were publicly viewable because not all of them were ADA compliant.

280
00:31:58,490 --> 00:32:07,890
Some of them didn't have transcripts, some of them didn't have colorblind friendly options, and they took down videos instead of fixing it.

281
00:32:07,890 --> 00:32:13,650
And that was a hullabaloo if you're in the UCLA world.

282
00:32:14,590 --> 00:32:18,190
I assume that's probably due to this court case or a related one.

283
00:32:18,190 --> 00:32:25,690
So when it comes to your home videos uploaded to YouTube, YouTube automatically creates a transcription.

284
00:32:26,270 --> 00:32:29,610
Unless it's singing, then it has trouble for that.

285
00:32:29,770 --> 00:32:37,790
But it creates a transcription and a lot of times it's really messed up and you can go on to fix it, but the transcription is enough to comply.

286
00:32:42,030 --> 00:32:46,150
Right, so I've actually used that a lot because I have trouble...

287
00:32:47,870 --> 00:32:53,150
YouTube refers to it as automatically generated text, right?

288
00:32:53,150 --> 00:32:58,510
I've used that a lot because I have trouble following large conversations in videos.

289
00:32:58,510 --> 00:33:01,250
And you already said it's quite bad.

290
00:33:01,250 --> 00:33:13,730
And if you're using a version of this for audio, where it's supposed to mess up what you're saying, it could absolutely trash the transcription.

291
00:33:15,830 --> 00:33:26,270
There's a way to comply with the ADA while protecting people from AI.

292
00:33:26,730 --> 00:33:36,910
The way you do that is you just need to put something in the alt field that gives a generic idea, like a picture, and then put by the name.

293
00:33:36,910 --> 00:33:40,090
That might be enough to comply, I'm not sure.

294
00:33:41,370 --> 00:33:45,230
I mean, that definitely doesn't comply with the spirit of the ADA.

295
00:33:45,230 --> 00:33:56,550
If the idea is that alt text is supposed to describe what's happening in a photo, and you just say, this is a photo posted by Stella Biederman, that's...

296
00:33:56,550 --> 00:34:01,190
It may not comply with the spirit, but it's better than nothing.

297
00:34:05,380 --> 00:34:12,720
Yeah, that's a very complicated question, because the ADA has very strong limits on it.

298
00:34:13,260 --> 00:34:18,900
If a company argues that complying with the ADA is too expensive, they don't have to.

299
00:34:20,060 --> 00:34:36,180
And the, well, you know, this isn't actually what it was supposed to be, but it's better than nothing, is a pretty contentious argument to make in the disability activism community, because that has been used... I'm not saying you're doing this, but that has been used a lot as an excuse by companies who don't want to be compliant.

300
00:34:37,200 --> 00:34:43,400
I'm just trying to come up with ideas how it can both be privacy and ADA at the same time.

301
00:34:44,300 --> 00:34:46,520
Oh, yeah, no, I understand that.

302
00:34:46,520 --> 00:34:49,160
I just wanted to share that.

303
00:34:49,160 --> 00:35:03,160
So, a blind employee at Facebook was the one that created their thing that tells you what colors and everything are in the images.

304
00:35:03,260 --> 00:35:13,040
So I'm wondering, maybe if there's a way to obscure the face, but leave in the details like green tree, blue water.

305
00:35:14,640 --> 00:35:17,060
Well, I mean, that's sort of the open question, right?

306
00:35:17,060 --> 00:35:26,720
Because as far as I know, one thing that hasn't seen a lot of work is how do like cross model attacks work, right?

307
00:35:26,720 --> 00:35:35,840
So if we apply Fox to a photo, right, is it going to completely break like object detection as well as facial recognition?

308
00:35:42,420 --> 00:35:45,840
Yeah, so I mean, it's a totally valid question, right?

309
00:35:45,840 --> 00:35:53,800
We don't know one way or the other that applying Fox to a photo would actually break the other object recognitions.

310
00:35:53,800 --> 00:36:00,900
It might very well, but, you know, that's an area for future research, as they say.

311
00:36:00,900 --> 00:36:09,300
Are the people who wrote the paper, are they on the DEF CON Discord or Twitch?

312
00:36:09,300 --> 00:36:14,560
Yeah, we tried to get Ben Zhao, I think, to join us.

313
00:36:14,560 --> 00:36:17,160
He's the, I think he's the group lead.

314
00:36:17,160 --> 00:36:19,020
He's one of the authors on the paper at any rate.

315
00:36:19,020 --> 00:36:20,940
Unfortunately, he couldn't make it today.

316
00:36:21,200 --> 00:36:22,080
Okay.

317
00:36:22,160 --> 00:36:23,380
Maybe tomorrow?

318
00:36:24,380 --> 00:36:26,540
Well, we're talking about a different paper tomorrow.

319
00:36:27,020 --> 00:36:27,900
Okay.

320
00:36:28,780 --> 00:36:29,660
Okay.

321
00:36:29,680 --> 00:36:31,100
You guys have a schedule?

322
00:36:31,300 --> 00:36:34,860
Like, do you talk about a paper an hour or how does it work?

323
00:36:34,860 --> 00:36:36,740
We got a schedule posted.

324
00:36:36,740 --> 00:36:42,540
If you go to the AIV General Text, we've got a schedule and that tells what journal clubs we're doing when.

325
00:36:42,860 --> 00:36:43,720
Okay.

326
00:36:43,720 --> 00:36:44,400
Yeah.

327
00:36:44,400 --> 00:36:49,500
And you can also find a lot of information on our website, AIVillage.org.

328
00:36:49,540 --> 00:36:55,720
AIVillage.org, I think, slash events is our schedule for the immediate future.

329
00:36:55,820 --> 00:36:58,720
I've never seen this village before, but I really like it.

330
00:36:58,720 --> 00:37:00,040
Thank you.

331
00:37:05,080 --> 00:37:08,000
So I see that there are some other people in the Discord.

332
00:37:08,000 --> 00:37:11,180
Do other people have questions or things they want to discuss?

333
00:37:49,360 --> 00:37:53,680
Couldn't you just remove the glasses with the program?

334
00:38:03,420 --> 00:38:05,160
It could just automate it.

335
00:38:05,160 --> 00:38:14,080
I mean, a lot of the problem with adversarial examples in general is that they don't tend to transfer terribly well between systems.

336
00:38:14,080 --> 00:38:25,480
So you can have... I can't remember the group that did the funny glasses thing off the top of my head, but I actually played around with that last year.

337
00:38:25,480 --> 00:38:28,780
And if you use it on one model, it works amazingly well.

338
00:38:28,780 --> 00:38:36,020
And if you try to put it on another model, like Amazon's recognition, they're just utterly useless.

339
00:38:36,020 --> 00:38:38,860
They do absolutely nothing to the detection rate.

340
00:38:40,700 --> 00:38:42,060
Oh, sorry.

341
00:38:42,160 --> 00:38:42,780
No, go ahead.

342
00:38:42,780 --> 00:38:43,860
Sorry, I'm rambling.

343
00:38:44,020 --> 00:38:44,680
Okay, yeah.

344
00:38:44,680 --> 00:38:49,560
So that's actually something that the Fox paper discusses.

345
00:38:50,080 --> 00:38:57,020
And at the end on page... scrolling down...

346
00:38:57,980 --> 00:39:03,580
Anyway, I don't actually see a table, but they have a table where they attack several open APIs.

347
00:39:03,580 --> 00:39:07,080
They attack Facebook's... Ah, there it is.

348
00:39:07,080 --> 00:39:13,560
It's on page 9, since I'm not the one controlling the screen that's on stream.

349
00:39:13,560 --> 00:39:28,480
But if you scroll down to page 9, they have a nice table there where they show you their attacks against Microsoft Azure Face Recognition API, against Amazon Recognition, and against Facebook Search API.

350
00:39:30,000 --> 00:39:40,080
And you're definitely right about the adversarial sunglasses paper, which has the extremely innocuous name of a general framework for adversarial examples with objectives.

351
00:39:40,860 --> 00:39:44,500
But fortunately, you can find it on Google if you search for adversarial sunglasses.

352
00:39:44,500 --> 00:39:48,140
I can also send a link of this paper to the Discord.

353
00:39:48,140 --> 00:39:50,280
Because it's a cool paper and worth reading.

354
00:39:50,280 --> 00:39:57,420
Basically, what they do is they 3D print sunglasses that can defeat some facial recognition algorithms.

355
00:39:59,380 --> 00:40:11,700
But that problem is something that there's at least been some progress on, as this paper shows, where they're able to successfully attack multiple open source APIs.

356
00:40:13,680 --> 00:40:19,200
Yeah, the one thing that always struck me about that was...

357
00:40:20,260 --> 00:40:27,460
So if you go back to the original adversarial examples paper, one of the things they were really big on was transferability and proxy models.

358
00:40:27,460 --> 00:40:35,820
So you can train a model off of this data, and any two models, like an adversarial example that evades one model frequently will evade another one.

359
00:40:35,820 --> 00:40:42,980
But all of a sudden you get into the facial recognition domain, and at least based on my experience, that tends to be...

360
00:40:44,060 --> 00:40:51,080
You don't get it for free, you actually have to work for it, unlike a lot of other adversarial example situations.

361
00:40:52,500 --> 00:41:06,820
And I wonder if it's because these object classifiers are looking at softmax versus the adverb, the facial recognition tends to be much more embedding oriented, or if there's some other difference about it that I'm not seeing.

362
00:41:06,820 --> 00:41:23,680
My impression, as someone who has played around with these things, but mostly does more theoretical research than actually trying to use these against real models in the real world, is that this is a problem of complexity.

363
00:41:23,680 --> 00:41:33,980
The more complex the classifier you're training is, the more peculiar its decision boundary is going to be.

364
00:41:33,980 --> 00:41:49,620
And basically what most adversarial attacks do is they exploit peculiarities in strangenesses in the decision boundary of neural networks, rather than quote unquote actually tracking it.

365
00:41:49,620 --> 00:42:03,480
But my understanding is that the more complex the problem is, the more different ways there are for an algorithm to be uniquely peculiar, you might say.

366
00:42:04,200 --> 00:42:19,760
So the peculiarities of a neural network solving a very complex problem are going to wind up less across more and more complex fields that use more and more larger and larger data sets, and especially private buildings.

367
00:42:21,100 --> 00:42:30,800
Yeah, especially for facial recognition, because facial recognition models tend to be looking at hundreds or even thousands of classes.

368
00:42:31,600 --> 00:42:41,800
Yeah, and it's on the face of it, the transferability of adversarial attacks is not something you should expect.

369
00:42:41,800 --> 00:42:44,960
Different models should have different strangenesses.

370
00:42:44,960 --> 00:43:03,560
It makes sense to me that the more complex the model seems to be, and the bigger the data set, and the more training time is required, what that is doing is making the decision boundary more and more convoluted, in a sense.

371
00:43:04,200 --> 00:43:14,800
It makes a lot of sense to me that that would decrease the transferability of adversarial examples, though I haven't seen any papers that specifically investigate that correlation.

372
00:43:16,560 --> 00:43:18,120
Mm-hmm.

373
00:44:15,120 --> 00:44:16,100
Yeah.

374
00:44:18,780 --> 00:44:36,300
How did you, when you ran into this problem, how did you decide whether it was a methodological problem that their method doesn't generalize versus a reproducibility problem where, you know, what you implemented wasn't what they really had, and that their model might generalize and yours doesn't?

375
00:44:36,300 --> 00:44:36,460
Mm

376
00:45:27,840 --> 00:45:47,500
-hmm.

377
00:45:47,720 --> 00:45:52,840
Hey, just real quick, is it okay that an 8-year-old joins?

378
00:45:58,900 --> 00:45:59,860
Sure.

379
00:46:15,590 --> 00:46:18,190
I am Symphony, I am the 8-year-old.

380
00:46:18,190 --> 00:46:19,430
I am the 8-year-old.

381
00:46:19,430 --> 00:46:25,930
I have hacked my uncle's tablet and a website once when I was young.

382
00:46:26,330 --> 00:46:30,090
She hacked one of my clients, she did a SQL injection on them.

383
00:46:30,870 --> 00:46:35,430
I don't even know what I was doing, I just did it.

384
00:46:35,430 --> 00:46:36,710
She was 3.

385
00:46:42,790 --> 00:46:45,490
Yet another reason to be scared.

386
00:48:15,450 --> 00:48:15,870
Um...

387
00:48:18,210 --> 00:48:19,350
Yeah, so...

388
00:48:21,090 --> 00:48:26,210
One thing that's really interesting about this paper is the way they do their targeted attack.

389
00:48:26,210 --> 00:48:30,570
And I'm not going to try to explain that over voice because I think it's better to just read it.

390
00:48:30,570 --> 00:48:43,530
But something you have to be careful about, they look at... there are a lot of different ways that Facebook or whoever can counter-ban systems like this.

391
00:48:43,530 --> 00:48:48,850
And so we've talked about the fact that they found that it was hard to detect their cloak.

392
00:48:48,850 --> 00:49:01,470
But an interesting and important question is if you know an image is cloaked, even if you can't detect the cloak in general, if you're told an image is cloaked, can you take that cloak off?

393
00:49:03,430 --> 00:49:14,330
And you need to be careful about considering a lot of different options when deploying systems like this because there are a lot of different ways they can suffer.

394
00:49:14,330 --> 00:49:20,050
I actually strongly suspect that if you're told which images are cloaked, you can invert them.

395
00:49:20,450 --> 00:49:24,350
Because the optimization problem they solve, you can basically solve inverse.

396
00:49:26,650 --> 00:49:29,810
Which is a... would be unfortunate.

397
00:49:29,810 --> 00:49:31,950
I'm actually currently trying to do that.

398
00:49:43,900 --> 00:49:48,040
Well, I mean, it depends on how the cloak works.

399
00:49:48,320 --> 00:49:58,760
Here they're assuming that you're solving a certain optimization problem against a publicly known and everybody set of reference images.

400
00:50:05,220 --> 00:50:09,320
I'm certainly not going to say any cloak can be undone.

401
00:50:09,320 --> 00:50:18,280
I certainly hope that's not the case, because I think technology like this is... and I hope that technology like this becomes very useful in the future.

402
00:50:31,450 --> 00:50:34,170
I believe that's an open question.

403
00:50:38,120 --> 00:50:43,060
This is something we've actually been discussing in the AI village recently, though.

404
00:50:43,060 --> 00:50:51,900
Can you reduce the... there isn't a good notion of hard for AI, the way there is hard for deterministic computing.

405
00:50:51,900 --> 00:51:02,880
But if you take the image recognition problem as an adversarial example, can you use that as a hard for AI benchmark?

406
00:51:02,920 --> 00:51:11,260
Can you design a system where detecting the cloak is exactly as hard as recognizing the image in the first place?

407
00:51:11,260 --> 00:51:16,020
There's been some interesting conversations along those lines that I'm sure we'll continue to have.

408
00:52:42,110 --> 00:52:45,410
Yeah, but Rich did mention a specific paper.

409
00:52:45,410 --> 00:52:47,590
Is he still on the line or did he drop?

410
00:52:58,880 --> 00:53:09,440
So Rich definitely talked about a paper that does exactly what you're curious about, EVS, and we can pull him up and get that paper sent out to the Journal Club text.

411
00:53:10,140 --> 00:53:10,660
For you.