WEBVTT 00:01.000 --> 00:03.040 Hello? 00:13.060 --> 00:16.060 Hey, how are you doing? 00:19.100 --> 00:21.140 Sorry? 00:30.640 --> 00:35.540 I can stream this thing easily to Twitch chat. 00:41.040 --> 00:42.280 Okay. 00:42.280 --> 00:44.360 I don't care if we do this or Zoom. 00:44.960 --> 00:46.080 Whichever. 00:46.080 --> 00:47.300 Whichever works. 00:52.790 --> 00:59.610 It's funny, I was talking about the Fox paper with a co-worker of mine on Friday. 00:59.870 --> 01:03.270 We were wondering about reverse engineering it. 01:03.270 --> 01:05.250 Is the author currently on? 01:05.790 --> 01:06.510 No. 01:06.510 --> 01:07.310 Do you know the author? 01:07.310 --> 01:09.010 Do you want to ping him quickly? 01:09.770 --> 01:10.370 No. 01:10.690 --> 01:15.790 I mean, I think I've met him once. 01:15.790 --> 01:20.470 I did my undergrad at Chicago, but I don't know any of them. 01:20.970 --> 01:27.270 By the way, we're also live on Twitch with MyUglyMug as the only visual reference. 01:27.790 --> 01:28.430 Gotcha. 01:28.430 --> 01:29.510 Hello, Twitch. 01:30.350 --> 01:35.510 So if you want to participate in this journal club, the link is in both... 01:36.270 --> 01:39.010 This is the experimental side of the village. 01:39.010 --> 01:41.450 So the link is in Twitch chat. 01:41.450 --> 01:43.910 The link is also in the Discord. 01:43.910 --> 01:55.490 If you are interested in using Adverse Fields examples productively and want to talk about that, join the AI village general voice and just ask away. 01:55.490 --> 02:01.090 It would be nice if you've read the paper, but don't worry about that. 02:01.090 --> 02:14.950 If you want to talk about this principle, like, you know, adversarial examples for good sort of thing, just come. 02:15.470 --> 02:16.990 We'd love to hear from you. 02:26.160 --> 02:30.700 So there are multiple ways for you to get on here. 02:34.420 --> 02:35.880 Hey, Rich. 02:39.460 --> 02:40.520 Hey. 02:40.940 --> 02:44.680 For too long. 02:44.900 --> 02:46.300 Yeah, welcome. 02:55.510 --> 02:57.110 Where are they? 02:58.630 --> 03:06.970 So I'm in the process of moving and I discovered my COVID panic beans. 03:09.140 --> 03:15.620 You know, when the whole thing was starting, I bought a bunch of beans because I like making bean curries. 03:15.620 --> 03:20.160 And I was worried that beans would be a little bit hard to find. 03:22.840 --> 03:28.020 And as I was moving, I found my giant bag of COVID panic beans. 03:30.980 --> 03:34.700 Isn't that like exactly what we were told to not do? 03:34.700 --> 03:37.540 I mean, I only... 03:38.700 --> 03:40.620 I use about two cups of that. 03:40.620 --> 03:42.700 So that would be like... 03:42.700 --> 03:51.240 If I was eating bean curries two, three times a week, that's about three weeks of beans. 03:52.200 --> 03:55.000 Some people bought like six months of beans. 03:58.290 --> 04:07.430 There was a while we actually had to get beans, flour and yeast essentially off of basically like the black market. 04:07.430 --> 04:17.970 It was a local company that was buying it in bulk and then like repackaging it and reselling it under the table. 04:19.830 --> 04:26.190 Nothing like people exploiting a pandemic to make a quick buck. 04:26.190 --> 04:28.650 That's true American ingenuity right there. 04:28.650 --> 04:30.750 Yeah, it's capitalism. 04:33.550 --> 04:36.950 This is the part where we say, yay, capitalism, right? 04:40.330 --> 04:41.890 Capitalism is the problem. 04:42.370 --> 04:45.590 Sometimes I get my speaking points confused. 04:47.690 --> 04:48.770 Yeah. 04:50.150 --> 04:53.590 Let's see how... well... 04:59.130 --> 04:59.850 How 05:08.700 --> 05:09.420 many... 05:09.420 --> 05:13.460 All right. 05:13.460 --> 05:26.560 So for everybody on Twitch that is new to Drone and Phone, basically what this is, is we run this once a week on Wednesdays usually at five o'clock Pacific Standard Time. 05:26.560 --> 05:28.560 Or 8 p.m. 05:28.640 --> 05:30.460 East Coast Time. 05:32.100 --> 05:35.060 And we have a paper that we're discussing. 05:35.060 --> 05:37.900 We usually get way off topic very quickly. 05:41.000 --> 05:47.800 And oftentimes we have the authors on and we basically talk about the paper and the surrounding topics. 05:53.060 --> 05:56.940 And it's a really interesting... they usually have really interesting discussions. 05:56.940 --> 06:03.580 There's some like motifs because of our... from our perspectives, like threat models. 06:03.580 --> 06:09.520 Whenever we have an academic paper with a threat model in it, we usually rag on it for a bit. 06:09.820 --> 06:12.000 Unless the threat model is really good. 06:14.520 --> 06:23.180 Because if you've never actually worked in security, building a threat model from a purely academic perspective is hard. 06:23.180 --> 06:27.840 And they do kind of... there's a lot of people who mess that up. 06:27.920 --> 06:30.560 But they at least try and try to figure it out. 06:30.560 --> 06:36.680 And when we have authors on, we'll talk about that all. 06:40.200 --> 06:44.980 And if you want to join us today, it's free, easy, open. 06:45.000 --> 06:49.340 You can come jump on the Discord voice chat. 06:49.340 --> 06:52.380 So we have AI Village General Voice. 06:52.620 --> 07:01.720 If you've got any questions about using adversarial examples to defend against facial recognition systems, please come jump on. 07:03.220 --> 07:07.580 Stella, since you've got like... you're invested in this paper a bit. 07:07.580 --> 07:12.440 Do you want to describe what it goes over and give us an overview of what's going on? 07:12.860 --> 07:13.820 Okay, sure. 07:13.820 --> 07:16.800 So is the author going to join us? 07:16.880 --> 07:19.680 Or does it look like no right now? 07:19.680 --> 07:22.220 No, it doesn't look like the author is going to join us. 07:23.720 --> 07:27.180 Someone had a note that they were still trying to get a hold of him. 07:27.180 --> 07:29.640 So that's the last update I heard. 07:29.640 --> 07:31.740 I don't think we're going to get him, unfortunately. 07:31.940 --> 07:34.460 Okay, gotcha. 07:34.600 --> 07:40.360 All right, so the paper we're discussing is called Fox Protecting Privacy Against Unauthorized Deep Learning Models. 07:40.360 --> 07:43.140 I hope I'm in the right talk, right? 07:43.140 --> 07:44.400 Yeah, that's the one. 07:45.640 --> 07:55.240 So the background concept of this paper is that there are a lot of ways that people upload photos to the internet. 07:55.240 --> 08:05.220 And companies like Facebook, like Google, process your photo through a machine learning algorithm and use it to identify you. 08:05.320 --> 08:09.100 And they can use this to identify you in future photos. 08:09.180 --> 08:16.960 A couple years ago, Facebook rolled out an automatic tagging process, where when you upload a new photo, it tags your friends in it automatically. 08:17.140 --> 08:19.300 And this creeped out a lot of people. 08:19.380 --> 08:23.480 They were surprised that Facebook was able to immediately identify who they were. 08:23.480 --> 08:41.360 And so the conceit of this paper is to give people a defense, a way to opt out of this kind of analysis without necessarily opting out of existing on the internet as a person. 08:41.360 --> 09:09.360 Another important connection that the authors draw specifically is Clearview AI, which is a now infamous company that did analytics drawing on what was technically speaking public data to draw all sorts of conclusions about people, where the users were not particularly comfortable with the amount of information that Clearview AI was able to obtain about them. 09:09.360 --> 09:15.380 And this is kind of a repeating motif in public use of technology. 09:15.380 --> 09:18.200 Nobody reads Terms of Service. 09:18.200 --> 09:23.100 Nobody has any idea what data agreements they're accepting and stuff like that. 09:23.340 --> 09:27.460 And even if you did, Facebook has a monopoly on being Facebook. 09:27.460 --> 09:29.060 Twitter has a monopoly on being Twitter. 09:29.060 --> 09:38.180 You either go along with whatever they demand of you, or you just don't exist on Twitter. 09:38.180 --> 09:46.860 And it's not like there's a Twitter 2 that you can go on to and have the same sort of experience, just not with the people whose data policies you don't like. 09:48.240 --> 10:03.400 So an idea that's been incubating over the past year or so in the adversarial AI community is, hey, defeating algorithms like this is exactly what adversarial AIs are supposed to do. 10:04.160 --> 10:18.880 Adversarial AI, or adversarial examples, I should say, is supposed to take photos, transform them in a way that they look normal to humans, but they can't be processed by AI algorithms correctly. 10:18.880 --> 10:30.000 So instead of using these things to fool... these things are usually conceived in the literature as a negative, as a bad thing. 10:30.000 --> 10:33.360 They're used to fool AI algorithms and do harm. 10:33.940 --> 10:37.980 But in terms of data privacy, you can view them as a very positive thing. 10:37.980 --> 10:47.820 You can take an image of yourself and transform it so that AIs that look at it don't think that you're a person, or think that you're a person who you're not. 10:47.820 --> 10:54.560 And then when you upload it, that photo that's been transformed onto the internet, it looks like you to humans. 10:54.740 --> 10:59.040 That's kind of the whole point, but it doesn't look like you to AI algorithms. 10:59.540 --> 11:06.120 And so that's the core idea at the center of this paper, and that's ultimately what they try to achieve. 11:06.400 --> 11:12.160 And what they do achieve with their approach. 11:12.460 --> 11:15.120 In the paper, they refer to it as a cloak. 11:33.760 --> 11:36.140 Yeah, that sounds about right. 12:30.420 --> 12:37.200 To cloak you, but they're going to cloak you in a way that makes you look like an image from this reference data set. 12:37.720 --> 12:42.380 So they're going to make you look like another person or look like another object. 12:42.900 --> 12:53.440 And the reason that they give for this in the paper is that they find that this helps it get past screening algorithms. 12:53.440 --> 12:58.860 That try to notice and reject adversarial examples. 12:59.580 --> 13:06.340 Typically, when you use adversarial examples, you're inserting a lot of noise, and it's not anything in particular. 13:06.420 --> 13:09.540 It just happens to make you look different to the AI. 13:09.840 --> 13:16.040 But what they're doing here is trying to inject some kind of pattern thing that makes me look like Gwyneth Paltrow. 13:16.040 --> 13:25.880 And then once it's made me look like Gwyneth Paltrow, the idea is that the algorithms are supposed to detect when this is going on. 13:25.880 --> 13:32.340 Aren't going to notice because they're going to say this looks like a non-adversarial photo of Gwyneth Paltrow. 13:34.340 --> 13:50.020 I'm kind of salty about this because I spent a huge amount of time last year before the talk I did on facial recognition trying to come up with a single universal adversarial vector that would make one model think everyone was John Malkovich. 13:50.860 --> 13:52.540 I'm super salty about this. 13:52.540 --> 13:56.160 I've literally written two-thirds of a paper on this exact topic. 13:56.340 --> 13:59.400 Yeah, definitely an idea that's in the air. 13:59.400 --> 14:03.100 No, yeah, this has been in the air for a long time. 14:03.760 --> 14:13.220 There's a, from an ethics point of view, because a lot of the framework for this is ethically oriented, you know, giving users control over their data. 14:13.220 --> 14:22.430 There's a really good paper from either earlier this year or last year called The Ethics of Adversarial... The Politics of Adversarial Examples. 14:23.620 --> 14:30.470 That kind of discusses the fact that the AI literature like almost universally treats adversarial examples as a bad thing. 14:30.880 --> 14:37.860 And how that's not really fair and discusses different ways that one could use them to achieve good. 14:37.860 --> 14:44.100 Though it's a it's a ethics policy oriented paper and it doesn't do any of them. 14:44.160 --> 14:48.860 So yeah, this is definitely an idea that's been around the block a couple times. 14:53.060 --> 14:54.240 For sure. 14:54.280 --> 14:58.720 I know a lot of people who have had thoughts along these lines. 14:59.660 --> 15:05.020 Yeah, I want to say there was a talk at, there was actually a talk at The Village last year. 15:05.020 --> 15:08.100 I'm drawing a complete blank on what they called it. 15:08.220 --> 15:15.460 I think they were specifically targeting Facebook's model and they were doing something a bit different. 15:15.460 --> 15:20.920 They were basically breaking the, they were basically breaking like the facial detection step. 15:20.920 --> 15:32.880 So, you know, skipping the, so the point where the system would actually like put the bounding box and clip out the face tile around the face so that like nothing looked like a face at all in it. 15:32.880 --> 15:39.000 And so it wouldn't even go to the next step of like finding the vector for the face and then doing the embedding and then trying to match it. 15:39.680 --> 15:48.840 Yeah, because a lot of a lot of these algorithms use some kind of bounding boxes or try to try to identify, you know, images are complex. 15:48.840 --> 15:53.580 So you kind of have to first pick out where there's something that you're going to look at before you do that. 15:53.580 --> 16:02.200 I don't think I've ever read that paper, but I assume you're talking about like disrupting its ability to find bounding boxes or what faces are. 16:02.200 --> 16:02.980 Yeah. 16:02.980 --> 16:08.380 So most, most of these facial recognition systems go through like a three step process, right? 16:08.380 --> 16:14.020 You, you first, you draw bounding boxes around the face and you get a lot of the time. 16:14.020 --> 16:18.720 Most of them, I think you still, still use like landmark based approaches. 16:18.900 --> 16:23.220 Then you use the landmarks to sort of like rotate and realign the face. 16:23.220 --> 16:27.480 So it's sort of like dead on and in a standard orientation for those faces. 16:27.480 --> 16:35.460 And then you put it through the actual facial recognition model, which will take the face and embed it into a vector space. 16:35.460 --> 16:38.260 And then you find like nearest matches within the vector space. 16:38.260 --> 16:40.360 And that's the actual recognition step. 16:40.420 --> 16:42.280 So this is what these guys are going after. 16:42.280 --> 16:44.220 They're saying, no, take this face. 16:44.380 --> 16:49.360 We'll let you find the face, but now we're going to make, you know, we're going to make Stella look like Gwyneth Paltrow. 16:49.360 --> 16:55.360 And what the guys at AI Village last year did was they said, we're not going to even let you find the face, right? 16:55.360 --> 17:00.180 It's going to look like, you know, like a landscape with no faces in it or something like that. 17:00.180 --> 17:01.540 But it was the same idea, right? 17:01.540 --> 17:07.860 It's trying to find like the minimal, like least disturbing perturbation you can use. 17:08.420 --> 17:13.680 That would still like, you know, so your picture looks nice and you can share it on the internet. 17:13.680 --> 17:20.080 But, you know, either you can't find any faces in it or you do find faces, but they're tagged as the wrong people. 17:20.080 --> 17:23.220 Yeah, that is very similar. 17:23.220 --> 17:26.900 And I think you just hit on something that's important to call out specifically. 17:26.960 --> 17:31.560 You know, there are a lot of different types of data that might benefit from this sort of masking. 17:31.560 --> 17:35.820 Here, they're using faces and they're doing that for a very important reason. 17:35.960 --> 17:46.200 Adversarial examples for faces seem to be... their performance is not better than it is on other kinds of data. 17:46.200 --> 17:51.450 But for facial analytics, you can make very difficult to perceive to human ones. 17:51.820 --> 18:01.340 So an adversarial example for images and especially for facial analytics looks to a human like you haven't edited the image at all. 18:01.340 --> 18:08.940 When you talk about like audio adversarial examples, like Carlini has a kind of the first major paper on that. 18:08.940 --> 18:12.640 You can hear in the background that there's a hum. 18:13.240 --> 18:17.500 And as the strength of the adversarial example increases, the hum increases. 18:17.500 --> 18:28.540 And I've played around with these and, you know, even at the loudest, the kind of noise in the background is, you can still clearly hear what's going on. 18:29.240 --> 18:35.140 But it is definitely perceptively modified to a human. 18:35.140 --> 18:46.540 And so, you know, when you're talking about these being imperceptible changes, because they're talking about, you know, editing your own data and then uploading it to Facebook as if it was an original photo. 18:46.540 --> 18:47.980 That's something that's really important here. 18:47.980 --> 18:50.280 You don't want people to have to... 18:50.280 --> 19:02.460 The ideal is for people to be able to upload things that to their, you know, their followers or their friends or their, I don't know, whatever you want from your favorite social media platform. 19:02.460 --> 19:09.040 You want it to be as close to the original content as possible to your desired audience. 19:09.040 --> 19:19.760 Let's say to refer to the people you want to be listening to this, despite being very different for your, you know, your undesired audience of Facebook's analytics algorithms. 19:39.200 --> 19:40.480 Correct. 19:40.680 --> 19:50.640 And it doesn't help much with the, however, 10-15 years that social media has been a thing where it already has tons of pictures of you. 19:50.640 --> 19:51.020 So there's... 19:53.800 --> 20:02.020 Yeah, I, as much as I love this line of research from a practical point of view, it really does feel like... 20:04.180 --> 20:15.800 Like, it's like the work is technically great, but in terms of making it, A, accessible to a wide range of people, right? 20:18.120 --> 20:18.640 Nobody... 20:18.640 --> 20:23.100 So, like, a lot of people, like, just don't care about this sort of privacy thing, right? 20:23.100 --> 20:29.220 And so they'll update photos of, upload photos of themselves or of you and kind of not give a shit. 20:29.220 --> 20:43.120 And then, you know, and then we've also got the issue that there are tons of places where you just don't have any control over the photo before it goes up, right? 20:43.120 --> 20:51.660 So we're, you know, if you're getting like an ID photo taken of yourself, right, that goes into the system as it goes into the system. 20:51.660 --> 20:56.920 And we know that that's like part of what was being collected by systems like Clearview, right? 20:56.920 --> 21:01.220 They're getting like, you know, state ID cards and passport photos and stuff like that. 21:01.220 --> 21:03.920 And those are all getting like dragged into this dragnet. 21:03.920 --> 21:12.980 And it doesn't take very many photos of a single person to get a good registration of them for a facial recognition system. 21:12.980 --> 21:15.840 I mean, in a lot of cases, you can get away with just one photo. 21:15.840 --> 21:36.120 So I understand that you can't go back and change the pre-existing photos, but couldn't you give the application permission to contact friends who have posted photos of you to ask them to use the application to blur you from them? 21:36.880 --> 21:39.940 By the application, what are you referring to? 21:39.940 --> 21:41.980 Because obviously Facebook isn't going to give you that option. 21:41.980 --> 21:44.500 Do you mean like the program you download to use this? 21:44.500 --> 21:51.760 Yeah, the one you were talking about blurring the image so Facebook can't identify you. 22:06.570 --> 22:10.670 So couldn't you send a message to your friends to use that? 22:11.170 --> 22:15.730 I mean, you could, but it's going to be another hoop for them to jump through. 22:15.730 --> 22:17.110 I mean, people are lazy, right? 22:17.110 --> 22:22.730 This is one of the biggest problems with doing any sort of security is people are lazy. 22:22.730 --> 22:24.690 They don't want to jump through extra hoops. 22:24.690 --> 22:37.070 I understand that people are lazy, but it's easier if you can just invite someone to use a product than having to find the link and send it to them. 22:37.070 --> 22:42.050 They're far more likely to use it if you just send them an invite that's really easy to use. 22:42.050 --> 22:44.590 Yes, that's definitely true. 22:45.090 --> 23:03.370 I believe that the authors are currently working on a... one of the authors has a note on their website that they're currently developing like a download an exe plug and play pre-trained model for people to use that will be like compatible with Windows OS and stuff like that. 23:03.370 --> 23:11.790 So I believe the best answer is that they are aware of distribution problems and accessibility problems. 23:11.890 --> 23:14.330 And that's something that they're actively working on. 23:14.790 --> 23:21.590 I don't know if invite links is something that they're building into their program or not. 23:21.590 --> 23:27.490 It's gonna be very interesting to see kind of what Sean, Emily, and the rest include. 23:28.590 --> 23:32.470 But I don't know if that's something they're developing, but I think you're right. 23:32.470 --> 23:45.830 Another question I have, if you don't mind me asking, is does this cause problems for the visually impaired for reading from like alt text? 24:16.610 --> 24:26.110 I don't think that this is something that these authors... I mean, it's a valid question. 24:26.110 --> 24:28.330 It's definitely something that should be considered. 24:28.330 --> 24:33.550 I don't think this is quite what these authors were focusing on in this paper, though. 24:33.550 --> 24:35.990 I understand their focus. 24:35.990 --> 24:41.950 But it's absolutely, yeah, no, like usability and accessibility concerns are major things that definitely... 24:41.950 --> 24:55.970 My thing is, there's currently the interpretation of the ADA has been changed by judges. 24:55.970 --> 25:03.490 And now if you can't read the image, whoever created that as the problem can be sued. 25:09.600 --> 25:21.660 It would be... I think it would be on the person who's making it impossible for Facebook to do their job of making it readable. 25:23.440 --> 25:27.620 The ADA doesn't apply to private individuals, though. 25:27.620 --> 25:28.840 There's no obligation... 25:29.780 --> 25:30.840 It didn't apply... 25:30.840 --> 25:45.580 If I, as myself, not as a company or as a non-profit or something, but if I just as myself upload a home video to YouTube, there's no requirement that that video be ADA compliant in any way. 25:45.580 --> 25:52.760 That was true until last year when that changed. 25:52.920 --> 25:54.200 Really? 25:54.460 --> 25:55.740 Yes. 25:56.300 --> 25:58.560 Do you know the court case or something? 25:58.560 --> 25:59.420 I'm very curious. 25:59.420 --> 26:00.960 Yeah, I can bring it up. 26:01.440 --> 26:03.240 Do we have a text chat? 26:03.240 --> 26:04.920 I can put it in a text chat. 26:04.920 --> 26:07.800 Yeah, put it in the AIG Journal Club text. 26:11.200 --> 26:14.380 Yeah, this is a topic I'm very interested in. 26:14.380 --> 26:15.620 I have disabilities myself. 26:15.620 --> 26:17.260 I care a lot about accessibility. 26:19.820 --> 26:22.860 Honestly, that wasn't something I thought about in this context. 26:22.860 --> 26:28.340 And I'm right now on Facebook trying to find out what the alt text for a photo of me is. 26:28.340 --> 26:30.760 I'm having trouble doing so. 26:31.460 --> 26:43.160 If anyone knows, like, you know, I don't know if the alt text says this is a photo of two people or if this is a photo of Stella and Madeline or a photo of Stella and her girlfriend Madeline. 26:45.580 --> 26:48.420 Or just like a user uploaded photo. 26:48.420 --> 27:01.300 But that is definitely a very interesting question as to how it affects people who rely on augmented computer devices. 27:03.420 --> 27:12.540 Yeah, I mean, I think it's like the first step in any of these things is, okay, can we solve the immediate problem that we're worried about? 27:12.540 --> 27:22.940 Can I actually successfully convince Facebook to reliably retag photos of me as photos of John Malkovich? 27:23.600 --> 27:33.880 If you can't do that, then worrying about the accessibility stuff, right, it's kind of like the cart before the horse. 27:33.880 --> 27:42.680 But yeah, before you roll this into a widespread thing, I mean, it's definitely worth doing a careful look at accessibility issues. 27:42.680 --> 27:51.660 And I think there's also other things that probably are worth considering, like, if you deploy this tool so that it can make everyone look like, you know, John Malkovich, whatever. 27:51.660 --> 28:02.800 I'm using John Malkovich as a silly example, but has John Malkovich been consulted about this, about whether he wants to be tagged in a million, you know, photos on Facebook all of a sudden? 28:02.800 --> 28:12.720 I don't know if, you know, like, again, John Malkovich is a made up example, but if you're going to do a targeted attack, like this paper is proposing, who's the target? 28:13.280 --> 28:17.640 And, like, do they really want to be the target? 28:17.640 --> 28:21.140 So it's interesting, because the paper kind of talks around that. 28:21.140 --> 28:36.400 The paper, like, specifically draws the distinction between private individuals, which is a phrase that uses to refer to the people they imagine using their photos, and celebrities and public individuals, who are the people they imagine being in their database. 28:36.400 --> 28:41.000 And the reason I use Gwyneth Paltrow as an example is because this is an example used in the paper. 28:41.580 --> 28:52.740 Her image is in the paper several times, and it shows, I'm belaying on his name, but the actor who plays Derek Shepard in Grey's Anatomy is another example in the paper. 28:52.740 --> 28:58.980 And so it shows you transforming Derek Shepard into Gwyneth Paltrow, and vice versa. 28:59.960 --> 29:05.520 But the idea that these famous people are also private individuals, I think, is really important. 29:05.520 --> 29:15.360 It is something that the paper kind of had a swing and a miss on, I think, because they certainly didn't ignore it, but I think you're right to question the way that they approached it. 29:16.200 --> 29:19.180 There is a way to do both. 29:19.180 --> 29:21.380 Can I jump in really quick? 29:21.380 --> 29:31.040 We've got someone on the stream who is asking... Tech Critter wants to jump in with a question. 29:31.060 --> 29:36.600 Tech Critter, if you're following on the stream, type it in the stream, otherwise type it in Discord. 29:36.700 --> 29:43.140 I want to make sure that we get other voices in here because we've got, I think it's been like three people talking most of the time. 29:44.220 --> 29:45.100 Definitely. 29:56.840 --> 30:00.980 Okay, or journalclub-text if you want to just drop a note in. 30:09.630 --> 30:14.610 Yeah, but we've got a couple other people in the Discord voice channel. 30:14.610 --> 30:18.010 I don't know if they want to chime in with any comments or questions. 30:55.680 --> 31:00.420 A slot is Robles versus Domino's Pizza. 31:03.710 --> 31:10.510 Yeah, so I just, if anyone's following on Discord, it just got posted in the aiv-offtopic-text channel. 31:10.510 --> 31:14.450 I'm going to post a newer one there. 31:14.450 --> 31:22.730 This is the newest one and there's been following cases after this was rolled. 31:23.370 --> 31:35.950 So the Domino's Pizza case says that the ADA applies to online venues such as websites, which was not necessarily obviously true before. 31:40.010 --> 31:49.090 This actually reminds me, there was definitely something that the University of California ran into trouble where they had to take down a bunch of their videos. 31:49.090 --> 31:50.230 I assume this must be related. 31:50.230 --> 31:58.490 A bunch of their online education videos that were publicly viewable because not all of them were ADA compliant. 31:58.490 --> 32:07.890 Some of them didn't have transcripts, some of them didn't have colorblind friendly options, and they took down videos instead of fixing it. 32:07.890 --> 32:13.650 And that was a hullabaloo if you're in the UCLA world. 32:14.590 --> 32:18.190 I assume that's probably due to this court case or a related one. 32:18.190 --> 32:25.690 So when it comes to your home videos uploaded to YouTube, YouTube automatically creates a transcription. 32:26.270 --> 32:29.610 Unless it's singing, then it has trouble for that. 32:29.770 --> 32:37.790 But it creates a transcription and a lot of times it's really messed up and you can go on to fix it, but the transcription is enough to comply. 32:42.030 --> 32:46.150 Right, so I've actually used that a lot because I have trouble... 32:47.870 --> 32:53.150 YouTube refers to it as automatically generated text, right? 32:53.150 --> 32:58.510 I've used that a lot because I have trouble following large conversations in videos. 32:58.510 --> 33:01.250 And you already said it's quite bad. 33:01.250 --> 33:13.730 And if you're using a version of this for audio, where it's supposed to mess up what you're saying, it could absolutely trash the transcription. 33:15.830 --> 33:26.270 There's a way to comply with the ADA while protecting people from AI. 33:26.730 --> 33:36.910 The way you do that is you just need to put something in the alt field that gives a generic idea, like a picture, and then put by the name. 33:36.910 --> 33:40.090 That might be enough to comply, I'm not sure. 33:41.370 --> 33:45.230 I mean, that definitely doesn't comply with the spirit of the ADA. 33:45.230 --> 33:56.550 If the idea is that alt text is supposed to describe what's happening in a photo, and you just say, this is a photo posted by Stella Biederman, that's... 33:56.550 --> 34:01.190 It may not comply with the spirit, but it's better than nothing. 34:05.380 --> 34:12.720 Yeah, that's a very complicated question, because the ADA has very strong limits on it. 34:13.260 --> 34:18.900 If a company argues that complying with the ADA is too expensive, they don't have to. 34:20.060 --> 34:36.180 And the, well, you know, this isn't actually what it was supposed to be, but it's better than nothing, is a pretty contentious argument to make in the disability activism community, because that has been used... I'm not saying you're doing this, but that has been used a lot as an excuse by companies who don't want to be compliant. 34:37.200 --> 34:43.400 I'm just trying to come up with ideas how it can both be privacy and ADA at the same time. 34:44.300 --> 34:46.520 Oh, yeah, no, I understand that. 34:46.520 --> 34:49.160 I just wanted to share that. 34:49.160 --> 35:03.160 So, a blind employee at Facebook was the one that created their thing that tells you what colors and everything are in the images. 35:03.260 --> 35:13.040 So I'm wondering, maybe if there's a way to obscure the face, but leave in the details like green tree, blue water. 35:14.640 --> 35:17.060 Well, I mean, that's sort of the open question, right? 35:17.060 --> 35:26.720 Because as far as I know, one thing that hasn't seen a lot of work is how do like cross model attacks work, right? 35:26.720 --> 35:35.840 So if we apply Fox to a photo, right, is it going to completely break like object detection as well as facial recognition? 35:42.420 --> 35:45.840 Yeah, so I mean, it's a totally valid question, right? 35:45.840 --> 35:53.800 We don't know one way or the other that applying Fox to a photo would actually break the other object recognitions. 35:53.800 --> 36:00.900 It might very well, but, you know, that's an area for future research, as they say. 36:00.900 --> 36:09.300 Are the people who wrote the paper, are they on the DEF CON Discord or Twitch? 36:09.300 --> 36:14.560 Yeah, we tried to get Ben Zhao, I think, to join us. 36:14.560 --> 36:17.160 He's the, I think he's the group lead. 36:17.160 --> 36:19.020 He's one of the authors on the paper at any rate. 36:19.020 --> 36:20.940 Unfortunately, he couldn't make it today. 36:21.200 --> 36:22.080 Okay. 36:22.160 --> 36:23.380 Maybe tomorrow? 36:24.380 --> 36:26.540 Well, we're talking about a different paper tomorrow. 36:27.020 --> 36:27.900 Okay. 36:28.780 --> 36:29.660 Okay. 36:29.680 --> 36:31.100 You guys have a schedule? 36:31.300 --> 36:34.860 Like, do you talk about a paper an hour or how does it work? 36:34.860 --> 36:36.740 We got a schedule posted. 36:36.740 --> 36:42.540 If you go to the AIV General Text, we've got a schedule and that tells what journal clubs we're doing when. 36:42.860 --> 36:43.720 Okay. 36:43.720 --> 36:44.400 Yeah. 36:44.400 --> 36:49.500 And you can also find a lot of information on our website, AIVillage.org. 36:49.540 --> 36:55.720 AIVillage.org, I think, slash events is our schedule for the immediate future. 36:55.820 --> 36:58.720 I've never seen this village before, but I really like it. 36:58.720 --> 37:00.040 Thank you. 37:05.080 --> 37:08.000 So I see that there are some other people in the Discord. 37:08.000 --> 37:11.180 Do other people have questions or things they want to discuss? 37:49.360 --> 37:53.680 Couldn't you just remove the glasses with the program? 38:03.420 --> 38:05.160 It could just automate it. 38:05.160 --> 38:14.080 I mean, a lot of the problem with adversarial examples in general is that they don't tend to transfer terribly well between systems. 38:14.080 --> 38:25.480 So you can have... I can't remember the group that did the funny glasses thing off the top of my head, but I actually played around with that last year. 38:25.480 --> 38:28.780 And if you use it on one model, it works amazingly well. 38:28.780 --> 38:36.020 And if you try to put it on another model, like Amazon's recognition, they're just utterly useless. 38:36.020 --> 38:38.860 They do absolutely nothing to the detection rate. 38:40.700 --> 38:42.060 Oh, sorry. 38:42.160 --> 38:42.780 No, go ahead. 38:42.780 --> 38:43.860 Sorry, I'm rambling. 38:44.020 --> 38:44.680 Okay, yeah. 38:44.680 --> 38:49.560 So that's actually something that the Fox paper discusses. 38:50.080 --> 38:57.020 And at the end on page... scrolling down... 38:57.980 --> 39:03.580 Anyway, I don't actually see a table, but they have a table where they attack several open APIs. 39:03.580 --> 39:07.080 They attack Facebook's... Ah, there it is. 39:07.080 --> 39:13.560 It's on page 9, since I'm not the one controlling the screen that's on stream. 39:13.560 --> 39:28.480 But if you scroll down to page 9, they have a nice table there where they show you their attacks against Microsoft Azure Face Recognition API, against Amazon Recognition, and against Facebook Search API. 39:30.000 --> 39:40.080 And you're definitely right about the adversarial sunglasses paper, which has the extremely innocuous name of a general framework for adversarial examples with objectives. 39:40.860 --> 39:44.500 But fortunately, you can find it on Google if you search for adversarial sunglasses. 39:44.500 --> 39:48.140 I can also send a link of this paper to the Discord. 39:48.140 --> 39:50.280 Because it's a cool paper and worth reading. 39:50.280 --> 39:57.420 Basically, what they do is they 3D print sunglasses that can defeat some facial recognition algorithms. 39:59.380 --> 40:11.700 But that problem is something that there's at least been some progress on, as this paper shows, where they're able to successfully attack multiple open source APIs. 40:13.680 --> 40:19.200 Yeah, the one thing that always struck me about that was... 40:20.260 --> 40:27.460 So if you go back to the original adversarial examples paper, one of the things they were really big on was transferability and proxy models. 40:27.460 --> 40:35.820 So you can train a model off of this data, and any two models, like an adversarial example that evades one model frequently will evade another one. 40:35.820 --> 40:42.980 But all of a sudden you get into the facial recognition domain, and at least based on my experience, that tends to be... 40:44.060 --> 40:51.080 You don't get it for free, you actually have to work for it, unlike a lot of other adversarial example situations. 40:52.500 --> 41:06.820 And I wonder if it's because these object classifiers are looking at softmax versus the adverb, the facial recognition tends to be much more embedding oriented, or if there's some other difference about it that I'm not seeing. 41:06.820 --> 41:23.680 My impression, as someone who has played around with these things, but mostly does more theoretical research than actually trying to use these against real models in the real world, is that this is a problem of complexity. 41:23.680 --> 41:33.980 The more complex the classifier you're training is, the more peculiar its decision boundary is going to be. 41:33.980 --> 41:49.620 And basically what most adversarial attacks do is they exploit peculiarities in strangenesses in the decision boundary of neural networks, rather than quote unquote actually tracking it. 41:49.620 --> 42:03.480 But my understanding is that the more complex the problem is, the more different ways there are for an algorithm to be uniquely peculiar, you might say. 42:04.200 --> 42:19.760 So the peculiarities of a neural network solving a very complex problem are going to wind up less across more and more complex fields that use more and more larger and larger data sets, and especially private buildings. 42:21.100 --> 42:30.800 Yeah, especially for facial recognition, because facial recognition models tend to be looking at hundreds or even thousands of classes. 42:31.600 --> 42:41.800 Yeah, and it's on the face of it, the transferability of adversarial attacks is not something you should expect. 42:41.800 --> 42:44.960 Different models should have different strangenesses. 42:44.960 --> 43:03.560 It makes sense to me that the more complex the model seems to be, and the bigger the data set, and the more training time is required, what that is doing is making the decision boundary more and more convoluted, in a sense. 43:04.200 --> 43:14.800 It makes a lot of sense to me that that would decrease the transferability of adversarial examples, though I haven't seen any papers that specifically investigate that correlation. 43:16.560 --> 43:18.120 Mm-hmm. 44:15.120 --> 44:16.100 Yeah. 44:18.780 --> 44:36.300 How did you, when you ran into this problem, how did you decide whether it was a methodological problem that their method doesn't generalize versus a reproducibility problem where, you know, what you implemented wasn't what they really had, and that their model might generalize and yours doesn't? 44:36.300 --> 44:36.460 Mm 45:27.840 --> 45:47.500 -hmm. 45:47.720 --> 45:52.840 Hey, just real quick, is it okay that an 8-year-old joins? 45:58.900 --> 45:59.860 Sure. 46:15.590 --> 46:18.190 I am Symphony, I am the 8-year-old. 46:18.190 --> 46:19.430 I am the 8-year-old. 46:19.430 --> 46:25.930 I have hacked my uncle's tablet and a website once when I was young. 46:26.330 --> 46:30.090 She hacked one of my clients, she did a SQL injection on them. 46:30.870 --> 46:35.430 I don't even know what I was doing, I just did it. 46:35.430 --> 46:36.710 She was 3. 46:42.790 --> 46:45.490 Yet another reason to be scared. 48:15.450 --> 48:15.870 Um... 48:18.210 --> 48:19.350 Yeah, so... 48:21.090 --> 48:26.210 One thing that's really interesting about this paper is the way they do their targeted attack. 48:26.210 --> 48:30.570 And I'm not going to try to explain that over voice because I think it's better to just read it. 48:30.570 --> 48:43.530 But something you have to be careful about, they look at... there are a lot of different ways that Facebook or whoever can counter-ban systems like this. 48:43.530 --> 48:48.850 And so we've talked about the fact that they found that it was hard to detect their cloak. 48:48.850 --> 49:01.470 But an interesting and important question is if you know an image is cloaked, even if you can't detect the cloak in general, if you're told an image is cloaked, can you take that cloak off? 49:03.430 --> 49:14.330 And you need to be careful about considering a lot of different options when deploying systems like this because there are a lot of different ways they can suffer. 49:14.330 --> 49:20.050 I actually strongly suspect that if you're told which images are cloaked, you can invert them. 49:20.450 --> 49:24.350 Because the optimization problem they solve, you can basically solve inverse. 49:26.650 --> 49:29.810 Which is a... would be unfortunate. 49:29.810 --> 49:31.950 I'm actually currently trying to do that. 49:43.900 --> 49:48.040 Well, I mean, it depends on how the cloak works. 49:48.320 --> 49:58.760 Here they're assuming that you're solving a certain optimization problem against a publicly known and everybody set of reference images. 50:05.220 --> 50:09.320 I'm certainly not going to say any cloak can be undone. 50:09.320 --> 50:18.280 I certainly hope that's not the case, because I think technology like this is... and I hope that technology like this becomes very useful in the future. 50:31.450 --> 50:34.170 I believe that's an open question. 50:38.120 --> 50:43.060 This is something we've actually been discussing in the AI village recently, though. 50:43.060 --> 50:51.900 Can you reduce the... there isn't a good notion of hard for AI, the way there is hard for deterministic computing. 50:51.900 --> 51:02.880 But if you take the image recognition problem as an adversarial example, can you use that as a hard for AI benchmark? 51:02.920 --> 51:11.260 Can you design a system where detecting the cloak is exactly as hard as recognizing the image in the first place? 51:11.260 --> 51:16.020 There's been some interesting conversations along those lines that I'm sure we'll continue to have. 52:42.110 --> 52:45.410 Yeah, but Rich did mention a specific paper. 52:45.410 --> 52:47.590 Is he still on the line or did he drop? 52:58.880 --> 53:09.440 So Rich definitely talked about a paper that does exactly what you're curious about, EVS, and we can pull him up and get that paper sent out to the Journal Club text. 53:10.140 --> 53:10.660 For you.