start	end	text
1000	3040	Hello?
13060	16060	Hey, how are you doing?
19100	21140	Sorry?
30640	35540	I can stream this thing easily to Twitch chat.
41040	44360	Okay. I don't care if we do this or Zoom.
44960	47300	Whichever. Whichever works.
52790	59610	It's funny, I was talking about the Fox paper with a co-worker of mine on Friday.
59870	65250	We were wondering about reverse engineering it. Is the author currently on?
65790	69010	No. Do you know the author? Do you want to ping him quickly?
69770	70370	No.
70690	80470	I mean, I think I've met him once. I did my undergrad at Chicago, but I don't know any of them.
80970	87270	By the way, we're also live on Twitch with MyUglyMug as the only visual reference.
87790	89510	Gotcha. Hello, Twitch.
90350	95510	So if you want to participate in this journal club, the link is in both...
96270	99010	This is the experimental side of the village.
99010	103910	So the link is in Twitch chat. The link is also in the Discord.
103910	109990	If you are interested in using Adverse Fields examples productively and want to talk about that,
109990	115490	join the AI village general voice and just ask away.
115490	121090	It would be nice if you've read the paper, but don't worry about that.
121090	130110	If you want to talk about this principle, like, you know, adversarial examples for good sort of thing,
130110	136990	just come. We'd love to hear from you.
146160	150700	So there are multiple ways for you to get on here.
154420	155880	Hey, Rich.
159460	160520	Hey.
160940	164680	For too long.
164900	166300	Yeah, welcome.
175510	177110	Where are they?
178630	186970	So I'm in the process of moving and I discovered my COVID panic beans.
189140	195620	You know, when the whole thing was starting, I bought a bunch of beans because I like making bean curries.
195620	200160	And I was worried that beans would be a little bit hard to find.
202840	208020	And as I was moving, I found my giant bag of COVID panic beans.
210980	214700	Isn't that like exactly what we were told to not do?
214700	217540	I mean, I only...
218700	222700	I use about two cups of that. So that would be like...
222700	231240	If I was eating bean curries two, three times a week, that's about three weeks of beans.
232200	235000	Some people bought like six months of beans.
238290	247430	There was a while we actually had to get beans, flour and yeast essentially off of basically like the black market.
247430	257970	It was a local company that was buying it in bulk and then like repackaging it and reselling it under the table.
259830	266190	Nothing like people exploiting a pandemic to make a quick buck.
266190	268650	That's true American ingenuity right there.
268650	270750	Yeah, it's capitalism.
273550	276950	This is the part where we say, yay, capitalism, right?
280330	281890	Capitalism is the problem.
282370	285590	Sometimes I get my speaking points confused.
287690	288770	Yeah.
290150	293590	Let's see how... well...
299130	309420	How many...
309420	326560	All right. So for everybody on Twitch that is new to Drone and Phone, basically what this is, is we run this once a week on Wednesdays usually at five o'clock Pacific Standard Time.
326560	330460	Or 8 p.m. East Coast Time.
332100	337900	And we have a paper that we're discussing. We usually get way off topic very quickly.
341000	347800	And oftentimes we have the authors on and we basically talk about the paper and the surrounding topics.
353060	356940	And it's a really interesting... they usually have really interesting discussions.
356940	363580	There's some like motifs because of our... from our perspectives, like threat models.
363580	369520	Whenever we have an academic paper with a threat model in it, we usually rag on it for a bit.
369820	372000	Unless the threat model is really good.
374520	383180	Because if you've never actually worked in security, building a threat model from a purely academic perspective is hard.
383180	387840	And they do kind of... there's a lot of people who mess that up.
387920	396680	But they at least try and try to figure it out. And when we have authors on, we'll talk about that all.
400200	404980	And if you want to join us today, it's free, easy, open.
405000	412380	You can come jump on the Discord voice chat. So we have AI Village General Voice.
412620	421720	If you've got any questions about using adversarial examples to defend against facial recognition systems, please come jump on.
423220	427580	Stella, since you've got like... you're invested in this paper a bit.
427580	432440	Do you want to describe what it goes over and give us an overview of what's going on?
432860	439680	Okay, sure. So is the author going to join us? Or does it look like no right now?
439680	442220	No, it doesn't look like the author is going to join us.
443720	447180	Someone had a note that they were still trying to get a hold of him.
447180	451740	So that's the last update I heard. I don't think we're going to get him, unfortunately.
451940	454460	Okay, gotcha.
454600	460360	All right, so the paper we're discussing is called Fox Protecting Privacy Against Unauthorized Deep Learning Models.
460360	463140	I hope I'm in the right talk, right?
463140	464400	Yeah, that's the one.
465640	475240	So the background concept of this paper is that there are a lot of ways that people upload photos to the internet.
475240	485220	And companies like Facebook, like Google, process your photo through a machine learning algorithm and use it to identify you.
485320	489100	And they can use this to identify you in future photos.
489180	496960	A couple years ago, Facebook rolled out an automatic tagging process, where when you upload a new photo, it tags your friends in it automatically.
497140	499300	And this creeped out a lot of people.
499380	503480	They were surprised that Facebook was able to immediately identify who they were.
503480	521360	And so the conceit of this paper is to give people a defense, a way to opt out of this kind of analysis without necessarily opting out of existing on the internet as a person.
521360	541280	Another important connection that the authors draw specifically is Clearview AI, which is a now infamous company that did analytics drawing on what was technically speaking public data to draw all sorts of conclusions about people,
541280	549360	where the users were not particularly comfortable with the amount of information that Clearview AI was able to obtain about them.
549360	555380	And this is kind of a repeating motif in public use of technology.
555380	558200	Nobody reads Terms of Service.
558200	563100	Nobody has any idea what data agreements they're accepting and stuff like that.
563340	567460	And even if you did, Facebook has a monopoly on being Facebook.
567460	569060	Twitter has a monopoly on being Twitter.
569060	578180	You either go along with whatever they demand of you, or you just don't exist on Twitter.
578180	586860	And it's not like there's a Twitter 2 that you can go on to and have the same sort of experience, just not with the people whose data policies you don't like.
588240	603400	So an idea that's been incubating over the past year or so in the adversarial AI community is, hey, defeating algorithms like this is exactly what adversarial AIs are supposed to do.
604160	618880	Adversarial AI, or adversarial examples, I should say, is supposed to take photos, transform them in a way that they look normal to humans, but they can't be processed by AI algorithms correctly.
618880	630000	So instead of using these things to fool... these things are usually conceived in the literature as a negative, as a bad thing.
630000	633360	They're used to fool AI algorithms and do harm.
633940	637980	But in terms of data privacy, you can view them as a very positive thing.
637980	647820	You can take an image of yourself and transform it so that AIs that look at it don't think that you're a person, or think that you're a person who you're not.
647820	654560	And then when you upload it, that photo that's been transformed onto the internet, it looks like you to humans.
654740	659040	That's kind of the whole point, but it doesn't look like you to AI algorithms.
659540	666120	And so that's the core idea at the center of this paper, and that's ultimately what they try to achieve.
666400	672160	And what they do achieve with their approach.
672460	675120	In the paper, they refer to it as a cloak.
693760	696140	Yeah, that sounds about right.
750420	757200	To cloak you, but they're going to cloak you in a way that makes you look like an image from this reference data set.
757720	762380	So they're going to make you look like another person or look like another object.
762900	773440	And the reason that they give for this in the paper is that they find that this helps it get past screening algorithms.
773440	778860	That try to notice and reject adversarial examples.
779580	786340	Typically, when you use adversarial examples, you're inserting a lot of noise, and it's not anything in particular.
786420	789540	It just happens to make you look different to the AI.
789840	796040	But what they're doing here is trying to inject some kind of pattern thing that makes me look like Gwyneth Paltrow.
796040	805880	And then once it's made me look like Gwyneth Paltrow, the idea is that the algorithms are supposed to detect when this is going on.
805880	812340	Aren't going to notice because they're going to say this looks like a non-adversarial photo of Gwyneth Paltrow.
814340	830020	I'm kind of salty about this because I spent a huge amount of time last year before the talk I did on facial recognition trying to come up with a single universal adversarial vector that would make one model think everyone was John Malkovich.
830860	836160	I'm super salty about this. I've literally written two-thirds of a paper on this exact topic.
836340	839400	Yeah, definitely an idea that's in the air.
839400	843100	No, yeah, this has been in the air for a long time.
843760	853220	There's a, from an ethics point of view, because a lot of the framework for this is ethically oriented, you know, giving users control over their data.
853220	862430	There's a really good paper from either earlier this year or last year called The Ethics of Adversarial... The Politics of Adversarial Examples.
863620	870470	That kind of discusses the fact that the AI literature like almost universally treats adversarial examples as a bad thing.
870880	877860	And how that's not really fair and discusses different ways that one could use them to achieve good.
877860	884100	Though it's a it's a ethics policy oriented paper and it doesn't do any of them.
884160	888860	So yeah, this is definitely an idea that's been around the block a couple times.
893060	898720	For sure. I know a lot of people who have had thoughts along these lines.
899660	908100	Yeah, I want to say there was a talk at, there was actually a talk at The Village last year. I'm drawing a complete blank on what they called it.
908220	915460	I think they were specifically targeting Facebook's model and they were doing something a bit different.
915460	920920	They were basically breaking the, they were basically breaking like the facial detection step.
920920	932880	So, you know, skipping the, so the point where the system would actually like put the bounding box and clip out the face tile around the face so that like nothing looked like a face at all in it.
932880	939000	And so it wouldn't even go to the next step of like finding the vector for the face and then doing the embedding and then trying to match it.
939680	948840	Yeah, because a lot of a lot of these algorithms use some kind of bounding boxes or try to try to identify, you know, images are complex.
948840	953580	So you kind of have to first pick out where there's something that you're going to look at before you do that.
953580	962200	I don't think I've ever read that paper, but I assume you're talking about like disrupting its ability to find bounding boxes or what faces are.
962200	968380	Yeah. So most, most of these facial recognition systems go through like a three step process, right?
968380	974020	You, you first, you draw bounding boxes around the face and you get a lot of the time.
974020	978720	Most of them, I think you still, still use like landmark based approaches.
978900	983220	Then you use the landmarks to sort of like rotate and realign the face.
983220	987480	So it's sort of like dead on and in a standard orientation for those faces.
987480	995460	And then you put it through the actual facial recognition model, which will take the face and embed it into a vector space.
995460	1000360	And then you find like nearest matches within the vector space. And that's the actual recognition step.
1000420	1004220	So this is what these guys are going after. They're saying, no, take this face.
1004380	1009360	We'll let you find the face, but now we're going to make, you know, we're going to make Stella look like Gwyneth Paltrow.
1009360	1015360	And what the guys at AI Village last year did was they said, we're not going to even let you find the face, right?
1015360	1020180	It's going to look like, you know, like a landscape with no faces in it or something like that.
1020180	1027860	But it was the same idea, right? It's trying to find like the minimal, like least disturbing perturbation you can use.
1028420	1033680	That would still like, you know, so your picture looks nice and you can share it on the internet.
1033680	1040080	But, you know, either you can't find any faces in it or you do find faces, but they're tagged as the wrong people.
1040080	1046900	Yeah, that is very similar. And I think you just hit on something that's important to call out specifically.
1046960	1051560	You know, there are a lot of different types of data that might benefit from this sort of masking.
1051560	1055820	Here, they're using faces and they're doing that for a very important reason.
1055960	1066200	Adversarial examples for faces seem to be... their performance is not better than it is on other kinds of data.
1066200	1071450	But for facial analytics, you can make very difficult to perceive to human ones.
1071820	1081340	So an adversarial example for images and especially for facial analytics looks to a human like you haven't edited the image at all.
1081340	1088940	When you talk about like audio adversarial examples, like Carlini has a kind of the first major paper on that.
1088940	1097500	You can hear in the background that there's a hum. And as the strength of the adversarial example increases, the hum increases.
1097500	1108540	And I've played around with these and, you know, even at the loudest, the kind of noise in the background is, you can still clearly hear what's going on.
1109240	1115140	But it is definitely perceptively modified to a human.
1115140	1126540	And so, you know, when you're talking about these being imperceptible changes, because they're talking about, you know, editing your own data and then uploading it to Facebook as if it was an original photo.
1126540	1130280	That's something that's really important here. You don't want people to have to...
1130280	1142460	The ideal is for people to be able to upload things that to their, you know, their followers or their friends or their, I don't know, whatever you want from your favorite social media platform.
1142460	1149040	You want it to be as close to the original content as possible to your desired audience.
1149040	1159760	Let's say to refer to the people you want to be listening to this, despite being very different for your, you know, your undesired audience of Facebook's analytics algorithms.
1179200	1180480	Correct.
1180680	1191020	And it doesn't help much with the, however, 10-15 years that social media has been a thing where it already has tons of pictures of you. So there's...
1193800	1202020	Yeah, I, as much as I love this line of research from a practical point of view, it really does feel like...
1204180	1215800	Like, it's like the work is technically great, but in terms of making it, A, accessible to a wide range of people, right?
1218120	1218640	Nobody...
1218640	1229220	So, like, a lot of people, like, just don't care about this sort of privacy thing, right? And so they'll update photos of, upload photos of themselves or of you and kind of not give a shit.
1229220	1243120	And then, you know, and then we've also got the issue that there are tons of places where you just don't have any control over the photo before it goes up, right?
1243120	1251660	So we're, you know, if you're getting like an ID photo taken of yourself, right, that goes into the system as it goes into the system.
1251660	1263920	And we know that that's like part of what was being collected by systems like Clearview, right? They're getting like, you know, state ID cards and passport photos and stuff like that. And those are all getting like dragged into this dragnet.
1263920	1275840	And it doesn't take very many photos of a single person to get a good registration of them for a facial recognition system. I mean, in a lot of cases, you can get away with just one photo.
1275840	1296120	So I understand that you can't go back and change the pre-existing photos, but couldn't you give the application permission to contact friends who have posted photos of you to ask them to use the application to blur you from them?
1296880	1304500	By the application, what are you referring to? Because obviously Facebook isn't going to give you that option. Do you mean like the program you download to use this?
1304500	1330670	Yeah, the one you were talking about blurring the image so Facebook can't identify you. So couldn't you send a message to your friends to use that?
1331170	1344690	I mean, you could, but it's going to be another hoop for them to jump through. I mean, people are lazy, right? This is one of the biggest problems with doing any sort of security is people are lazy. They don't want to jump through extra hoops.
1344690	1362050	I understand that people are lazy, but it's easier if you can just invite someone to use a product than having to find the link and send it to them. They're far more likely to use it if you just send them an invite that's really easy to use.
1362050	1383370	Yes, that's definitely true. I believe that the authors are currently working on a... one of the authors has a note on their website that they're currently developing like a download an exe plug and play pre-trained model for people to use that will be like compatible with Windows OS and stuff like that.
1383370	1412470	So I believe the best answer is that they are aware of distribution problems and accessibility problems. And that's something that they're actively working on. I don't know if invite links is something that they're building into their program or not. It's gonna be very interesting to see kind of what Sean, Emily, and the rest include. But I don't know if that's something they're developing, but I think you're right.
1412470	1425830	Another question I have, if you don't mind me asking, is does this cause problems for the visually impaired for reading from like alt text?
1456610	1468330	I don't think that this is something that these authors... I mean, it's a valid question. It's definitely something that should be considered.
1468330	1473550	I don't think this is quite what these authors were focusing on in this paper, though.
1473550	1475990	I understand their focus.
1475990	1481950	But it's absolutely, yeah, no, like usability and accessibility concerns are major things that definitely...
1481950	1503490	My thing is, there's currently the interpretation of the ADA has been changed by judges. And now if you can't read the image, whoever created that as the problem can be sued.
1509600	1521660	It would be... I think it would be on the person who's making it impossible for Facebook to do their job of making it readable.
1523440	1528840	The ADA doesn't apply to private individuals, though. There's no obligation...
1529780	1530840	It didn't apply...
1530840	1545580	If I, as myself, not as a company or as a non-profit or something, but if I just as myself upload a home video to YouTube, there's no requirement that that video be ADA compliant in any way.
1545580	1552760	That was true until last year when that changed.
1552920	1554200	Really?
1554460	1555740	Yes.
1556300	1559420	Do you know the court case or something? I'm very curious.
1559420	1560960	Yeah, I can bring it up.
1561440	1564920	Do we have a text chat? I can put it in a text chat.
1564920	1567800	Yeah, put it in the AIG Journal Club text.
1571200	1577260	Yeah, this is a topic I'm very interested in. I have disabilities myself. I care a lot about accessibility.
1579820	1588340	Honestly, that wasn't something I thought about in this context. And I'm right now on Facebook trying to find out what the alt text for a photo of me is.
1588340	1590760	I'm having trouble doing so.
1591460	1603160	If anyone knows, like, you know, I don't know if the alt text says this is a photo of two people or if this is a photo of Stella and Madeline or a photo of Stella and her girlfriend Madeline.
1605580	1608420	Or just like a user uploaded photo.
1608420	1621300	But that is definitely a very interesting question as to how it affects people who rely on augmented computer devices.
1623420	1632540	Yeah, I mean, I think it's like the first step in any of these things is, okay, can we solve the immediate problem that we're worried about?
1632540	1642940	Can I actually successfully convince Facebook to reliably retag photos of me as photos of John Malkovich?
1643600	1653880	If you can't do that, then worrying about the accessibility stuff, right, it's kind of like the cart before the horse.
1653880	1662680	But yeah, before you roll this into a widespread thing, I mean, it's definitely worth doing a careful look at accessibility issues.
1662680	1671660	And I think there's also other things that probably are worth considering, like, if you deploy this tool so that it can make everyone look like, you know, John Malkovich, whatever.
1671660	1682800	I'm using John Malkovich as a silly example, but has John Malkovich been consulted about this, about whether he wants to be tagged in a million, you know, photos on Facebook all of a sudden?
1682800	1697640	I don't know if, you know, like, again, John Malkovich is a made up example, but if you're going to do a targeted attack, like this paper is proposing, who's the target? And, like, do they really want to be the target?
1697640	1716400	So it's interesting, because the paper kind of talks around that. The paper, like, specifically draws the distinction between private individuals, which is a phrase that uses to refer to the people they imagine using their photos, and celebrities and public individuals, who are the people they imagine being in their database.
1716400	1721000	And the reason I use Gwyneth Paltrow as an example is because this is an example used in the paper.
1721580	1738980	Her image is in the paper several times, and it shows, I'm belaying on his name, but the actor who plays Derek Shepard in Grey's Anatomy is another example in the paper. And so it shows you transforming Derek Shepard into Gwyneth Paltrow, and vice versa.
1739960	1755360	But the idea that these famous people are also private individuals, I think, is really important. It is something that the paper kind of had a swing and a miss on, I think, because they certainly didn't ignore it, but I think you're right to question the way that they approached it.
1756200	1759180	There is a way to do both.
1759180	1783140	Can I jump in really quick? We've got someone on the stream who is asking... Tech Critter wants to jump in with a question. Tech Critter, if you're following on the stream, type it in the stream, otherwise type it in Discord. I want to make sure that we get other voices in here because we've got, I think it's been like three people talking most of the time.
1784220	1785100	Definitely.
1796840	1800980	Okay, or journalclub-text if you want to just drop a note in.
1809630	1818010	Yeah, but we've got a couple other people in the Discord voice channel. I don't know if they want to chime in with any comments or questions.
1855680	1860420	A slot is Robles versus Domino's Pizza.
1863710	1870510	Yeah, so I just, if anyone's following on Discord, it just got posted in the aiv-offtopic-text channel.
1870510	1882730	I'm going to post a newer one there. This is the newest one and there's been following cases after this was rolled.
1883370	1895950	So the Domino's Pizza case says that the ADA applies to online venues such as websites, which was not necessarily obviously true before.
1900010	1909090	This actually reminds me, there was definitely something that the University of California ran into trouble where they had to take down a bunch of their videos.
1909090	1918490	I assume this must be related. A bunch of their online education videos that were publicly viewable because not all of them were ADA compliant.
1918490	1927890	Some of them didn't have transcripts, some of them didn't have colorblind friendly options, and they took down videos instead of fixing it.
1927890	1938190	And that was a hullabaloo if you're in the UCLA world. I assume that's probably due to this court case or a related one.
1938190	1945690	So when it comes to your home videos uploaded to YouTube, YouTube automatically creates a transcription.
1946270	1957790	Unless it's singing, then it has trouble for that. But it creates a transcription and a lot of times it's really messed up and you can go on to fix it, but the transcription is enough to comply.
1962030	1966150	Right, so I've actually used that a lot because I have trouble...
1967870	1978510	YouTube refers to it as automatically generated text, right? I've used that a lot because I have trouble following large conversations in videos.
1978510	1993730	And you already said it's quite bad. And if you're using a version of this for audio, where it's supposed to mess up what you're saying, it could absolutely trash the transcription.
1995830	2020090	There's a way to comply with the ADA while protecting people from AI. The way you do that is you just need to put something in the alt field that gives a generic idea, like a picture, and then put by the name. That might be enough to comply, I'm not sure.
2021370	2036550	I mean, that definitely doesn't comply with the spirit of the ADA. If the idea is that alt text is supposed to describe what's happening in a photo, and you just say, this is a photo posted by Stella Biederman, that's...
2036550	2041190	It may not comply with the spirit, but it's better than nothing.
2045380	2058900	Yeah, that's a very complicated question, because the ADA has very strong limits on it. If a company argues that complying with the ADA is too expensive, they don't have to.
2060060	2076180	And the, well, you know, this isn't actually what it was supposed to be, but it's better than nothing, is a pretty contentious argument to make in the disability activism community, because that has been used... I'm not saying you're doing this, but that has been used a lot as an excuse by companies who don't want to be compliant.
2077200	2083400	I'm just trying to come up with ideas how it can both be privacy and ADA at the same time.
2084300	2089160	Oh, yeah, no, I understand that. I just wanted to share that.
2089160	2113040	So, a blind employee at Facebook was the one that created their thing that tells you what colors and everything are in the images. So I'm wondering, maybe if there's a way to obscure the face, but leave in the details like green tree, blue water.
2114640	2135840	Well, I mean, that's sort of the open question, right? Because as far as I know, one thing that hasn't seen a lot of work is how do like cross model attacks work, right? So if we apply Fox to a photo, right, is it going to completely break like object detection as well as facial recognition?
2142420	2160900	Yeah, so I mean, it's a totally valid question, right? We don't know one way or the other that applying Fox to a photo would actually break the other object recognitions. It might very well, but, you know, that's an area for future research, as they say.
2160900	2169300	Are the people who wrote the paper, are they on the DEF CON Discord or Twitch?
2169300	2180940	Yeah, we tried to get Ben Zhao, I think, to join us. He's the, I think he's the group lead. He's one of the authors on the paper at any rate. Unfortunately, he couldn't make it today.
2181200	2183380	Okay. Maybe tomorrow?
2184380	2186540	Well, we're talking about a different paper tomorrow.
2187020	2194860	Okay. Okay. You guys have a schedule? Like, do you talk about a paper an hour or how does it work?
2194860	2202540	We got a schedule posted. If you go to the AIV General Text, we've got a schedule and that tells what journal clubs we're doing when.
2202860	2215720	Okay. Yeah. And you can also find a lot of information on our website, AIVillage.org. AIVillage.org, I think, slash events is our schedule for the immediate future.
2215820	2218720	I've never seen this village before, but I really like it.
2218720	2231180	Thank you. So I see that there are some other people in the Discord. Do other people have questions or things they want to discuss?
2269360	2285160	Couldn't you just remove the glasses with the program? It could just automate it.
2285160	2294080	I mean, a lot of the problem with adversarial examples in general is that they don't tend to transfer terribly well between systems.
2294080	2305480	So you can have... I can't remember the group that did the funny glasses thing off the top of my head, but I actually played around with that last year.
2305480	2316020	And if you use it on one model, it works amazingly well. And if you try to put it on another model, like Amazon's recognition, they're just utterly useless.
2316020	2318860	They do absolutely nothing to the detection rate.
2320700	2322060	Oh, sorry.
2322160	2323860	No, go ahead. Sorry, I'm rambling.
2324020	2337020	Okay, yeah. So that's actually something that the Fox paper discusses. And at the end on page... scrolling down...
2337980	2343580	Anyway, I don't actually see a table, but they have a table where they attack several open APIs.
2343580	2353560	They attack Facebook's... Ah, there it is. It's on page 9, since I'm not the one controlling the screen that's on stream.
2353560	2368480	But if you scroll down to page 9, they have a nice table there where they show you their attacks against Microsoft Azure Face Recognition API, against Amazon Recognition, and against Facebook Search API.
2370000	2380080	And you're definitely right about the adversarial sunglasses paper, which has the extremely innocuous name of a general framework for adversarial examples with objectives.
2380860	2388140	But fortunately, you can find it on Google if you search for adversarial sunglasses. I can also send a link of this paper to the Discord.
2388140	2397420	Because it's a cool paper and worth reading. Basically, what they do is they 3D print sunglasses that can defeat some facial recognition algorithms.
2399380	2411700	But that problem is something that there's at least been some progress on, as this paper shows, where they're able to successfully attack multiple open source APIs.
2413680	2419200	Yeah, the one thing that always struck me about that was...
2420260	2427460	So if you go back to the original adversarial examples paper, one of the things they were really big on was transferability and proxy models.
2427460	2435820	So you can train a model off of this data, and any two models, like an adversarial example that evades one model frequently will evade another one.
2435820	2442980	But all of a sudden you get into the facial recognition domain, and at least based on my experience, that tends to be...
2444060	2451080	You don't get it for free, you actually have to work for it, unlike a lot of other adversarial example situations.
2452500	2466820	And I wonder if it's because these object classifiers are looking at softmax versus the adverb, the facial recognition tends to be much more embedding oriented, or if there's some other difference about it that I'm not seeing.
2466820	2483680	My impression, as someone who has played around with these things, but mostly does more theoretical research than actually trying to use these against real models in the real world, is that this is a problem of complexity.
2483680	2493980	The more complex the classifier you're training is, the more peculiar its decision boundary is going to be.
2493980	2509620	And basically what most adversarial attacks do is they exploit peculiarities in strangenesses in the decision boundary of neural networks, rather than quote unquote actually tracking it.
2509620	2523480	But my understanding is that the more complex the problem is, the more different ways there are for an algorithm to be uniquely peculiar, you might say.
2524200	2539760	So the peculiarities of a neural network solving a very complex problem are going to wind up less across more and more complex fields that use more and more larger and larger data sets, and especially private buildings.
2541100	2550800	Yeah, especially for facial recognition, because facial recognition models tend to be looking at hundreds or even thousands of classes.
2551600	2564960	Yeah, and it's on the face of it, the transferability of adversarial attacks is not something you should expect. Different models should have different strangenesses.
2564960	2583560	It makes sense to me that the more complex the model seems to be, and the bigger the data set, and the more training time is required, what that is doing is making the decision boundary more and more convoluted, in a sense.
2584200	2594800	It makes a lot of sense to me that that would decrease the transferability of adversarial examples, though I haven't seen any papers that specifically investigate that correlation.
2596560	2598120	Mm-hmm.
2655120	2656100	Yeah.
2658780	2676300	How did you, when you ran into this problem, how did you decide whether it was a methodological problem that their method doesn't generalize versus a reproducibility problem where, you know, what you implemented wasn't what they really had, and that their model might generalize and yours doesn't?
2676300	2747500	Mm-hmm.
2747720	2752840	Hey, just real quick, is it okay that an 8-year-old joins?
2758900	2759860	Sure.
2775590	2785930	I am Symphony, I am the 8-year-old. I am the 8-year-old. I have hacked my uncle's tablet and a website once when I was young.
2786330	2790090	She hacked one of my clients, she did a SQL injection on them.
2790870	2795430	I don't even know what I was doing, I just did it.
2795430	2796710	She was 3.
2802790	2805490	Yet another reason to be scared.
2895450	2895870	Um...
2898210	2899350	Yeah, so...
2901090	2910570	One thing that's really interesting about this paper is the way they do their targeted attack. And I'm not going to try to explain that over voice because I think it's better to just read it.
2910570	2923530	But something you have to be careful about, they look at... there are a lot of different ways that Facebook or whoever can counter-ban systems like this.
2923530	2928850	And so we've talked about the fact that they found that it was hard to detect their cloak.
2928850	2941470	But an interesting and important question is if you know an image is cloaked, even if you can't detect the cloak in general, if you're told an image is cloaked, can you take that cloak off?
2943430	2954330	And you need to be careful about considering a lot of different options when deploying systems like this because there are a lot of different ways they can suffer.
2954330	2960050	I actually strongly suspect that if you're told which images are cloaked, you can invert them.
2960450	2964350	Because the optimization problem they solve, you can basically solve inverse.
2966650	2971950	Which is a... would be unfortunate. I'm actually currently trying to do that.
2983900	2988040	Well, I mean, it depends on how the cloak works.
2988320	2998760	Here they're assuming that you're solving a certain optimization problem against a publicly known and everybody set of reference images.
3005220	3009320	I'm certainly not going to say any cloak can be undone.
3009320	3018280	I certainly hope that's not the case, because I think technology like this is... and I hope that technology like this becomes very useful in the future.
3031450	3034170	I believe that's an open question.
3038120	3043060	This is something we've actually been discussing in the AI village recently, though.
3043060	3051900	Can you reduce the... there isn't a good notion of hard for AI, the way there is hard for deterministic computing.
3051900	3062880	But if you take the image recognition problem as an adversarial example, can you use that as a hard for AI benchmark?
3062920	3071260	Can you design a system where detecting the cloak is exactly as hard as recognizing the image in the first place?
3071260	3076020	There's been some interesting conversations along those lines that I'm sure we'll continue to have.
3162110	3167590	Yeah, but Rich did mention a specific paper. Is he still on the line or did he drop?
3178880	3189440	So Rich definitely talked about a paper that does exactly what you're curious about, EVS, and we can pull him up and get that paper sent out to the Journal Club text.
3190140	3190660	For you.