Hello?
Hey, how are you doing?
Sorry?
I can stream this thing easily to Twitch chat.
Okay. I don't care if we do this or Zoom.
Whichever. Whichever works.
It's funny, I was talking about the Fox paper with a co-worker of mine on Friday.
We were wondering about reverse engineering it. Is the author currently on?
No. Do you know the author? Do you want to ping him quickly?
No.
I mean, I think I've met him once. I did my undergrad at Chicago, but I don't know any of them.
By the way, we're also live on Twitch with MyUglyMug as the only visual reference.
Gotcha. Hello, Twitch.
So if you want to participate in this journal club, the link is in both...
This is the experimental side of the village.
So the link is in Twitch chat. The link is also in the Discord.
If you are interested in using Adverse Fields examples productively and want to talk about that,
join the AI village general voice and just ask away.
It would be nice if you've read the paper, but don't worry about that.
If you want to talk about this principle, like, you know, adversarial examples for good sort of thing,
just come. We'd love to hear from you.
So there are multiple ways for you to get on here.
Hey, Rich.
Hey.
For too long.
Yeah, welcome.
Where are they?
So I'm in the process of moving and I discovered my COVID panic beans.
You know, when the whole thing was starting, I bought a bunch of beans because I like making bean curries.
And I was worried that beans would be a little bit hard to find.
And as I was moving, I found my giant bag of COVID panic beans.
Isn't that like exactly what we were told to not do?
I mean, I only...
I use about two cups of that. So that would be like...
If I was eating bean curries two, three times a week, that's about three weeks of beans.
Some people bought like six months of beans.
There was a while we actually had to get beans, flour and yeast essentially off of basically like the black market.
It was a local company that was buying it in bulk and then like repackaging it and reselling it under the table.
Nothing like people exploiting a pandemic to make a quick buck.
That's true American ingenuity right there.
Yeah, it's capitalism.
This is the part where we say, yay, capitalism, right?
Capitalism is the problem.
Sometimes I get my speaking points confused.
Yeah.
Let's see how... well...
How many...
All right. So for everybody on Twitch that is new to Drone and Phone, basically what this is, is we run this once a week on Wednesdays usually at five o'clock Pacific Standard Time.
Or 8 p.m. East Coast Time.
And we have a paper that we're discussing. We usually get way off topic very quickly.
And oftentimes we have the authors on and we basically talk about the paper and the surrounding topics.
And it's a really interesting... they usually have really interesting discussions.
There's some like motifs because of our... from our perspectives, like threat models.
Whenever we have an academic paper with a threat model in it, we usually rag on it for a bit.
Unless the threat model is really good.
Because if you've never actually worked in security, building a threat model from a purely academic perspective is hard.
And they do kind of... there's a lot of people who mess that up.
But they at least try and try to figure it out. And when we have authors on, we'll talk about that all.
And if you want to join us today, it's free, easy, open.
You can come jump on the Discord voice chat. So we have AI Village General Voice.
If you've got any questions about using adversarial examples to defend against facial recognition systems, please come jump on.
Stella, since you've got like... you're invested in this paper a bit.
Do you want to describe what it goes over and give us an overview of what's going on?
Okay, sure. So is the author going to join us? Or does it look like no right now?
No, it doesn't look like the author is going to join us.
Someone had a note that they were still trying to get a hold of him.
So that's the last update I heard. I don't think we're going to get him, unfortunately.
Okay, gotcha.
All right, so the paper we're discussing is called Fox Protecting Privacy Against Unauthorized Deep Learning Models.
I hope I'm in the right talk, right?
Yeah, that's the one.
So the background concept of this paper is that there are a lot of ways that people upload photos to the internet.
And companies like Facebook, like Google, process your photo through a machine learning algorithm and use it to identify you.
And they can use this to identify you in future photos.
A couple years ago, Facebook rolled out an automatic tagging process, where when you upload a new photo, it tags your friends in it automatically.
And this creeped out a lot of people.
They were surprised that Facebook was able to immediately identify who they were.
And so the conceit of this paper is to give people a defense, a way to opt out of this kind of analysis without necessarily opting out of existing on the internet as a person.
Another important connection that the authors draw specifically is Clearview AI, which is a now infamous company that did analytics drawing on what was technically speaking public data to draw all sorts of conclusions about people,
where the users were not particularly comfortable with the amount of information that Clearview AI was able to obtain about them.
And this is kind of a repeating motif in public use of technology.
Nobody reads Terms of Service.
Nobody has any idea what data agreements they're accepting and stuff like that.
And even if you did, Facebook has a monopoly on being Facebook.
Twitter has a monopoly on being Twitter.
You either go along with whatever they demand of you, or you just don't exist on Twitter.
And it's not like there's a Twitter 2 that you can go on to and have the same sort of experience, just not with the people whose data policies you don't like.
So an idea that's been incubating over the past year or so in the adversarial AI community is, hey, defeating algorithms like this is exactly what adversarial AIs are supposed to do.
Adversarial AI, or adversarial examples, I should say, is supposed to take photos, transform them in a way that they look normal to humans, but they can't be processed by AI algorithms correctly.
So instead of using these things to fool... these things are usually conceived in the literature as a negative, as a bad thing.
They're used to fool AI algorithms and do harm.
But in terms of data privacy, you can view them as a very positive thing.
You can take an image of yourself and transform it so that AIs that look at it don't think that you're a person, or think that you're a person who you're not.
And then when you upload it, that photo that's been transformed onto the internet, it looks like you to humans.
That's kind of the whole point, but it doesn't look like you to AI algorithms.
And so that's the core idea at the center of this paper, and that's ultimately what they try to achieve.
And what they do achieve with their approach.
In the paper, they refer to it as a cloak.
Yeah, that sounds about right.
To cloak you, but they're going to cloak you in a way that makes you look like an image from this reference data set.
So they're going to make you look like another person or look like another object.
And the reason that they give for this in the paper is that they find that this helps it get past screening algorithms.
That try to notice and reject adversarial examples.
Typically, when you use adversarial examples, you're inserting a lot of noise, and it's not anything in particular.
It just happens to make you look different to the AI.
But what they're doing here is trying to inject some kind of pattern thing that makes me look like Gwyneth Paltrow.
And then once it's made me look like Gwyneth Paltrow, the idea is that the algorithms are supposed to detect when this is going on.
Aren't going to notice because they're going to say this looks like a non-adversarial photo of Gwyneth Paltrow.
I'm kind of salty about this because I spent a huge amount of time last year before the talk I did on facial recognition trying to come up with a single universal adversarial vector that would make one model think everyone was John Malkovich.
I'm super salty about this. I've literally written two-thirds of a paper on this exact topic.
Yeah, definitely an idea that's in the air.
No, yeah, this has been in the air for a long time.
There's a, from an ethics point of view, because a lot of the framework for this is ethically oriented, you know, giving users control over their data.
There's a really good paper from either earlier this year or last year called The Ethics of Adversarial... The Politics of Adversarial Examples.
That kind of discusses the fact that the AI literature like almost universally treats adversarial examples as a bad thing.
And how that's not really fair and discusses different ways that one could use them to achieve good.
Though it's a it's a ethics policy oriented paper and it doesn't do any of them.
So yeah, this is definitely an idea that's been around the block a couple times.
For sure. I know a lot of people who have had thoughts along these lines.
Yeah, I want to say there was a talk at, there was actually a talk at The Village last year. I'm drawing a complete blank on what they called it.
I think they were specifically targeting Facebook's model and they were doing something a bit different.
They were basically breaking the, they were basically breaking like the facial detection step.
So, you know, skipping the, so the point where the system would actually like put the bounding box and clip out the face tile around the face so that like nothing looked like a face at all in it.
And so it wouldn't even go to the next step of like finding the vector for the face and then doing the embedding and then trying to match it.
Yeah, because a lot of a lot of these algorithms use some kind of bounding boxes or try to try to identify, you know, images are complex.
So you kind of have to first pick out where there's something that you're going to look at before you do that.
I don't think I've ever read that paper, but I assume you're talking about like disrupting its ability to find bounding boxes or what faces are.
Yeah. So most, most of these facial recognition systems go through like a three step process, right?
You, you first, you draw bounding boxes around the face and you get a lot of the time.
Most of them, I think you still, still use like landmark based approaches.
Then you use the landmarks to sort of like rotate and realign the face.
So it's sort of like dead on and in a standard orientation for those faces.
And then you put it through the actual facial recognition model, which will take the face and embed it into a vector space.
And then you find like nearest matches within the vector space. And that's the actual recognition step.
So this is what these guys are going after. They're saying, no, take this face.
We'll let you find the face, but now we're going to make, you know, we're going to make Stella look like Gwyneth Paltrow.
And what the guys at AI Village last year did was they said, we're not going to even let you find the face, right?
It's going to look like, you know, like a landscape with no faces in it or something like that.
But it was the same idea, right? It's trying to find like the minimal, like least disturbing perturbation you can use.
That would still like, you know, so your picture looks nice and you can share it on the internet.
But, you know, either you can't find any faces in it or you do find faces, but they're tagged as the wrong people.
Yeah, that is very similar. And I think you just hit on something that's important to call out specifically.
You know, there are a lot of different types of data that might benefit from this sort of masking.
Here, they're using faces and they're doing that for a very important reason.
Adversarial examples for faces seem to be... their performance is not better than it is on other kinds of data.
But for facial analytics, you can make very difficult to perceive to human ones.
So an adversarial example for images and especially for facial analytics looks to a human like you haven't edited the image at all.
When you talk about like audio adversarial examples, like Carlini has a kind of the first major paper on that.
You can hear in the background that there's a hum. And as the strength of the adversarial example increases, the hum increases.
And I've played around with these and, you know, even at the loudest, the kind of noise in the background is, you can still clearly hear what's going on.
But it is definitely perceptively modified to a human.
And so, you know, when you're talking about these being imperceptible changes, because they're talking about, you know, editing your own data and then uploading it to Facebook as if it was an original photo.
That's something that's really important here. You don't want people to have to...
The ideal is for people to be able to upload things that to their, you know, their followers or their friends or their, I don't know, whatever you want from your favorite social media platform.
You want it to be as close to the original content as possible to your desired audience.
Let's say to refer to the people you want to be listening to this, despite being very different for your, you know, your undesired audience of Facebook's analytics algorithms.
Correct.
And it doesn't help much with the, however, 10-15 years that social media has been a thing where it already has tons of pictures of you. So there's...
Yeah, I, as much as I love this line of research from a practical point of view, it really does feel like...
Like, it's like the work is technically great, but in terms of making it, A, accessible to a wide range of people, right?
Nobody...
So, like, a lot of people, like, just don't care about this sort of privacy thing, right? And so they'll update photos of, upload photos of themselves or of you and kind of not give a shit.
And then, you know, and then we've also got the issue that there are tons of places where you just don't have any control over the photo before it goes up, right?
So we're, you know, if you're getting like an ID photo taken of yourself, right, that goes into the system as it goes into the system.
And we know that that's like part of what was being collected by systems like Clearview, right? They're getting like, you know, state ID cards and passport photos and stuff like that. And those are all getting like dragged into this dragnet.
And it doesn't take very many photos of a single person to get a good registration of them for a facial recognition system. I mean, in a lot of cases, you can get away with just one photo.
So I understand that you can't go back and change the pre-existing photos, but couldn't you give the application permission to contact friends who have posted photos of you to ask them to use the application to blur you from them?
By the application, what are you referring to? Because obviously Facebook isn't going to give you that option. Do you mean like the program you download to use this?
Yeah, the one you were talking about blurring the image so Facebook can't identify you. So couldn't you send a message to your friends to use that?
I mean, you could, but it's going to be another hoop for them to jump through. I mean, people are lazy, right? This is one of the biggest problems with doing any sort of security is people are lazy. They don't want to jump through extra hoops.
I understand that people are lazy, but it's easier if you can just invite someone to use a product than having to find the link and send it to them. They're far more likely to use it if you just send them an invite that's really easy to use.
Yes, that's definitely true. I believe that the authors are currently working on a... one of the authors has a note on their website that they're currently developing like a download an exe plug and play pre-trained model for people to use that will be like compatible with Windows OS and stuff like that.
So I believe the best answer is that they are aware of distribution problems and accessibility problems. And that's something that they're actively working on. I don't know if invite links is something that they're building into their program or not. It's gonna be very interesting to see kind of what Sean, Emily, and the rest include. But I don't know if that's something they're developing, but I think you're right.
Another question I have, if you don't mind me asking, is does this cause problems for the visually impaired for reading from like alt text?
I don't think that this is something that these authors... I mean, it's a valid question. It's definitely something that should be considered.
I don't think this is quite what these authors were focusing on in this paper, though.
I understand their focus.
But it's absolutely, yeah, no, like usability and accessibility concerns are major things that definitely...
My thing is, there's currently the interpretation of the ADA has been changed by judges. And now if you can't read the image, whoever created that as the problem can be sued.
It would be... I think it would be on the person who's making it impossible for Facebook to do their job of making it readable.
The ADA doesn't apply to private individuals, though. There's no obligation...
It didn't apply...
If I, as myself, not as a company or as a non-profit or something, but if I just as myself upload a home video to YouTube, there's no requirement that that video be ADA compliant in any way.
That was true until last year when that changed.
Really?
Yes.
Do you know the court case or something? I'm very curious.
Yeah, I can bring it up.
Do we have a text chat? I can put it in a text chat.
Yeah, put it in the AIG Journal Club text.
Yeah, this is a topic I'm very interested in. I have disabilities myself. I care a lot about accessibility.
Honestly, that wasn't something I thought about in this context. And I'm right now on Facebook trying to find out what the alt text for a photo of me is.
I'm having trouble doing so.
If anyone knows, like, you know, I don't know if the alt text says this is a photo of two people or if this is a photo of Stella and Madeline or a photo of Stella and her girlfriend Madeline.
Or just like a user uploaded photo.
But that is definitely a very interesting question as to how it affects people who rely on augmented computer devices.
Yeah, I mean, I think it's like the first step in any of these things is, okay, can we solve the immediate problem that we're worried about?
Can I actually successfully convince Facebook to reliably retag photos of me as photos of John Malkovich?
If you can't do that, then worrying about the accessibility stuff, right, it's kind of like the cart before the horse.
But yeah, before you roll this into a widespread thing, I mean, it's definitely worth doing a careful look at accessibility issues.
And I think there's also other things that probably are worth considering, like, if you deploy this tool so that it can make everyone look like, you know, John Malkovich, whatever.
I'm using John Malkovich as a silly example, but has John Malkovich been consulted about this, about whether he wants to be tagged in a million, you know, photos on Facebook all of a sudden?
I don't know if, you know, like, again, John Malkovich is a made up example, but if you're going to do a targeted attack, like this paper is proposing, who's the target? And, like, do they really want to be the target?
So it's interesting, because the paper kind of talks around that. The paper, like, specifically draws the distinction between private individuals, which is a phrase that uses to refer to the people they imagine using their photos, and celebrities and public individuals, who are the people they imagine being in their database.
And the reason I use Gwyneth Paltrow as an example is because this is an example used in the paper.
Her image is in the paper several times, and it shows, I'm belaying on his name, but the actor who plays Derek Shepard in Grey's Anatomy is another example in the paper. And so it shows you transforming Derek Shepard into Gwyneth Paltrow, and vice versa.
But the idea that these famous people are also private individuals, I think, is really important. It is something that the paper kind of had a swing and a miss on, I think, because they certainly didn't ignore it, but I think you're right to question the way that they approached it.
There is a way to do both.
Can I jump in really quick? We've got someone on the stream who is asking... Tech Critter wants to jump in with a question. Tech Critter, if you're following on the stream, type it in the stream, otherwise type it in Discord. I want to make sure that we get other voices in here because we've got, I think it's been like three people talking most of the time.
Definitely.
Okay, or journalclub-text if you want to just drop a note in.
Yeah, but we've got a couple other people in the Discord voice channel. I don't know if they want to chime in with any comments or questions.
A slot is Robles versus Domino's Pizza.
Yeah, so I just, if anyone's following on Discord, it just got posted in the aiv-offtopic-text channel.
I'm going to post a newer one there. This is the newest one and there's been following cases after this was rolled.
So the Domino's Pizza case says that the ADA applies to online venues such as websites, which was not necessarily obviously true before.
This actually reminds me, there was definitely something that the University of California ran into trouble where they had to take down a bunch of their videos.
I assume this must be related. A bunch of their online education videos that were publicly viewable because not all of them were ADA compliant.
Some of them didn't have transcripts, some of them didn't have colorblind friendly options, and they took down videos instead of fixing it.
And that was a hullabaloo if you're in the UCLA world. I assume that's probably due to this court case or a related one.
So when it comes to your home videos uploaded to YouTube, YouTube automatically creates a transcription.
Unless it's singing, then it has trouble for that. But it creates a transcription and a lot of times it's really messed up and you can go on to fix it, but the transcription is enough to comply.
Right, so I've actually used that a lot because I have trouble...
YouTube refers to it as automatically generated text, right? I've used that a lot because I have trouble following large conversations in videos.
And you already said it's quite bad. And if you're using a version of this for audio, where it's supposed to mess up what you're saying, it could absolutely trash the transcription.
There's a way to comply with the ADA while protecting people from AI. The way you do that is you just need to put something in the alt field that gives a generic idea, like a picture, and then put by the name. That might be enough to comply, I'm not sure.
I mean, that definitely doesn't comply with the spirit of the ADA. If the idea is that alt text is supposed to describe what's happening in a photo, and you just say, this is a photo posted by Stella Biederman, that's...
It may not comply with the spirit, but it's better than nothing.
Yeah, that's a very complicated question, because the ADA has very strong limits on it. If a company argues that complying with the ADA is too expensive, they don't have to.
And the, well, you know, this isn't actually what it was supposed to be, but it's better than nothing, is a pretty contentious argument to make in the disability activism community, because that has been used... I'm not saying you're doing this, but that has been used a lot as an excuse by companies who don't want to be compliant.
I'm just trying to come up with ideas how it can both be privacy and ADA at the same time.
Oh, yeah, no, I understand that. I just wanted to share that.
So, a blind employee at Facebook was the one that created their thing that tells you what colors and everything are in the images. So I'm wondering, maybe if there's a way to obscure the face, but leave in the details like green tree, blue water.
Well, I mean, that's sort of the open question, right? Because as far as I know, one thing that hasn't seen a lot of work is how do like cross model attacks work, right? So if we apply Fox to a photo, right, is it going to completely break like object detection as well as facial recognition?
Yeah, so I mean, it's a totally valid question, right? We don't know one way or the other that applying Fox to a photo would actually break the other object recognitions. It might very well, but, you know, that's an area for future research, as they say.
Are the people who wrote the paper, are they on the DEF CON Discord or Twitch?
Yeah, we tried to get Ben Zhao, I think, to join us. He's the, I think he's the group lead. He's one of the authors on the paper at any rate. Unfortunately, he couldn't make it today.
Okay. Maybe tomorrow?
Well, we're talking about a different paper tomorrow.
Okay. Okay. You guys have a schedule? Like, do you talk about a paper an hour or how does it work?
We got a schedule posted. If you go to the AIV General Text, we've got a schedule and that tells what journal clubs we're doing when.
Okay. Yeah. And you can also find a lot of information on our website, AIVillage.org. AIVillage.org, I think, slash events is our schedule for the immediate future.
I've never seen this village before, but I really like it.
Thank you. So I see that there are some other people in the Discord. Do other people have questions or things they want to discuss?
Couldn't you just remove the glasses with the program? It could just automate it.
I mean, a lot of the problem with adversarial examples in general is that they don't tend to transfer terribly well between systems.
So you can have... I can't remember the group that did the funny glasses thing off the top of my head, but I actually played around with that last year.
And if you use it on one model, it works amazingly well. And if you try to put it on another model, like Amazon's recognition, they're just utterly useless.
They do absolutely nothing to the detection rate.
Oh, sorry.
No, go ahead. Sorry, I'm rambling.
Okay, yeah. So that's actually something that the Fox paper discusses. And at the end on page... scrolling down...
Anyway, I don't actually see a table, but they have a table where they attack several open APIs.
They attack Facebook's... Ah, there it is. It's on page 9, since I'm not the one controlling the screen that's on stream.
But if you scroll down to page 9, they have a nice table there where they show you their attacks against Microsoft Azure Face Recognition API, against Amazon Recognition, and against Facebook Search API.
And you're definitely right about the adversarial sunglasses paper, which has the extremely innocuous name of a general framework for adversarial examples with objectives.
But fortunately, you can find it on Google if you search for adversarial sunglasses. I can also send a link of this paper to the Discord.
Because it's a cool paper and worth reading. Basically, what they do is they 3D print sunglasses that can defeat some facial recognition algorithms.
But that problem is something that there's at least been some progress on, as this paper shows, where they're able to successfully attack multiple open source APIs.
Yeah, the one thing that always struck me about that was...
So if you go back to the original adversarial examples paper, one of the things they were really big on was transferability and proxy models.
So you can train a model off of this data, and any two models, like an adversarial example that evades one model frequently will evade another one.
But all of a sudden you get into the facial recognition domain, and at least based on my experience, that tends to be...
You don't get it for free, you actually have to work for it, unlike a lot of other adversarial example situations.
And I wonder if it's because these object classifiers are looking at softmax versus the adverb, the facial recognition tends to be much more embedding oriented, or if there's some other difference about it that I'm not seeing.
My impression, as someone who has played around with these things, but mostly does more theoretical research than actually trying to use these against real models in the real world, is that this is a problem of complexity.
The more complex the classifier you're training is, the more peculiar its decision boundary is going to be.
And basically what most adversarial attacks do is they exploit peculiarities in strangenesses in the decision boundary of neural networks, rather than quote unquote actually tracking it.
But my understanding is that the more complex the problem is, the more different ways there are for an algorithm to be uniquely peculiar, you might say.
So the peculiarities of a neural network solving a very complex problem are going to wind up less across more and more complex fields that use more and more larger and larger data sets, and especially private buildings.
Yeah, especially for facial recognition, because facial recognition models tend to be looking at hundreds or even thousands of classes.
Yeah, and it's on the face of it, the transferability of adversarial attacks is not something you should expect. Different models should have different strangenesses.
It makes sense to me that the more complex the model seems to be, and the bigger the data set, and the more training time is required, what that is doing is making the decision boundary more and more convoluted, in a sense.
It makes a lot of sense to me that that would decrease the transferability of adversarial examples, though I haven't seen any papers that specifically investigate that correlation.
Mm-hmm.
Yeah.
How did you, when you ran into this problem, how did you decide whether it was a methodological problem that their method doesn't generalize versus a reproducibility problem where, you know, what you implemented wasn't what they really had, and that their model might generalize and yours doesn't?
Mm-hmm.
Hey, just real quick, is it okay that an 8-year-old joins?
Sure.
I am Symphony, I am the 8-year-old. I am the 8-year-old. I have hacked my uncle's tablet and a website once when I was young.
She hacked one of my clients, she did a SQL injection on them.
I don't even know what I was doing, I just did it.
She was 3.
Yet another reason to be scared.
Um...
Yeah, so...
One thing that's really interesting about this paper is the way they do their targeted attack. And I'm not going to try to explain that over voice because I think it's better to just read it.
But something you have to be careful about, they look at... there are a lot of different ways that Facebook or whoever can counter-ban systems like this.
And so we've talked about the fact that they found that it was hard to detect their cloak.
But an interesting and important question is if you know an image is cloaked, even if you can't detect the cloak in general, if you're told an image is cloaked, can you take that cloak off?
And you need to be careful about considering a lot of different options when deploying systems like this because there are a lot of different ways they can suffer.
I actually strongly suspect that if you're told which images are cloaked, you can invert them.
Because the optimization problem they solve, you can basically solve inverse.
Which is a... would be unfortunate. I'm actually currently trying to do that.
Well, I mean, it depends on how the cloak works.
Here they're assuming that you're solving a certain optimization problem against a publicly known and everybody set of reference images.
I'm certainly not going to say any cloak can be undone.
I certainly hope that's not the case, because I think technology like this is... and I hope that technology like this becomes very useful in the future.
I believe that's an open question.
This is something we've actually been discussing in the AI village recently, though.
Can you reduce the... there isn't a good notion of hard for AI, the way there is hard for deterministic computing.
But if you take the image recognition problem as an adversarial example, can you use that as a hard for AI benchmark?
Can you design a system where detecting the cloak is exactly as hard as recognizing the image in the first place?
There's been some interesting conversations along those lines that I'm sure we'll continue to have.
Yeah, but Rich did mention a specific paper. Is he still on the line or did he drop?
So Rich definitely talked about a paper that does exactly what you're curious about, EVS, and we can pull him up and get that paper sent out to the Journal Club text.
For you.