Now we have some audio. Okay, so everyone somewhat speak?
Okay, now try again. Yeah, now we have audio.
Hello, good. Can you hear me, Sven?
I can hear you. Hey, Irem.
Oh my gosh, it was just a simple Windows reboot fixes everything.
Hey, I unfortunately, I actually have a meeting right now. So I'm going to drop and join again.
What time do I need to be on for speaker time?
In 30 minutes.
But only in Discord or also in Zoom?
Only in Discord.
Okay.
Yep, so we've got people in the Slack chat or in the Twitch chat saying they can hear us.
So I think we're good to go.
Sweet.
Well, hi, everyone. Sorry about the audio issues.
That seems to be the start of every stream is figuring out the audio.
The only people I've seen get it right are like Critical Role.
They pay a lot of people to get it right.
Yeah. Welcome to AI Village at DEF CON 28 safe mode.
We're all doing this from home.
I hope that this can be as fulfilling to you guys as it is to us.
And we just want to start off the village with some like general questions about the
state of the field as it is right now at the end, halfway through 2020.
So Jag has got a bunch of questions prepared and the rest of us are going to try to answer them.
Okay.
Welcome, everyone. Thanks.
Thank you for being here.
The first question that I have listed is...
Let's just knock this one out of the way real first.
Facial recognition is a security concern for us.
In the ML field, of course.
And before you answer, please introduce yourself.
Yeah. We should probably do a who is who.
Sven, do you want to kick us off?
Sure.
So I'm Sven.
I am a senior data scientist at Elastic.
Before that, I was a topologist who accidentally got into machine learning and security.
And I kind of helped run the AI village.
John, do you want to go next?
Yeah, sure.
Hey, everyone.
DeltaZero here.
I do data science for intrusion detection.
Yeah, I've been in the field for maybe, I don't know, four years now.
But pretty awesome.
Glad to be here.
Rich?
Hi, I'm Rich Huerang.
I'm a director of data science research at Sophos.
I've been working at the intersection of machine learning, security, and privacy for probably
about 10 years now.
Got my PhD from UCSB.
Spent some time with the US Army Research Laboratory, and I've been at Invencion Sophos
ever since.
Yaga?
Hi, I'm Yaga.
I am a professor at a small university in Virginia.
I've got about 25 years of experience in security, but relatively new to machine learning.
My research is focused on securing machine learning, and I like playing video games too.
So there's something interesting about it.
Anita?
Hi, I'm Anita.
I'm a research scientist at University of Illinois.
Started life as a cryptographer.
Been in security for almost 30 years.
Okay, and Will's the last guy.
Certainly not least.
Yeah.
Sorry, my morning's just starting.
Will, I am a network operator, and I guess attack machine learning models and use AI
for offensive purposes.
Phishing, pathfinding, and stuff.
Yeah, Will's being modest.
He actually has the first CVE ever assigned against a machine learning system.
If you attend my lab later, there'll be another one.
I think that's a challenge for everyone.
Can you get one file before Will drops number two?
Okay, thank you everyone for the introductions.
Sorry, I should have thought about that.
Please be sure to ask any questions over in the Discord server.
There is a talks channel.
Yes, there's a channel for talks.
You can ask in there or in the general, and we'll try to catch it.
So back to the first question.
This was one that was both easy and hard to throw out there.
But is facial recognition a security concern when we're talking about privacy?
For sure we know about, but in the general machine learning terms,
is facial recognition a security issue?
So I think just to make the point in our village, one of the things we're trying to do
is expand from just thinking of this as a technical kind of algorithm problem.
If you look at the broader scope of facial recognition, there are issues not just with
security, with data sets, with data integrity, with these kind of choices.
So I think my answer would be it is a security issue.
It's a machine learning issue.
It's a data issue.
It's a human issue.
And for us, we need to think about all those things as technology people.
Yeah, and just to add to that, a lot of the issues around facial recognition
are less to do with the technology and more to do with the processes that that technology is
embedded in.
So if you have a facial recognition system that works, you have issues around privacy, right?
Who gets access to that system?
Who gets to know who was seen where on which camera?
And then when it doesn't work, you have bias issues that sneak in.
So how do you control for false positives, right?
Someone gets reported as being, you know, facially recognized as a bank robber or something
when it's just like an average person.
And this actually extends beyond issues of just facial recognition.
Just computer vision in general has problems like this.
There was a story recently about a license plate recognition system, which led to someone
being pulled over and detained, you know, face down on the side of the road for hours
before they figured out that it was the wrong license plate number, the wrong make of car,
the wrong model year, et cetera, et cetera, et cetera.
So yeah, it's definitely a case of not only does the ML work, but do you have the right
controls around the ML to make it actually usable?
And have you, you know, assessed what happens when it actually fails so that it can fail
gracefully without hurting people?
Anyone else?
I think the last thing, not to dominate this, but the last thing to throw in is I feel like
in our village, part of our job is to also translate all this technobabble sometimes
to policy speak and to real world speak.
And that could be very challenging.
We know the technical parts.
What ends up on the other end, we need to bridge that gap.
I think that's important to understand that security isn't just the technical side.
There's a human side to it too, and that extends to policy.
So definitely something to think about.
Thank you.
Staying along the same lines, one of the things that I wanted to ask about is the FOX program.
Have you heard about it?
And how does it affect our ability for the end user to take back some of that control
over their data?
It's technically, again, this is, it's the gap between technical aspects of it and
implementation and social aspects of it.
Technically, it's a brilliant idea.
And, you know, it does what it says on the tin.
In practice, there are so many ways that people get registered into facial recognition systems
from, you know, collecting ID cards to photos that they uploaded before FOX was a big thing
to photos that other people take of them and maybe tag them in later, that you really can't
just find a quick technical band-aid for security problems that are really a combination of
social and technical problems.
The root cause here isn't like, oh, we just need better technology to fix this.
The root cause is society and sort of laws and norms haven't kept up with the technology.
And that's the angle that we need to address it from.
Technical band-aids are only ever going to be available to people that know about them,
know how to apply them, and can get over that barrier to entry.
Really, we need to be thinking broader about impact to society as a whole.
It makes me think a little bit of this field of, you know, usable privacy and, you know,
privacy by design.
See, that's analogous here because people don't understand that long-term impact of
having a photo op.
So, enabling people to use this tool is great.
I think the average person isn't going to even think about it because it doesn't affect
their day-to-day life.
So, again, I think the issue is, how do you express to them the long-term potential damage
of having their pictures out there?
Okay, we're going to move into a different arena.
What do you say the latest advances that your typical security team could use
in terms of machine learning?
You know, a lot of vendors are out there saying that this is going to be the panacea for them
to be able to fix things.
Some of the actual products that will help them or techniques that will help them.
I just want to take this opportunity to plug the panel that we have coming up on Saturday
on whether or not ML and InfoSec lives up to the hype.
So, we're going to be diving, hopefully, pretty deep into those questions.
But I'll kick it over to someone else since I've been talking a lot.
I really don't know what the boots on the ground could be using.
But there are some promising things coming down the pipe from researchers.
But I...
So, I'll jump in real quick. Sorry to dominate again.
And I'll let Will speak because he's more an operator.
For me, federated learning, federated and privacy-preserving machine learning is very
important for a bunch of reasons.
Not the least of which is being able to learn across edge devices that are in multiple disparate
places and being able to do it more efficiently.
And being able to use new types of data because there's privacy-preserving ways to do it.
So, I think I see federated learning as something that's promising.
Yeah, I'd say from a product standpoint, it's just like the simple stuff.
So, you know, on a network, clients will still miss password spraying internally.
And that's just something that shouldn't be happening with the logs that are available.
So, I think in terms of just instant response and stuff on networks, it's the simple stuff
that is going to give you the most bang for the buck.
Obviously, there's a lot of competition in the machine learning space, or it would seem.
And half of me, at least 40% of me, thinks in two years, we're going to be moving on
from machine learning as a solution in networks.
And we'll probably replace it with like zero trust or something like that.
But there's also the other 10% that thinks like it will always have a place.
But just like with the facial recognition stuff and the bias, if we don't put it out
in a way that's useful, and in a way that people can understand,
the minute it breaks, they're likely to just throw it out the window and say it doesn't work.
Even though, you know, it is a perfectly viable solution. But yeah, I'd say simple stuff.
Yeah. Okay.
To me, that's the more important side of the question is like,
practitioners on the ground, we don't need any new math or anything new interesting.
But there's some basics that they can get their hands on that will probably help them out when
they have to deal with a deluge of alerts from some sim that they have to deal with.
A little bit of clustering could go a long way in that situation.
Yeah, I think that's perfect.
That's perfect for what I was going to say is, I didn't mean to say products,
I probably wrote that down wrong. But like, what can we tell our audience here that maybe
practitioners that they can do on their own? So I think something like our workshops,
if you take a look at them, they'll be able to help you
see that you may not need to go get these expensive products. There's some simple things
you can do. Yeah.
Yeah, and I just add to that, but from my own experience as someone who's
used ML to tackle a lot of security problems, very often you don't even need like the big
complicated deep neural networks or gradient boosted decision trees or pick your flavor of
the day. Very often, it's enough to have a good idea of the problem you're trying to solve and
a good set of representative data and then simple methods like, you know, linear logistic
regression will often be enough to tackle it for you. So, you know, you don't need to think that
you've got to have tons of like Amazon EC2 credits to run a big beefy GPU machine.
A lot of times you can get really like 80% solutions, really good solutions just on your
laptop. Okay.
For most security to be able to defend yourself, you have to be able to identify
the issue or identify the attack. We discuss, you know, adversarial attacks quite a bit
in the academic realm. How could someone tell that they're seeing an adversarial attack in practice
today? You mean on the AI? I'm throwing out some hard ones. You mean on the AI portion?
Well, how could you just notice an attack? Because to defend against something,
you typically have to be able to define what that attack looks like, right?
At least to some degree. Can we do that today?
It's kind of too bad Hiram's not here because I feel like he'd have a lot to say on this topic.
Let's hear it.
So for the traditional adversarial attacks, we basically are trying to modify an image
or something slightly to bypass the machine learning system. That doesn't really work
with security because our data is much more structured. You can't like do a gradient descent
on a binary and then hope it still compiles or runs because you're going to get byte codes that
are just wrong. So I don't think like adversarial attacks as academics that you read up in a big
year paper are going to really apply. There are some modifications to that that might apply that
might be very helpful. But what we do see already and have been seeing for like a decade is people
just trying a bunch of stuff and then they get past it. And the more they get informed they are
about how the machine learning model works, the more effective and more like gradient descent it
is. You know, like the more effective they are. Like for like the spam filters from back in 2001
when people were packing Wikipedia articles at the bottom of their spam messages to bypass the
naive Bayesian classifiers, they understood that it wasn't a naive Bayesian classifier. They
understood that they needed to pack the thing. And then they did that. And that's going to happen
with next-gen AV products too. It may already be happening. I mean, that was what we saw in
the silence bypass was effectively a stuffing attack. Yeah. That was researchers being nice to
us and telling us about the problem that's happening. So this one is more directed at Will,
but definitely want everyone's opinion. And it gets back to something we talked about in a journal
club, which for those of you that might not have seen it, we may be replaying it during the weekend.
Will received a CVE for the vulnerability he discovered.
But do we need actual CVEs specifically for machine learning or should we keep them more general?
So I guess first I'd like to say it was our, so Nick Landers and I did it.
I just talk about it more so. So the CVE system is designed to communicate risk.
And I think anywhere that risk is present, then a CVE is appropriate.
You know, I just, I submitted it. So I'm not the one that gets to say whether or not it's a CVE.
So I guess that the question would be for the NIST. And I know they've come out with a paper
that kind of highlights machine learning attacks or there's a taxonomy. It's like 15 taxonomies
right now. But yeah, I think in any way there's risk that is at least measurable, a CVE would
be appropriate or any standardized score. I know there's operators like...
And I'm sorry, I don't mean should it be a CVE, but more should there be a class just for machine
learning, you know, because they break it down. Yeah, yeah, I think so. It's fundamentally
different. You know, so for example, like with a traditional exploit, you know, there's like
a patch and then you can push it out. But with, for example, like with Proofpoint's model,
there's some residual, there's some like lingering theft going on. So
once you have that copycat model, you're never going to get 100% of their model. But
for some, there's some decay into the future, but you'll have some portion of that model.
Your copycat model will be right, you know, up to some future time until they've retrained enough.
But in their response, you know, unless they do things to fix it or prevent, you know,
people from stealing outputs, then you can always create a new model. So I think that
piece is important. But yeah, it's a little, it's fundamentally different beast than
than like an exploit. Yeah, I don't know, you guys probably have a better idea of like,
how you're supposed to patch a machine learning model or you take it out. I don't know.
Long and expensive retraining process.
Do you guys agree, though, that it's good to have that separate machine learning
kind of category to highlight the fact that this is a vulnerability, a specific type of vulnerability?
Yeah, I think most, so there are some exceptions. And, you know, again, I'm going to sort of like
promote the Journal Club on the Summoning Demons paper later this weekend. But most machine learning
exploits don't translate into exploits as we usually think of them in InfoSec. They are much
more either model stealing or model bypass, or maybe like a denial of service type thing. So
in my mind, keep in mind, I'm coming at this from the ML side of things and much less from the
InfoSec side, I do think it makes a lot of sense to have separate categories that reflect the
different kinds of risks that we currently think of ML models as being exposed to.
Okay. So that's kind of all the questions I had. The next was, I was going to turn it back over to
Sven to kind of give you an overview of the weekend. And thanks for being here, everyone.
And thank you for your questions. Or answers. Yeah, cool. I'm really looking forward to the
talks. This is going to be a great DEF CON. Cool. I'm out. Later, folks.
Have fun. So about this weekend, the way we're structuring it is everyone, all the talks are
recorded already and on my desktop. So the speakers won't be talking live. The reason why we're doing
that is so that the speakers can focus on answering your questions in Discord. And so during the talk,
speakers will be in the Discord and hopefully afterwards. So any questions you have about the
thing, you can just ask it in the Discord channel, and we'll answer it. The speaker will answer it,
or maybe someone else who's knowledgeable on the subject will help try to answer your question.
So we're going to, in a few minutes, launch into the first one, which is we've got Zoltan here.
Zahak, who's in Hyrum. They'll be going over to the Discord to answer any questions.
And then, yeah, then we've got Eriks and a few others. Oh,
Sarah, do you want to introduce yourself, too?
Oh, god. SJTurk, ethics chair, so I guess the buck stops here.
I guess data nerd who's working on disinformation.
I think that's it for the intro stuff.
Yeah, that's it. We were trying to leave some time to make sure the transition to Hyrum.
Yeah, I've got that all ready to hit play.
Okay, yeah. Oh, workshops. Yeah, so there's a few workshops.
They're mostly focused on sort of bypasses and interesting techniques on...
The hard part for us is text data. So, GtKondike, Gavin Kondike, well, has a lovely workshop on
spam builders. And we have... Will here is talking about a new, basically, CVE2.
Model stealing stuff.
And the way those work are they'll be streamed on Twitch live. They're not recorded. And then
there'll be a few people who are TAing who will help the workshop presenter help relay questions
from Discord over to the workshop presenter and help resolve any minor issues you guys might have.
There are Jupyter notebooks and stuff posted on the website.
So, hopefully this all works out. It's a pretty normal machine learning workshop structure.
Just online versus in person so that you can't have someone spy over your shoulder to tell you
your typo is. Well, with that, let's start Hyrum's...
Okay. I'm gonna go ahead and drop off. Hi, everyone. Have a great DevCon.
We'll do. Have fun. Awesome. Thanks all.