Okay, say something.
Nope, the audio is definitely broken again.
Now say something.
Time with the audio issues.
There we go. The audio seems to be working now.
All right, awesome.
Yeah, I almost feel like we got to have... whoever
is the poor soul stuck with jockeying this in the future almost needs to have two laptops,
one with Twitch and everything going and one with just OBS and Zoom, yeah.
Yeah, that's...
Nothing can confuse OBS's tiny little brain.
But then the problem is when you forget to mute that thing and then this thing
and then there's one of your tabs is doing the wrong noise and you're like, oh, this is wrong.
Yep.
Sven, you're being heroic. It's just...
Yep, we are live, by the way. The internet is listening to us talk, so enjoying the banter.
You're still being heroic.
Yes. Yeah, so for those of you watching, Sven has been the main guy troubleshooting
all of the audio issues and fighting with the stream for basically all of DEF CON.
And it's a testament to his hard work that things have gone as smoothly as they have.
So you've made it a lot easier that things were pre-recorded.
Can you imagine trying to hunt people down,
hunt speakers down, and then their sharing is not quite right and all that?
Yeah, no, I mean...
I mean, this year, at least they're not passed out in the bar.
How many speakers have been passed out in the bar?
None of ours that I know of, but we've definitely hooked a few of them out from the nearest bar.
Yeah, well, you know.
Are you expecting anyone else or we want to get this show on the road?
Yeah, let me... well, let me pull up this thing for myself.
I'm going to try a new layout as soon as I can find the thing.
So while I'm looking for it, Rich, do you want to introduce the paper to everyone?
Oh, sorry. So we're going to do a couple of things.
Hi, Barton.
So we're going to start with just the four of us talking about it for a bit.
And then I'm going to open up the...
and join the voice chat in the Defcon Discord.
And so you guys will be able to talk on stream and talk with us and everything.
So for the first, like, 20 minutes, it's going to be just us.
And then we'll bring everyone we can in
and have a bit more of a Wild West thing.
We're going to try out how that works.
And the other thing is I'm just going to play with the layouts a little bit.
So that's the structure of it.
We've also got a journal club coming up on this Wednesday.
We will put the link to that paper in the Defcon Discord.
And then if you want to participate, you should join our Discord.
Link in the Twitter bio.
Yeah, it's in the Twitter bio.
And you can just DM one of us.
And we'll send you an invite to the AI Village Discord.
Please don't use the AI Village Discord while Defcon is going on.
Please be respectful of the hard work they have put into building their Discord
and getting it running.
And only use it, like, tomorrow or Tuesday.
So take it away, Rich.
Okay. Cool.
So this is actually a slightly older paper.
It's from 2017.
It is called Summoning Demons, the Pursuit of Exploitable Bugs in Machine Learning,
which is probably one of the coolest paper titles I've come across in a while.
And it's by Rock Stevens and Tudor Dimitrescu's group.
So the basic idea of the paper, it's kind of an interesting spin
on the adversarial ML papers that I think started coming out about the same time.
And in this, rather than actually, like, using the ML model itself to sort of guide inputs,
they're looking for more...
they're more looking for flaws in how the different components of the system hook up.
And to do that, they actually crack out an old classic American fuzzy lump, the fuzzer,
and they re-instrument a lot of it so that weird actions from the ML components of the system
register as crashes to AFL.
And so that lets them essentially do sort of like a black box adversarial attack,
if you want to think of it that way, against the ML system.
And what they find is that in addition to sort of the usual stuff you can get,
where you have, like, inputs that look like faces that don't register as faces,
because they've had some pixels perturbed,
they also run into, like, image format parsing errors,
where a component of the ML system won't parse, for example, a face correctly,
completely fails to find the face.
But if you open it up in other programs, it actually looks good.
So I think this is kind of in the vein of the same sort of things
that Ariel Herbert-Boss talks about a lot,
where it's not just attacking the ML system itself that's important.
You've got all of these other components in the pipeline,
from ingesting the actual image all the way through feature extraction,
pre-processing, and so on.
And each of those is actually a point that you can attack.
And so in this paper, they're basically using fuzzing as a way to find
some of those, you know, broken assumptions or breakdowns
in the link from one stage of processing to the next.
And I thought it was a super clever paper with a really good name.
And they dig a couple of CVs out of it, too.
So super cool application of sort of more classic security methodology
to what, from my perspective, is kind of an adversarial ML problem.
So that's my, I guess, like two-minute summary of the paper.
So I guess maybe to kick off conversation,
so does this, do you feel like, does anyone have an opinion,
strong opinion one way or the other,
if this falls into the same sort of general field of adversarial ML
as sort of like gradient-based attacks?
Like I'm trying to think of, the name escapes me off the top of my head,
like fast gradient sign method, or like these Jacobian-based methods,
or saliency-based methods, or maybe even like,
you could almost argue it's like a genetic algorithm-based attack
against a particular ML pipeline.
Well, to be honest, I thought that adversarial ML covered all this already.
I was a bit surprised that it was new.
Yeah, it's genetic algorithm-based because they're using FuzzyLock,
which is genetic algorithm-based, which is also interesting
because there were network extensions to genetic algorithms
that might be more appropriate if you're searching spaces.
So there's some interesting stuff on the front that could be done.
I kept wanting to say, what happened next?
Likewise, in the context of healthcare,
practical realities of securing systems,
it seems like a lot more emphasis right now is placed on more theoretical attacks,
where it's more so once you have uninhibited access
to being able to query this API that's being explored.
So it's nice to see a paper like this that's a bit more
in an all-systems set of things,
because at least from healthcare IT,
it seems like it's a more likely encounter that you'll find in the industry.
Yeah, one thing that I did really like was their focus on implementations
rather than just theoretical properties, right?
Because in some sense, something like the FastGradientSign method
would apply to almost any of these models.
But here, they're sort of like, okay, this model was instantiated in this way
and is doing these things.
And now how can we attack sort of that specific version of the model?
Because it's a lot less academic in some ways,
and a lot of the adversarial ML research seems to be very academic still.
This was like down and dirty in how do we attack this thing.
It's kind of in my vein of stuff.
Just look at the system, work out how to break the system,
and take it to pieces.
I loved the way that they hacked that little hack on FuzzyLop,
where they went, okay, we're going to take out the things that actually broke,
and then we're going to induce breaks for the things we care about.
Yeah, changing the target.
Yeah, changing the target.
It was just beautifully done.
I thought it would be useful for defense as well,
because we're also looking now at things like Strafe's work,
so Brittany Prostakoff's work on attacking with robots.
What could we do to screw up their vision systems?
So how could we mess that up?
There's some beautiful adversarial there too.
They can't see, they can't move.
Yeah.
Yeah, so actually, I thought table one in this was...
I don't know, I'm surprised.
Flips to table one.
Flips to table one, yeah.
Waves to table one.
It's, I think it's on page four.
Yeah, yeah.
Yeah, page four.
Yeah, they actually like, I feel like, again, this is very much like
Ariel's jam, right?
They go over sort of the attack surface and the kinds of things you can do to it.
And it reminds me a little bit of the contest there was...
So I think it was a late entry to the contest last year
that was announced at the AI Village where they were trying to do ML evasions.
And one guy actually found essentially a bunch of ways
to just completely trash the feature extraction
so that the ML prediction just wouldn't run on the samples he uploaded.
And so, yeah, he was a little upset that that didn't count as a win,
which I think is justifiable, right?
Because if you can just like stop the analysis cold, right?
That's almost as good as evading it.
But yeah.
It depends what you're trying to do if you kind of sneak into a system
and it's a bunch of systems, right?
We just want to, you know, the usual thing.
Get in, do your thing, get out without no one noticing.
And we see enough of that in real life of the stuff.
It's like, oh crap, we didn't realize that was a problem.
Before someone else adds them.
There's also another thing to that.
Like if you wanted to attack the Ember model with the Ember dataset,
it only looks at your headers and you could just change other parts.
Like don't even think about it in terms of adversarial ML.
Just copy like a notepad.exe's header and just put your own
entry point function in and start launching weird stuff from there.
And that's a different way to bypass the thing that is also not...
Academics also not aware of.
Yeah, so that's sort of like...
I guess that's maybe like a...
You can classify that if you're gonna...
I'm gonna screw this up.
So if I'm getting the terminology wrong, security people, please like yell at me.
But it's almost sort of like a logic attack on the feature extraction
rather than what they're doing here,
which seems to be more going for crashes, right?
If you know how the feature extraction process works,
then you can sort of write your way around it deliberately
rather than just throwing shit at it until something blows up.
Some of it's blowing it up, some of it's sneaking in the back.
Some of it's like sneaking in the back conventionally depending on your system.
The other thing that interests me is that only some of these became CVEs.
That the response to the people who are responsible for the algorithms
for some of this was like, won't fix and who cares?
And that's part of our journey, I guess.
It is just like every other security thing
is trying to get people to accept that yes,
you may have to drop a little bit of responsiveness in your system.
But you'll get it more secure and it matters.
Yeah, I'm kind of reminded a little bit of some of the argument
that went around for the proof point CVE,
where people were arguing whether or not
that actually counted as like an information leak CVE.
Because so if people aren't familiar,
the short version is the people who did it,
Will Pierce and someone whose name I'm forgetting, I'm sorry.
They found a way to essentially like send an email
through proof points, spam detection service
that would then bounce back to them.
And in the header of the email, it had a score for how spammy it looked.
And so that essentially was the information leak.
The fact that they could get that score back
and that was enough for them to rebuild essentially a proxy model for it.
And yeah, over on in various like AI village discussion channels,
we had very strong debates over whether or not
that actually should have counted as like an information leak CVE.
Because, you know, it's in a lot of cases,
it's useful contextual information.
But when you look at it in an ML context,
you're actually enabling like a model stealing attack.
Which again, it's one of these, we're kind of in new territory
as far as the security of these ML based systems goes.
And a lot of people that are more familiar
with conventional security, maybe don't quite get
what these various information leaks or attack surfaces could do
once you wrap it in like how ML based systems work.
And I think another thing we get out of this is by doing this sort of attack,
we start showing where the things that break up.
And one of the things I'd love about the fact that MLSec exists
is it makes machine learning better.
Just by dint of the spotlight we throw on it.
You chuck hackers at it, they take it to pieces,
they show you where it's broken.
And hell, we spend most of our life trying to get our damn systems to work.
But I'm going to give a little bit of history.
I'm old.
Jump in for just a second.
I just saw in the Discord chat.
So it was Will Pierce and Nick Landers who got the Proofpoint CVE.
So I'm very sorry that I forgot your name, Nick.
Okay, sorry, carry on.
I was just going to say about sort of some things, the findings,
that it's double float precision thing.
So way back before we had, I think we had Python,
but we sure as hell didn't have scikit-learn and friends.
We had matplotlib.
I'm not... matmap.
We use MATLAB for a hell of a lot of machine learning stuff.
And there was a neural network kit in the MATLAB.
And that neural network kit suffered horrendously from underflow errors.
You could break it really, really, really easily.
So Aston University built its own.
They built a separate kit because they couldn't get the first one fixed.
I mean, this sort of stuff, like the float to double errors.
I mean, crossing my fingers that actually all of those hidden faults
that are probably still going on with neural network stuff and deep learning,
we're not even noticing because we just trust the systems.
Yeah. I mean, even in Python, right?
Like under the hood, a ton of these neural network libraries,
and PyTorch, at least, sklearn as well,
use pickle, which is just broken kind of by default, right?
It's a stack machine that you can write arbitrary code for.
And yeah, people still share wait files that are essentially pickles.
And so you're downloading this off the internet and running code.
And I keep threatening.
One of these days, I'm actually going to write a pickle poisoner
that just replaces calls to the relu function
with some sort of stochastic function
so that half the time it works and half the time it gives you garbage.
Comment from Discord.
Yeah, numpy as well uses pickles.
So it's everywhere.
Keras, yeah.
Is that what the numpy save file is?
I thought that was secure.
I'm not sure.
I know, I'm pretty sure it has one...
One pickle thing in there.
One or two.
It's got one or two formats that are not.
Apparently, there's a couple that...
The scikit-learn form that these guys found, numpy was part of that.
I'm just flicking to the page.
Page seven at the bottom, if you're reading along with us.
One interesting thing is that they weren't able to induce misclassification
or from the precision loss problem.
So they tried different values of epsilon,
and basically said that it's possible in theory.
But it's interesting to consider how would you model that epsilon
based on the syntax to induce this misclassification.
How much of this is induced by...
Basically, all the machine learning code is written by academics
and over-enthusiastic academics.
Guilty.
Who aren't experienced in...
It's incredibly hard to write numerically efficient code.
You don't have time to really learn how to multiply matrices incredibly efficiently
and make sure you open files correctly.
So we've chosen a lot of lazy paths to formatting and opening things
versus secure paths.
And also, you've got tiny numbers and people don't...
I mean, you go wading on in, build your stuff,
and don't think about how those numbers interact.
You just assume the system is going to magically deal with it.
Yeah. And I mean, there's also the fact that
writing secure code is very much its own skill set, right?
Some people are writing research code
because they don't expect it to be used in Angular,
they don't expect it to be used in a production context.
Some people are writing numerically efficient code
because that's their jam and God bless them.
And then you've got sort of security people
who are probably gazing at a lot of this
with sort of a feeling of vague horror that they're actually going into production.
Yeah, I know even in my day job, right,
a lot of what we've been doing recently
is sort of going back and rethinking how all of this stuff is put together
so that we can feel confident in actually pushing out products containing it.
There's a middle ground where you can enable basically more flexible code,
but also execute it in a way that perhaps minimizes the blast radius.
I don't mean put it in a container, obviously,
but it's something more to the point of running code in secure enclaves.
For instance, the work with silo type of workflows
or do you see anything where you try to mitigate the impacts of this code?
And could we have code checkers?
I mean, literally just some way, some sort of test data sets.
But we know this stuff is bad. Run it through your system.
Yeah.
Yeah, I mean, I had a great amount of fun, again, at my day job,
actually like taking our feature abstraction code
and fuzzing it just to see what would happen with it.
Fortunately, it turned out that I spent more time than I care to think of beating on it,
and it turned out fine.
But yeah, if it hadn't, what would we have done then, right?
And I think the idea of building test vectors of just really bizarre,
but technically conformant files for this sort of thing is an awesome idea.
But extending that to the actual data science part,
how often have you issued a PR about some data science thing
and sent it to someone and they're gone?
It looks good to me.
But there's some stuff that you can check easily,
but a lot of the code that goes into BERT is incredibly complicated.
Very hard to check.
When you make a small change,
you could have broken things fundamentally because it's software.
How do you write a unit test for I broke BERT when you input this one slight thing
and now it causes this whole class of things to be false positives, false negatives?
Like, how do you unit test machine learning?
There's a... I mean, you can do...
I mean, essentially, you do test vectors, right?
You say, you know, if I load...
You can consider the weights are obviously part of the model.
You say, I've got this set of weights.
If I load in, you know, this set of features,
I should get this exact set of outputs up to numerical precision,
which, again, numerical precision is kind of its own nightmare.
Yeah, but then if you change the weights, right, then kind of all bets are off.
And that's one of the painful things about debugging ML,
even when you're not under adversarial conditions,
is just, I retrained the model, do I still trust it?
I mean, there's a difference between I messed up my models,
which is like, we do this all the time.
And somebody is actively trying to attack my models
or may actively try to attack my models using known vulnerabilities.
Because sure as hell, half the systems out there haven't patched any of this.
How many machine learning developers even know unit testing
or even know good software practices?
Like, a lot of us came from academia.
I have a PhD in math, not software engineering.
So I have a philosophical question I want to pose to the panel from Discord.
And Yaga asks, what is a terror and...
Sorry, what's a terror? This is terror right here.
What is an error and what's an attack?
And like, is there any meaningful distinction between the two?
I guess I would follow up.
I think if you're looking at adversarial images,
I mean, the fact you have like the little image patches
to screw up your system is an attack.
It's not an error.
You just don't randomly get sort of extra patches on a stop sign.
No matter how much undergrowth you got on it.
I think the difference is intention.
Yeah, I mean, that's the same thing in security, right?
Bugs can be... I mean, exploits are essentially software bugs, right?
Maybe that's being too general, but if you...
You know, a subset of software bugs or exploits
and you use them to make the system do things you don't want it,
it shouldn't be doing under normal circumstances.
Or wasn't intended to be doing.
Right.
And sometimes they're useful.
Would you consider a well-informed person building an attack
against a machine learning system
where they understand the machine learning system,
they understand the feature extract and all that stuff,
and they've built a false positive.
Is that an adversarial attack against the machine learning
or is that like an exploit?
Like, where would you classify the mistake?
Is it like a software mistake or a exploit or what?
I think that's the tricky bit about ML models, right?
I mean, they're inherently statistical.
So it kind of gets back to testing after you retrain it, right?
So if you have a single example
and it turns out to be a false positive,
and maybe that's one of the bridges
that I feel like as a data scientist,
I have to cross a lot when I'm talking to security specialists.
If I have a model and it FPs on a single file,
that doesn't call into question the efficacy of the model, right?
You have to look at the statistics of what the model does, right?
And if you say, well, yeah, it got this FP
and that was a pretty boneheaded FP,
but for 99% detection, you get one FP in 10,000 negative samples.
That's a pretty good model.
And so being able to sort of drive that switch from,
hey, here's the good, bad to,
hey, you have to think of this sort of heuristically and statistically.
Is a big part of just like communicating as a data scientist, I think.
Go ahead, Sarah.
Yeah, if you've got somebody actively attacking,
then maybe some pre-processing is enough to cut down the attacks.
Yeah, yeah, for sure.
All of that goes out the window when you've got someone maliciously trying to...
Yeah, but so my question is,
so if you have like a fancy ML from an academic paper bypass of your model,
versus someone trying things, what would be...
When is it like a machine learning mistake
or versus a mistake in your featurizer and stuff?
If you look at the decision boundary,
basically for something and seeing how the perturbations relate to that?
Yeah, I kind of feel like maybe to pull it back to this paper a little bit,
if it gets through your feature extraction process in one shape,
in one piece, then maybe it's a modeling issue.
And if it crashes your feature extraction process
or it gives you something that you wouldn't expect
from your feature extraction process, then it's a featurization bug, right?
If you feed garbage into a model,
you can't blame the model for producing garbage outputs.
How would you basically...
What are some methods for quantifying unexpectedness in this scenario?
Is it more like error counts or latencies in response?
What would you pay attention to if you were trying to understand
if something was misbehaving?
You know what, when you see it.
Yeah, like if you take a look in the paper, right?
They've got an example. I want to find it.
Figure two, right?
So they show an example of two different images.
One is an image of sort of the very top of a person's head
and then the rest of it is gray.
And one is a picture of a person's head.
Yes, there you go. Thank you.
Page six, if you're following along.
Right, so OpenCV has a bug that these guys found
by their guided fuzzing technique,
which means that OpenCV can't properly load the file
and then you run it...
Obviously, you run it through feature extraction
and it doesn't get anywhere.
So it's like all these different individual components
that you got to worry about, right?
And I think on that topic, there was a...
From Hacker Factor in Discord, he's got a question.
Are you making a distinction between the ML model
versus software that drives the ML system?
He says fundamental error versus algorithmic.
And I think here, right?
This is what this paper is going after, right?
It's saying the software that drives the ML system,
the implementation of the system also has bugs
and they find them here.
And, you know, if you go to table two, right?
They've got in OpenCV, they've got like two heap corruptions.
They've got a weird rendering bug that they found.
Page six, thank you, yeah.
Right, so all these different...
Like they literally have like heap corruption bugs
that they're discovering with these funky inputs.
And that has nothing to do with the model.
That's you're attacking the plumbing around the model, which is...
But you can use that plumbing to go and tweak the model.
Right, exactly, yeah.
So that's what the third line down does.
That's the figure one, figure two?
I forget, figure two.
Yeah, so you break the rendering
and then the rendering goes to the feature extraction
and the feature extraction can't get anything
because it's just got a bunch of gray pixels to look at.
And then the model, right?
Garbage in, garbage out.
The model can't do anything because it doesn't have good features.
So I'm going to try to open this up to Discord.
If you are in the DevCon Discord,
go to airvillage-general-voice.
We'll try unmute you and get you so you can talk.
If you're human plus, that's the correct way to do it,
but we have a workaround.
So head over to airvillage-general-voice to ask any questions.
If you can't, we'll still be paying attention to the text chat
in airvillage-journal-text.
The moment of truth.
I'm watching journal club text.
Yeah, me too.
Audio, managing audio is...
I'm so glad I don't have to deal with it.
I wish I had seen this paper before I started, like,
really trying to solve adversarial machine learning as a security problem.
Because there's a lot...
this seems a lot easier, like, to attack for a lot of attack people.
I think a lot of attackers have more knowledge
on how to break a parser than how to break a neural network.
So this, to me, feels like more of a realistic threat model
of, like, your pipeline was... they broke your pipeline.
You did something wrong in AWS.
You did something wrong with your container setup.
And it's parsing your images incorrectly now.
Yeah, I mean, that's tons of the bugs that you find
in sort of real systems, right, as parsing bugs.
And yeah, a lot of the research right now focuses less on that
and more on sort of these theoretical properties of machine learning models,
which, I mean, I don't want to dig on those.
They're fascinating, right? I love these papers.
But they're also, again, keep coming back to this paper.
This paper makes the really good point that these sort of theoretical models
that have these nice properties are embedded in real world systems.
And we know that real world systems are always kind of
full of bugs that can be attacked and exploited.
And so it gives you sort of multiple entries into the problem, right?
On the one hand, you can exploit the preprocessor,
and maybe that gives you some sort of heap corruption
or some sort of like direct, you know, denial of service or even code execution.
And on the other hand, you can break the feature extraction enough
so that everything downstream is getting garbage data.
Or mildly corrupted. I mean, I say again, with like the sneaking in,
you want to do low and slows on some of this.
Again, I am really like the idea of using this as defense.
I mean, there's a whole bunch of applications where the bad guys
are using machine learning based systems.
I like the idea of breaking their systems.
Yeah, as I would. Great hats on.
Yeah, just like yesterday's paper with the Forbes masks.
Mm-hmm. Yeah, so Will has a comment.
He says, I'm surprised a paper is the vehicle for this knowledge.
There's a whole security industry that has advice for this stuff.
And I think, yeah, this is part of what we're trying to do, right, with AI Village
is draw a link between security and machine learning, right?
So we have a lot of like wicked smart security people
who know about all about these implementation bugs.
And then we've got a bunch of like really smart academics
and machine learning people who know about like sort of these,
you know, feature based attacks or weird properties of ML systems.
And getting sort of that knowledge transfer going on
so that we can find out where those intersect
and how those two, you know, sort of sets of attacks inform each other.
I think that's like a really rich area for future development.
And it kind of makes me sad a little bit that this paper came out in 2017
and got comparatively little attention
when it really is going right to the heart of a lot of really big questions
in this, you know, in ML slash security.
So in practice, you see a lot of data science teams
disconnected from both like infrastructure application
network security sort of teams.
And so like the whole MLOps sort of formation of teams
that can interface between data scientists
and implementation has been emerging.
But I'm curious what the panel thinks as far as
what's the appropriate like role, a person, a cross-functional team?
Do we want to have security people sitting with data scientists
as these things are being developed?
And...
Yeah, I mean, weirdly enough,
I used to be one of the people going into large companies
working out where to put the data scientists.
And there was always this argument between
you want to embed the data scientists out into the rest of the team.
And have them work with because you're informing
the rest of the team everywhere.
But then you've got data scientists out on their own.
And we're kind of, you know, you see in this paper
that we get ignored quite a bit when we're on our own.
Or having these unicorn pens full of data scientists
who talk to each other, but not really to anybody else.
And there's this sense of you put people out,
but you make sure they're still part of TRIPE.
So you literally build TRIPE
or whatever the good word now is for that.
It used to be called TRIPE, but cross...
Village?
Cross, yeah, village.
So you build villages for the data scientists
to keep connected to each other,
but they need to be out in the rest of the teams.
Because otherwise, you know, our work is everywhere.
We should be everywhere too.
That's strongly our belief though.
Yeah, I mean, I think it's, but it's got to be
sort of a two-way street, right?
On the one hand, yeah, there is kind of,
for data scientists and sort of ML practitioners,
I think that, yeah, we do sort of tend to stay
in our own little bubble and don't think about
these sort of externalities.
I don't know what the right word for it is,
but like these other ways, you know,
it's not just going to be like
fast, radiant, sign-based attacks.
It's going to be like, no, someone broke the parsing.
But then at the same time,
and this is kind of going back to maybe
a little bit of a hype panel yesterday,
having people that aren't data scientists
understand what the models can and can't do
and what kind of behavior to expect from them
so that they don't freak out
when they see something weird happen
and be like, oh, you know, whatever.
We flagged this DLL as being malware when it's not.
We're clearly under attack, right?
And you're just like, it's statistics.
Sometimes it makes mistakes.
Well, just explaining it's not magic.
It's hard, but it's not magic.
Yeah, it's hard. It's useful.
It does cool stuff, but it's a tool.
Jager's talking about implementing
a security development lifecycle
and a secure software development lifecycle
into model creation and upkeep.
I'm going to make a data scientist cry.
Yeah.
I had issues with just being like, cool,
we need to just write a few unit tests
to make sure that the functions
all kind of do what they're supposed to do.
There's a bit of pushback
from our more academically minded folk
of just a little bit of basics like that,
which may have solved some of these open CV issues.
Interesting that if you basically treat
the lifecycle of a data project
as more of an internal product for a company
and treat it as a product which serves a purpose,
provides value, and then is integrated
with other ways of developing software products,
then data science and machine learning,
they just become tools in the engineering toolbox,
but then everything else falls within methods
of continuous integration and testing and delivery.
So then perhaps the risk,
rather than the unicorn, depends on data scientists.
Yeah.
That's what we're doing.
We're doing things like hypothesis-based development,
which is like the next thing up
from behavior-based development,
which is next thing up for test-driven development.
So you very much work out what the universe
you'd like to see is, go do the maths on it,
build the systems in it.
So there are ways of working.
Yeah, so maybe looping...
Sorry, go ahead, Sven.
Go ahead.
Yeah, so I was going to loop back
to the paper one more time
and talk about the disclosure experience
that they mentioned towards the end of the article.
So basically, three of the vulnerabilities
got new CVEs because they enabled
arbitrary code execution
or denial-of-service attacks, right?
So those are both very firmly
within the wheelhouse of what people...
Page seven.
Page seven, thank you.
Very much within the wheelhouse
of what security people think of as,
oh yeah, that's a bug, that's an issue,
that's a security gap that we need to fix.
But there were a couple of other ones
that they found where they could impact
or manipulate the prediction.
And then they call out in particular
the mouthier memory corruption they found,
where basically they can rewrite
the feature vector,
which allows you to basically
make the model give you
any output you want, essentially.
Doesn't have to have anything to do
with what the actual file
that you were analyzing was.
And those specifically are the ones
that didn't get a CVE,
didn't get called out,
and mostly got labeled as won't fix.
Or as won't fix.
So that's sort of like,
we were kind of bashing
the data scientists earlier saying,
oh, we got to think of this
as part of a security system.
But then it goes back the other way, right?
We need to be able to communicate to people
that as data scientists,
as machine learning experts,
look, this is bad, right?
And how do we convince people that
something that allows
arbitrary misclassification
or arbitrary predictions
to be produced is as serious as,
or maybe not as serious,
but has some degree of severity,
just like a remote code execution
or a denial of service.
I like the example that Eric brought up
in his talk of a turtle rifle.
Like that is a very effective way
of communicating potential problems.
Whereas for those who haven't seen it,
like a turtle 3D printed object
could be applied to a turtle
to make it misclassified as a rifle.
You can imagine like systems
that track security
and stuff to do certain events.
I mean, it's like
any other ethics discussion.
We have to talk about consequences
and then track it back
and just keep pushing
and pushing examples,
real examples if we've got them.
It's hard.
Yeah, I mean, I guess like that's,
you have a joke,
but we're not really a joke,
that like airline regulations
are written in blood, right?
And so when you've got
all of these safety requirements,
it's because a lot of people got hurt
before someone was like,
we ought to do something about that.
And it would be nice to think
that we could find a way
to not have like ML disclosure requirements
written in blood, right?
Like maybe, you know,
we're at a place now
where we could maybe like
jump ahead of the game a little bit
and get an understanding
of this out there
so that we don't have to see people hurt
before people are like,
yeah, we should maybe
take that seriously.
I actually have a historical example
for that too, for being old.
So I ran one of the
unmanned air vehicle safety teams
back in the day before UAVs were sexy.
And we didn't have the
privilege, I guess,
that the rest of the airspace industry had.
So aerospace regulations
are literally written in blood.
So an aircraft would crash
into something else
or hit the ground
and you would change the regulations.
And that's literally
how the regulations are written.
Same with fire safety.
Fire happens, change the regulations.
And we were told explicitly
and understood explicitly
that we couldn't just put UAVs
into manned airspace,
have them crash into stuff
and then rewrite the rules.
We literally had to think
all the scenarios
and write the rules
for how to do it safely.
And it's whole space safety,
not just individual aircraft safety
before we were allowed
to fly in the same spaces.
So there's some precedent.
I mean, there's maybe
some looking in those spaces
to see how it was done well.
That's a good way to look.
But a lot of it's just getting the will.
Now, nothing in focus
gets your attention
like a whole bunch of airliners
about to crash.
But yeah, we could maybe
get there before that point.
So one more question from Discord,
which is good
because that's where I wanted to go next.
So Yaga has a question.
How do we force open source software
to fix itself?
We don't fix it ourselves.
That's why it's open source.
And the authors actually make a comment
in the future work
where they say that it's unclear
who should be responsible
for fixing them as well.
So when they found a bug
in the Malheur feature processing,
that was because Malheur relied
on libarchive.
And the bug was actually
in libarchive.
And again, this gets back
to this notion of like,
you have all these different components
that feed into the system.
So even though the bug
was in libarchive,
it affected Malheur.
So where do you put
the responsibility for fixing it?
And how do you convince,
you know, like whoever's
maintaining libarchive,
look, this is serious enough that,
you know, yes, I don't think
it doesn't affect you directly,
but it does affect
this other system
and it could be a critical impact.
Like, how do we,
how can we navigate
that sort of uncertainty?
How does Linux and Ubuntu
and other open source
software ecosystems do it?
Do they have liability?
I mean, I know for a lot of systems,
the end users,
especially if they're big end users,
just get involved
with the open source communities
and work in there.
But if you're using
even like Python
and your physical system crashes,
who's responsible?
Yeah. I mean, a lot of licensing argument,
a lot of licensing agreements
basically say like,
yeah, you agree that you're using this
at your own risk too.
So, you know, basically
everyone's going around
not me, which, you know, it's fair, right?
And people have, like, they have,
they can't assume liability
for everything that they do.
But, you know,
they can't agree to fix
every single thing.
But at some point, right,
there's a balance, right?
If something's widely adopted,
it's widely used,
and there's like a bug
three steps up the chain
that's causing it to behave weirdly.
It feels like somebody ought to,
there ought to be like
some impetus to fix it somehow.
The low tail of companies like Fortune 500
is more bothered to use
these open source libraries
than in theory, you know, this is AI,
so you have a lot of, like,
boost in productivity,
like, and there's a lot of value
that's being derived,
and especially if they've started
to use it for critical applications
or decisions.
So I think the onus and responsibility
also needs to fall on the end users
of, like, widely adopted packages,
and then somehow making,
making it very clear to decision makers
in those companies that,
hey, if you're using this to
predict insurance rates or something
and it fails,
that's what you could potentially face,
and then somehow channeling that
into the projects themselves.
Are you proposing that large companies
that make a lot of money
using open source tools
should actually give back
to the open source community?
Smells like communism to me.
It's kind of terrifying
that we all just laughed
when you said that.
Yep.
Sorry, I got to get my,
got to get my commentary
in once in a while.
These are complex systems,
and, like, just communicating
the impact of some small,
like, overflow bug
to somebody who only cares about,
like, the bottom line is a problem.
Maybe there needs to be, like,
awesome ML failures,
GitHub repository,
or something like that,
where you can find it.
We haven't had this.
Sounds like, actually,
a pretty good weekend project
for someone.
Yeah, I mean, like, you know,
when we started up
the ethics stuff years ago,
it just, just takes
some people determined
to make this thing a thing,
a visible thing.
Yeah.
Hey, community.
Anyone want to take this one on?
So, how many, like,
solid examples of ML failure
do we have?
If we want to...
I mean, a lot of it is.
Compile those, yeah.
How much do we actually know?
Yeah, that's, I want to say,
Andrew Davis over
in different Discord
had a pretty good comment on it,
which is that a lot of time
when ML fails,
it kind of fails silently, right?
It just, you don't flag something,
and it comes back to
a lot of the earlier discussion
about is it a bug,
or is it, like, normal,
just sort of, like,
statistical failure, right?
A 1% failure rate means that
if it's not failing 1% of the time,
you should be vaguely surprised, right?
So, yeah, like,
I think it's possible
that there's a lot of ML failures
that we just kind of assume
are in the statistical noise.
A lot of them are probably kept quiet
because, you know,
there's potentially, like,
PR risks or PR damage, right?
Or, you know, like,
like, actual, like, gross,
you know, liability attached to it.
Yeah, I mean, it's an open question
how much is actually sort of
flying beneath the radar out there.
I mentioned a rental company saying,
or a background check company
who uses ML going,
so we didn't give a lot of people
their approved, you know,
their credit check approval
because of a mistake.
And we want to publicly
apologize for that
when it's an ML mistake.
And they can just be like,
well, the model just was,
yeah, sweep it under the rug,
fix it quietly, never mention it.
Like, that would be an ethical nightmare.
Yeah, I mean, there's lots of startups
that have proposed to do things.
There was one,
what was it, about two, three years ago,
oh, we'll use ML to pre-screen your babysitters
by digging through their social media profiles and stuff.
And I have my own opinions
on how likely that was to succeed
versus how likely it was
to simply just recommend white people.
But how do you even measure
or quantify failures in that case?
Because it's all going to be completely internal
and all you can do is essentially try
and get a proxy model to be like,
aha, here are the features, right?
I've submitted a thousand different profiles
and here are the features that seem to be triggering, right?
And you have to affirmatively dig for it, right?
Those failures aren't going to be obvious
unless you're actually doing something like that.
A recent streak of sort of facial recognition failures
that lead to characteristics
that elicits a visceral gut reaction like,
yes, this is wrong.
So I think in some cases,
just seeing the outcomes, like Claire said,
helps understand whether it's a failure or not.
This gets back to the bias question.
Like, how do you tell if something's biased
or just like for this one instance, it was wrong?
Oh, man.
Yeah, it kind of works.
We're going to have to clean that one out, Tommy.
Yeah.
In images is, oh, God.
How much longer do we have for this conversation?
Yeah, we can solve it in eight minutes.
We've got eight minutes.
Yeah, yeah, no problem.
Yeah, I mean, the problem in a lot of the bias stuff, right,
is like the terms aren't even defined sometimes.
So like the example I keep coming back to
is like recidivism and like predicting re-arrest rates, right?
And someone might point out,
hey, look, this is biased, right?
It's clearly predicting re-arrest rates
that are much higher for this group, right?
And one argument would be, well, that's clearly wrong, right?
It's biased because it's trained on biased data.
And the counter argument would be like,
look, it's reproducing the data, right?
We don't live in a perfect society.
The data is what it is.
And within that box, the ML has done as much
as you can ask it to do.
And so it always boils down to these normative questions.
And this is my own totally personal opinion
that talking about unbiased systems,
unbiased ML is almost inherently impossible, right?
It's a question of how is it producing
the kinds of outcomes that you want
and how is the ML system moving power around, right?
Who is it empowering? Who is it disenfranchising?
And who is it helping and who is it hurting?
And chasing after some vague notion of fairness,
a fair system or an unbiased system
is kind of like a distraction
from more fundamental questions of who's being helped,
who's being hurt, what's being reinforced
or whose concerns are being downplayed.
And who's accountable, I guess,
also because of the reference recent Twitter story
between Timothy Kimbrough and Yann LeCun.
Basically, is it the ML engineer
or is it actually the data scientist
or is it throughout the entire chain?
Yep.
Yeah, and I mean...
Sorry, go ahead. I'm talking too much.
I'm seeing over in the chat
that we already have awesome ML failures
in the AI village as a repo.
And Stellar Athena seems to be getting in there already.
Outstanding.
Suggestions for implementation failures,
ethical failures, HCI failures and security failures.
You got that Stellar?
Yeah.
Yeah, I think...
Do we have a category for...
Because we were playing around with
this cat does not exist a couple days ago.
I think we maybe need a category.
This ML should not exist.
Yeah, nightmare fuel.
Yeah, well, I mean, I'm thinking of things like...
Yeah, you're right.
There are a whole bunch of machine learning systems
that just should not be on this planet.
Yeah.
And anyone asked to contribute to it
should have been like, perhaps not.
Yeah, as far as the responsibility...
And we would never have had Facebook
if anyone had that call.
Anyway.
I mean, hindsight.
But you can look at really short-term stuff
like even just like genderify, right?
It was should have been pretty obvious
to everyone involved that this was just
like a losing proposition from the start.
Yeah, and the recidivism through face images
stuff from China.
Oh, God.
Yeah. Okay, so...
Way to replicate somebody's bias.
So we do have a workshop coming up
in a couple of minutes.
So I think I want to kind of like
put a pin in this.
I think if everyone's enjoying this conversation,
having an awesome ethics in AI panel,
which is coming up in a couple of hours.
I'm not sure someone fill me in.
Two o'clock, two o'clock.
Thank you.
So everyone should absolutely turn,
tune in for that
if they're enjoying this discussion.
But yeah, I guess I want to just like go around
just to wrap this up
and see if we have any like last minute,
like big idea takeaways from this paper
that people thought were really, really cool.
I love the way they did it.
The idea of attacking the system
and looking at taking it apart
is just kind of my thing.
And I like the idea of using it to attack systems
that shouldn't exist.
Yep. Sven?
I really like the idea of attacking
the feature extractors.
And I think we should also add
like bypassing the feature extractors
by just doing something
you know the ML system is going to miss.
Because you can understand it well
to this whole thing.
Cool. Martin?
Yeah, that also really enjoyed the paper
and discussion.
I think to me what was clear
is that it can occur at many levels
of the basically stack.
So just instrumentation and stuff
become very, very important
to monitoring what happens.
And I would really like to see
follow on work that applies this
to TensorFlow, PyTorch,
basically the learning libraries
and seeing how you can play
with intermediate representations
and things like that too.
Yeah, for sure.
Yeah, the reminder that like
these ML things, ML projects
don't exist in some abstract theoretical space
and are implemented in real code
and real systems that come
with their own flaws.
Yeah, like you know it
but being reminded of it
sort of viscerally by,
look, we have CVEs
is always a great thing.
So cool.
So I think this is
we probably want to leave a few minutes
for people to fight with the audio
as we transition over to the workshop.
So I think we want to
call that a panel or a discussion.
So awesome. Thanks, everyone.
Thanks, Rich.
One second.
Before we jump off,
we have one on Wednesday
at 5 p.m. Pacific time.
And we have the, I'm guessing,
I think we should go with your paper, Rich.
The slide deck on...
Yeah, the anomaly detection
outside the closed world
on using machine learning
for network confusion detection
by Robin Subban and Vern Paxson.
So we'll post that in the Defcon Discord
and also be posted to our Discord and Twitter.
So if you want to participate,
you should find your way over to our Discord.
We can't post a link to that in the Defcon Discord,
but the links are places
and you can just DM us for an invite.
And we'll be discussing that
for probably an hour and a half on Wednesday.
And we might be able to get one of the authors in,
even though this is a 10-year-old paper.
How far we have come.
All right.
Thanks, everyone.
Catch you later.