[00:00.820 --> 00:03.180]  Okay, say something.
[00:05.280 --> 00:09.420]  Nope, the audio is definitely broken again.
[00:10.220 --> 00:11.680]  Now say something.
[00:19.120 --> 00:20.920]  Time with the audio issues.
[00:20.920 --> 00:24.040]  There we go. The audio seems to be working now.
[00:24.040 --> 00:25.140]  All right, awesome.
[00:26.280 --> 00:29.100]  Yeah, I almost feel like we got to have... whoever
[00:30.900 --> 00:35.200]  is the poor soul stuck with jockeying this in the future almost needs to have two laptops,
[00:35.200 --> 00:42.700]  one with Twitch and everything going and one with just OBS and Zoom, yeah.
[00:43.880 --> 00:45.260]  Yeah, that's...
[00:45.260 --> 00:49.840]  Nothing can confuse OBS's tiny little brain.
[00:50.060 --> 00:54.320]  But then the problem is when you forget to mute that thing and then this thing
[00:54.320 --> 00:59.860]  and then there's one of your tabs is doing the wrong noise and you're like, oh, this is wrong.
[01:00.220 --> 01:00.920]  Yep.
[01:00.920 --> 01:03.300]  Sven, you're being heroic. It's just...
[01:04.020 --> 01:09.880]  Yep, we are live, by the way. The internet is listening to us talk, so enjoying the banter.
[01:09.920 --> 01:11.240]  You're still being heroic.
[01:11.420 --> 01:17.780]  Yes. Yeah, so for those of you watching, Sven has been the main guy troubleshooting
[01:17.780 --> 01:23.120]  all of the audio issues and fighting with the stream for basically all of DEF CON.
[01:23.120 --> 01:29.860]  And it's a testament to his hard work that things have gone as smoothly as they have.
[01:29.860 --> 01:35.060]  So you've made it a lot easier that things were pre-recorded.
[01:35.060 --> 01:38.040]  Can you imagine trying to hunt people down,
[01:38.040 --> 01:42.840]  hunt speakers down, and then their sharing is not quite right and all that?
[01:43.400 --> 01:45.420]  Yeah, no, I mean...
[01:45.420 --> 01:47.880]  I mean, this year, at least they're not passed out in the bar.
[01:49.660 --> 01:52.260]  How many speakers have been passed out in the bar?
[01:53.360 --> 01:57.720]  None of ours that I know of, but we've definitely hooked a few of them out from the nearest bar.
[01:58.240 --> 01:59.600]  Yeah, well, you know.
[02:04.250 --> 02:08.810]  Are you expecting anyone else or we want to get this show on the road?
[02:09.130 --> 02:16.930]  Yeah, let me... well, let me pull up this thing for myself.
[02:18.090 --> 02:24.970]  I'm going to try a new layout as soon as I can find the thing.
[02:25.210 --> 02:31.210]  So while I'm looking for it, Rich, do you want to introduce the paper to everyone?
[02:33.470 --> 02:36.270]  Oh, sorry. So we're going to do a couple of things.
[02:36.290 --> 02:37.430]  Hi, Barton.
[02:37.510 --> 02:40.810]  So we're going to start with just the four of us talking about it for a bit.
[02:40.810 --> 02:42.410]  And then I'm going to open up the...
[02:43.770 --> 02:48.710]  and join the voice chat in the Defcon Discord.
[02:48.710 --> 02:52.310]  And so you guys will be able to talk on stream and talk with us and everything.
[02:53.870 --> 02:56.950]  So for the first, like, 20 minutes, it's going to be just us.
[02:56.950 --> 02:59.310]  And then we'll bring everyone we can in
[02:59.310 --> 03:02.070]  and have a bit more of a Wild West thing.
[03:02.070 --> 03:03.790]  We're going to try out how that works.
[03:04.430 --> 03:07.530]  And the other thing is I'm just going to play with the layouts a little bit.
[03:07.750 --> 03:10.750]  So that's the structure of it.
[03:10.750 --> 03:13.670]  We've also got a journal club coming up on this Wednesday.
[03:14.370 --> 03:17.770]  We will put the link to that paper in the Defcon Discord.
[03:17.770 --> 03:21.190]  And then if you want to participate, you should join our Discord.
[03:22.810 --> 03:24.370]  Link in the Twitter bio.
[03:24.810 --> 03:26.410]  Yeah, it's in the Twitter bio.
[03:26.410 --> 03:28.690]  And you can just DM one of us.
[03:28.690 --> 03:32.730]  And we'll send you an invite to the AI Village Discord.
[03:32.730 --> 03:36.050]  Please don't use the AI Village Discord while Defcon is going on.
[03:36.050 --> 03:40.350]  Please be respectful of the hard work they have put into building their Discord
[03:40.350 --> 03:41.770]  and getting it running.
[03:41.970 --> 03:45.330]  And only use it, like, tomorrow or Tuesday.
[03:46.730 --> 03:48.910]  So take it away, Rich.
[03:49.490 --> 03:51.070]  Okay. Cool.
[03:51.070 --> 03:54.690]  So this is actually a slightly older paper.
[03:54.690 --> 03:56.950]  It's from 2017.
[03:58.010 --> 04:02.130]  It is called Summoning Demons, the Pursuit of Exploitable Bugs in Machine Learning,
[04:02.130 --> 04:06.890]  which is probably one of the coolest paper titles I've come across in a while.
[04:07.310 --> 04:12.310]  And it's by Rock Stevens and Tudor Dimitrescu's group.
[04:12.310 --> 04:15.870]  So the basic idea of the paper, it's kind of an interesting spin
[04:15.870 --> 04:22.050]  on the adversarial ML papers that I think started coming out about the same time.
[04:22.050 --> 04:29.810]  And in this, rather than actually, like, using the ML model itself to sort of guide inputs,
[04:29.810 --> 04:32.130]  they're looking for more...
[04:32.770 --> 04:37.930]  they're more looking for flaws in how the different components of the system hook up.
[04:37.930 --> 04:44.050]  And to do that, they actually crack out an old classic American fuzzy lump, the fuzzer,
[04:44.050 --> 04:51.710]  and they re-instrument a lot of it so that weird actions from the ML components of the system
[04:51.710 --> 04:54.370]  register as crashes to AFL.
[04:54.570 --> 04:59.750]  And so that lets them essentially do sort of like a black box adversarial attack,
[04:59.750 --> 05:02.530]  if you want to think of it that way, against the ML system.
[05:02.570 --> 05:05.890]  And what they find is that in addition to sort of the usual stuff you can get,
[05:05.890 --> 05:09.050]  where you have, like, inputs that look like faces that don't register as faces,
[05:09.050 --> 05:10.890]  because they've had some pixels perturbed,
[05:10.890 --> 05:13.950]  they also run into, like, image format parsing errors,
[05:13.950 --> 05:20.250]  where a component of the ML system won't parse, for example, a face correctly,
[05:20.250 --> 05:21.730]  completely fails to find the face.
[05:21.730 --> 05:25.830]  But if you open it up in other programs, it actually looks good.
[05:25.990 --> 05:32.630]  So I think this is kind of in the vein of the same sort of things
[05:32.630 --> 05:35.310]  that Ariel Herbert-Boss talks about a lot,
[05:35.310 --> 05:39.330]  where it's not just attacking the ML system itself that's important.
[05:39.330 --> 05:42.850]  You've got all of these other components in the pipeline,
[05:42.850 --> 05:46.870]  from ingesting the actual image all the way through feature extraction,
[05:47.470 --> 05:48.830]  pre-processing, and so on.
[05:48.830 --> 05:51.650]  And each of those is actually a point that you can attack.
[05:51.650 --> 05:57.590]  And so in this paper, they're basically using fuzzing as a way to find
[05:57.590 --> 06:01.270]  some of those, you know, broken assumptions or breakdowns
[06:01.270 --> 06:04.510]  in the link from one stage of processing to the next.
[06:05.350 --> 06:08.590]  And I thought it was a super clever paper with a really good name.
[06:08.590 --> 06:11.250]  And they dig a couple of CVs out of it, too.
[06:11.250 --> 06:17.530]  So super cool application of sort of more classic security methodology
[06:17.530 --> 06:23.510]  to what, from my perspective, is kind of an adversarial ML problem.
[06:26.210 --> 06:30.530]  So that's my, I guess, like two-minute summary of the paper.
[06:35.440 --> 06:38.140]  So I guess maybe to kick off conversation,
[06:41.340 --> 06:44.820]  so does this, do you feel like, does anyone have an opinion,
[06:44.820 --> 06:46.540]  strong opinion one way or the other,
[06:46.540 --> 06:51.800]  if this falls into the same sort of general field of adversarial ML
[06:51.800 --> 06:55.180]  as sort of like gradient-based attacks?
[06:55.180 --> 06:58.160]  Like I'm trying to think of, the name escapes me off the top of my head,
[06:58.160 --> 07:02.740]  like fast gradient sign method, or like these Jacobian-based methods,
[07:02.740 --> 07:07.160]  or saliency-based methods, or maybe even like,
[07:07.160 --> 07:10.440]  you could almost argue it's like a genetic algorithm-based attack
[07:10.440 --> 07:14.520]  against a particular ML pipeline.
[07:14.580 --> 07:18.420]  Well, to be honest, I thought that adversarial ML covered all this already.
[07:18.420 --> 07:20.540]  I was a bit surprised that it was new.
[07:21.800 --> 07:26.300]  Yeah, it's genetic algorithm-based because they're using FuzzyLock,
[07:26.300 --> 07:28.780]  which is genetic algorithm-based, which is also interesting
[07:28.780 --> 07:33.740]  because there were network extensions to genetic algorithms
[07:33.740 --> 07:36.440]  that might be more appropriate if you're searching spaces.
[07:36.980 --> 07:40.100]  So there's some interesting stuff on the front that could be done.
[07:40.100 --> 07:41.980]  I kept wanting to say, what happened next?
[07:50.890 --> 07:54.430]  Likewise, in the context of healthcare,
[07:54.970 --> 07:56.690]  practical realities of securing systems,
[07:56.690 --> 08:03.910]  it seems like a lot more emphasis right now is placed on more theoretical attacks,
[08:03.910 --> 08:07.750]  where it's more so once you have uninhibited access
[08:07.750 --> 08:10.710]  to being able to query this API that's being explored.
[08:10.710 --> 08:12.970]  So it's nice to see a paper like this that's a bit more
[08:12.970 --> 08:14.950]  in an all-systems set of things,
[08:14.950 --> 08:19.390]  because at least from healthcare IT,
[08:19.390 --> 08:23.450]  it seems like it's a more likely encounter that you'll find in the industry.
[08:24.670 --> 08:30.610]  Yeah, one thing that I did really like was their focus on implementations
[08:30.610 --> 08:33.090]  rather than just theoretical properties, right?
[08:33.090 --> 08:39.030]  Because in some sense, something like the FastGradientSign method
[08:39.030 --> 08:42.710]  would apply to almost any of these models.
[08:42.710 --> 08:46.690]  But here, they're sort of like, okay, this model was instantiated in this way
[08:46.690 --> 08:48.170]  and is doing these things.
[08:48.170 --> 08:52.850]  And now how can we attack sort of that specific version of the model?
[08:53.450 --> 08:58.550]  Because it's a lot less academic in some ways,
[08:58.550 --> 09:02.890]  and a lot of the adversarial ML research seems to be very academic still.
[09:03.150 --> 09:06.190]  This was like down and dirty in how do we attack this thing.
[09:06.310 --> 09:09.150]  It's kind of in my vein of stuff.
[09:09.270 --> 09:12.330]  Just look at the system, work out how to break the system,
[09:12.330 --> 09:13.810]  and take it to pieces.
[09:14.050 --> 09:18.310]  I loved the way that they hacked that little hack on FuzzyLop,
[09:18.310 --> 09:21.750]  where they went, okay, we're going to take out the things that actually broke,
[09:21.750 --> 09:25.070]  and then we're going to induce breaks for the things we care about.
[09:25.490 --> 09:27.430]  Yeah, changing the target.
[09:27.430 --> 09:28.570]  Yeah, changing the target.
[09:29.290 --> 09:31.250]  It was just beautifully done.
[09:31.650 --> 09:34.530]  I thought it would be useful for defense as well,
[09:35.250 --> 09:41.830]  because we're also looking now at things like Strafe's work,
[09:41.830 --> 09:45.510]  so Brittany Prostakoff's work on attacking with robots.
[09:45.990 --> 09:50.290]  What could we do to screw up their vision systems?
[09:51.210 --> 09:53.710]  So how could we mess that up?
[09:53.710 --> 09:55.850]  There's some beautiful adversarial there too.
[09:56.910 --> 09:58.870]  They can't see, they can't move.
[09:59.190 --> 09:59.750]  Yeah.
[10:00.470 --> 10:03.650]  Yeah, so actually, I thought table one in this was...
[10:04.490 --> 10:06.130]  I don't know, I'm surprised.
[10:06.830 --> 10:08.270]  Flips to table one.
[10:08.270 --> 10:09.010]  Flips to table one, yeah.
[10:09.010 --> 10:10.510]  Waves to table one.
[10:11.170 --> 10:13.190]  It's, I think it's on page four.
[10:13.390 --> 10:14.310]  Yeah, yeah.
[10:14.310 --> 10:15.270]  Yeah, page four.
[10:15.330 --> 10:18.550]  Yeah, they actually like, I feel like, again, this is very much like
[10:19.210 --> 10:20.330]  Ariel's jam, right?
[10:20.330 --> 10:23.650]  They go over sort of the attack surface and the kinds of things you can do to it.
[10:23.650 --> 10:28.130]  And it reminds me a little bit of the contest there was...
[10:28.130 --> 10:31.670]  So I think it was a late entry to the contest last year
[10:31.670 --> 10:35.670]  that was announced at the AI Village where they were trying to do ML evasions.
[10:35.830 --> 10:40.230]  And one guy actually found essentially a bunch of ways
[10:40.230 --> 10:42.370]  to just completely trash the feature extraction
[10:42.370 --> 10:47.650]  so that the ML prediction just wouldn't run on the samples he uploaded.
[10:48.550 --> 10:53.010]  And so, yeah, he was a little upset that that didn't count as a win,
[10:53.010 --> 10:54.310]  which I think is justifiable, right?
[10:54.310 --> 10:57.290]  Because if you can just like stop the analysis cold, right?
[10:57.470 --> 10:59.530]  That's almost as good as evading it.
[10:59.530 --> 11:00.230]  But yeah.
[11:01.170 --> 11:03.670]  It depends what you're trying to do if you kind of sneak into a system
[11:03.670 --> 11:05.610]  and it's a bunch of systems, right?
[11:05.610 --> 11:07.830]  We just want to, you know, the usual thing.
[11:07.830 --> 11:10.930]  Get in, do your thing, get out without no one noticing.
[11:12.090 --> 11:14.850]  And we see enough of that in real life of the stuff.
[11:14.850 --> 11:17.350]  It's like, oh crap, we didn't realize that was a problem.
[11:18.070 --> 11:19.770]  Before someone else adds them.
[11:21.050 --> 11:23.650]  There's also another thing to that.
[11:23.770 --> 11:29.030]  Like if you wanted to attack the Ember model with the Ember dataset,
[11:29.030 --> 11:33.350]  it only looks at your headers and you could just change other parts.
[11:33.350 --> 11:36.570]  Like don't even think about it in terms of adversarial ML.
[11:36.570 --> 11:42.250]  Just copy like a notepad.exe's header and just put your own
[11:44.030 --> 11:48.190]  entry point function in and start launching weird stuff from there.
[11:48.190 --> 11:51.870]  And that's a different way to bypass the thing that is also not...
[11:52.810 --> 11:54.670]  Academics also not aware of.
[11:56.330 --> 11:58.110]  Yeah, so that's sort of like...
[11:58.110 --> 12:03.010]  I guess that's maybe like a...
[12:03.010 --> 12:04.910]  You can classify that if you're gonna...
[12:04.910 --> 12:05.910]  I'm gonna screw this up.
[12:05.910 --> 12:10.110]  So if I'm getting the terminology wrong, security people, please like yell at me.
[12:10.110 --> 12:15.010]  But it's almost sort of like a logic attack on the feature extraction
[12:15.010 --> 12:18.150]  rather than what they're doing here,
[12:18.150 --> 12:20.570]  which seems to be more going for crashes, right?
[12:20.570 --> 12:23.490]  If you know how the feature extraction process works,
[12:24.250 --> 12:27.550]  then you can sort of write your way around it deliberately
[12:27.550 --> 12:31.830]  rather than just throwing shit at it until something blows up.
[12:32.930 --> 12:35.490]  Some of it's blowing it up, some of it's sneaking in the back.
[12:35.670 --> 12:39.630]  Some of it's like sneaking in the back conventionally depending on your system.
[12:40.770 --> 12:46.710]  The other thing that interests me is that only some of these became CVEs.
[12:46.710 --> 12:50.530]  That the response to the people who are responsible for the algorithms
[12:50.530 --> 12:53.830]  for some of this was like, won't fix and who cares?
[12:54.330 --> 12:57.990]  And that's part of our journey, I guess.
[12:58.370 --> 13:00.990]  It is just like every other security thing
[13:00.990 --> 13:04.710]  is trying to get people to accept that yes,
[13:04.710 --> 13:09.630]  you may have to drop a little bit of responsiveness in your system.
[13:10.110 --> 13:13.410]  But you'll get it more secure and it matters.
[13:14.570 --> 13:17.990]  Yeah, I'm kind of reminded a little bit of some of the argument
[13:17.990 --> 13:21.130]  that went around for the proof point CVE,
[13:21.130 --> 13:23.290]  where people were arguing whether or not
[13:23.290 --> 13:26.670]  that actually counted as like an information leak CVE.
[13:26.670 --> 13:29.190]  Because so if people aren't familiar,
[13:29.190 --> 13:32.910]  the short version is the people who did it,
[13:32.910 --> 13:36.510]  Will Pierce and someone whose name I'm forgetting, I'm sorry.
[13:37.040 --> 13:40.150]  They found a way to essentially like send an email
[13:40.150 --> 13:43.270]  through proof points, spam detection service
[13:43.270 --> 13:45.030]  that would then bounce back to them.
[13:45.030 --> 13:49.650]  And in the header of the email, it had a score for how spammy it looked.
[13:50.110 --> 13:53.750]  And so that essentially was the information leak.
[13:53.750 --> 13:55.190]  The fact that they could get that score back
[13:55.190 --> 13:59.270]  and that was enough for them to rebuild essentially a proxy model for it.
[13:59.750 --> 14:04.910]  And yeah, over on in various like AI village discussion channels,
[14:04.910 --> 14:08.570]  we had very strong debates over whether or not
[14:08.570 --> 14:12.790]  that actually should have counted as like an information leak CVE.
[14:12.830 --> 14:16.090]  Because, you know, it's in a lot of cases,
[14:16.090 --> 14:17.890]  it's useful contextual information.
[14:17.890 --> 14:20.390]  But when you look at it in an ML context,
[14:20.390 --> 14:22.950]  you're actually enabling like a model stealing attack.
[14:23.590 --> 14:26.670]  Which again, it's one of these, we're kind of in new territory
[14:26.670 --> 14:30.110]  as far as the security of these ML based systems goes.
[14:30.130 --> 14:33.190]  And a lot of people that are more familiar
[14:33.190 --> 14:36.450]  with conventional security, maybe don't quite get
[14:36.450 --> 14:41.970]  what these various information leaks or attack surfaces could do
[14:41.970 --> 14:47.630]  once you wrap it in like how ML based systems work.
[14:47.790 --> 14:52.310]  And I think another thing we get out of this is by doing this sort of attack,
[14:52.310 --> 14:56.750]  we start showing where the things that break up.
[14:56.750 --> 14:59.630]  And one of the things I'd love about the fact that MLSec exists
[14:59.630 --> 15:01.590]  is it makes machine learning better.
[15:02.530 --> 15:05.650]  Just by dint of the spotlight we throw on it.
[15:05.650 --> 15:07.790]  You chuck hackers at it, they take it to pieces,
[15:07.790 --> 15:09.370]  they show you where it's broken.
[15:09.990 --> 15:14.290]  And hell, we spend most of our life trying to get our damn systems to work.
[15:14.450 --> 15:16.210]  But I'm going to give a little bit of history.
[15:16.210 --> 15:17.170]  I'm old.
[15:17.510 --> 15:20.590]  Jump in for just a second.
[15:20.630 --> 15:22.470]  I just saw in the Discord chat.
[15:22.470 --> 15:27.170]  So it was Will Pierce and Nick Landers who got the Proofpoint CVE.
[15:27.170 --> 15:30.010]  So I'm very sorry that I forgot your name, Nick.
[15:30.010 --> 15:31.410]  Okay, sorry, carry on.
[15:31.590 --> 15:35.110]  I was just going to say about sort of some things, the findings,
[15:35.110 --> 15:36.870]  that it's double float precision thing.
[15:37.250 --> 15:42.110]  So way back before we had, I think we had Python,
[15:42.110 --> 15:44.710]  but we sure as hell didn't have scikit-learn and friends.
[15:44.710 --> 15:46.590]  We had matplotlib.
[15:46.650 --> 15:48.270]  I'm not... matmap.
[15:48.390 --> 15:51.670]  We use MATLAB for a hell of a lot of machine learning stuff.
[15:51.910 --> 15:54.770]  And there was a neural network kit in the MATLAB.
[15:55.010 --> 15:59.170]  And that neural network kit suffered horrendously from underflow errors.
[15:59.690 --> 16:02.090]  You could break it really, really, really easily.
[16:02.090 --> 16:03.950]  So Aston University built its own.
[16:04.510 --> 16:08.170]  They built a separate kit because they couldn't get the first one fixed.
[16:08.810 --> 16:11.910]  I mean, this sort of stuff, like the float to double errors.
[16:12.890 --> 16:18.550]  I mean, crossing my fingers that actually all of those hidden faults
[16:19.410 --> 16:22.910]  that are probably still going on with neural network stuff and deep learning,
[16:22.910 --> 16:25.950]  we're not even noticing because we just trust the systems.
[16:25.950 --> 16:29.670]  Yeah. I mean, even in Python, right?
[16:29.670 --> 16:33.450]  Like under the hood, a ton of these neural network libraries,
[16:33.450 --> 16:37.030]  and PyTorch, at least, sklearn as well,
[16:37.550 --> 16:43.070]  use pickle, which is just broken kind of by default, right?
[16:43.070 --> 16:46.310]  It's a stack machine that you can write arbitrary code for.
[16:46.570 --> 16:53.090]  And yeah, people still share wait files that are essentially pickles.
[16:53.090 --> 16:56.850]  And so you're downloading this off the internet and running code.
[16:56.850 --> 16:57.690]  And I keep threatening.
[16:57.690 --> 17:01.050]  One of these days, I'm actually going to write a pickle poisoner
[17:01.050 --> 17:05.170]  that just replaces calls to the relu function
[17:05.170 --> 17:07.710]  with some sort of stochastic function
[17:07.710 --> 17:11.470]  so that half the time it works and half the time it gives you garbage.
[17:12.030 --> 17:14.670]  Comment from Discord.
[17:14.670 --> 17:16.510]  Yeah, numpy as well uses pickles.
[17:16.510 --> 17:17.810]  So it's everywhere.
[17:17.810 --> 17:18.890]  Keras, yeah.
[17:18.890 --> 17:22.710]  Is that what the numpy save file is?
[17:24.330 --> 17:26.050]  I thought that was secure.
[17:26.350 --> 17:27.570]  I'm not sure.
[17:27.570 --> 17:29.130]  I know, I'm pretty sure it has one...
[17:30.570 --> 17:31.970]  One pickle thing in there.
[17:32.030 --> 17:33.110]  One or two.
[17:33.110 --> 17:37.890]  It's got one or two formats that are not.
[17:37.890 --> 17:39.390]  Apparently, there's a couple that...
[17:39.390 --> 17:44.210]  The scikit-learn form that these guys found, numpy was part of that.
[17:44.210 --> 17:45.490]  I'm just flicking to the page.
[17:45.490 --> 17:47.990]  Page seven at the bottom, if you're reading along with us.
[17:49.210 --> 17:53.110]  One interesting thing is that they weren't able to induce misclassification
[17:53.110 --> 17:56.650]  or from the precision loss problem.
[17:56.650 --> 17:58.610]  So they tried different values of epsilon,
[17:58.610 --> 18:00.610]  and basically said that it's possible in theory.
[18:00.610 --> 18:04.390]  But it's interesting to consider how would you model that epsilon
[18:05.490 --> 18:09.330]  based on the syntax to induce this misclassification.
[18:15.120 --> 18:16.840]  How much of this is induced by...
[18:18.440 --> 18:21.220]  Basically, all the machine learning code is written by academics
[18:21.220 --> 18:25.560]  and over-enthusiastic academics.
[18:25.560 --> 18:26.200]  Guilty.
[18:27.860 --> 18:29.540]  Who aren't experienced in...
[18:30.820 --> 18:35.180]  It's incredibly hard to write numerically efficient code.
[18:35.180 --> 18:39.960]  You don't have time to really learn how to multiply matrices incredibly efficiently
[18:40.660 --> 18:43.440]  and make sure you open files correctly.
[18:43.960 --> 18:50.300]  So we've chosen a lot of lazy paths to formatting and opening things
[18:50.300 --> 18:52.260]  versus secure paths.
[18:53.720 --> 18:56.260]  And also, you've got tiny numbers and people don't...
[18:56.260 --> 18:58.180]  I mean, you go wading on in, build your stuff,
[18:58.180 --> 19:00.640]  and don't think about how those numbers interact.
[19:00.640 --> 19:03.260]  You just assume the system is going to magically deal with it.
[19:03.900 --> 19:05.960]  Yeah. And I mean, there's also the fact that
[19:07.240 --> 19:12.200]  writing secure code is very much its own skill set, right?
[19:14.440 --> 19:16.400]  Some people are writing research code
[19:16.400 --> 19:18.900]  because they don't expect it to be used in Angular,
[19:18.900 --> 19:21.040]  they don't expect it to be used in a production context.
[19:21.040 --> 19:23.600]  Some people are writing numerically efficient code
[19:23.600 --> 19:26.160]  because that's their jam and God bless them.
[19:26.900 --> 19:31.260]  And then you've got sort of security people
[19:31.260 --> 19:33.920]  who are probably gazing at a lot of this
[19:33.920 --> 19:38.120]  with sort of a feeling of vague horror that they're actually going into production.
[19:39.180 --> 19:42.600]  Yeah, I know even in my day job, right,
[19:42.800 --> 19:45.260]  a lot of what we've been doing recently
[19:45.260 --> 19:50.300]  is sort of going back and rethinking how all of this stuff is put together
[19:50.300 --> 19:55.260]  so that we can feel confident in actually pushing out products containing it.
[19:56.120 --> 19:59.460]  There's a middle ground where you can enable basically more flexible code,
[19:59.460 --> 20:04.060]  but also execute it in a way that perhaps minimizes the blast radius.
[20:04.140 --> 20:05.940]  I don't mean put it in a container, obviously,
[20:05.940 --> 20:10.420]  but it's something more to the point of running code in secure enclaves.
[20:10.420 --> 20:14.500]  For instance, the work with silo type of workflows
[20:14.500 --> 20:19.980]  or do you see anything where you try to mitigate the impacts of this code?
[20:20.560 --> 20:22.060]  And could we have code checkers?
[20:22.060 --> 20:26.100]  I mean, literally just some way, some sort of test data sets.
[20:26.180 --> 20:29.360]  But we know this stuff is bad. Run it through your system.
[20:30.020 --> 20:30.560]  Yeah.
[20:31.660 --> 20:38.420]  Yeah, I mean, I had a great amount of fun, again, at my day job,
[20:38.420 --> 20:40.840]  actually like taking our feature abstraction code
[20:40.840 --> 20:43.120]  and fuzzing it just to see what would happen with it.
[20:43.120 --> 20:49.880]  Fortunately, it turned out that I spent more time than I care to think of beating on it,
[20:49.880 --> 20:51.580]  and it turned out fine.
[20:51.580 --> 20:54.840]  But yeah, if it hadn't, what would we have done then, right?
[20:55.660 --> 21:01.680]  And I think the idea of building test vectors of just really bizarre,
[21:01.680 --> 21:06.700]  but technically conformant files for this sort of thing is an awesome idea.
[21:07.820 --> 21:12.020]  But extending that to the actual data science part,
[21:12.020 --> 21:16.160]  how often have you issued a PR about some data science thing
[21:16.160 --> 21:18.940]  and sent it to someone and they're gone?
[21:18.940 --> 21:20.540]  It looks good to me.
[21:22.360 --> 21:24.920]  But there's some stuff that you can check easily,
[21:24.920 --> 21:32.160]  but a lot of the code that goes into BERT is incredibly complicated.
[21:32.520 --> 21:34.180]  Very hard to check.
[21:34.280 --> 21:35.660]  When you make a small change,
[21:35.660 --> 21:40.480]  you could have broken things fundamentally because it's software.
[21:41.720 --> 21:46.800]  How do you write a unit test for I broke BERT when you input this one slight thing
[21:46.800 --> 21:53.380]  and now it causes this whole class of things to be false positives, false negatives?
[21:55.680 --> 21:58.140]  Like, how do you unit test machine learning?
[21:58.580 --> 22:01.740]  There's a... I mean, you can do...
[22:01.740 --> 22:04.600]  I mean, essentially, you do test vectors, right?
[22:04.800 --> 22:07.980]  You say, you know, if I load...
[22:08.660 --> 22:11.340]  You can consider the weights are obviously part of the model.
[22:11.340 --> 22:12.520]  You say, I've got this set of weights.
[22:12.520 --> 22:15.420]  If I load in, you know, this set of features,
[22:15.420 --> 22:19.600]  I should get this exact set of outputs up to numerical precision,
[22:19.600 --> 22:23.580]  which, again, numerical precision is kind of its own nightmare.
[22:25.020 --> 22:29.660]  Yeah, but then if you change the weights, right, then kind of all bets are off.
[22:29.660 --> 22:35.440]  And that's one of the painful things about debugging ML,
[22:35.440 --> 22:38.620]  even when you're not under adversarial conditions,
[22:38.620 --> 22:44.040]  is just, I retrained the model, do I still trust it?
[22:44.180 --> 22:46.780]  I mean, there's a difference between I messed up my models,
[22:46.780 --> 22:48.440]  which is like, we do this all the time.
[22:48.440 --> 22:51.180]  And somebody is actively trying to attack my models
[22:51.180 --> 22:55.360]  or may actively try to attack my models using known vulnerabilities.
[22:55.860 --> 22:59.480]  Because sure as hell, half the systems out there haven't patched any of this.
[23:00.060 --> 23:03.880]  How many machine learning developers even know unit testing
[23:03.880 --> 23:06.420]  or even know good software practices?
[23:06.420 --> 23:08.580]  Like, a lot of us came from academia.
[23:08.580 --> 23:11.300]  I have a PhD in math, not software engineering.
[23:11.740 --> 23:17.520]  So I have a philosophical question I want to pose to the panel from Discord.
[23:18.080 --> 23:21.580]  And Yaga asks, what is a terror and...
[23:21.580 --> 23:25.420]  Sorry, what's a terror? This is terror right here.
[23:25.420 --> 23:28.100]  What is an error and what's an attack?
[23:28.100 --> 23:32.980]  And like, is there any meaningful distinction between the two?
[23:32.980 --> 23:35.100]  I guess I would follow up.
[23:36.080 --> 23:38.540]  I think if you're looking at adversarial images,
[23:38.540 --> 23:40.740]  I mean, the fact you have like the little image patches
[23:40.740 --> 23:43.200]  to screw up your system is an attack.
[23:43.280 --> 23:44.440]  It's not an error.
[23:44.700 --> 23:48.620]  You just don't randomly get sort of extra patches on a stop sign.
[23:49.280 --> 23:51.960]  No matter how much undergrowth you got on it.
[23:52.140 --> 23:54.280]  I think the difference is intention.
[23:55.540 --> 23:57.840]  Yeah, I mean, that's the same thing in security, right?
[23:57.840 --> 24:02.660]  Bugs can be... I mean, exploits are essentially software bugs, right?
[24:02.980 --> 24:05.300]  Maybe that's being too general, but if you...
[24:06.140 --> 24:08.500]  You know, a subset of software bugs or exploits
[24:08.500 --> 24:12.580]  and you use them to make the system do things you don't want it,
[24:12.580 --> 24:15.060]  it shouldn't be doing under normal circumstances.
[24:16.400 --> 24:18.360]  Or wasn't intended to be doing.
[24:18.360 --> 24:18.880]  Right.
[24:18.880 --> 24:20.380]  And sometimes they're useful.
[24:21.740 --> 24:25.640]  Would you consider a well-informed person building an attack
[24:25.640 --> 24:28.080]  against a machine learning system
[24:28.080 --> 24:29.820]  where they understand the machine learning system,
[24:29.820 --> 24:31.820]  they understand the feature extract and all that stuff,
[24:31.820 --> 24:35.640]  and they've built a false positive.
[24:36.460 --> 24:39.300]  Is that an adversarial attack against the machine learning
[24:39.300 --> 24:41.180]  or is that like an exploit?
[24:42.120 --> 24:44.700]  Like, where would you classify the mistake?
[24:44.700 --> 24:50.020]  Is it like a software mistake or a exploit or what?
[24:51.580 --> 24:54.240]  I think that's the tricky bit about ML models, right?
[24:54.240 --> 24:57.180]  I mean, they're inherently statistical.
[24:59.620 --> 25:03.140]  So it kind of gets back to testing after you retrain it, right?
[25:03.140 --> 25:08.280]  So if you have a single example
[25:08.280 --> 25:10.460]  and it turns out to be a false positive,
[25:10.460 --> 25:12.300]  and maybe that's one of the bridges
[25:12.300 --> 25:14.600]  that I feel like as a data scientist,
[25:14.600 --> 25:18.320]  I have to cross a lot when I'm talking to security specialists.
[25:18.320 --> 25:21.500]  If I have a model and it FPs on a single file,
[25:21.500 --> 25:25.380]  that doesn't call into question the efficacy of the model, right?
[25:25.380 --> 25:28.560]  You have to look at the statistics of what the model does, right?
[25:28.560 --> 25:30.800]  And if you say, well, yeah, it got this FP
[25:30.800 --> 25:33.300]  and that was a pretty boneheaded FP,
[25:33.300 --> 25:41.560]  but for 99% detection, you get one FP in 10,000 negative samples.
[25:41.720 --> 25:43.380]  That's a pretty good model.
[25:43.520 --> 25:47.880]  And so being able to sort of drive that switch from,
[25:47.880 --> 25:52.200]  hey, here's the good, bad to,
[25:52.200 --> 25:55.360]  hey, you have to think of this sort of heuristically and statistically.
[25:55.980 --> 26:01.140]  Is a big part of just like communicating as a data scientist, I think.
[26:02.360 --> 26:03.940]  Go ahead, Sarah.
[26:03.940 --> 26:05.760]  Yeah, if you've got somebody actively attacking,
[26:05.760 --> 26:09.840]  then maybe some pre-processing is enough to cut down the attacks.
[26:09.840 --> 26:11.540]  Yeah, yeah, for sure.
[26:12.020 --> 26:15.860]  All of that goes out the window when you've got someone maliciously trying to...
[26:17.280 --> 26:20.320]  Yeah, but so my question is,
[26:20.320 --> 26:28.780]  so if you have like a fancy ML from an academic paper bypass of your model,
[26:28.780 --> 26:34.980]  versus someone trying things, what would be...
[26:37.100 --> 26:39.320]  When is it like a machine learning mistake
[26:39.320 --> 26:43.360]  or versus a mistake in your featurizer and stuff?
[26:44.040 --> 26:47.020]  If you look at the decision boundary,
[26:47.020 --> 26:51.640]  basically for something and seeing how the perturbations relate to that?
[26:51.900 --> 26:58.760]  Yeah, I kind of feel like maybe to pull it back to this paper a little bit,
[26:59.720 --> 27:07.260]  if it gets through your feature extraction process in one shape,
[27:07.260 --> 27:12.640]  in one piece, then maybe it's a modeling issue.
[27:12.640 --> 27:15.340]  And if it crashes your feature extraction process
[27:15.340 --> 27:19.760]  or it gives you something that you wouldn't expect
[27:19.760 --> 27:24.680]  from your feature extraction process, then it's a featurization bug, right?
[27:25.140 --> 27:27.680]  If you feed garbage into a model,
[27:27.680 --> 27:30.740]  you can't blame the model for producing garbage outputs.
[27:31.660 --> 27:33.020]  How would you basically...
[27:33.020 --> 27:36.360]  What are some methods for quantifying unexpectedness in this scenario?
[27:36.360 --> 27:40.660]  Is it more like error counts or latencies in response?
[27:40.660 --> 27:43.100]  What would you pay attention to if you were trying to understand
[27:44.280 --> 27:46.040]  if something was misbehaving?
[27:47.960 --> 27:49.720]  You know what, when you see it.
[27:49.820 --> 27:52.240]  Yeah, like if you take a look in the paper, right?
[27:52.240 --> 27:54.780]  They've got an example. I want to find it.
[27:55.440 --> 27:56.620]  Figure two, right?
[27:56.620 --> 27:59.740]  So they show an example of two different images.
[27:59.840 --> 28:05.560]  One is an image of sort of the very top of a person's head
[28:05.560 --> 28:07.320]  and then the rest of it is gray.
[28:07.580 --> 28:11.040]  And one is a picture of a person's head.
[28:11.040 --> 28:12.340]  Yes, there you go. Thank you.
[28:13.480 --> 28:15.340]  Page six, if you're following along.
[28:15.340 --> 28:17.960]  Right, so OpenCV has a bug that these guys found
[28:17.960 --> 28:21.020]  by their guided fuzzing technique,
[28:21.020 --> 28:25.820]  which means that OpenCV can't properly load the file
[28:25.820 --> 28:26.740]  and then you run it...
[28:26.740 --> 28:28.180]  Obviously, you run it through feature extraction
[28:28.180 --> 28:30.300]  and it doesn't get anywhere.
[28:30.300 --> 28:33.240]  So it's like all these different individual components
[28:33.240 --> 28:34.980]  that you got to worry about, right?
[28:36.300 --> 28:38.940]  And I think on that topic, there was a...
[28:38.940 --> 28:41.500]  From Hacker Factor in Discord, he's got a question.
[28:41.500 --> 28:43.680]  Are you making a distinction between the ML model
[28:43.680 --> 28:46.740]  versus software that drives the ML system?
[28:46.880 --> 28:49.020]  He says fundamental error versus algorithmic.
[28:49.020 --> 28:51.240]  And I think here, right?
[28:51.240 --> 28:53.900]  This is what this paper is going after, right?
[28:53.900 --> 28:56.360]  It's saying the software that drives the ML system,
[28:56.360 --> 28:59.440]  the implementation of the system also has bugs
[28:59.440 --> 29:00.600]  and they find them here.
[29:00.600 --> 29:04.840]  And, you know, if you go to table two, right?
[29:04.840 --> 29:09.020]  They've got in OpenCV, they've got like two heap corruptions.
[29:09.020 --> 29:11.180]  They've got a weird rendering bug that they found.
[29:11.180 --> 29:13.980]  Page six, thank you, yeah.
[29:14.880 --> 29:16.700]  Right, so all these different...
[29:16.700 --> 29:18.980]  Like they literally have like heap corruption bugs
[29:18.980 --> 29:21.760]  that they're discovering with these funky inputs.
[29:22.380 --> 29:24.440]  And that has nothing to do with the model.
[29:24.440 --> 29:28.220]  That's you're attacking the plumbing around the model, which is...
[29:28.220 --> 29:31.660]  But you can use that plumbing to go and tweak the model.
[29:32.480 --> 29:33.700]  Right, exactly, yeah.
[29:33.700 --> 29:36.740]  So that's what the third line down does.
[29:36.740 --> 29:39.320]  That's the figure one, figure two?
[29:39.320 --> 29:40.680]  I forget, figure two.
[29:40.680 --> 29:43.880]  Yeah, so you break the rendering
[29:43.880 --> 29:47.120]  and then the rendering goes to the feature extraction
[29:47.120 --> 29:48.660]  and the feature extraction can't get anything
[29:48.660 --> 29:51.720]  because it's just got a bunch of gray pixels to look at.
[29:51.720 --> 29:53.160]  And then the model, right?
[29:53.160 --> 29:54.040]  Garbage in, garbage out.
[29:54.040 --> 29:56.780]  The model can't do anything because it doesn't have good features.
[29:57.860 --> 30:01.260]  So I'm going to try to open this up to Discord.
[30:01.740 --> 30:06.700]  If you are in the DevCon Discord,
[30:06.700 --> 30:09.140]  go to airvillage-general-voice.
[30:09.760 --> 30:14.960]  We'll try unmute you and get you so you can talk.
[30:16.180 --> 30:20.420]  If you're human plus, that's the correct way to do it,
[30:20.420 --> 30:22.080]  but we have a workaround.
[30:22.480 --> 30:27.220]  So head over to airvillage-general-voice to ask any questions.
[30:27.360 --> 30:30.600]  If you can't, we'll still be paying attention to the text chat
[30:30.600 --> 30:32.960]  in airvillage-journal-text.
[30:35.900 --> 30:37.940]  The moment of truth.
[30:39.940 --> 30:42.260]  I'm watching journal club text.
[30:42.360 --> 30:43.980]  Yeah, me too.
[30:45.220 --> 30:48.080]  Audio, managing audio is...
[30:48.080 --> 30:50.280]  I'm so glad I don't have to deal with it.
[30:57.170 --> 31:01.410]  I wish I had seen this paper before I started, like,
[31:01.410 --> 31:05.690]  really trying to solve adversarial machine learning as a security problem.
[31:05.890 --> 31:07.340]  Because there's a lot...
[31:07.860 --> 31:13.540]  this seems a lot easier, like, to attack for a lot of attack people.
[31:14.240 --> 31:16.680]  I think a lot of attackers have more knowledge
[31:16.680 --> 31:22.400]  on how to break a parser than how to break a neural network.
[31:23.880 --> 31:28.860]  So this, to me, feels like more of a realistic threat model
[31:28.860 --> 31:32.640]  of, like, your pipeline was... they broke your pipeline.
[31:32.640 --> 31:34.400]  You did something wrong in AWS.
[31:34.400 --> 31:38.940]  You did something wrong with your container setup.
[31:38.940 --> 31:42.200]  And it's parsing your images incorrectly now.
[31:44.140 --> 31:47.980]  Yeah, I mean, that's tons of the bugs that you find
[31:47.980 --> 31:52.100]  in sort of real systems, right, as parsing bugs.
[31:52.100 --> 32:00.320]  And yeah, a lot of the research right now focuses less on that
[32:00.320 --> 32:05.320]  and more on sort of these theoretical properties of machine learning models,
[32:05.320 --> 32:07.380]  which, I mean, I don't want to dig on those.
[32:07.380 --> 32:09.400]  They're fascinating, right? I love these papers.
[32:09.400 --> 32:13.320]  But they're also, again, keep coming back to this paper.
[32:13.320 --> 32:16.780]  This paper makes the really good point that these sort of theoretical models
[32:16.780 --> 32:20.420]  that have these nice properties are embedded in real world systems.
[32:20.420 --> 32:23.140]  And we know that real world systems are always kind of
[32:23.140 --> 32:26.680]  full of bugs that can be attacked and exploited.
[32:27.380 --> 32:32.040]  And so it gives you sort of multiple entries into the problem, right?
[32:32.040 --> 32:34.500]  On the one hand, you can exploit the preprocessor,
[32:34.500 --> 32:36.200]  and maybe that gives you some sort of heap corruption
[32:36.200 --> 32:42.360]  or some sort of like direct, you know, denial of service or even code execution.
[32:42.500 --> 32:45.220]  And on the other hand, you can break the feature extraction enough
[32:45.220 --> 32:48.720]  so that everything downstream is getting garbage data.
[32:50.160 --> 32:55.060]  Or mildly corrupted. I mean, I say again, with like the sneaking in,
[32:55.060 --> 32:57.480]  you want to do low and slows on some of this.
[32:59.000 --> 33:02.520]  Again, I am really like the idea of using this as defense.
[33:02.520 --> 33:06.180]  I mean, there's a whole bunch of applications where the bad guys
[33:06.180 --> 33:09.120]  are using machine learning based systems.
[33:09.520 --> 33:12.020]  I like the idea of breaking their systems.
[33:12.160 --> 33:15.200]  Yeah, as I would. Great hats on.
[33:16.780 --> 33:26.170]  Yeah, just like yesterday's paper with the Forbes masks.
[33:27.390 --> 33:31.730]  Mm-hmm. Yeah, so Will has a comment.
[33:31.730 --> 33:34.350]  He says, I'm surprised a paper is the vehicle for this knowledge.
[33:34.350 --> 33:36.710]  There's a whole security industry that has advice for this stuff.
[33:36.710 --> 33:43.010]  And I think, yeah, this is part of what we're trying to do, right, with AI Village
[33:43.010 --> 33:47.670]  is draw a link between security and machine learning, right?
[33:47.670 --> 33:52.750]  So we have a lot of like wicked smart security people
[33:53.370 --> 33:56.690]  who know about all about these implementation bugs.
[33:56.690 --> 33:59.990]  And then we've got a bunch of like really smart academics
[33:59.990 --> 34:02.990]  and machine learning people who know about like sort of these,
[34:02.990 --> 34:06.350]  you know, feature based attacks or weird properties of ML systems.
[34:06.350 --> 34:08.630]  And getting sort of that knowledge transfer going on
[34:08.630 --> 34:11.990]  so that we can find out where those intersect
[34:11.990 --> 34:15.270]  and how those two, you know, sort of sets of attacks inform each other.
[34:15.270 --> 34:20.270]  I think that's like a really rich area for future development.
[34:20.270 --> 34:25.590]  And it kind of makes me sad a little bit that this paper came out in 2017
[34:25.590 --> 34:28.450]  and got comparatively little attention
[34:29.650 --> 34:34.130]  when it really is going right to the heart of a lot of really big questions
[34:34.130 --> 34:37.650]  in this, you know, in ML slash security.
[34:37.650 --> 34:41.710]  So in practice, you see a lot of data science teams
[34:41.710 --> 34:43.930]  disconnected from both like infrastructure application
[34:43.930 --> 34:46.090]  network security sort of teams.
[34:46.090 --> 34:51.070]  And so like the whole MLOps sort of formation of teams
[34:51.070 --> 34:53.310]  that can interface between data scientists
[34:53.310 --> 34:56.070]  and implementation has been emerging.
[34:56.070 --> 34:57.790]  But I'm curious what the panel thinks as far as
[34:58.730 --> 35:02.250]  what's the appropriate like role, a person, a cross-functional team?
[35:02.250 --> 35:04.730]  Do we want to have security people sitting with data scientists
[35:04.730 --> 35:07.170]  as these things are being developed?
[35:08.270 --> 35:08.730]  And...
[35:08.730 --> 35:11.430]  Yeah, I mean, weirdly enough,
[35:11.430 --> 35:14.570]  I used to be one of the people going into large companies
[35:14.570 --> 35:16.730]  working out where to put the data scientists.
[35:16.750 --> 35:19.450]  And there was always this argument between
[35:19.450 --> 35:22.770]  you want to embed the data scientists out into the rest of the team.
[35:23.310 --> 35:25.150]  And have them work with because you're informing
[35:25.150 --> 35:26.550]  the rest of the team everywhere.
[35:27.710 --> 35:29.870]  But then you've got data scientists out on their own.
[35:29.870 --> 35:32.350]  And we're kind of, you know, you see in this paper
[35:32.350 --> 35:35.490]  that we get ignored quite a bit when we're on our own.
[35:35.670 --> 35:38.290]  Or having these unicorn pens full of data scientists
[35:38.290 --> 35:41.290]  who talk to each other, but not really to anybody else.
[35:41.810 --> 35:44.990]  And there's this sense of you put people out,
[35:44.990 --> 35:46.790]  but you make sure they're still part of TRIPE.
[35:46.790 --> 35:48.870]  So you literally build TRIPE
[35:48.870 --> 35:51.570]  or whatever the good word now is for that.
[35:51.570 --> 35:53.650]  It used to be called TRIPE, but cross...
[35:53.650 --> 35:54.510]  Village?
[35:54.610 --> 35:55.850]  Cross, yeah, village.
[35:56.450 --> 35:58.850]  So you build villages for the data scientists
[35:58.850 --> 36:00.070]  to keep connected to each other,
[36:00.070 --> 36:04.330]  but they need to be out in the rest of the teams.
[36:04.770 --> 36:08.270]  Because otherwise, you know, our work is everywhere.
[36:08.290 --> 36:09.630]  We should be everywhere too.
[36:10.350 --> 36:11.970]  That's strongly our belief though.
[36:12.510 --> 36:14.550]  Yeah, I mean, I think it's, but it's got to be
[36:14.550 --> 36:15.690]  sort of a two-way street, right?
[36:15.690 --> 36:19.550]  On the one hand, yeah, there is kind of,
[36:19.550 --> 36:23.310]  for data scientists and sort of ML practitioners,
[36:23.310 --> 36:26.290]  I think that, yeah, we do sort of tend to stay
[36:26.290 --> 36:28.090]  in our own little bubble and don't think about
[36:29.250 --> 36:32.210]  these sort of externalities.
[36:32.210 --> 36:33.450]  I don't know what the right word for it is,
[36:33.450 --> 36:35.550]  but like these other ways, you know,
[36:35.550 --> 36:36.710]  it's not just going to be like
[36:36.710 --> 36:38.190]  fast, radiant, sign-based attacks.
[36:38.190 --> 36:41.110]  It's going to be like, no, someone broke the parsing.
[36:41.110 --> 36:43.870]  But then at the same time,
[36:43.870 --> 36:45.250]  and this is kind of going back to maybe
[36:45.470 --> 36:47.410]  a little bit of a hype panel yesterday,
[36:48.730 --> 36:50.790]  having people that aren't data scientists
[36:51.350 --> 36:53.750]  understand what the models can and can't do
[36:53.750 --> 36:55.710]  and what kind of behavior to expect from them
[36:55.710 --> 36:57.130]  so that they don't freak out
[36:57.130 --> 36:58.930]  when they see something weird happen
[36:58.930 --> 37:01.710]  and be like, oh, you know, whatever.
[37:01.710 --> 37:04.430]  We flagged this DLL as being malware when it's not.
[37:04.430 --> 37:06.530]  We're clearly under attack, right?
[37:06.530 --> 37:08.250]  And you're just like, it's statistics.
[37:08.250 --> 37:09.730]  Sometimes it makes mistakes.
[37:10.590 --> 37:13.470]  Well, just explaining it's not magic.
[37:13.630 --> 37:15.330]  It's hard, but it's not magic.
[37:15.330 --> 37:16.570]  Yeah, it's hard. It's useful.
[37:16.570 --> 37:19.210]  It does cool stuff, but it's a tool.
[37:26.530 --> 37:28.370]  Jager's talking about implementing
[37:28.570 --> 37:29.770]  a security development lifecycle
[37:30.350 --> 37:32.630]  and a secure software development lifecycle
[37:32.630 --> 37:34.590]  into model creation and upkeep.
[37:34.750 --> 37:36.770]  I'm going to make a data scientist cry.
[37:38.570 --> 37:39.410]  Yeah.
[37:40.030 --> 37:43.650]  I had issues with just being like, cool,
[37:43.650 --> 37:45.910]  we need to just write a few unit tests
[37:45.910 --> 37:48.330]  to make sure that the functions
[37:48.330 --> 37:50.730]  all kind of do what they're supposed to do.
[37:51.390 --> 37:52.950]  There's a bit of pushback
[37:52.950 --> 37:57.670]  from our more academically minded folk
[37:57.670 --> 38:01.290]  of just a little bit of basics like that,
[38:01.920 --> 38:05.430]  which may have solved some of these open CV issues.
[38:09.550 --> 38:12.630]  Interesting that if you basically treat
[38:13.160 --> 38:15.330]  the lifecycle of a data project
[38:15.330 --> 38:17.890]  as more of an internal product for a company
[38:17.890 --> 38:20.730]  and treat it as a product which serves a purpose,
[38:20.730 --> 38:23.410]  provides value, and then is integrated
[38:23.410 --> 38:25.770]  with other ways of developing software products,
[38:25.770 --> 38:27.950]  then data science and machine learning,
[38:27.950 --> 38:30.030]  they just become tools in the engineering toolbox,
[38:30.030 --> 38:31.990]  but then everything else falls within methods
[38:31.990 --> 38:34.950]  of continuous integration and testing and delivery.
[38:34.950 --> 38:36.630]  So then perhaps the risk,
[38:36.630 --> 38:38.810]  rather than the unicorn, depends on data scientists.
[38:38.810 --> 38:39.370]  Yeah.
[38:39.770 --> 38:40.690]  That's what we're doing.
[38:40.690 --> 38:45.250]  We're doing things like hypothesis-based development,
[38:45.250 --> 38:46.750]  which is like the next thing up
[38:46.750 --> 38:48.830]  from behavior-based development,
[38:48.830 --> 38:52.690]  which is next thing up for test-driven development.
[38:52.910 --> 38:55.890]  So you very much work out what the universe
[38:55.890 --> 38:59.330]  you'd like to see is, go do the maths on it,
[38:59.330 --> 39:00.650]  build the systems in it.
[39:00.970 --> 39:03.090]  So there are ways of working.
[39:05.090 --> 39:06.350]  Yeah, so maybe looping...
[39:06.990 --> 39:07.850]  Sorry, go ahead, Sven.
[39:07.850 --> 39:08.790]  Go ahead.
[39:09.690 --> 39:11.370]  Yeah, so I was going to loop back
[39:11.370 --> 39:12.450]  to the paper one more time
[39:13.130 --> 39:15.570]  and talk about the disclosure experience
[39:15.570 --> 39:18.410]  that they mentioned towards the end of the article.
[39:18.830 --> 39:21.790]  So basically, three of the vulnerabilities
[39:21.790 --> 39:25.570]  got new CVEs because they enabled
[39:25.570 --> 39:26.950]  arbitrary code execution
[39:26.950 --> 39:28.570]  or denial-of-service attacks, right?
[39:28.570 --> 39:29.730]  So those are both very firmly
[39:29.730 --> 39:31.770]  within the wheelhouse of what people...
[39:31.770 --> 39:32.550]  Page seven.
[39:32.550 --> 39:33.490]  Page seven, thank you.
[39:34.270 --> 39:35.510]  Very much within the wheelhouse
[39:35.510 --> 39:36.910]  of what security people think of as,
[39:36.910 --> 39:38.930]  oh yeah, that's a bug, that's an issue,
[39:38.930 --> 39:40.930]  that's a security gap that we need to fix.
[39:40.930 --> 39:42.690]  But there were a couple of other ones
[39:42.690 --> 39:45.870]  that they found where they could impact
[39:45.870 --> 39:47.350]  or manipulate the prediction.
[39:47.350 --> 39:48.850]  And then they call out in particular
[39:48.850 --> 39:51.070]  the mouthier memory corruption they found,
[39:51.070 --> 39:52.730]  where basically they can rewrite
[39:52.730 --> 39:54.450]  the feature vector,
[39:54.450 --> 39:57.470]  which allows you to basically
[39:57.470 --> 39:58.750]  make the model give you
[39:58.750 --> 40:00.930]  any output you want, essentially.
[40:01.150 --> 40:03.110]  Doesn't have to have anything to do
[40:03.110 --> 40:04.510]  with what the actual file
[40:04.510 --> 40:06.150]  that you were analyzing was.
[40:06.250 --> 40:08.050]  And those specifically are the ones
[40:08.050 --> 40:12.510]  that didn't get a CVE,
[40:12.510 --> 40:13.310]  didn't get called out,
[40:13.310 --> 40:15.730]  and mostly got labeled as won't fix.
[40:15.730 --> 40:17.030]  Or as won't fix.
[40:17.030 --> 40:18.270]  So that's sort of like,
[40:18.270 --> 40:19.410]  we were kind of bashing
[40:19.410 --> 40:20.870]  the data scientists earlier saying,
[40:20.870 --> 40:22.350]  oh, we got to think of this
[40:22.350 --> 40:23.650]  as part of a security system.
[40:23.650 --> 40:25.770]  But then it goes back the other way, right?
[40:25.770 --> 40:27.570]  We need to be able to communicate to people
[40:27.570 --> 40:28.870]  that as data scientists,
[40:28.870 --> 40:29.990]  as machine learning experts,
[40:29.990 --> 40:32.570]  look, this is bad, right?
[40:32.570 --> 40:34.510]  And how do we convince people that
[40:35.270 --> 40:36.310]  something that allows
[40:36.310 --> 40:37.790]  arbitrary misclassification
[40:37.790 --> 40:38.790]  or arbitrary predictions
[40:38.790 --> 40:43.250]  to be produced is as serious as,
[40:43.250 --> 40:44.770]  or maybe not as serious,
[40:45.550 --> 40:47.650]  but has some degree of severity,
[40:47.650 --> 40:48.930]  just like a remote code execution
[40:48.930 --> 40:50.710]  or a denial of service.
[40:50.710 --> 40:52.390]  I like the example that Eric brought up
[40:52.390 --> 40:54.190]  in his talk of a turtle rifle.
[40:54.370 --> 40:56.110]  Like that is a very effective way
[40:56.110 --> 40:57.930]  of communicating potential problems.
[40:58.430 --> 41:00.550]  Whereas for those who haven't seen it,
[41:00.550 --> 41:03.810]  like a turtle 3D printed object
[41:03.810 --> 41:05.550]  could be applied to a turtle
[41:05.550 --> 41:08.510]  to make it misclassified as a rifle.
[41:08.570 --> 41:09.650]  You can imagine like systems
[41:09.650 --> 41:11.690]  that track security
[41:11.690 --> 41:16.210]  and stuff to do certain events.
[41:17.230 --> 41:18.050]  I mean, it's like
[41:18.050 --> 41:19.290]  any other ethics discussion.
[41:19.290 --> 41:20.610]  We have to talk about consequences
[41:21.190 --> 41:22.490]  and then track it back
[41:22.490 --> 41:24.350]  and just keep pushing
[41:24.350 --> 41:25.710]  and pushing examples,
[41:25.710 --> 41:27.390]  real examples if we've got them.
[41:28.830 --> 41:30.270]  It's hard.
[41:31.450 --> 41:34.490]  Yeah, I mean, I guess like that's,
[41:34.490 --> 41:36.410]  you have a joke,
[41:36.410 --> 41:37.590]  but we're not really a joke,
[41:37.590 --> 41:38.870]  that like airline regulations
[41:38.870 --> 41:40.990]  are written in blood, right?
[41:40.990 --> 41:41.790]  And so when you've got
[41:41.790 --> 41:43.210]  all of these safety requirements,
[41:43.210 --> 41:44.950]  it's because a lot of people got hurt
[41:44.950 --> 41:45.890]  before someone was like,
[41:45.890 --> 41:47.710]  we ought to do something about that.
[41:47.750 --> 41:50.050]  And it would be nice to think
[41:50.050 --> 41:52.690]  that we could find a way
[41:52.690 --> 41:56.870]  to not have like ML disclosure requirements
[41:56.870 --> 41:58.390]  written in blood, right?
[41:58.390 --> 41:59.610]  Like maybe, you know,
[42:00.610 --> 42:01.830]  we're at a place now
[42:01.830 --> 42:02.810]  where we could maybe like
[42:02.810 --> 42:04.390]  jump ahead of the game a little bit
[42:04.390 --> 42:07.710]  and get an understanding
[42:07.710 --> 42:08.330]  of this out there
[42:08.330 --> 42:10.030]  so that we don't have to see people hurt
[42:10.030 --> 42:10.910]  before people are like,
[42:10.910 --> 42:11.970]  yeah, we should maybe
[42:11.970 --> 42:13.290]  take that seriously.
[42:13.290 --> 42:14.710]  I actually have a historical example
[42:14.710 --> 42:17.010]  for that too, for being old.
[42:17.010 --> 42:19.550]  So I ran one of the
[42:19.550 --> 42:21.190]  unmanned air vehicle safety teams
[42:21.190 --> 42:23.900]  back in the day before UAVs were sexy.
[42:25.230 --> 42:28.030]  And we didn't have the
[42:30.870 --> 42:32.230]  privilege, I guess,
[42:32.230 --> 42:34.810]  that the rest of the airspace industry had.
[42:34.810 --> 42:36.630]  So aerospace regulations
[42:36.630 --> 42:38.090]  are literally written in blood.
[42:38.090 --> 42:39.930]  So an aircraft would crash
[42:39.930 --> 42:40.930]  into something else
[42:40.930 --> 42:42.570]  or hit the ground
[42:42.570 --> 42:44.690]  and you would change the regulations.
[42:44.710 --> 42:45.510]  And that's literally
[42:45.510 --> 42:46.990]  how the regulations are written.
[42:46.990 --> 42:48.210]  Same with fire safety.
[42:48.550 --> 42:51.270]  Fire happens, change the regulations.
[42:53.690 --> 42:58.010]  And we were told explicitly
[42:58.010 --> 42:59.230]  and understood explicitly
[42:59.230 --> 43:02.010]  that we couldn't just put UAVs
[43:02.010 --> 43:03.030]  into manned airspace,
[43:03.030 --> 43:04.090]  have them crash into stuff
[43:04.090 --> 43:05.570]  and then rewrite the rules.
[43:06.050 --> 43:07.470]  We literally had to think
[43:07.470 --> 43:08.290]  all the scenarios
[43:09.590 --> 43:11.350]  and write the rules
[43:11.350 --> 43:13.090]  for how to do it safely.
[43:13.610 --> 43:15.090]  And it's whole space safety,
[43:15.090 --> 43:16.950]  not just individual aircraft safety
[43:16.950 --> 43:18.190]  before we were allowed
[43:18.190 --> 43:20.050]  to fly in the same spaces.
[43:21.330 --> 43:23.330]  So there's some precedent.
[43:23.350 --> 43:24.670]  I mean, there's maybe
[43:24.670 --> 43:26.610]  some looking in those spaces
[43:26.610 --> 43:28.490]  to see how it was done well.
[43:28.830 --> 43:30.330]  That's a good way to look.
[43:31.670 --> 43:33.770]  But a lot of it's just getting the will.
[43:34.970 --> 43:35.670]  Now, nothing in focus
[43:35.670 --> 43:36.230]  gets your attention
[43:36.230 --> 43:37.330]  like a whole bunch of airliners
[43:37.330 --> 43:38.270]  about to crash.
[43:38.290 --> 43:42.370]  But yeah, we could maybe
[43:42.370 --> 43:43.770]  get there before that point.
[43:46.650 --> 43:49.230]  So one more question from Discord,
[43:49.230 --> 43:50.230]  which is good
[43:50.230 --> 43:51.950]  because that's where I wanted to go next.
[43:52.430 --> 43:53.750]  So Yaga has a question.
[43:53.750 --> 43:55.350]  How do we force open source software
[43:55.350 --> 43:56.230]  to fix itself?
[43:56.230 --> 43:57.210]  We don't fix it ourselves.
[43:57.210 --> 43:58.530]  That's why it's open source.
[43:58.570 --> 44:00.170]  And the authors actually make a comment
[44:00.170 --> 44:01.390]  in the future work
[44:02.290 --> 44:04.970]  where they say that it's unclear
[44:04.970 --> 44:06.050]  who should be responsible
[44:06.050 --> 44:07.390]  for fixing them as well.
[44:07.390 --> 44:09.730]  So when they found a bug
[44:09.730 --> 44:11.630]  in the Malheur feature processing,
[44:11.630 --> 44:13.310]  that was because Malheur relied
[44:13.310 --> 44:15.170]  on libarchive.
[44:15.610 --> 44:17.130]  And the bug was actually
[44:17.130 --> 44:18.370]  in libarchive.
[44:18.370 --> 44:19.190]  And again, this gets back
[44:19.190 --> 44:20.510]  to this notion of like,
[44:20.510 --> 44:22.330]  you have all these different components
[44:22.330 --> 44:24.090]  that feed into the system.
[44:25.130 --> 44:26.470]  So even though the bug
[44:26.470 --> 44:27.550]  was in libarchive,
[44:27.550 --> 44:29.150]  it affected Malheur.
[44:29.150 --> 44:31.510]  So where do you put
[44:31.510 --> 44:33.090]  the responsibility for fixing it?
[44:33.090 --> 44:34.950]  And how do you convince,
[44:34.950 --> 44:36.430]  you know, like whoever's
[44:36.430 --> 44:37.490]  maintaining libarchive,
[44:37.490 --> 44:39.690]  look, this is serious enough that,
[44:39.690 --> 44:40.630]  you know, yes, I don't think
[44:40.630 --> 44:42.070]  it doesn't affect you directly,
[44:42.070 --> 44:43.710]  but it does affect
[44:43.710 --> 44:44.970]  this other system
[44:44.970 --> 44:46.210]  and it could be a critical impact.
[44:46.210 --> 44:47.530]  Like, how do we,
[44:47.530 --> 44:48.270]  how can we navigate
[44:48.270 --> 44:50.590]  that sort of uncertainty?
[44:52.170 --> 44:55.030]  How does Linux and Ubuntu
[44:55.030 --> 44:56.750]  and other open source
[44:56.750 --> 44:59.150]  software ecosystems do it?
[45:02.130 --> 45:03.330]  Do they have liability?
[45:03.870 --> 45:05.990]  I mean, I know for a lot of systems,
[45:05.990 --> 45:07.310]  the end users,
[45:07.310 --> 45:08.570]  especially if they're big end users,
[45:08.570 --> 45:09.130]  just get involved
[45:09.130 --> 45:10.230]  with the open source communities
[45:10.870 --> 45:12.710]  and work in there.
[45:12.710 --> 45:15.410]  But if you're using
[45:16.870 --> 45:18.330]  even like Python
[45:18.330 --> 45:21.050]  and your physical system crashes,
[45:21.590 --> 45:22.730]  who's responsible?
[45:26.030 --> 45:28.790]  Yeah. I mean, a lot of licensing argument,
[45:29.010 --> 45:30.110]  a lot of licensing agreements
[45:30.110 --> 45:31.350]  basically say like,
[45:31.350 --> 45:33.330]  yeah, you agree that you're using this
[45:33.330 --> 45:34.990]  at your own risk too.
[45:35.030 --> 45:37.470]  So, you know, basically
[45:37.470 --> 45:38.650]  everyone's going around
[45:38.650 --> 45:42.290]  not me, which, you know, it's fair, right?
[45:42.290 --> 45:43.850]  And people have, like, they have,
[45:43.850 --> 45:44.830]  they can't assume liability
[45:44.830 --> 45:46.510]  for everything that they do.
[45:46.510 --> 45:49.590]  But, you know,
[45:49.590 --> 45:50.770]  they can't agree to fix
[45:50.770 --> 45:52.630]  every single thing.
[45:52.910 --> 45:54.030]  But at some point, right,
[45:54.030 --> 45:55.110]  there's a balance, right?
[45:55.110 --> 45:56.430]  If something's widely adopted,
[45:56.430 --> 45:57.270]  it's widely used,
[45:57.270 --> 45:58.190]  and there's like a bug
[45:58.190 --> 45:59.410]  three steps up the chain
[45:59.410 --> 46:01.510]  that's causing it to behave weirdly.
[46:01.910 --> 46:04.350]  It feels like somebody ought to,
[46:04.350 --> 46:05.190]  there ought to be like
[46:05.190 --> 46:07.170]  some impetus to fix it somehow.
[46:07.170 --> 46:08.770]  The low tail of companies like Fortune 500
[46:08.770 --> 46:09.730]  is more bothered to use
[46:09.730 --> 46:10.730]  these open source libraries
[46:10.730 --> 46:12.450]  than in theory, you know, this is AI,
[46:12.450 --> 46:13.530]  so you have a lot of, like,
[46:13.530 --> 46:15.410]  boost in productivity,
[46:15.410 --> 46:16.370]  like, and there's a lot of value
[46:16.370 --> 46:17.090]  that's being derived,
[46:17.090 --> 46:18.070]  and especially if they've started
[46:18.070 --> 46:19.570]  to use it for critical applications
[46:19.570 --> 46:20.830]  or decisions.
[46:20.890 --> 46:22.570]  So I think the onus and responsibility
[46:22.570 --> 46:23.910]  also needs to fall on the end users
[46:23.910 --> 46:25.410]  of, like, widely adopted packages,
[46:25.410 --> 46:26.950]  and then somehow making,
[46:26.950 --> 46:28.290]  making it very clear to decision makers
[46:28.290 --> 46:29.230]  in those companies that,
[46:29.230 --> 46:30.370]  hey, if you're using this to
[46:31.010 --> 46:32.310]  predict insurance rates or something
[46:32.310 --> 46:33.430]  and it fails,
[46:33.430 --> 46:35.030]  that's what you could potentially face,
[46:35.030 --> 46:36.530]  and then somehow channeling that
[46:36.530 --> 46:37.830]  into the projects themselves.
[46:38.010 --> 46:39.810]  Are you proposing that large companies
[46:39.810 --> 46:40.730]  that make a lot of money
[46:40.730 --> 46:41.830]  using open source tools
[46:41.830 --> 46:42.730]  should actually give back
[46:42.730 --> 46:44.270]  to the open source community?
[46:48.330 --> 46:50.730]  Smells like communism to me.
[46:51.850 --> 46:53.250]  It's kind of terrifying
[46:53.250 --> 46:54.190]  that we all just laughed
[46:54.190 --> 46:55.550]  when you said that.
[46:57.390 --> 46:58.270]  Yep.
[46:58.330 --> 46:59.550]  Sorry, I got to get my,
[46:59.550 --> 47:00.770]  got to get my commentary
[47:00.770 --> 47:02.070]  in once in a while.
[47:03.110 --> 47:04.290]  These are complex systems,
[47:04.290 --> 47:05.130]  and, like, just communicating
[47:05.130 --> 47:07.190]  the impact of some small,
[47:07.190 --> 47:07.890]  like, overflow bug
[47:07.890 --> 47:09.430]  to somebody who only cares about,
[47:09.430 --> 47:11.390]  like, the bottom line is a problem.
[47:11.390 --> 47:12.930]  Maybe there needs to be, like,
[47:12.930 --> 47:14.130]  awesome ML failures,
[47:14.130 --> 47:14.870]  GitHub repository,
[47:14.870 --> 47:15.490]  or something like that,
[47:15.490 --> 47:16.630]  where you can find it.
[47:17.070 --> 47:18.810]  We haven't had this.
[47:18.950 --> 47:20.270]  Sounds like, actually,
[47:20.390 --> 47:21.150]  a pretty good weekend project
[47:21.150 --> 47:21.990]  for someone.
[47:21.990 --> 47:22.870]  Yeah, I mean, like, you know,
[47:22.870 --> 47:23.490]  when we started up
[47:23.490 --> 47:24.830]  the ethics stuff years ago,
[47:24.830 --> 47:26.050]  it just, just takes
[47:26.050 --> 47:26.910]  some people determined
[47:26.910 --> 47:28.590]  to make this thing a thing,
[47:28.790 --> 47:29.810]  a visible thing.
[47:29.950 --> 47:30.550]  Yeah.
[47:30.610 --> 47:31.630]  Hey, community.
[47:32.030 --> 47:33.550]  Anyone want to take this one on?
[47:35.130 --> 47:36.370]  So, how many, like,
[47:36.370 --> 47:38.210]  solid examples of ML failure
[47:38.210 --> 47:39.510]  do we have?
[47:41.470 --> 47:42.130]  If we want to...
[47:42.130 --> 47:43.230]  I mean, a lot of it is.
[47:43.310 --> 47:44.710]  Compile those, yeah.
[47:44.710 --> 47:46.410]  How much do we actually know?
[47:46.670 --> 47:50.490]  Yeah, that's, I want to say,
[47:50.490 --> 47:53.030]  Andrew Davis over
[47:53.030 --> 47:54.170]  in different Discord
[47:54.170 --> 47:56.370]  had a pretty good comment on it,
[47:56.370 --> 47:57.630]  which is that a lot of time
[47:57.630 --> 47:59.230]  when ML fails,
[47:59.230 --> 48:01.170]  it kind of fails silently, right?
[48:01.170 --> 48:03.170]  It just, you don't flag something,
[48:03.170 --> 48:04.730]  and it comes back to
[48:05.870 --> 48:06.910]  a lot of the earlier discussion
[48:06.910 --> 48:08.730]  about is it a bug,
[48:08.730 --> 48:10.210]  or is it, like, normal,
[48:10.210 --> 48:10.870]  just sort of, like,
[48:10.870 --> 48:12.990]  statistical failure, right?
[48:12.990 --> 48:16.070]  A 1% failure rate means that
[48:16.070 --> 48:18.430]  if it's not failing 1% of the time,
[48:18.430 --> 48:20.770]  you should be vaguely surprised, right?
[48:20.830 --> 48:23.430]  So, yeah, like,
[48:23.430 --> 48:24.230]  I think it's possible
[48:24.230 --> 48:26.010]  that there's a lot of ML failures
[48:26.010 --> 48:28.030]  that we just kind of assume
[48:28.030 --> 48:29.570]  are in the statistical noise.
[48:29.570 --> 48:31.590]  A lot of them are probably kept quiet
[48:31.590 --> 48:32.870]  because, you know,
[48:32.870 --> 48:34.270]  there's potentially, like,
[48:34.270 --> 48:36.550]  PR risks or PR damage, right?
[48:36.550 --> 48:37.990]  Or, you know, like,
[48:37.990 --> 48:39.250]  like, actual, like, gross,
[48:39.250 --> 48:42.370]  you know, liability attached to it.
[48:43.150 --> 48:46.090]  Yeah, I mean, it's an open question
[48:46.090 --> 48:48.490]  how much is actually sort of
[48:48.490 --> 48:50.730]  flying beneath the radar out there.
[48:50.790 --> 48:52.610]  I mentioned a rental company saying,
[48:52.610 --> 48:54.090]  or a background check company
[48:54.090 --> 48:56.130]  who uses ML going,
[48:56.130 --> 48:58.510]  so we didn't give a lot of people
[48:58.510 --> 49:00.630]  their approved, you know,
[49:00.630 --> 49:01.710]  their credit check approval
[49:02.790 --> 49:04.410]  because of a mistake.
[49:04.410 --> 49:05.350]  And we want to publicly
[49:05.350 --> 49:06.390]  apologize for that
[49:07.050 --> 49:08.870]  when it's an ML mistake.
[49:08.870 --> 49:09.670]  And they can just be like,
[49:09.670 --> 49:11.310]  well, the model just was,
[49:11.310 --> 49:12.610]  yeah, sweep it under the rug,
[49:12.610 --> 49:14.830]  fix it quietly, never mention it.
[49:17.390 --> 49:19.650]  Like, that would be an ethical nightmare.
[49:21.050 --> 49:24.490]  Yeah, I mean, there's lots of startups
[49:25.510 --> 49:27.590]  that have proposed to do things.
[49:27.590 --> 49:28.910]  There was one,
[49:28.910 --> 49:30.410]  what was it, about two, three years ago,
[49:30.750 --> 49:33.830]  oh, we'll use ML to pre-screen your babysitters
[49:33.830 --> 49:37.750]  by digging through their social media profiles and stuff.
[49:38.050 --> 49:40.630]  And I have my own opinions
[49:40.630 --> 49:42.710]  on how likely that was to succeed
[49:42.710 --> 49:45.070]  versus how likely it was
[49:45.070 --> 49:49.090]  to simply just recommend white people.
[49:49.090 --> 49:53.610]  But how do you even measure
[49:53.610 --> 49:57.230]  or quantify failures in that case?
[49:57.230 --> 49:58.930]  Because it's all going to be completely internal
[49:58.930 --> 50:04.470]  and all you can do is essentially try
[50:04.470 --> 50:06.630]  and get a proxy model to be like,
[50:06.630 --> 50:08.010]  aha, here are the features, right?
[50:08.010 --> 50:09.590]  I've submitted a thousand different profiles
[50:09.590 --> 50:12.610]  and here are the features that seem to be triggering, right?
[50:12.610 --> 50:16.410]  And you have to affirmatively dig for it, right?
[50:16.410 --> 50:18.050]  Those failures aren't going to be obvious
[50:18.050 --> 50:20.170]  unless you're actually doing something like that.
[50:21.170 --> 50:23.910]  A recent streak of sort of facial recognition failures
[50:23.910 --> 50:26.470]  that lead to characteristics
[50:26.470 --> 50:30.290]  that elicits a visceral gut reaction like,
[50:30.290 --> 50:31.330]  yes, this is wrong.
[50:31.730 --> 50:33.590]  So I think in some cases,
[50:33.590 --> 50:36.170]  just seeing the outcomes, like Claire said,
[50:36.170 --> 50:39.450]  helps understand whether it's a failure or not.
[50:39.670 --> 50:43.130]  This gets back to the bias question.
[50:43.130 --> 50:44.670]  Like, how do you tell if something's biased
[50:44.670 --> 50:49.070]  or just like for this one instance, it was wrong?
[50:49.070 --> 50:49.890]  Oh, man.
[50:51.530 --> 50:52.890]  Yeah, it kind of works.
[50:52.890 --> 50:54.710]  We're going to have to clean that one out, Tommy.
[50:54.770 --> 50:55.170]  Yeah.
[50:55.170 --> 50:58.210]  In images is, oh, God.
[50:58.910 --> 51:01.690]  How much longer do we have for this conversation?
[51:02.530 --> 51:04.550]  Yeah, we can solve it in eight minutes.
[51:04.550 --> 51:05.070]  We've got eight minutes.
[51:05.070 --> 51:06.130]  Yeah, yeah, no problem.
[51:06.170 --> 51:10.050]  Yeah, I mean, the problem in a lot of the bias stuff, right,
[51:10.050 --> 51:13.690]  is like the terms aren't even defined sometimes.
[51:13.690 --> 51:17.210]  So like the example I keep coming back to
[51:17.210 --> 51:22.030]  is like recidivism and like predicting re-arrest rates, right?
[51:22.030 --> 51:23.290]  And someone might point out,
[51:23.290 --> 51:26.670]  hey, look, this is biased, right?
[51:26.670 --> 51:28.430]  It's clearly predicting re-arrest rates
[51:28.430 --> 51:30.790]  that are much higher for this group, right?
[51:30.790 --> 51:34.090]  And one argument would be, well, that's clearly wrong, right?
[51:35.150 --> 51:40.130]  It's biased because it's trained on biased data.
[51:40.130 --> 51:41.810]  And the counter argument would be like,
[51:41.810 --> 51:44.270]  look, it's reproducing the data, right?
[51:44.270 --> 51:46.090]  We don't live in a perfect society.
[51:46.090 --> 51:47.570]  The data is what it is.
[51:47.570 --> 51:51.170]  And within that box, the ML has done as much
[51:51.170 --> 51:52.530]  as you can ask it to do.
[51:53.290 --> 51:59.090]  And so it always boils down to these normative questions.
[51:59.090 --> 52:02.330]  And this is my own totally personal opinion
[52:02.330 --> 52:07.910]  that talking about unbiased systems,
[52:07.910 --> 52:12.070]  unbiased ML is almost inherently impossible, right?
[52:12.070 --> 52:16.850]  It's a question of how is it producing
[52:16.850 --> 52:19.210]  the kinds of outcomes that you want
[52:19.210 --> 52:24.690]  and how is the ML system moving power around, right?
[52:24.690 --> 52:27.080]  Who is it empowering? Who is it disenfranchising?
[52:27.890 --> 52:31.470]  And who is it helping and who is it hurting?
[52:31.470 --> 52:37.890]  And chasing after some vague notion of fairness,
[52:38.170 --> 52:39.810]  a fair system or an unbiased system
[52:39.810 --> 52:42.170]  is kind of like a distraction
[52:42.170 --> 52:45.550]  from more fundamental questions of who's being helped,
[52:45.550 --> 52:49.330]  who's being hurt, what's being reinforced
[52:49.330 --> 52:51.730]  or whose concerns are being downplayed.
[52:52.010 --> 52:53.250]  And who's accountable, I guess,
[52:53.250 --> 52:55.710]  also because of the reference recent Twitter story
[52:55.710 --> 52:57.450]  between Timothy Kimbrough and Yann LeCun.
[52:57.450 --> 52:59.170]  Basically, is it the ML engineer
[52:59.170 --> 53:00.550]  or is it actually the data scientist
[53:00.550 --> 53:02.210]  or is it throughout the entire chain?
[53:02.410 --> 53:02.950]  Yep.
[53:04.630 --> 53:05.590]  Yeah, and I mean...
[53:06.370 --> 53:07.830]  Sorry, go ahead. I'm talking too much.
[53:07.830 --> 53:09.970]  I'm seeing over in the chat
[53:09.970 --> 53:12.470]  that we already have awesome ML failures
[53:12.470 --> 53:14.270]  in the AI village as a repo.
[53:14.270 --> 53:19.030]  And Stellar Athena seems to be getting in there already.
[53:19.710 --> 53:20.690]  Outstanding.
[53:20.690 --> 53:22.350]  Suggestions for implementation failures,
[53:22.350 --> 53:25.710]  ethical failures, HCI failures and security failures.
[53:26.010 --> 53:27.730]  You got that Stellar?
[53:28.130 --> 53:28.930]  Yeah.
[53:28.930 --> 53:29.790]  Yeah, I think...
[53:30.250 --> 53:32.170]  Do we have a category for...
[53:32.170 --> 53:33.530]  Because we were playing around with
[53:33.530 --> 53:36.570]  this cat does not exist a couple days ago.
[53:36.570 --> 53:38.270]  I think we maybe need a category.
[53:38.270 --> 53:40.210]  This ML should not exist.
[53:40.370 --> 53:41.810]  Yeah, nightmare fuel.
[53:42.810 --> 53:45.090]  Yeah, well, I mean, I'm thinking of things like...
[53:45.090 --> 53:45.590]  Yeah, you're right.
[53:45.590 --> 53:47.290]  There are a whole bunch of machine learning systems
[53:47.290 --> 53:49.070]  that just should not be on this planet.
[53:49.190 --> 53:49.730]  Yeah.
[53:49.790 --> 53:51.990]  And anyone asked to contribute to it
[53:51.990 --> 53:54.650]  should have been like, perhaps not.
[53:56.370 --> 53:58.810]  Yeah, as far as the responsibility...
[53:58.810 --> 53:59.950]  And we would never have had Facebook
[53:59.950 --> 54:01.790]  if anyone had that call.
[54:01.910 --> 54:02.590]  Anyway.
[54:02.590 --> 54:03.810]  I mean, hindsight.
[54:03.810 --> 54:08.550]  But you can look at really short-term stuff
[54:08.550 --> 54:11.870]  like even just like genderify, right?
[54:11.870 --> 54:13.530]  It was should have been pretty obvious
[54:13.530 --> 54:15.430]  to everyone involved that this was just
[54:15.430 --> 54:18.130]  like a losing proposition from the start.
[54:18.930 --> 54:21.090]  Yeah, and the recidivism through face images
[54:21.090 --> 54:22.350]  stuff from China.
[54:22.350 --> 54:23.070]  Oh, God.
[54:23.070 --> 54:24.470]  Yeah. Okay, so...
[54:24.470 --> 54:26.070]  Way to replicate somebody's bias.
[54:26.070 --> 54:30.590]  So we do have a workshop coming up
[54:30.590 --> 54:31.190]  in a couple of minutes.
[54:31.190 --> 54:32.430]  So I think I want to kind of like
[54:33.470 --> 54:34.550]  put a pin in this.
[54:34.550 --> 54:36.650]  I think if everyone's enjoying this conversation,
[54:36.650 --> 54:39.870]  having an awesome ethics in AI panel,
[54:39.870 --> 54:42.550]  which is coming up in a couple of hours.
[54:42.550 --> 54:44.850]  I'm not sure someone fill me in.
[54:45.030 --> 54:46.510]  Two o'clock, two o'clock.
[54:46.590 --> 54:47.530]  Thank you.
[54:47.530 --> 54:50.170]  So everyone should absolutely turn,
[54:50.170 --> 54:50.890]  tune in for that
[54:50.890 --> 54:52.750]  if they're enjoying this discussion.
[54:53.210 --> 54:55.810]  But yeah, I guess I want to just like go around
[54:55.810 --> 54:57.470]  just to wrap this up
[54:58.130 --> 55:00.290]  and see if we have any like last minute,
[55:00.290 --> 55:02.190]  like big idea takeaways from this paper
[55:02.190 --> 55:04.150]  that people thought were really, really cool.
[55:04.990 --> 55:06.630]  I love the way they did it.
[55:06.650 --> 55:08.470]  The idea of attacking the system
[55:08.470 --> 55:09.850]  and looking at taking it apart
[55:09.850 --> 55:11.250]  is just kind of my thing.
[55:11.250 --> 55:14.850]  And I like the idea of using it to attack systems
[55:14.850 --> 55:17.510]  that shouldn't exist.
[55:18.310 --> 55:20.390]  Yep. Sven?
[55:21.050 --> 55:23.270]  I really like the idea of attacking
[55:23.270 --> 55:24.670]  the feature extractors.
[55:24.670 --> 55:26.710]  And I think we should also add
[55:26.710 --> 55:28.670]  like bypassing the feature extractors
[55:28.670 --> 55:29.670]  by just doing something
[55:29.670 --> 55:34.030]  you know the ML system is going to miss.
[55:35.110 --> 55:37.370]  Because you can understand it well
[55:37.950 --> 55:40.010]  to this whole thing.
[55:41.030 --> 55:42.250]  Cool. Martin?
[55:43.310 --> 55:45.070]  Yeah, that also really enjoyed the paper
[55:45.070 --> 55:45.970]  and discussion.
[55:46.190 --> 55:49.170]  I think to me what was clear
[55:49.170 --> 55:50.670]  is that it can occur at many levels
[55:50.670 --> 55:53.010]  of the basically stack.
[55:53.010 --> 55:55.150]  So just instrumentation and stuff
[55:55.150 --> 55:56.230]  become very, very important
[55:56.230 --> 55:57.390]  to monitoring what happens.
[55:57.390 --> 55:58.570]  And I would really like to see
[55:58.570 --> 56:00.510]  follow on work that applies this
[56:00.510 --> 56:01.870]  to TensorFlow, PyTorch,
[56:01.870 --> 56:02.910]  basically the learning libraries
[56:02.910 --> 56:04.450]  and seeing how you can play
[56:04.450 --> 56:06.310]  with intermediate representations
[56:06.310 --> 56:07.390]  and things like that too.
[56:08.410 --> 56:09.630]  Yeah, for sure.
[56:09.630 --> 56:10.850]  Yeah, the reminder that like
[56:10.850 --> 56:13.570]  these ML things, ML projects
[56:13.570 --> 56:16.430]  don't exist in some abstract theoretical space
[56:16.430 --> 56:17.990]  and are implemented in real code
[56:17.990 --> 56:19.450]  and real systems that come
[56:19.450 --> 56:20.770]  with their own flaws.
[56:21.190 --> 56:22.890]  Yeah, like you know it
[56:22.890 --> 56:24.150]  but being reminded of it
[56:24.150 --> 56:25.170]  sort of viscerally by,
[56:25.170 --> 56:26.350]  look, we have CVEs
[56:26.350 --> 56:28.250]  is always a great thing.
[56:28.530 --> 56:29.970]  So cool.
[56:30.050 --> 56:31.050]  So I think this is
[56:31.050 --> 56:32.550]  we probably want to leave a few minutes
[56:32.550 --> 56:33.970]  for people to fight with the audio
[56:33.970 --> 56:36.150]  as we transition over to the workshop.
[56:36.170 --> 56:37.670]  So I think we want to
[56:37.670 --> 56:40.230]  call that a panel or a discussion.
[56:40.230 --> 56:41.890]  So awesome. Thanks, everyone.
[56:41.890 --> 56:42.330]  Thanks, Rich.
[56:42.330 --> 56:43.170]  One second.
[56:43.450 --> 56:44.910]  Before we jump off,
[56:44.910 --> 56:46.550]  we have one on Wednesday
[56:46.550 --> 56:49.030]  at 5 p.m. Pacific time.
[56:50.510 --> 56:53.310]  And we have the, I'm guessing,
[56:53.310 --> 56:55.310]  I think we should go with your paper, Rich.
[56:56.310 --> 56:57.510]  The slide deck on...
[56:58.770 --> 57:00.470]  Yeah, the anomaly detection
[57:01.030 --> 57:02.030]  outside the closed world
[57:02.030 --> 57:03.010]  on using machine learning
[57:03.010 --> 57:04.290]  for network confusion detection
[57:04.290 --> 57:07.470]  by Robin Subban and Vern Paxson.
[57:08.050 --> 57:11.230]  So we'll post that in the Defcon Discord
[57:11.230 --> 57:13.710]  and also be posted to our Discord and Twitter.
[57:13.710 --> 57:15.470]  So if you want to participate,
[57:15.470 --> 57:17.930]  you should find your way over to our Discord.
[57:17.930 --> 57:21.130]  We can't post a link to that in the Defcon Discord,
[57:21.130 --> 57:22.170]  but the links are places
[57:22.170 --> 57:24.530]  and you can just DM us for an invite.
[57:24.770 --> 57:25.930]  And we'll be discussing that
[57:25.930 --> 57:29.610]  for probably an hour and a half on Wednesday.
[57:29.610 --> 57:31.310]  And we might be able to get one of the authors in,
[57:31.310 --> 57:33.650]  even though this is a 10-year-old paper.
[57:36.050 --> 57:37.750]  How far we have come.
[57:39.130 --> 57:40.010]  All right.
[57:40.010 --> 57:40.730]  Thanks, everyone.
[57:40.730 --> 57:41.390]  Catch you later.