WEBVTT 00:00.820 --> 00:03.180 Okay, say something. 00:05.280 --> 00:09.420 Nope, the audio is definitely broken again. 00:10.220 --> 00:11.680 Now say something. 00:19.120 --> 00:20.920 Time with the audio issues. 00:20.920 --> 00:22.100 There we go. 00:22.180 --> 00:24.040 The audio seems to be working now. 00:24.040 --> 00:25.140 All right, awesome. 00:26.280 --> 00:42.700 Yeah, I almost feel like we got to have... whoever is the poor soul stuck with jockeying this in the future almost needs to have two laptops, one with Twitch and everything going and one with just OBS and Zoom, yeah. 00:43.880 --> 00:45.260 Yeah, that's... 00:45.260 --> 00:49.840 Nothing can confuse OBS's tiny little brain. 00:50.060 --> 00:59.860 But then the problem is when you forget to mute that thing and then this thing and then there's one of your tabs is doing the wrong noise and you're like, oh, this is wrong. 01:00.220 --> 01:00.920 Yep. 01:00.920 --> 01:02.720 Sven, you're being heroic. 01:02.760 --> 01:03.300 It's just... 01:04.020 --> 01:05.960 Yep, we are live, by the way. 01:05.960 --> 01:09.880 The internet is listening to us talk, so enjoying the banter. 01:09.920 --> 01:11.240 You're still being heroic. 01:11.420 --> 01:12.200 Yes. 01:12.420 --> 01:23.120 Yeah, so for those of you watching, Sven has been the main guy troubleshooting all of the audio issues and fighting with the stream for basically all of DEF CON. 01:23.120 --> 01:29.860 And it's a testament to his hard work that things have gone as smoothly as they have. 01:29.860 --> 01:35.060 So you've made it a lot easier that things were pre-recorded. 01:35.060 --> 01:42.840 Can you imagine trying to hunt people down, hunt speakers down, and then their sharing is not quite right and all that? 01:43.400 --> 01:45.420 Yeah, no, I mean... 01:45.420 --> 01:47.880 I mean, this year, at least they're not passed out in the bar. 01:49.660 --> 01:52.260 How many speakers have been passed out in the bar? 01:53.360 --> 01:57.720 None of ours that I know of, but we've definitely hooked a few of them out from the nearest bar. 01:58.240 --> 01:59.600 Yeah, well, you know. 02:04.250 --> 02:08.810 Are you expecting anyone else or we want to get this show on the road? 02:09.130 --> 02:16.930 Yeah, let me... well, let me pull up this thing for myself. 02:18.090 --> 02:24.970 I'm going to try a new layout as soon as I can find the thing. 02:25.210 --> 02:31.210 So while I'm looking for it, Rich, do you want to introduce the paper to everyone? 02:33.470 --> 02:34.670 Oh, sorry. 02:34.670 --> 02:36.270 So we're going to do a couple of things. 02:36.290 --> 02:37.430 Hi, Barton. 02:37.510 --> 02:40.810 So we're going to start with just the four of us talking about it for a bit. 02:40.810 --> 02:42.410 And then I'm going to open up the... 02:43.770 --> 02:48.710 and join the voice chat in the Defcon Discord. 02:48.710 --> 02:52.310 And so you guys will be able to talk on stream and talk with us and everything. 02:53.870 --> 02:56.950 So for the first, like, 20 minutes, it's going to be just us. 02:56.950 --> 03:02.070 And then we'll bring everyone we can in and have a bit more of a Wild West thing. 03:02.070 --> 03:03.790 We're going to try out how that works. 03:04.430 --> 03:07.530 And the other thing is I'm just going to play with the layouts a little bit. 03:07.750 --> 03:10.750 So that's the structure of it. 03:10.750 --> 03:13.670 We've also got a journal club coming up on this Wednesday. 03:14.370 --> 03:17.770 We will put the link to that paper in the Defcon Discord. 03:17.770 --> 03:21.190 And then if you want to participate, you should join our Discord. 03:22.810 --> 03:24.370 Link in the Twitter bio. 03:24.810 --> 03:26.410 Yeah, it's in the Twitter bio. 03:26.410 --> 03:28.690 And you can just DM one of us. 03:28.690 --> 03:32.730 And we'll send you an invite to the AI Village Discord. 03:32.730 --> 03:36.050 Please don't use the AI Village Discord while Defcon is going on. 03:36.050 --> 03:41.770 Please be respectful of the hard work they have put into building their Discord and getting it running. 03:41.970 --> 03:45.330 And only use it, like, tomorrow or Tuesday. 03:46.730 --> 03:48.910 So take it away, Rich. 03:49.490 --> 03:50.530 Okay. 03:50.590 --> 03:51.070 Cool. 03:51.070 --> 03:54.690 So this is actually a slightly older paper. 03:54.690 --> 03:56.950 It's from 2017. 03:58.010 --> 04:06.890 It is called Summoning Demons, the Pursuit of Exploitable Bugs in Machine Learning, which is probably one of the coolest paper titles I've come across in a while. 04:07.310 --> 04:12.310 And it's by Rock Stevens and Tudor Dimitrescu's group. 04:12.310 --> 04:22.050 So the basic idea of the paper, it's kind of an interesting spin on the adversarial ML papers that I think started coming out about the same time. 04:22.050 --> 04:32.130 And in this, rather than actually, like, using the ML model itself to sort of guide inputs, they're looking for more... 04:32.770 --> 04:37.930 they're more looking for flaws in how the different components of the system hook up. 04:37.930 --> 04:54.370 And to do that, they actually crack out an old classic American fuzzy lump, the fuzzer, and they re-instrument a lot of it so that weird actions from the ML components of the system register as crashes to AFL. 04:54.570 --> 05:02.530 And so that lets them essentially do sort of like a black box adversarial attack, if you want to think of it that way, against the ML system. 05:02.570 --> 05:13.950 And what they find is that in addition to sort of the usual stuff you can get, where you have, like, inputs that look like faces that don't register as faces, because they've had some pixels perturbed, they also run into, like, image format parsing errors, 05:13.950 --> 05:21.730 where a component of the ML system won't parse, for example, a face correctly, completely fails to find the face. 05:21.730 --> 05:25.830 But if you open it up in other programs, it actually looks good. 05:25.990 --> 05:39.330 So I think this is kind of in the vein of the same sort of things that Ariel Herbert-Boss talks about a lot, where it's not just attacking the ML system itself that's important. 05:39.330 --> 05:48.830 You've got all of these other components in the pipeline, from ingesting the actual image all the way through feature extraction, pre-processing, and so on. 05:48.830 --> 05:51.650 And each of those is actually a point that you can attack. 05:51.650 --> 06:04.510 And so in this paper, they're basically using fuzzing as a way to find some of those, you know, broken assumptions or breakdowns in the link from one stage of processing to the next. 06:05.350 --> 06:08.590 And I thought it was a super clever paper with a really good name. 06:08.590 --> 06:11.250 And they dig a couple of CVs out of it, too. 06:11.250 --> 06:23.510 So super cool application of sort of more classic security methodology to what, from my perspective, is kind of an adversarial ML problem. 06:26.210 --> 06:30.530 So that's my, I guess, like two-minute summary of the paper. 06:35.440 --> 06:38.140 So I guess maybe to kick off conversation, 06:41.340 --> 06:55.180 so does this, do you feel like, does anyone have an opinion, strong opinion one way or the other, if this falls into the same sort of general field of adversarial ML as sort of like gradient-based attacks? 06:55.180 --> 07:14.520 Like I'm trying to think of, the name escapes me off the top of my head, like fast gradient sign method, or like these Jacobian-based methods, or saliency-based methods, or maybe even like, you could almost argue it's like a genetic algorithm-based attack against a particular ML pipeline. 07:14.580 --> 07:18.420 Well, to be honest, I thought that adversarial ML covered all this already. 07:18.420 --> 07:20.540 I was a bit surprised that it was new. 07:21.800 --> 07:36.440 Yeah, it's genetic algorithm-based because they're using FuzzyLock, which is genetic algorithm-based, which is also interesting because there were network extensions to genetic algorithms that might be more appropriate if you're searching spaces. 07:36.980 --> 07:40.100 So there's some interesting stuff on the front that could be done. 07:40.100 --> 07:41.980 I kept wanting to say, what happened next? 07:50.890 --> 08:10.710 Likewise, in the context of healthcare, practical realities of securing systems, it seems like a lot more emphasis right now is placed on more theoretical attacks, where it's more so once you have uninhibited access to being able to query this API that's being explored. 08:10.710 --> 08:23.450 So it's nice to see a paper like this that's a bit more in an all-systems set of things, because at least from healthcare IT, it seems like it's a more likely encounter that you'll find in the industry. 08:24.670 --> 08:33.090 Yeah, one thing that I did really like was their focus on implementations rather than just theoretical properties, right? 08:33.090 --> 08:42.710 Because in some sense, something like the FastGradientSign method would apply to almost any of these models. 08:42.710 --> 08:48.170 But here, they're sort of like, okay, this model was instantiated in this way and is doing these things. 08:48.170 --> 08:52.850 And now how can we attack sort of that specific version of the model? 08:53.450 --> 09:02.890 Because it's a lot less academic in some ways, and a lot of the adversarial ML research seems to be very academic still. 09:03.150 --> 09:06.190 This was like down and dirty in how do we attack this thing. 09:06.310 --> 09:09.150 It's kind of in my vein of stuff. 09:09.270 --> 09:13.810 Just look at the system, work out how to break the system, and take it to pieces. 09:14.050 --> 09:25.070 I loved the way that they hacked that little hack on FuzzyLop, where they went, okay, we're going to take out the things that actually broke, and then we're going to induce breaks for the things we care about. 09:25.490 --> 09:27.430 Yeah, changing the target. 09:27.430 --> 09:28.570 Yeah, changing the target. 09:29.290 --> 09:31.250 It was just beautifully done. 09:31.650 --> 09:45.510 I thought it would be useful for defense as well, because we're also looking now at things like Strafe's work, so Brittany Prostakoff's work on attacking with robots. 09:45.990 --> 09:50.290 What could we do to screw up their vision systems? 09:51.210 --> 09:53.710 So how could we mess that up? 09:53.710 --> 09:55.850 There's some beautiful adversarial there too. 09:56.910 --> 09:58.870 They can't see, they can't move. 09:59.190 --> 09:59.750 Yeah. 10:00.470 --> 10:03.650 Yeah, so actually, I thought table one in this was... 10:04.490 --> 10:06.130 I don't know, I'm surprised. 10:06.830 --> 10:08.270 Flips to table one. 10:08.270 --> 10:09.010 Flips to table one, yeah. 10:09.010 --> 10:10.510 Waves to table one. 10:11.170 --> 10:13.190 It's, I think it's on page four. 10:13.390 --> 10:14.310 Yeah, yeah. 10:14.310 --> 10:15.270 Yeah, page four. 10:15.330 --> 10:20.330 Yeah, they actually like, I feel like, again, this is very much like Ariel's jam, right? 10:20.330 --> 10:23.650 They go over sort of the attack surface and the kinds of things you can do to it. 10:23.650 --> 10:28.130 And it reminds me a little bit of the contest there was... 10:28.130 --> 10:35.670 So I think it was a late entry to the contest last year that was announced at the AI Village where they were trying to do ML evasions. 10:35.830 --> 10:47.650 And one guy actually found essentially a bunch of ways to just completely trash the feature extraction so that the ML prediction just wouldn't run on the samples he uploaded. 10:48.550 --> 10:54.310 And so, yeah, he was a little upset that that didn't count as a win, which I think is justifiable, right? 10:54.310 --> 10:57.290 Because if you can just like stop the analysis cold, right? 10:57.470 --> 10:59.530 That's almost as good as evading it. 10:59.530 --> 11:00.230 But yeah. 11:01.170 --> 11:05.610 It depends what you're trying to do if you kind of sneak into a system and it's a bunch of systems, right? 11:05.610 --> 11:07.830 We just want to, you know, the usual thing. 11:07.830 --> 11:10.930 Get in, do your thing, get out without no one noticing. 11:12.090 --> 11:14.850 And we see enough of that in real life of the stuff. 11:14.850 --> 11:17.350 It's like, oh crap, we didn't realize that was a problem. 11:18.070 --> 11:19.770 Before someone else adds them. 11:21.050 --> 11:23.650 There's also another thing to that. 11:23.770 --> 11:33.350 Like if you wanted to attack the Ember model with the Ember dataset, it only looks at your headers and you could just change other parts. 11:33.350 --> 11:36.570 Like don't even think about it in terms of adversarial ML. 11:36.570 --> 11:48.190 Just copy like a notepad.exe's header and just put your own entry point function in and start launching weird stuff from there. 11:48.190 --> 11:51.870 And that's a different way to bypass the thing that is also not... 11:52.810 --> 11:54.670 Academics also not aware of. 11:56.330 --> 11:58.110 Yeah, so that's sort of like... 11:58.110 --> 12:03.010 I guess that's maybe like a... 12:03.010 --> 12:04.910 You can classify that if you're gonna... 12:04.910 --> 12:05.910 I'm gonna screw this up. 12:05.910 --> 12:10.110 So if I'm getting the terminology wrong, security people, please like yell at me. 12:10.110 --> 12:20.570 But it's almost sort of like a logic attack on the feature extraction rather than what they're doing here, which seems to be more going for crashes, right? 12:20.570 --> 12:31.830 If you know how the feature extraction process works, then you can sort of write your way around it deliberately rather than just throwing shit at it until something blows up. 12:32.930 --> 12:35.490 Some of it's blowing it up, some of it's sneaking in the back. 12:35.670 --> 12:39.630 Some of it's like sneaking in the back conventionally depending on your system. 12:40.770 --> 12:46.710 The other thing that interests me is that only some of these became CVEs. 12:46.710 --> 12:53.830 That the response to the people who are responsible for the algorithms for some of this was like, won't fix and who cares? 12:54.330 --> 12:57.990 And that's part of our journey, I guess. 12:58.370 --> 13:09.630 It is just like every other security thing is trying to get people to accept that yes, you may have to drop a little bit of responsiveness in your system. 13:10.110 --> 13:13.410 But you'll get it more secure and it matters. 13:14.570 --> 13:26.670 Yeah, I'm kind of reminded a little bit of some of the argument that went around for the proof point CVE, where people were arguing whether or not that actually counted as like an information leak CVE. 13:26.670 --> 13:36.510 Because so if people aren't familiar, the short version is the people who did it, Will Pierce and someone whose name I'm forgetting, I'm sorry. 13:37.040 --> 13:45.030 They found a way to essentially like send an email through proof points, spam detection service that would then bounce back to them. 13:45.030 --> 13:49.650 And in the header of the email, it had a score for how spammy it looked. 13:50.110 --> 13:53.750 And so that essentially was the information leak. 13:53.750 --> 13:59.270 The fact that they could get that score back and that was enough for them to rebuild essentially a proxy model for it. 13:59.750 --> 14:12.790 And yeah, over on in various like AI village discussion channels, we had very strong debates over whether or not that actually should have counted as like an information leak CVE. 14:12.830 --> 14:17.890 Because, you know, it's in a lot of cases, it's useful contextual information. 14:17.890 --> 14:22.950 But when you look at it in an ML context, you're actually enabling like a model stealing attack. 14:23.590 --> 14:30.110 Which again, it's one of these, we're kind of in new territory as far as the security of these ML based systems goes. 14:30.130 --> 14:47.630 And a lot of people that are more familiar with conventional security, maybe don't quite get what these various information leaks or attack surfaces could do once you wrap it in like how ML based systems work. 14:47.790 --> 14:56.750 And I think another thing we get out of this is by doing this sort of attack, we start showing where the things that break up. 14:56.750 --> 15:01.590 And one of the things I'd love about the fact that MLSec exists is it makes machine learning better. 15:02.530 --> 15:05.650 Just by dint of the spotlight we throw on it. 15:05.650 --> 15:09.370 You chuck hackers at it, they take it to pieces, they show you where it's broken. 15:09.990 --> 15:14.290 And hell, we spend most of our life trying to get our damn systems to work. 15:14.450 --> 15:16.210 But I'm going to give a little bit of history. 15:16.210 --> 15:17.170 I'm old. 15:17.510 --> 15:20.590 Jump in for just a second. 15:20.630 --> 15:22.470 I just saw in the Discord chat. 15:22.470 --> 15:27.170 So it was Will Pierce and Nick Landers who got the Proofpoint CVE. 15:27.170 --> 15:30.010 So I'm very sorry that I forgot your name, Nick. 15:30.010 --> 15:31.410 Okay, sorry, carry on. 15:31.590 --> 15:36.870 I was just going to say about sort of some things, the findings, that it's double float precision thing. 15:37.250 --> 15:44.710 So way back before we had, I think we had Python, but we sure as hell didn't have scikit-learn and friends. 15:44.710 --> 15:46.590 We had matplotlib. 15:46.650 --> 15:48.270 I'm not... matmap. 15:48.390 --> 15:51.670 We use MATLAB for a hell of a lot of machine learning stuff. 15:51.910 --> 15:54.770 And there was a neural network kit in the MATLAB. 15:55.010 --> 15:59.170 And that neural network kit suffered horrendously from underflow errors. 15:59.690 --> 16:02.090 You could break it really, really, really easily. 16:02.090 --> 16:03.950 So Aston University built its own. 16:04.510 --> 16:08.170 They built a separate kit because they couldn't get the first one fixed. 16:08.810 --> 16:11.910 I mean, this sort of stuff, like the float to double errors. 16:12.890 --> 16:25.950 I mean, crossing my fingers that actually all of those hidden faults that are probably still going on with neural network stuff and deep learning, we're not even noticing because we just trust the systems. 16:25.950 --> 16:26.890 Yeah. 16:26.890 --> 16:29.670 I mean, even in Python, right? 16:29.670 --> 16:43.070 Like under the hood, a ton of these neural network libraries, and PyTorch, at least, sklearn as well, use pickle, which is just broken kind of by default, right? 16:43.070 --> 16:46.310 It's a stack machine that you can write arbitrary code for. 16:46.570 --> 16:53.090 And yeah, people still share wait files that are essentially pickles. 16:53.090 --> 16:56.850 And so you're downloading this off the internet and running code. 16:56.850 --> 16:57.690 And I keep threatening. 16:57.690 --> 17:11.470 One of these days, I'm actually going to write a pickle poisoner that just replaces calls to the relu function with some sort of stochastic function so that half the time it works and half the time it gives you garbage. 17:12.030 --> 17:14.670 Comment from Discord. 17:14.670 --> 17:16.510 Yeah , numpy as well uses pickles. 17:16.510 --> 17:17.810 So it's everywhere. 17:17.810 --> 17:18.890 Keras, yeah. 17:18.890 --> 17:22.710 Is that what the numpy save file is? 17:24.330 --> 17:26.050 I thought that was secure. 17:26.350 --> 17:27.570 I'm not sure. 17:27.570 --> 17:29.130 I know, I'm pretty sure it has one... 17:30.570 --> 17:31.970 One pickle thing in there. 17:32.030 --> 17:33.110 One or two. 17:33.110 --> 17:37.890 It's got one or two formats that are not. 17:37.890 --> 17:39.390 Apparently, there's a couple that... 17:39.390 --> 17:44.210 The scikit-learn form that these guys found, numpy was part of that. 17:44.210 --> 17:45.490 I'm just flicking to the page. 17:45.490 --> 17:47.990 Page seven at the bottom, if you're reading along with us. 17:49.210 --> 17:56.650 One interesting thing is that they weren't able to induce misclassification or from the precision loss problem. 17:56.650 --> 18:00.610 So they tried different values of epsilon, and basically said that it's possible in theory. 18:00.610 --> 18:09.330 But it's interesting to consider how would you model that epsilon based on the syntax to induce this misclassification. 18:15.120 --> 18:16.840 How much of this is induced by... 18:18.440 --> 18:25.560 Basically, all the machine learning code is written by academics and over-enthusiastic academics. 18:25.560 --> 18:26.200 Guilty. 18:27.860 --> 18:29.540 Who aren't experienced in... 18:30.820 --> 18:35.180 It's incredibly hard to write numerically efficient code. 18:35.180 --> 18:43.440 You don't have time to really learn how to multiply matrices incredibly efficiently and make sure you open files correctly. 18:43.960 --> 18:52.260 So we've chosen a lot of lazy paths to formatting and opening things versus secure paths. 18:53.720 --> 18:56.260 And also, you've got tiny numbers and people don't... 18:56.260 --> 19:00.640 I mean, you go wading on in, build your stuff, and don't think about how those numbers interact. 19:00.640 --> 19:03.260 You just assume the system is going to magically deal with it. 19:03.900 --> 19:04.220 Yeah. 19:04.220 --> 19:12.200 And I mean, there's also the fact that writing secure code is very much its own skill set, right? 19:14.440 --> 19:21.040 Some people are writing research code because they don't expect it to be used in Angular, they don't expect it to be used in a production context. 19:21.040 --> 19:26.160 Some people are writing numerically efficient code because that's their jam and God bless them. 19:26.900 --> 19:38.120 And then you've got sort of security people who are probably gazing at a lot of this with sort of a feeling of vague horror that they're actually going into production. 19:39.180 --> 19:55.260 Yeah, I know even in my day job, right, a lot of what we've been doing recently is sort of going back and rethinking how all of this stuff is put together so that we can feel confident in actually pushing out products containing it. 19:56.120 --> 20:04.060 There's a middle ground where you can enable basically more flexible code, but also execute it in a way that perhaps minimizes the blast radius. 20:04.140 --> 20:10.420 I don't mean put it in a container, obviously, but it's something more to the point of running code in secure enclaves. 20:10.420 --> 20:19.980 For instance, the work with silo type of workflows or do you see anything where you try to mitigate the impacts of this code? 20:20.560 --> 20:22.060 And could we have code checkers? 20:22.060 --> 20:26.100 I mean, literally just some way, some sort of test data sets. 20:26.180 --> 20:27.740 But we know this stuff is bad. 20:27.740 --> 20:29.360 Run it through your system. 20:30.020 --> 20:30.560 Yeah. 20:31.660 --> 20:43.120 Yeah, I mean, I had a great amount of fun, again, at my day job, actually like taking our feature abstraction code and fuzzing it just to see what would happen with it. 20:43.120 --> 20:51.580 Fortunately, it turned out that I spent more time than I care to think of beating on it, and it turned out fine. 20:51.580 --> 20:54.840 But yeah, if it hadn't, what would we have done then, right? 20:55.660 --> 21:06.700 And I think the idea of building test vectors of just really bizarre, but technically conformant files for this sort of thing is an awesome idea. 21:07.820 --> 21:18.940 But extending that to the actual data science part, how often have you issued a PR about some data science thing and sent it to someone and they're gone? 21:18.940 --> 21:20.540 It looks good to me. 21:22.360 --> 21:32.160 But there's some stuff that you can check easily, but a lot of the code that goes into BERT is incredibly complicated. 21:32.520 --> 21:34.180 Very hard to check. 21:34.280 --> 21:40.480 When you make a small change, you could have broken things fundamentally because it's software. 21:41.720 --> 21:53.380 How do you write a unit test for I broke BERT when you input this one slight thing and now it causes this whole class of things to be false positives, false negatives? 21:55.680 --> 21:58.140 Like, how do you unit test machine learning? 21:58.580 --> 22:01.740 There's a... I mean, you can do... 22:01.740 --> 22:04.600 I mean, essentially, you do test vectors, right? 22:04.800 --> 22:07.980 You say, you know, if I load... 22:08.660 --> 22:11.340 You can consider the weights are obviously part of the model. 22:11.340 --> 22:12.520 You say, I've got this set of weights. 22:12.520 --> 22:23.580 If I load in, you know, this set of features, I should get this exact set of outputs up to numerical precision, which, again, numerical precision is kind of its own nightmare. 22:25.020 --> 22:29.660 Yeah, but then if you change the weights, right, then kind of all bets are off. 22:29.660 --> 22:44.040 And that's one of the painful things about debugging ML, even when you're not under adversarial conditions, is just, I retrained the model, do I still trust it? 22:44.180 --> 22:48.440 I mean, there's a difference between I messed up my models, which is like, we do this all the time. 22:48.440 --> 22:55.360 And somebody is actively trying to attack my models or may actively try to attack my models using known vulnerabilities. 22:55.860 --> 22:59.480 Because sure as hell, half the systems out there haven't patched any of this. 23:00.060 --> 23:06.420 How many machine learning developers even know unit testing or even know good software practices? 23:06.420 --> 23:08.580 Like, a lot of us came from academia. 23:08.580 --> 23:11.300 I have a PhD in math, not software engineering. 23:11.740 --> 23:17.520 So I have a philosophical question I want to pose to the panel from Discord. 23:18.080 --> 23:21.580 And Yaga asks, what is a terror and... 23:21.580 --> 23:23.560 Sorry, what's a terror? 23:23.560 --> 23:25.420 This is terror right here. 23:25.420 --> 23:28.100 What is an error and what's an attack? 23:28.100 --> 23:32.980 And like, is there any meaningful distinction between the two? 23:32.980 --> 23:35.100 I guess I would follow up. 23:36.080 --> 23:43.200 I think if you're looking at adversarial images, I mean, the fact you have like the little image patches to screw up your system is an attack. 23:43.280 --> 23:44.440 It's not an error. 23:44.700 --> 23:48.620 You just don't randomly get sort of extra patches on a stop sign. 23:49.280 --> 23:51.960 No matter how much undergrowth you got on it. 23:52.140 --> 23:54.280 I think the difference is intention. 23:55.540 --> 23:57.840 Yeah, I mean, that's the same thing in security, right? 23:57.840 --> 24:02.660 Bugs can be... I mean, exploits are essentially software bugs, right? 24:02.980 --> 24:05.300 Maybe that's being too general, but if you... 24:06.140 --> 24:15.060 You know, a subset of software bugs or exploits and you use them to make the system do things you don't want it, it shouldn't be doing under normal circumstances. 24:16.400 --> 24:18.360 Or wasn't intended to be doing. 24:18.360 --> 24:18.880 Right. 24:18.880 --> 24:20.380 And sometimes they're useful. 24:21.740 --> 24:35.640 Would you consider a well-informed person building an attack against a machine learning system where they understand the machine learning system, they understand the feature extract and all that stuff, and they've built a false positive. 24:36.460 --> 24:41.180 Is that an adversarial attack against the machine learning or is that like an exploit? 24:42.120 --> 24:44.700 Like, where would you classify the mistake? 24:44.700 --> 24:50.020 Is it like a software mistake or a exploit or what? 24:51.580 --> 24:54.240 I think that's the tricky bit about ML models, right? 24:54.240 --> 24:57.180 I mean, they're inherently statistical. 24:59.620 --> 25:03.140 So it kind of gets back to testing after you retrain it, right? 25:03.140 --> 25:18.320 So if you have a single example and it turns out to be a false positive, and maybe that's one of the bridges that I feel like as a data scientist, I have to cross a lot when I'm talking to security specialists. 25:18.320 --> 25:25.380 If I have a model and it FPs on a single file, that doesn't call into question the efficacy of the model, right? 25:25.380 --> 25:28.560 You have to look at the statistics of what the model does, right? 25:28.560 --> 25:41.560 And if you say, well, yeah, it got this FP and that was a pretty boneheaded FP, but for 99% detection, you get one FP in 10,000 negative samples. 25:41.720 --> 25:43.380 That's a pretty good model. 25:43.520 --> 25:55.360 And so being able to sort of drive that switch from, hey, here's the good, bad to, hey, you have to think of this sort of heuristically and statistically. 25:55.980 --> 26:01.140 Is a big part of just like communicating as a data scientist, I think. 26:02.360 --> 26:03.940 Go ahead, Sarah. 26:03.940 --> 26:09.840 Yeah, if you've got somebody actively attacking, then maybe some pre-processing is enough to cut down the attacks. 26:09.840 --> 26:11.540 Yeah, yeah, for sure. 26:12.020 --> 26:15.860 All of that goes out the window when you've got someone maliciously trying to... 26:17.280 --> 26:34.980 Yeah, but so my question is, so if you have like a fancy ML from an academic paper bypass of your model, versus someone trying things, what would be... 26:37.100 --> 26:43.360 When is it like a machine learning mistake or versus a mistake in your featurizer and stuff? 26:44.040 --> 26:51.640 If you look at the decision boundary, basically for something and seeing how the perturbations relate to that? 26:51.900 --> 27:12.640 Yeah, I kind of feel like maybe to pull it back to this paper a little bit, if it gets through your feature extraction process in one shape, in one piece, then maybe it's a modeling issue. 27:12.640 --> 27:24.680 And if it crashes your feature extraction process or it gives you something that you wouldn't expect from your feature extraction process, then it's a featurization bug, right? 27:25.140 --> 27:30.740 If you feed garbage into a model, you can't blame the model for producing garbage outputs. 27:31.660 --> 27:33.020 How would you basically... 27:33.020 --> 27:36.360 What are some methods for quantifying unexpectedness in this scenario? 27:36.360 --> 27:40.660 Is it more like error counts or latencies in response? 27:40.660 --> 27:46.040 What would you pay attention to if you were trying to understand if something was misbehaving? 27:47.960 --> 27:49.720 You know what, when you see it. 27:49.820 --> 27:52.240 Yeah, like if you take a look in the paper, right? 27:52.240 --> 27:53.520 They've got an example. 27:53.520 --> 27:54.780 I want to find it. 27:55.440 --> 27:56.620 Figure two, right? 27:56.620 --> 27:59.740 So they show an example of two different images. 27:59.840 --> 28:07.320 One is an image of sort of the very top of a person's head and then the rest of it is gray. 28:07.580 --> 28:11.040 And one is a picture of a person's head. 28:11.040 --> 28:11.660 Yes, there you go. 28:11.660 --> 28:12.340 Thank you. 28:13.480 --> 28:15.340 Page six, if you're following along. 28:15.340 --> 28:26.740 Right, so OpenCV has a bug that these guys found by their guided fuzzing technique, which means that OpenCV can't properly load the file and then you run it... 28:26.740 --> 28:30.300 Obviously, you run it through feature extraction and it doesn't get anywhere. 28:30.300 --> 28:34.980 So it's like all these different individual components that you got to worry about, right? 28:36.300 --> 28:38.940 And I think on that topic, there was a... 28:38.940 --> 28:41.500 From Hacker Factor in Discord, he's got a question. 28:41.500 --> 28:46.740 Are you making a distinction between the ML model versus software that drives the ML system? 28:46.880 --> 28:49.020 He says fundamental error versus algorithmic. 28:49.020 --> 28:51.240 And I think here, right? 28:51.240 --> 28:53.900 This is what this paper is going after, right? 28:53.900 --> 29:00.600 It's saying the software that drives the ML system, the implementation of the system also has bugs and they find them here. 29:00.600 --> 29:04.840 And , you know, if you go to table two, right? 29:04.840 --> 29:09.020 They've got in OpenCV, they've got like two heap corruptions. 29:09.020 --> 29:11.180 They've got a weird rendering bug that they found. 29:11.180 --> 29:13.980 Page six, thank you, yeah. 29:14.880 --> 29:16.700 Right, so all these different... 29:16.700 --> 29:21.760 Like they literally have like heap corruption bugs that they're discovering with these funky inputs. 29:22.380 --> 29:24.440 And that has nothing to do with the model. 29:24.440 --> 29:28.220 That's you're attacking the plumbing around the model, which is... 29:28.220 --> 29:31.660 But you can use that plumbing to go and tweak the model. 29:32.480 --> 29:33.700 Right, exactly, yeah. 29:33.700 --> 29:36.740 So that's what the third line down does. 29:36.740 --> 29:39.320 That's the figure one, figure two? 29:39.320 --> 29:40.680 I forget, figure two. 29:40.680 --> 29:51.720 Yeah, so you break the rendering and then the rendering goes to the feature extraction and the feature extraction can't get anything because it's just got a bunch of gray pixels to look at. 29:51.720 --> 29:53.160 And then the model, right? 29:53.160 --> 29:54.040 Garbage in, garbage out. 29:54.040 --> 29:56.780 The model can't do anything because it doesn't have good features. 29:57.860 --> 30:01.260 So I'm going to try to open this up to Discord. 30:01.740 --> 30:09.140 If you are in the DevCon Discord, go to airvillage-general-voice. 30:09.760 --> 30:14.960 We'll try unmute you and get you so you can talk. 30:16.180 --> 30:22.080 If you're human plus, that's the correct way to do it, but we have a workaround. 30:22.480 --> 30:27.220 So head over to airvillage-general-voice to ask any questions. 30:27.360 --> 30:32.960 If you can't, we'll still be paying attention to the text chat in airvillage-journal-text. 30:35.900 --> 30:37.940 The moment of truth. 30:39.940 --> 30:42.260 I'm watching journal club text. 30:42.360 --> 30:43.980 Yeah, me too. 30:45.220 --> 30:48.080 Audio, managing audio is... 30:48.080 --> 30:50.280 I'm so glad I don't have to deal with it. 30:57.170 --> 31:05.690 I wish I had seen this paper before I started, like, really trying to solve adversarial machine learning as a security problem. 31:05.890 --> 31:07.340 Because there's a lot... 31:07.860 --> 31:13.540 this seems a lot easier, like, to attack for a lot of attack people. 31:14.240 --> 31:22.400 I think a lot of attackers have more knowledge on how to break a parser than how to break a neural network. 31:23.880 --> 31:32.640 So this, to me, feels like more of a realistic threat model of, like, your pipeline was... they broke your pipeline. 31:32.640 --> 31:34.400 You did something wrong in AWS. 31:34.400 --> 31:38.940 You did something wrong with your container setup. 31:38.940 --> 31:42.200 And it's parsing your images incorrectly now. 31:44.140 --> 31:52.100 Yeah, I mean, that's tons of the bugs that you find in sort of real systems, right, as parsing bugs. 31:52.100 --> 32:07.380 And yeah, a lot of the research right now focuses less on that and more on sort of these theoretical properties of machine learning models, which, I mean, I don't want to dig on those. 32:07.380 --> 32:08.200 They're fascinating, right? 32:08.200 --> 32:09.400 I love these papers. 32:09.400 --> 32:13.320 But they're also, again, keep coming back to this paper. 32:13.320 --> 32:20.420 This paper makes the really good point that these sort of theoretical models that have these nice properties are embedded in real world systems. 32:20.420 --> 32:26.680 And we know that real world systems are always kind of full of bugs that can be attacked and exploited. 32:27.380 --> 32:32.040 And so it gives you sort of multiple entries into the problem, right? 32:32.040 --> 32:42.360 On the one hand, you can exploit the preprocessor, and maybe that gives you some sort of heap corruption or some sort of like direct, you know, denial of service or even code execution. 32:42.500 --> 32:48.720 And on the other hand, you can break the feature extraction enough so that everything downstream is getting garbage data. 32:50.160 --> 32:51.600 Or mildly corrupted. 32:51.600 --> 32:57.480 I mean, I say again, with like the sneaking in, you want to do low and slows on some of this. 32:59.000 --> 33:02.520 Again, I am really like the idea of using this as defense. 33:02.520 --> 33:09.120 I mean, there's a whole bunch of applications where the bad guys are using machine learning based systems. 33:09.520 --> 33:12.020 I like the idea of breaking their systems. 33:12.160 --> 33:14.060 Yeah, as I would. 33:14.060 --> 33:15.200 Great hats on. 33:16.780 --> 33:26.170 Yeah, just like yesterday's paper with the Forbes masks. 33:27.390 --> 33:28.290 Mm-hmm. 33:28.690 --> 33:31.730 Yeah, so Will has a comment. 33:31.730 --> 33:34.350 He says, I'm surprised a paper is the vehicle for this knowledge. 33:34.350 --> 33:36.710 There's a whole security industry that has advice for this stuff. 33:36.710 --> 33:47.670 And I think, yeah, this is part of what we're trying to do, right, with AI Village is draw a link between security and machine learning, right? 33:47.670 --> 33:56.690 So we have a lot of like wicked smart security people who know about all about these implementation bugs. 33:56.690 --> 34:06.350 And then we've got a bunch of like really smart academics and machine learning people who know about like sort of these, you know, feature based attacks or weird properties of ML systems. 34:06.350 --> 34:15.270 And getting sort of that knowledge transfer going on so that we can find out where those intersect and how those two, you know, sort of sets of attacks inform each other. 34:15.270 --> 34:20.270 I think that's like a really rich area for future development. 34:20.270 --> 34:37.650 And it kind of makes me sad a little bit that this paper came out in 2017 and got comparatively little attention when it really is going right to the heart of a lot of really big questions in this, you know, in ML slash security. 34:37.650 --> 34:46.090 So in practice, you see a lot of data science teams disconnected from both like infrastructure application network security sort of teams. 34:46.090 --> 34:56.070 And so like the whole MLOps sort of formation of teams that can interface between data scientists and implementation has been emerging. 34:56.070 --> 35:02.250 But I'm curious what the panel thinks as far as what's the appropriate like role, a person, a cross-functional team? 35:02.250 --> 35:07.170 Do we want to have security people sitting with data scientists as these things are being developed? 35:08.270 --> 35:08.730 And... 35:08.730 --> 35:16.730 Yeah, I mean, weirdly enough, I used to be one of the people going into large companies working out where to put the data scientists. 35:16.750 --> 35:22.770 And there was always this argument between you want to embed the data scientists out into the rest of the team. 35:23.310 --> 35:26.550 And have them work with because you're informing the rest of the team everywhere. 35:27.710 --> 35:29.870 But then you've got data scientists out on their own. 35:29.870 --> 35:35.490 And we're kind of, you know, you see in this paper that we get ignored quite a bit when we're on our own. 35:35.670 --> 35:41.290 Or having these unicorn pens full of data scientists who talk to each other, but not really to anybody else. 35:41.810 --> 35:46.790 And there's this sense of you put people out, but you make sure they're still part of TRIPE. 35:46.790 --> 35:51.570 So you literally build TRIPE or whatever the good word now is for that. 35:51.570 --> 35:53.650 It used to be called TRIPE, but cross... 35:53.650 --> 35:54.510 Village? 35:54.610 --> 35:55.850 Cross, yeah, village. 35:56.450 --> 36:04.330 So you build villages for the data scientists to keep connected to each other, but they need to be out in the rest of the teams. 36:04.770 --> 36:08.270 Because otherwise, you know, our work is everywhere. 36:08.290 --> 36:09.630 We should be everywhere too. 36:10.350 --> 36:11.970 That's strongly our belief though. 36:12.510 --> 36:15.690 Yeah, I mean, I think it's, but it's got to be sort of a two-way street, right? 36:15.690 --> 36:32.210 On the one hand, yeah, there is kind of, for data scientists and sort of ML practitioners, I think that, yeah, we do sort of tend to stay in our own little bubble and don't think about these sort of externalities. 36:32.210 --> 36:38.190 I don't know what the right word for it is, but like these other ways, you know, it's not just going to be like fast, radiant, sign-based attacks. 36:38.190 --> 36:41.110 It's going to be like, no, someone broke the parsing. 36:41.110 --> 36:59.450 But then at the same time, and this is kind of going back to maybe a little bit of a hype panel yesterday, having people that aren't data scientists understand what the models can and can't do and what kind of behavior to expect from them so that they don't freak out when they see something weird happen and be like, 36:59.450 --> 37:01.710 oh, you know, whatever. 37:01.710 --> 37:04.430 We flagged this DLL as being malware when it's not. 37:04.430 --> 37:06.530 We're clearly under attack, right? 37:06.530 --> 37:08.250 And you're just like, it's statistics. 37:08.250 --> 37:09.730 Sometimes it makes mistakes. 37:10.590 --> 37:13.470 Well, just explaining it's not magic. 37:13.630 --> 37:15.330 It's hard, but it's not magic. 37:15.330 --> 37:16.030 Yeah, it's hard. 37:16.030 --> 37:16.570 It's useful. 37:16.570 --> 37:19.210 It does cool stuff, but it's a tool. 37:26.530 --> 37:34.590 Jager's talking about implementing a security development lifecycle and a secure software development lifecycle into model creation and upkeep. 37:34.750 --> 37:36.770 I'm going to make a data scientist cry. 37:38.570 --> 37:39.410 Yeah. 37:40.030 --> 37:50.730 I had issues with just being like, cool, we need to just write a few unit tests to make sure that the functions all kind of do what they're supposed to do. 37:51.390 --> 38:05.430 There's a bit of pushback from our more academically minded folk of just a little bit of basics like that, which may have solved some of these open CV issues. 38:09.550 --> 38:25.770 Interesting that if you basically treat the lifecycle of a data project as more of an internal product for a company and treat it as a product which serves a purpose, provides value, and then is integrated with other ways of developing software products, 38:25.770 --> 38:34.950 then data science and machine learning, they just become tools in the engineering toolbox, but then everything else falls within methods of continuous integration and testing and delivery. 38:34.950 --> 38:38.810 So then perhaps the risk, rather than the unicorn, depends on data scientists. 38:38.810 --> 38:39.370 Yeah. 38:39.770 --> 38:40.690 That's what we're doing. 38:40.690 --> 38:52.690 We're doing things like hypothesis-based development, which is like the next thing up from behavior-based development, which is next thing up for test-driven development. 38:52.910 --> 39:00.650 So you very much work out what the universe you'd like to see is, go do the maths on it, build the systems in it. 39:00.970 --> 39:03.090 So there are ways of working. 39:05.090 --> 39:06.350 Yeah, so maybe looping... 39:06.990 --> 39:07.850 Sorry, go ahead, Sven. 39:07.850 --> 39:08.790 Go ahead. 39:09.690 --> 39:18.410 Yeah, so I was going to loop back to the paper one more time and talk about the disclosure experience that they mentioned towards the end of the article. 39:18.830 --> 39:28.570 So basically, three of the vulnerabilities got new CVEs because they enabled arbitrary code execution or denial-of-service attacks, right? 39:28.570 --> 39:31.770 So those are both very firmly within the wheelhouse of what people... 39:31.770 --> 39:32.550 Page seven. 39:32.550 --> 39:33.490 Page seven, thank you. 39:34.270 --> 39:40.930 Very much within the wheelhouse of what security people think of as, oh yeah, that's a bug, that's an issue, that's a security gap that we need to fix. 39:40.930 --> 39:47.350 But there were a couple of other ones that they found where they could impact or manipulate the prediction. 39:47.350 --> 40:00.930 And then they call out in particular the mouthier memory corruption they found, where basically they can rewrite the feature vector, which allows you to basically make the model give you any output you want, essentially. 40:01.150 --> 40:06.150 Doesn't have to have anything to do with what the actual file that you were analyzing was. 40:06.250 --> 40:15.730 And those specifically are the ones that didn't get a CVE, didn't get called out, and mostly got labeled as won't fix. 40:15.730 --> 40:17.030 Or as won't fix. 40:17.030 --> 40:23.650 So that's sort of like, we were kind of bashing the data scientists earlier saying, oh, we got to think of this as part of a security system. 40:23.650 --> 40:25.770 But then it goes back the other way, right? 40:25.770 --> 40:32.570 We need to be able to communicate to people that as data scientists, as machine learning experts, look, this is bad, right? 40:32.570 --> 40:50.710 And how do we convince people that something that allows arbitrary misclassification or arbitrary predictions to be produced is as serious as, or maybe not as serious, but has some degree of severity, just like a remote code execution or a denial of service. 40:50.710 --> 40:54.190 I like the example that Eric brought up in his talk of a turtle rifle. 40:54.370 --> 40:57.930 Like that is a very effective way of communicating potential problems. 40:58.430 --> 41:08.510 Whereas for those who haven't seen it, like a turtle 3D printed object could be applied to a turtle to make it misclassified as a rifle. 41:08.570 --> 41:16.210 You can imagine like systems that track security and stuff to do certain events. 41:17.230 --> 41:19.290 I mean, it's like any other ethics discussion. 41:19.290 --> 41:27.390 We have to talk about consequences and then track it back and just keep pushing and pushing examples, real examples if we've got them. 41:28.830 --> 41:30.270 It's hard. 41:31.450 --> 41:40.990 Yeah, I mean, I guess like that's, you have a joke, but we're not really a joke, that like airline regulations are written in blood, right? 41:40.990 --> 41:47.710 And so when you've got all of these safety requirements, it's because a lot of people got hurt before someone was like, we ought to do something about that. 41:47.750 --> 41:58.390 And it would be nice to think that we could find a way to not have like ML disclosure requirements written in blood, right? 41:58.390 --> 42:13.290 Like maybe, you know, we're at a place now where we could maybe like jump ahead of the game a little bit and get an understanding of this out there so that we don't have to see people hurt before people are like, yeah, we should maybe take that seriously. 42:13.290 --> 42:17.010 I actually have a historical example for that too, for being old. 42:17.010 --> 42:23.900 So I ran one of the unmanned air vehicle safety teams back in the day before UAVs were sexy. 42:25.230 --> 42:34.810 And we didn't have the privilege, I guess, that the rest of the airspace industry had. 42:34.810 --> 42:38.090 So aerospace regulations are literally written in blood. 42:38.090 --> 42:44.690 So an aircraft would crash into something else or hit the ground and you would change the regulations. 42:44.710 --> 42:46.990 And that's literally how the regulations are written. 42:46.990 --> 42:48.210 Same with fire safety. 42:48.550 --> 42:51.270 Fire happens, change the regulations. 42:53.690 --> 43:05.570 And we were told explicitly and understood explicitly that we couldn't just put UAVs into manned airspace, have them crash into stuff and then rewrite the rules. 43:06.050 --> 43:13.090 We literally had to think all the scenarios and write the rules for how to do it safely. 43:13.610 --> 43:20.050 And it's whole space safety, not just individual aircraft safety before we were allowed to fly in the same spaces. 43:21.330 --> 43:23.330 So there's some precedent. 43:23.350 --> 43:28.490 I mean, there's maybe some looking in those spaces to see how it was done well. 43:28.830 --> 43:30.330 That's a good way to look. 43:31.670 --> 43:33.770 But a lot of it's just getting the will. 43:34.970 --> 43:38.270 Now, nothing in focus gets your attention like a whole bunch of airliners about to crash. 43:38.290 --> 43:43.770 But yeah, we could maybe get there before that point. 43:46.650 --> 43:51.950 So one more question from Discord, which is good because that's where I wanted to go next. 43:52.430 --> 43:53.750 So Yaga has a question. 43:53.750 --> 43:56.230 How do we force open source software to fix itself? 43:56.230 --> 43:57.210 We don't fix it ourselves. 43:57.210 --> 43:58.530 That's why it's open source. 43:58.570 --> 44:07.390 And the authors actually make a comment in the future work where they say that it's unclear who should be responsible for fixing them as well. 44:07.390 --> 44:15.170 So when they found a bug in the Malheur feature processing, that was because Malheur relied on libarchive. 44:15.610 --> 44:18.370 And the bug was actually in libarchive. 44:18.370 --> 44:24.090 And again, this gets back to this notion of like, you have all these different components that feed into the system. 44:25.130 --> 44:29.150 So even though the bug was in libarchive, it affected Malheur. 44:29.150 --> 44:33.090 So where do you put the responsibility for fixing it? 44:33.090 --> 44:46.210 And how do you convince, you know, like whoever's maintaining libarchive, look, this is serious enough that, you know, yes, I don't think it doesn't affect you directly, but it does affect this other system and it could be a critical impact. 44:46.210 --> 44:50.590 Like, how do we, how can we navigate that sort of uncertainty? 44:52.170 --> 44:59.150 How does Linux and Ubuntu and other open source software ecosystems do it? 45:02.130 --> 45:03.330 Do they have liability? 45:03.870 --> 45:12.710 I mean, I know for a lot of systems, the end users, especially if they're big end users, just get involved with the open source communities and work in there. 45:12.710 --> 45:22.730 But if you're using even like Python and your physical system crashes, who's responsible? 45:26.030 --> 45:26.570 Yeah. 45:26.570 --> 45:34.990 I mean, a lot of licensing argument, a lot of licensing agreements basically say like, yeah, you agree that you're using this at your own risk too. 45:35.030 --> 45:42.290 So, you know, basically everyone's going around not me, which, you know, it's fair, right? 45:42.290 --> 45:46.510 And people have, like, they have, they can't assume liability for everything that they do. 45:46.510 --> 45:52.630 But, you know, they can't agree to fix every single thing. 45:52.910 --> 45:55.110 But at some point, right, there's a balance, right? 45:55.110 --> 46:01.510 If something's widely adopted, it's widely used, and there's like a bug three steps up the chain that's causing it to behave weirdly. 46:01.910 --> 46:07.170 It feels like somebody ought to, there ought to be like some impetus to fix it somehow. 46:07.170 --> 46:20.830 The low tail of companies like Fortune 500 is more bothered to use these open source libraries than in theory, you know, this is AI, so you have a lot of, like, boost in productivity, like, and there's a lot of value that's being derived, and especially if they've started to use it for critical applications or decisions. 46:20.890 --> 46:33.430 So I think the onus and responsibility also needs to fall on the end users of, like, widely adopted packages, and then somehow making, making it very clear to decision makers in those companies that, hey, if you're using this to predict insurance rates or something and it fails, 46:33.430 --> 46:37.830 that's what you could potentially face, and then somehow channeling that into the projects themselves. 46:38.010 --> 46:44.270 Are you proposing that large companies that make a lot of money using open source tools should actually give back to the open source community? 46:48.330 --> 46:50.730 Smells like communism to me. 46:51.850 --> 46:55.550 It's kind of terrifying that we all just laughed when you said that. 46:57.390 --> 46:58.270 Yep. 46:58.330 --> 47:02.070 Sorry, I got to get my, got to get my commentary in once in a while. 47:03.110 --> 47:11.390 These are complex systems, and, like, just communicating the impact of some small, like, overflow bug to somebody who only cares about, like, the bottom line is a problem. 47:11.390 --> 47:16.630 Maybe there needs to be, like, awesome ML failures, GitHub repository, or something like that, where you can find it. 47:17.070 --> 47:18.810 We haven't had this. 47:18.950 --> 47:21.990 Sounds like, actually, a pretty good weekend project for someone. 47:21.990 --> 47:29.810 Yeah, I mean, like, you know, when we started up the ethics stuff years ago, it just, just takes some people determined to make this thing a thing, a visible thing. 47:29.950 --> 47:30.550 Yeah. 47:30.610 --> 47:31.630 Hey, community. 47:32.030 --> 47:33.550 Anyone want to take this one on? 47:35.130 --> 47:39.510 So, how many, like, solid examples of ML failure do we have? 47:41.470 --> 47:42.130 If we want to... 47:42.130 --> 47:43.230 I mean, a lot of it is. 47:43.310 --> 47:44.710 Compile those, yeah. 47:44.710 --> 47:46.410 How much do we actually know? 47:46.670 --> 48:01.170 Yeah, that's, I want to say, Andrew Davis over in different Discord had a pretty good comment on it, which is that a lot of time when ML fails, it kind of fails silently, right? 48:01.170 --> 48:12.990 It just, you don't flag something, and it comes back to a lot of the earlier discussion about is it a bug, or is it, like, normal, just sort of, like, statistical failure, right? 48:12.990 --> 48:20.770 A 1% failure rate means that if it's not failing 1% of the time, you should be vaguely surprised, right? 48:20.830 --> 48:29.570 So, yeah, like, I think it's possible that there's a lot of ML failures that we just kind of assume are in the statistical noise. 48:29.570 --> 48:36.550 A lot of them are probably kept quiet because, you know, there's potentially, like, PR risks or PR damage, right? 48:36.550 --> 48:42.370 Or, you know, like, like, actual, like, gross, you know, liability attached to it. 48:43.150 --> 48:50.730 Yeah, I mean, it's an open question how much is actually sort of flying beneath the radar out there. 48:50.790 --> 49:04.410 I mentioned a rental company saying, or a background check company who uses ML going, so we didn't give a lot of people their approved, you know, their credit check approval because of a mistake. 49:04.410 --> 49:08.870 And we want to publicly apologize for that when it's an ML mistake. 49:08.870 --> 49:14.830 And they can just be like, well, the model just was, yeah, sweep it under the rug, fix it quietly, never mention it. 49:17.390 --> 49:19.650 Like, that would be an ethical nightmare. 49:21.050 --> 49:27.590 Yeah, I mean, there's lots of startups that have proposed to do things. 49:27.590 --> 49:37.750 There was one, what was it, about two, three years ago, oh, we'll use ML to pre-screen your babysitters by digging through their social media profiles and stuff. 49:38.050 --> 49:49.090 And I have my own opinions on how likely that was to succeed versus how likely it was to simply just recommend white people. 49:49.090 --> 49:57.230 But how do you even measure or quantify failures in that case? 49:57.230 --> 50:08.010 Because it's all going to be completely internal and all you can do is essentially try and get a proxy model to be like, aha, here are the features, right? 50:08.010 --> 50:12.610 I've submitted a thousand different profiles and here are the features that seem to be triggering, right? 50:12.610 --> 50:16.410 And you have to affirmatively dig for it, right? 50:16.410 --> 50:20.170 Those failures aren't going to be obvious unless you're actually doing something like that. 50:21.170 --> 50:31.330 A recent streak of sort of facial recognition failures that lead to characteristics that elicits a visceral gut reaction like, yes, this is wrong. 50:31.730 --> 50:39.450 So I think in some cases, just seeing the outcomes, like Claire said, helps understand whether it's a failure or not. 50:39.670 --> 50:43.130 This gets back to the bias question. 50:43.130 --> 50:49.070 Like, how do you tell if something's biased or just like for this one instance, it was wrong? 50:49.070 --> 50:49.890 Oh, man. 50:51.530 --> 50:52.890 Yeah, it kind of works. 50:52.890 --> 50:54.710 We're going to have to clean that one out, Tommy. 50:54.770 --> 50:55.170 Yeah . 50:55.170 --> 50:58.210 In images is, oh, God. 50:58.910 --> 51:01.690 How much longer do we have for this conversation? 51:02.530 --> 51:04.550 Yeah, we can solve it in eight minutes. 51:04.550 --> 51:05.070 We've got eight minutes. 51:05.070 --> 51:06.130 Yeah, yeah, no problem. 51:06.170 --> 51:13.690 Yeah, I mean, the problem in a lot of the bias stuff, right, is like the terms aren't even defined sometimes. 51:13.690 --> 51:22.030 So like the example I keep coming back to is like recidivism and like predicting re-arrest rates, right? 51:22.030 --> 51:26.670 And someone might point out, hey, look, this is biased, right? 51:26.670 --> 51:30.790 It's clearly predicting re-arrest rates that are much higher for this group, right? 51:30.790 --> 51:34.090 And one argument would be, well, that's clearly wrong, right? 51:35.150 --> 51:40.130 It's biased because it's trained on biased data. 51:40.130 --> 51:44.270 And the counter argument would be like, look, it's reproducing the data, right? 51:44.270 --> 51:46.090 We don't live in a perfect society. 51:46.090 --> 51:47.570 The data is what it is. 51:47.570 --> 51:52.530 And within that box, the ML has done as much as you can ask it to do. 51:53.290 --> 51:59.090 And so it always boils down to these normative questions. 51:59.090 --> 52:12.070 And this is my own totally personal opinion that talking about unbiased systems, unbiased ML is almost inherently impossible, right? 52:12.070 --> 52:24.690 It's a question of how is it producing the kinds of outcomes that you want and how is the ML system moving power around, right? 52:24.690 --> 52:25.650 Who is it empowering? 52:25.650 --> 52:27.080 Who is it disenfranchising? 52:27.890 --> 52:31.470 And who is it helping and who is it hurting? 52:31.470 --> 52:51.730 And chasing after some vague notion of fairness, a fair system or an unbiased system is kind of like a distraction from more fundamental questions of who's being helped, who's being hurt, what's being reinforced or whose concerns are being downplayed. 52:52.010 --> 52:57.450 And who's accountable, I guess, also because of the reference recent Twitter story between Timothy Kimbrough and Yann LeCun. 52:57.450 --> 53:02.210 Basically, is it the ML engineer or is it actually the data scientist or is it throughout the entire chain? 53:02.410 --> 53:02.950 Yep. 53:04.630 --> 53:05.590 Yeah, and I mean... 53:06.370 --> 53:06.930 Sorry, go ahead. 53:06.930 --> 53:07.830 I'm talking too much. 53:07.830 --> 53:14.270 I'm seeing over in the chat that we already have awesome ML failures in the AI village as a repo. 53:14.270 --> 53:19.030 And Stellar Athena seems to be getting in there already. 53:19.710 --> 53:20.690 Outstanding. 53:20.690 --> 53:25.710 Suggestions for implementation failures, ethical failures, HCI failures and security failures. 53:26.010 --> 53:27.730 You got that Stellar? 53:28.130 --> 53:28.930 Yeah. 53:28.930 --> 53:29.790 Yeah, I think... 53:30.250 --> 53:32.170 Do we have a category for... 53:32.170 --> 53:36.570 Because we were playing around with this cat does not exist a couple days ago. 53:36.570 --> 53:38.270 I think we maybe need a category. 53:38.270 --> 53:40.210 This ML should not exist. 53:40.370 --> 53:41.810 Yeah, nightmare fuel. 53:42.810 --> 53:45.090 Yeah, well, I mean, I'm thinking of things like... 53:45.090 --> 53:45.590 Yeah , you're right. 53:45.590 --> 53:49.070 There are a whole bunch of machine learning systems that just should not be on this planet. 53:49.190 --> 53:49.730 Yeah. 53:49.790 --> 53:54.650 And anyone asked to contribute to it should have been like, perhaps not. 53:56.370 --> 53:58.810 Yeah, as far as the responsibility... 53:58.810 --> 54:01.790 And we would never have had Facebook if anyone had that call. 54:01.910 --> 54:02.590 Anyway. 54:02.590 --> 54:03.810 I mean, hindsight. 54:03.810 --> 54:11.870 But you can look at really short-term stuff like even just like genderify, right? 54:11.870 --> 54:18.130 It was should have been pretty obvious to everyone involved that this was just like a losing proposition from the start. 54:18.930 --> 54:22.350 Yeah, and the recidivism through face images stuff from China. 54:22.350 --> 54:23.070 Oh , God. 54:23.070 --> 54:23.390 Yeah. 54:23.390 --> 54:24.470 Okay, so... 54:24.470 --> 54:26.070 Way to replicate somebody's bias. 54:26.070 --> 54:31.190 So we do have a workshop coming up in a couple of minutes. 54:31.190 --> 54:34.550 So I think I want to kind of like put a pin in this. 54:34.550 --> 54:42.550 I think if everyone's enjoying this conversation, having an awesome ethics in AI panel, which is coming up in a couple of hours. 54:42.550 --> 54:44.850 I'm not sure someone fill me in. 54:45.030 --> 54:46.510 Two o'clock, two o'clock. 54:46.590 --> 54:47.530 Thank you. 54:47.530 --> 54:52.750 So everyone should absolutely turn, tune in for that if they're enjoying this discussion. 54:53.210 --> 55:04.150 But yeah, I guess I want to just like go around just to wrap this up and see if we have any like last minute, like big idea takeaways from this paper that people thought were really, really cool. 55:04.990 --> 55:06.630 I love the way they did it. 55:06.650 --> 55:11.250 The idea of attacking the system and looking at taking it apart is just kind of my thing. 55:11.250 --> 55:17.510 And I like the idea of using it to attack systems that shouldn't exist. 55:18.310 --> 55:19.250 Yep. 55:19.430 --> 55:20.390 Sven? 55:21.050 --> 55:24.670 I really like the idea of attacking the feature extractors. 55:24.670 --> 55:34.030 And I think we should also add like bypassing the feature extractors by just doing something you know the ML system is going to miss. 55:35.110 --> 55:40.010 Because you can understand it well to this whole thing. 55:41.030 --> 55:41.610 Cool. 55:41.610 --> 55:42.250 Martin? 55:43.310 --> 55:45.970 Yeah, that also really enjoyed the paper and discussion. 55:46.190 --> 55:53.010 I think to me what was clear is that it can occur at many levels of the basically stack. 55:53.010 --> 55:57.390 So just instrumentation and stuff become very, very important to monitoring what happens. 55:57.390 --> 56:07.390 And I would really like to see follow on work that applies this to TensorFlow, PyTorch, basically the learning libraries and seeing how you can play with intermediate representations and things like that too. 56:08.410 --> 56:09.630 Yeah, for sure. 56:09.630 --> 56:20.770 Yeah, the reminder that like these ML things, ML projects don't exist in some abstract theoretical space and are implemented in real code and real systems that come with their own flaws. 56:21.190 --> 56:28.250 Yeah, like you know it but being reminded of it sort of viscerally by, look, we have CVEs is always a great thing. 56:28.530 --> 56:29.970 So cool. 56:30.050 --> 56:36.150 So I think this is we probably want to leave a few minutes for people to fight with the audio as we transition over to the workshop. 56:36.170 --> 56:40.230 So I think we want to call that a panel or a discussion. 56:40.230 --> 56:41.050 So awesome. 56:41.050 --> 56:41.890 Thanks, everyone. 56:41.890 --> 56:42.330 Thanks, Rich. 56:42.330 --> 56:43.170 One second. 56:43.450 --> 56:47.890 Before we jump off, we have one on Wednesday at 5 p.m. 56:47.890 --> 56:49.030 Pacific time. 56:50.510 --> 56:55.310 And we have the, I'm guessing, I think we should go with your paper, Rich. 56:56.310 --> 56:57.510 The slide deck on... 56:58.770 --> 57:07.470 Yeah, the anomaly detection outside the closed world on using machine learning for network confusion detection by Robin Subban and Vern Paxson. 57:08.050 --> 57:13.710 So we'll post that in the Defcon Discord and also be posted to our Discord and Twitter. 57:13.710 --> 57:17.930 So if you want to participate, you should find your way over to our Discord. 57:17.930 --> 57:24.530 We can't post a link to that in the Defcon Discord, but the links are places and you can just DM us for an invite. 57:24.770 --> 57:29.610 And we'll be discussing that for probably an hour and a half on Wednesday. 57:29.610 --> 57:33.650 And we might be able to get one of the authors in, even though this is a 10-year-old paper. 57:36.050 --> 57:37.750 How far we have come. 57:39.130 --> 57:40.010 All right. 57:40.010 --> 57:40.730 Thanks, everyone. 57:40.730 --> 57:41.390 Catch you later.