start	end	text
820	3180	Okay, say something.
5280	9420	Nope, the audio is definitely broken again.
10220	11680	Now say something.
19120	20920	Time with the audio issues.
20920	24040	There we go. The audio seems to be working now.
24040	25140	All right, awesome.
26280	29100	Yeah, I almost feel like we got to have... whoever
30900	35200	is the poor soul stuck with jockeying this in the future almost needs to have two laptops,
35200	42700	one with Twitch and everything going and one with just OBS and Zoom, yeah.
43880	45260	Yeah, that's...
45260	49840	Nothing can confuse OBS's tiny little brain.
50060	54320	But then the problem is when you forget to mute that thing and then this thing
54320	59860	and then there's one of your tabs is doing the wrong noise and you're like, oh, this is wrong.
60220	60920	Yep.
60920	63300	Sven, you're being heroic. It's just...
64020	69880	Yep, we are live, by the way. The internet is listening to us talk, so enjoying the banter.
69920	71240	You're still being heroic.
71420	77780	Yes. Yeah, so for those of you watching, Sven has been the main guy troubleshooting
77780	83120	all of the audio issues and fighting with the stream for basically all of DEF CON.
83120	89860	And it's a testament to his hard work that things have gone as smoothly as they have.
89860	95060	So you've made it a lot easier that things were pre-recorded.
95060	98040	Can you imagine trying to hunt people down,
98040	102840	hunt speakers down, and then their sharing is not quite right and all that?
103400	105420	Yeah, no, I mean...
105420	107880	I mean, this year, at least they're not passed out in the bar.
109660	112260	How many speakers have been passed out in the bar?
113360	117720	None of ours that I know of, but we've definitely hooked a few of them out from the nearest bar.
118240	119600	Yeah, well, you know.
124250	128810	Are you expecting anyone else or we want to get this show on the road?
129130	136930	Yeah, let me... well, let me pull up this thing for myself.
138090	144970	I'm going to try a new layout as soon as I can find the thing.
145210	151210	So while I'm looking for it, Rich, do you want to introduce the paper to everyone?
153470	156270	Oh, sorry. So we're going to do a couple of things.
156290	157430	Hi, Barton.
157510	160810	So we're going to start with just the four of us talking about it for a bit.
160810	162410	And then I'm going to open up the...
163770	168710	and join the voice chat in the Defcon Discord.
168710	172310	And so you guys will be able to talk on stream and talk with us and everything.
173870	176950	So for the first, like, 20 minutes, it's going to be just us.
176950	179310	And then we'll bring everyone we can in
179310	182070	and have a bit more of a Wild West thing.
182070	183790	We're going to try out how that works.
184430	187530	And the other thing is I'm just going to play with the layouts a little bit.
187750	190750	So that's the structure of it.
190750	193670	We've also got a journal club coming up on this Wednesday.
194370	197770	We will put the link to that paper in the Defcon Discord.
197770	201190	And then if you want to participate, you should join our Discord.
202810	204370	Link in the Twitter bio.
204810	206410	Yeah, it's in the Twitter bio.
206410	208690	And you can just DM one of us.
208690	212730	And we'll send you an invite to the AI Village Discord.
212730	216050	Please don't use the AI Village Discord while Defcon is going on.
216050	220350	Please be respectful of the hard work they have put into building their Discord
220350	221770	and getting it running.
221970	225330	And only use it, like, tomorrow or Tuesday.
226730	228910	So take it away, Rich.
229490	231070	Okay. Cool.
231070	234690	So this is actually a slightly older paper.
234690	236950	It's from 2017.
238010	242130	It is called Summoning Demons, the Pursuit of Exploitable Bugs in Machine Learning,
242130	246890	which is probably one of the coolest paper titles I've come across in a while.
247310	252310	And it's by Rock Stevens and Tudor Dimitrescu's group.
252310	255870	So the basic idea of the paper, it's kind of an interesting spin
255870	262050	on the adversarial ML papers that I think started coming out about the same time.
262050	269810	And in this, rather than actually, like, using the ML model itself to sort of guide inputs,
269810	272130	they're looking for more...
272770	277930	they're more looking for flaws in how the different components of the system hook up.
277930	284050	And to do that, they actually crack out an old classic American fuzzy lump, the fuzzer,
284050	291710	and they re-instrument a lot of it so that weird actions from the ML components of the system
291710	294370	register as crashes to AFL.
294570	299750	And so that lets them essentially do sort of like a black box adversarial attack,
299750	302530	if you want to think of it that way, against the ML system.
302570	305890	And what they find is that in addition to sort of the usual stuff you can get,
305890	309050	where you have, like, inputs that look like faces that don't register as faces,
309050	310890	because they've had some pixels perturbed,
310890	313950	they also run into, like, image format parsing errors,
313950	320250	where a component of the ML system won't parse, for example, a face correctly,
320250	321730	completely fails to find the face.
321730	325830	But if you open it up in other programs, it actually looks good.
325990	332630	So I think this is kind of in the vein of the same sort of things
332630	335310	that Ariel Herbert-Boss talks about a lot,
335310	339330	where it's not just attacking the ML system itself that's important.
339330	342850	You've got all of these other components in the pipeline,
342850	346870	from ingesting the actual image all the way through feature extraction,
347470	348830	pre-processing, and so on.
348830	351650	And each of those is actually a point that you can attack.
351650	357590	And so in this paper, they're basically using fuzzing as a way to find
357590	361270	some of those, you know, broken assumptions or breakdowns
361270	364510	in the link from one stage of processing to the next.
365350	368590	And I thought it was a super clever paper with a really good name.
368590	371250	And they dig a couple of CVs out of it, too.
371250	377530	So super cool application of sort of more classic security methodology
377530	383510	to what, from my perspective, is kind of an adversarial ML problem.
386210	390530	So that's my, I guess, like two-minute summary of the paper.
395440	398140	So I guess maybe to kick off conversation,
401340	404820	so does this, do you feel like, does anyone have an opinion,
404820	406540	strong opinion one way or the other,
406540	411800	if this falls into the same sort of general field of adversarial ML
411800	415180	as sort of like gradient-based attacks?
415180	418160	Like I'm trying to think of, the name escapes me off the top of my head,
418160	422740	like fast gradient sign method, or like these Jacobian-based methods,
422740	427160	or saliency-based methods, or maybe even like,
427160	430440	you could almost argue it's like a genetic algorithm-based attack
430440	434520	against a particular ML pipeline.
434580	438420	Well, to be honest, I thought that adversarial ML covered all this already.
438420	440540	I was a bit surprised that it was new.
441800	446300	Yeah, it's genetic algorithm-based because they're using FuzzyLock,
446300	448780	which is genetic algorithm-based, which is also interesting
448780	453740	because there were network extensions to genetic algorithms
453740	456440	that might be more appropriate if you're searching spaces.
456980	460100	So there's some interesting stuff on the front that could be done.
460100	461980	I kept wanting to say, what happened next?
470890	474430	Likewise, in the context of healthcare,
474970	476690	practical realities of securing systems,
476690	483910	it seems like a lot more emphasis right now is placed on more theoretical attacks,
483910	487750	where it's more so once you have uninhibited access
487750	490710	to being able to query this API that's being explored.
490710	492970	So it's nice to see a paper like this that's a bit more
492970	494950	in an all-systems set of things,
494950	499390	because at least from healthcare IT,
499390	503450	it seems like it's a more likely encounter that you'll find in the industry.
504670	510610	Yeah, one thing that I did really like was their focus on implementations
510610	513090	rather than just theoretical properties, right?
513090	519030	Because in some sense, something like the FastGradientSign method
519030	522710	would apply to almost any of these models.
522710	526690	But here, they're sort of like, okay, this model was instantiated in this way
526690	528170	and is doing these things.
528170	532850	And now how can we attack sort of that specific version of the model?
533450	538550	Because it's a lot less academic in some ways,
538550	542890	and a lot of the adversarial ML research seems to be very academic still.
543150	546190	This was like down and dirty in how do we attack this thing.
546310	549150	It's kind of in my vein of stuff.
549270	552330	Just look at the system, work out how to break the system,
552330	553810	and take it to pieces.
554050	558310	I loved the way that they hacked that little hack on FuzzyLop,
558310	561750	where they went, okay, we're going to take out the things that actually broke,
561750	565070	and then we're going to induce breaks for the things we care about.
565490	567430	Yeah, changing the target.
567430	568570	Yeah, changing the target.
569290	571250	It was just beautifully done.
571650	574530	I thought it would be useful for defense as well,
575250	581830	because we're also looking now at things like Strafe's work,
581830	585510	so Brittany Prostakoff's work on attacking with robots.
585990	590290	What could we do to screw up their vision systems?
591210	593710	So how could we mess that up?
593710	595850	There's some beautiful adversarial there too.
596910	598870	They can't see, they can't move.
599190	599750	Yeah.
600470	603650	Yeah, so actually, I thought table one in this was...
604490	606130	I don't know, I'm surprised.
606830	608270	Flips to table one.
608270	609010	Flips to table one, yeah.
609010	610510	Waves to table one.
611170	613190	It's, I think it's on page four.
613390	614310	Yeah, yeah.
614310	615270	Yeah, page four.
615330	618550	Yeah, they actually like, I feel like, again, this is very much like
619210	620330	Ariel's jam, right?
620330	623650	They go over sort of the attack surface and the kinds of things you can do to it.
623650	628130	And it reminds me a little bit of the contest there was...
628130	631670	So I think it was a late entry to the contest last year
631670	635670	that was announced at the AI Village where they were trying to do ML evasions.
635830	640230	And one guy actually found essentially a bunch of ways
640230	642370	to just completely trash the feature extraction
642370	647650	so that the ML prediction just wouldn't run on the samples he uploaded.
648550	653010	And so, yeah, he was a little upset that that didn't count as a win,
653010	654310	which I think is justifiable, right?
654310	657290	Because if you can just like stop the analysis cold, right?
657470	659530	That's almost as good as evading it.
659530	660230	But yeah.
661170	663670	It depends what you're trying to do if you kind of sneak into a system
663670	665610	and it's a bunch of systems, right?
665610	667830	We just want to, you know, the usual thing.
667830	670930	Get in, do your thing, get out without no one noticing.
672090	674850	And we see enough of that in real life of the stuff.
674850	677350	It's like, oh crap, we didn't realize that was a problem.
678070	679770	Before someone else adds them.
681050	683650	There's also another thing to that.
683770	689030	Like if you wanted to attack the Ember model with the Ember dataset,
689030	693350	it only looks at your headers and you could just change other parts.
693350	696570	Like don't even think about it in terms of adversarial ML.
696570	702250	Just copy like a notepad.exe's header and just put your own
704030	708190	entry point function in and start launching weird stuff from there.
708190	711870	And that's a different way to bypass the thing that is also not...
712810	714670	Academics also not aware of.
716330	718110	Yeah, so that's sort of like...
718110	723010	I guess that's maybe like a...
723010	724910	You can classify that if you're gonna...
724910	725910	I'm gonna screw this up.
725910	730110	So if I'm getting the terminology wrong, security people, please like yell at me.
730110	735010	But it's almost sort of like a logic attack on the feature extraction
735010	738150	rather than what they're doing here,
738150	740570	which seems to be more going for crashes, right?
740570	743490	If you know how the feature extraction process works,
744250	747550	then you can sort of write your way around it deliberately
747550	751830	rather than just throwing shit at it until something blows up.
752930	755490	Some of it's blowing it up, some of it's sneaking in the back.
755670	759630	Some of it's like sneaking in the back conventionally depending on your system.
760770	766710	The other thing that interests me is that only some of these became CVEs.
766710	770530	That the response to the people who are responsible for the algorithms
770530	773830	for some of this was like, won't fix and who cares?
774330	777990	And that's part of our journey, I guess.
778370	780990	It is just like every other security thing
780990	784710	is trying to get people to accept that yes,
784710	789630	you may have to drop a little bit of responsiveness in your system.
790110	793410	But you'll get it more secure and it matters.
794570	797990	Yeah, I'm kind of reminded a little bit of some of the argument
797990	801130	that went around for the proof point CVE,
801130	803290	where people were arguing whether or not
803290	806670	that actually counted as like an information leak CVE.
806670	809190	Because so if people aren't familiar,
809190	812910	the short version is the people who did it,
812910	816510	Will Pierce and someone whose name I'm forgetting, I'm sorry.
817040	820150	They found a way to essentially like send an email
820150	823270	through proof points, spam detection service
823270	825030	that would then bounce back to them.
825030	829650	And in the header of the email, it had a score for how spammy it looked.
830110	833750	And so that essentially was the information leak.
833750	835190	The fact that they could get that score back
835190	839270	and that was enough for them to rebuild essentially a proxy model for it.
839750	844910	And yeah, over on in various like AI village discussion channels,
844910	848570	we had very strong debates over whether or not
848570	852790	that actually should have counted as like an information leak CVE.
852830	856090	Because, you know, it's in a lot of cases,
856090	857890	it's useful contextual information.
857890	860390	But when you look at it in an ML context,
860390	862950	you're actually enabling like a model stealing attack.
863590	866670	Which again, it's one of these, we're kind of in new territory
866670	870110	as far as the security of these ML based systems goes.
870130	873190	And a lot of people that are more familiar
873190	876450	with conventional security, maybe don't quite get
876450	881970	what these various information leaks or attack surfaces could do
881970	887630	once you wrap it in like how ML based systems work.
887790	892310	And I think another thing we get out of this is by doing this sort of attack,
892310	896750	we start showing where the things that break up.
896750	899630	And one of the things I'd love about the fact that MLSec exists
899630	901590	is it makes machine learning better.
902530	905650	Just by dint of the spotlight we throw on it.
905650	907790	You chuck hackers at it, they take it to pieces,
907790	909370	they show you where it's broken.
909990	914290	And hell, we spend most of our life trying to get our damn systems to work.
914450	916210	But I'm going to give a little bit of history.
916210	917170	I'm old.
917510	920590	Jump in for just a second.
920630	922470	I just saw in the Discord chat.
922470	927170	So it was Will Pierce and Nick Landers who got the Proofpoint CVE.
927170	930010	So I'm very sorry that I forgot your name, Nick.
930010	931410	Okay, sorry, carry on.
931590	935110	I was just going to say about sort of some things, the findings,
935110	936870	that it's double float precision thing.
937250	942110	So way back before we had, I think we had Python,
942110	944710	but we sure as hell didn't have scikit-learn and friends.
944710	946590	We had matplotlib.
946650	948270	I'm not... matmap.
948390	951670	We use MATLAB for a hell of a lot of machine learning stuff.
951910	954770	And there was a neural network kit in the MATLAB.
955010	959170	And that neural network kit suffered horrendously from underflow errors.
959690	962090	You could break it really, really, really easily.
962090	963950	So Aston University built its own.
964510	968170	They built a separate kit because they couldn't get the first one fixed.
968810	971910	I mean, this sort of stuff, like the float to double errors.
972890	978550	I mean, crossing my fingers that actually all of those hidden faults
979410	982910	that are probably still going on with neural network stuff and deep learning,
982910	985950	we're not even noticing because we just trust the systems.
985950	989670	Yeah. I mean, even in Python, right?
989670	993450	Like under the hood, a ton of these neural network libraries,
993450	997030	and PyTorch, at least, sklearn as well,
997550	1003070	use pickle, which is just broken kind of by default, right?
1003070	1006310	It's a stack machine that you can write arbitrary code for.
1006570	1013090	And yeah, people still share wait files that are essentially pickles.
1013090	1016850	And so you're downloading this off the internet and running code.
1016850	1017690	And I keep threatening.
1017690	1021050	One of these days, I'm actually going to write a pickle poisoner
1021050	1025170	that just replaces calls to the relu function
1025170	1027710	with some sort of stochastic function
1027710	1031470	so that half the time it works and half the time it gives you garbage.
1032030	1034670	Comment from Discord.
1034670	1036510	Yeah, numpy as well uses pickles.
1036510	1037810	So it's everywhere.
1037810	1038890	Keras, yeah.
1038890	1042710	Is that what the numpy save file is?
1044330	1046050	I thought that was secure.
1046350	1047570	I'm not sure.
1047570	1049130	I know, I'm pretty sure it has one...
1050570	1051970	One pickle thing in there.
1052030	1053110	One or two.
1053110	1057890	It's got one or two formats that are not.
1057890	1059390	Apparently, there's a couple that...
1059390	1064210	The scikit-learn form that these guys found, numpy was part of that.
1064210	1065490	I'm just flicking to the page.
1065490	1067990	Page seven at the bottom, if you're reading along with us.
1069210	1073110	One interesting thing is that they weren't able to induce misclassification
1073110	1076650	or from the precision loss problem.
1076650	1078610	So they tried different values of epsilon,
1078610	1080610	and basically said that it's possible in theory.
1080610	1084390	But it's interesting to consider how would you model that epsilon
1085490	1089330	based on the syntax to induce this misclassification.
1095120	1096840	How much of this is induced by...
1098440	1101220	Basically, all the machine learning code is written by academics
1101220	1105560	and over-enthusiastic academics.
1105560	1106200	Guilty.
1107860	1109540	Who aren't experienced in...
1110820	1115180	It's incredibly hard to write numerically efficient code.
1115180	1119960	You don't have time to really learn how to multiply matrices incredibly efficiently
1120660	1123440	and make sure you open files correctly.
1123960	1130300	So we've chosen a lot of lazy paths to formatting and opening things
1130300	1132260	versus secure paths.
1133720	1136260	And also, you've got tiny numbers and people don't...
1136260	1138180	I mean, you go wading on in, build your stuff,
1138180	1140640	and don't think about how those numbers interact.
1140640	1143260	You just assume the system is going to magically deal with it.
1143900	1145960	Yeah. And I mean, there's also the fact that
1147240	1152200	writing secure code is very much its own skill set, right?
1154440	1156400	Some people are writing research code
1156400	1158900	because they don't expect it to be used in Angular,
1158900	1161040	they don't expect it to be used in a production context.
1161040	1163600	Some people are writing numerically efficient code
1163600	1166160	because that's their jam and God bless them.
1166900	1171260	And then you've got sort of security people
1171260	1173920	who are probably gazing at a lot of this
1173920	1178120	with sort of a feeling of vague horror that they're actually going into production.
1179180	1182600	Yeah, I know even in my day job, right,
1182800	1185260	a lot of what we've been doing recently
1185260	1190300	is sort of going back and rethinking how all of this stuff is put together
1190300	1195260	so that we can feel confident in actually pushing out products containing it.
1196120	1199460	There's a middle ground where you can enable basically more flexible code,
1199460	1204060	but also execute it in a way that perhaps minimizes the blast radius.
1204140	1205940	I don't mean put it in a container, obviously,
1205940	1210420	but it's something more to the point of running code in secure enclaves.
1210420	1214500	For instance, the work with silo type of workflows
1214500	1219980	or do you see anything where you try to mitigate the impacts of this code?
1220560	1222060	And could we have code checkers?
1222060	1226100	I mean, literally just some way, some sort of test data sets.
1226180	1229360	But we know this stuff is bad. Run it through your system.
1230020	1230560	Yeah.
1231660	1238420	Yeah, I mean, I had a great amount of fun, again, at my day job,
1238420	1240840	actually like taking our feature abstraction code
1240840	1243120	and fuzzing it just to see what would happen with it.
1243120	1249880	Fortunately, it turned out that I spent more time than I care to think of beating on it,
1249880	1251580	and it turned out fine.
1251580	1254840	But yeah, if it hadn't, what would we have done then, right?
1255660	1261680	And I think the idea of building test vectors of just really bizarre,
1261680	1266700	but technically conformant files for this sort of thing is an awesome idea.
1267820	1272020	But extending that to the actual data science part,
1272020	1276160	how often have you issued a PR about some data science thing
1276160	1278940	and sent it to someone and they're gone?
1278940	1280540	It looks good to me.
1282360	1284920	But there's some stuff that you can check easily,
1284920	1292160	but a lot of the code that goes into BERT is incredibly complicated.
1292520	1294180	Very hard to check.
1294280	1295660	When you make a small change,
1295660	1300480	you could have broken things fundamentally because it's software.
1301720	1306800	How do you write a unit test for I broke BERT when you input this one slight thing
1306800	1313380	and now it causes this whole class of things to be false positives, false negatives?
1315680	1318140	Like, how do you unit test machine learning?
1318580	1321740	There's a... I mean, you can do...
1321740	1324600	I mean, essentially, you do test vectors, right?
1324800	1327980	You say, you know, if I load...
1328660	1331340	You can consider the weights are obviously part of the model.
1331340	1332520	You say, I've got this set of weights.
1332520	1335420	If I load in, you know, this set of features,
1335420	1339600	I should get this exact set of outputs up to numerical precision,
1339600	1343580	which, again, numerical precision is kind of its own nightmare.
1345020	1349660	Yeah, but then if you change the weights, right, then kind of all bets are off.
1349660	1355440	And that's one of the painful things about debugging ML,
1355440	1358620	even when you're not under adversarial conditions,
1358620	1364040	is just, I retrained the model, do I still trust it?
1364180	1366780	I mean, there's a difference between I messed up my models,
1366780	1368440	which is like, we do this all the time.
1368440	1371180	And somebody is actively trying to attack my models
1371180	1375360	or may actively try to attack my models using known vulnerabilities.
1375860	1379480	Because sure as hell, half the systems out there haven't patched any of this.
1380060	1383880	How many machine learning developers even know unit testing
1383880	1386420	or even know good software practices?
1386420	1388580	Like, a lot of us came from academia.
1388580	1391300	I have a PhD in math, not software engineering.
1391740	1397520	So I have a philosophical question I want to pose to the panel from Discord.
1398080	1401580	And Yaga asks, what is a terror and...
1401580	1405420	Sorry, what's a terror? This is terror right here.
1405420	1408100	What is an error and what's an attack?
1408100	1412980	And like, is there any meaningful distinction between the two?
1412980	1415100	I guess I would follow up.
1416080	1418540	I think if you're looking at adversarial images,
1418540	1420740	I mean, the fact you have like the little image patches
1420740	1423200	to screw up your system is an attack.
1423280	1424440	It's not an error.
1424700	1428620	You just don't randomly get sort of extra patches on a stop sign.
1429280	1431960	No matter how much undergrowth you got on it.
1432140	1434280	I think the difference is intention.
1435540	1437840	Yeah, I mean, that's the same thing in security, right?
1437840	1442660	Bugs can be... I mean, exploits are essentially software bugs, right?
1442980	1445300	Maybe that's being too general, but if you...
1446140	1448500	You know, a subset of software bugs or exploits
1448500	1452580	and you use them to make the system do things you don't want it,
1452580	1455060	it shouldn't be doing under normal circumstances.
1456400	1458360	Or wasn't intended to be doing.
1458360	1458880	Right.
1458880	1460380	And sometimes they're useful.
1461740	1465640	Would you consider a well-informed person building an attack
1465640	1468080	against a machine learning system
1468080	1469820	where they understand the machine learning system,
1469820	1471820	they understand the feature extract and all that stuff,
1471820	1475640	and they've built a false positive.
1476460	1479300	Is that an adversarial attack against the machine learning
1479300	1481180	or is that like an exploit?
1482120	1484700	Like, where would you classify the mistake?
1484700	1490020	Is it like a software mistake or a exploit or what?
1491580	1494240	I think that's the tricky bit about ML models, right?
1494240	1497180	I mean, they're inherently statistical.
1499620	1503140	So it kind of gets back to testing after you retrain it, right?
1503140	1508280	So if you have a single example
1508280	1510460	and it turns out to be a false positive,
1510460	1512300	and maybe that's one of the bridges
1512300	1514600	that I feel like as a data scientist,
1514600	1518320	I have to cross a lot when I'm talking to security specialists.
1518320	1521500	If I have a model and it FPs on a single file,
1521500	1525380	that doesn't call into question the efficacy of the model, right?
1525380	1528560	You have to look at the statistics of what the model does, right?
1528560	1530800	And if you say, well, yeah, it got this FP
1530800	1533300	and that was a pretty boneheaded FP,
1533300	1541560	but for 99% detection, you get one FP in 10,000 negative samples.
1541720	1543380	That's a pretty good model.
1543520	1547880	And so being able to sort of drive that switch from,
1547880	1552200	hey, here's the good, bad to,
1552200	1555360	hey, you have to think of this sort of heuristically and statistically.
1555980	1561140	Is a big part of just like communicating as a data scientist, I think.
1562360	1563940	Go ahead, Sarah.
1563940	1565760	Yeah, if you've got somebody actively attacking,
1565760	1569840	then maybe some pre-processing is enough to cut down the attacks.
1569840	1571540	Yeah, yeah, for sure.
1572020	1575860	All of that goes out the window when you've got someone maliciously trying to...
1577280	1580320	Yeah, but so my question is,
1580320	1588780	so if you have like a fancy ML from an academic paper bypass of your model,
1588780	1594980	versus someone trying things, what would be...
1597100	1599320	When is it like a machine learning mistake
1599320	1603360	or versus a mistake in your featurizer and stuff?
1604040	1607020	If you look at the decision boundary,
1607020	1611640	basically for something and seeing how the perturbations relate to that?
1611900	1618760	Yeah, I kind of feel like maybe to pull it back to this paper a little bit,
1619720	1627260	if it gets through your feature extraction process in one shape,
1627260	1632640	in one piece, then maybe it's a modeling issue.
1632640	1635340	And if it crashes your feature extraction process
1635340	1639760	or it gives you something that you wouldn't expect
1639760	1644680	from your feature extraction process, then it's a featurization bug, right?
1645140	1647680	If you feed garbage into a model,
1647680	1650740	you can't blame the model for producing garbage outputs.
1651660	1653020	How would you basically...
1653020	1656360	What are some methods for quantifying unexpectedness in this scenario?
1656360	1660660	Is it more like error counts or latencies in response?
1660660	1663100	What would you pay attention to if you were trying to understand
1664280	1666040	if something was misbehaving?
1667960	1669720	You know what, when you see it.
1669820	1672240	Yeah, like if you take a look in the paper, right?
1672240	1674780	They've got an example. I want to find it.
1675440	1676620	Figure two, right?
1676620	1679740	So they show an example of two different images.
1679840	1685560	One is an image of sort of the very top of a person's head
1685560	1687320	and then the rest of it is gray.
1687580	1691040	And one is a picture of a person's head.
1691040	1692340	Yes, there you go. Thank you.
1693480	1695340	Page six, if you're following along.
1695340	1697960	Right, so OpenCV has a bug that these guys found
1697960	1701020	by their guided fuzzing technique,
1701020	1705820	which means that OpenCV can't properly load the file
1705820	1706740	and then you run it...
1706740	1708180	Obviously, you run it through feature extraction
1708180	1710300	and it doesn't get anywhere.
1710300	1713240	So it's like all these different individual components
1713240	1714980	that you got to worry about, right?
1716300	1718940	And I think on that topic, there was a...
1718940	1721500	From Hacker Factor in Discord, he's got a question.
1721500	1723680	Are you making a distinction between the ML model
1723680	1726740	versus software that drives the ML system?
1726880	1729020	He says fundamental error versus algorithmic.
1729020	1731240	And I think here, right?
1731240	1733900	This is what this paper is going after, right?
1733900	1736360	It's saying the software that drives the ML system,
1736360	1739440	the implementation of the system also has bugs
1739440	1740600	and they find them here.
1740600	1744840	And, you know, if you go to table two, right?
1744840	1749020	They've got in OpenCV, they've got like two heap corruptions.
1749020	1751180	They've got a weird rendering bug that they found.
1751180	1753980	Page six, thank you, yeah.
1754880	1756700	Right, so all these different...
1756700	1758980	Like they literally have like heap corruption bugs
1758980	1761760	that they're discovering with these funky inputs.
1762380	1764440	And that has nothing to do with the model.
1764440	1768220	That's you're attacking the plumbing around the model, which is...
1768220	1771660	But you can use that plumbing to go and tweak the model.
1772480	1773700	Right, exactly, yeah.
1773700	1776740	So that's what the third line down does.
1776740	1779320	That's the figure one, figure two?
1779320	1780680	I forget, figure two.
1780680	1783880	Yeah, so you break the rendering
1783880	1787120	and then the rendering goes to the feature extraction
1787120	1788660	and the feature extraction can't get anything
1788660	1791720	because it's just got a bunch of gray pixels to look at.
1791720	1793160	And then the model, right?
1793160	1794040	Garbage in, garbage out.
1794040	1796780	The model can't do anything because it doesn't have good features.
1797860	1801260	So I'm going to try to open this up to Discord.
1801740	1806700	If you are in the DevCon Discord,
1806700	1809140	go to airvillage-general-voice.
1809760	1814960	We'll try unmute you and get you so you can talk.
1816180	1820420	If you're human plus, that's the correct way to do it,
1820420	1822080	but we have a workaround.
1822480	1827220	So head over to airvillage-general-voice to ask any questions.
1827360	1830600	If you can't, we'll still be paying attention to the text chat
1830600	1832960	in airvillage-journal-text.
1835900	1837940	The moment of truth.
1839940	1842260	I'm watching journal club text.
1842360	1843980	Yeah, me too.
1845220	1848080	Audio, managing audio is...
1848080	1850280	I'm so glad I don't have to deal with it.
1857170	1861410	I wish I had seen this paper before I started, like,
1861410	1865690	really trying to solve adversarial machine learning as a security problem.
1865890	1867340	Because there's a lot...
1867860	1873540	this seems a lot easier, like, to attack for a lot of attack people.
1874240	1876680	I think a lot of attackers have more knowledge
1876680	1882400	on how to break a parser than how to break a neural network.
1883880	1888860	So this, to me, feels like more of a realistic threat model
1888860	1892640	of, like, your pipeline was... they broke your pipeline.
1892640	1894400	You did something wrong in AWS.
1894400	1898940	You did something wrong with your container setup.
1898940	1902200	And it's parsing your images incorrectly now.
1904140	1907980	Yeah, I mean, that's tons of the bugs that you find
1907980	1912100	in sort of real systems, right, as parsing bugs.
1912100	1920320	And yeah, a lot of the research right now focuses less on that
1920320	1925320	and more on sort of these theoretical properties of machine learning models,
1925320	1927380	which, I mean, I don't want to dig on those.
1927380	1929400	They're fascinating, right? I love these papers.
1929400	1933320	But they're also, again, keep coming back to this paper.
1933320	1936780	This paper makes the really good point that these sort of theoretical models
1936780	1940420	that have these nice properties are embedded in real world systems.
1940420	1943140	And we know that real world systems are always kind of
1943140	1946680	full of bugs that can be attacked and exploited.
1947380	1952040	And so it gives you sort of multiple entries into the problem, right?
1952040	1954500	On the one hand, you can exploit the preprocessor,
1954500	1956200	and maybe that gives you some sort of heap corruption
1956200	1962360	or some sort of like direct, you know, denial of service or even code execution.
1962500	1965220	And on the other hand, you can break the feature extraction enough
1965220	1968720	so that everything downstream is getting garbage data.
1970160	1975060	Or mildly corrupted. I mean, I say again, with like the sneaking in,
1975060	1977480	you want to do low and slows on some of this.
1979000	1982520	Again, I am really like the idea of using this as defense.
1982520	1986180	I mean, there's a whole bunch of applications where the bad guys
1986180	1989120	are using machine learning based systems.
1989520	1992020	I like the idea of breaking their systems.
1992160	1995200	Yeah, as I would. Great hats on.
1996780	2006170	Yeah, just like yesterday's paper with the Forbes masks.
2007390	2011730	Mm-hmm. Yeah, so Will has a comment.
2011730	2014350	He says, I'm surprised a paper is the vehicle for this knowledge.
2014350	2016710	There's a whole security industry that has advice for this stuff.
2016710	2023010	And I think, yeah, this is part of what we're trying to do, right, with AI Village
2023010	2027670	is draw a link between security and machine learning, right?
2027670	2032750	So we have a lot of like wicked smart security people
2033370	2036690	who know about all about these implementation bugs.
2036690	2039990	And then we've got a bunch of like really smart academics
2039990	2042990	and machine learning people who know about like sort of these,
2042990	2046350	you know, feature based attacks or weird properties of ML systems.
2046350	2048630	And getting sort of that knowledge transfer going on
2048630	2051990	so that we can find out where those intersect
2051990	2055270	and how those two, you know, sort of sets of attacks inform each other.
2055270	2060270	I think that's like a really rich area for future development.
2060270	2065590	And it kind of makes me sad a little bit that this paper came out in 2017
2065590	2068450	and got comparatively little attention
2069650	2074130	when it really is going right to the heart of a lot of really big questions
2074130	2077650	in this, you know, in ML slash security.
2077650	2081710	So in practice, you see a lot of data science teams
2081710	2083930	disconnected from both like infrastructure application
2083930	2086090	network security sort of teams.
2086090	2091070	And so like the whole MLOps sort of formation of teams
2091070	2093310	that can interface between data scientists
2093310	2096070	and implementation has been emerging.
2096070	2097790	But I'm curious what the panel thinks as far as
2098730	2102250	what's the appropriate like role, a person, a cross-functional team?
2102250	2104730	Do we want to have security people sitting with data scientists
2104730	2107170	as these things are being developed?
2108270	2108730	And...
2108730	2111430	Yeah, I mean, weirdly enough,
2111430	2114570	I used to be one of the people going into large companies
2114570	2116730	working out where to put the data scientists.
2116750	2119450	And there was always this argument between
2119450	2122770	you want to embed the data scientists out into the rest of the team.
2123310	2125150	And have them work with because you're informing
2125150	2126550	the rest of the team everywhere.
2127710	2129870	But then you've got data scientists out on their own.
2129870	2132350	And we're kind of, you know, you see in this paper
2132350	2135490	that we get ignored quite a bit when we're on our own.
2135670	2138290	Or having these unicorn pens full of data scientists
2138290	2141290	who talk to each other, but not really to anybody else.
2141810	2144990	And there's this sense of you put people out,
2144990	2146790	but you make sure they're still part of TRIPE.
2146790	2148870	So you literally build TRIPE
2148870	2151570	or whatever the good word now is for that.
2151570	2153650	It used to be called TRIPE, but cross...
2153650	2154510	Village?
2154610	2155850	Cross, yeah, village.
2156450	2158850	So you build villages for the data scientists
2158850	2160070	to keep connected to each other,
2160070	2164330	but they need to be out in the rest of the teams.
2164770	2168270	Because otherwise, you know, our work is everywhere.
2168290	2169630	We should be everywhere too.
2170350	2171970	That's strongly our belief though.
2172510	2174550	Yeah, I mean, I think it's, but it's got to be
2174550	2175690	sort of a two-way street, right?
2175690	2179550	On the one hand, yeah, there is kind of,
2179550	2183310	for data scientists and sort of ML practitioners,
2183310	2186290	I think that, yeah, we do sort of tend to stay
2186290	2188090	in our own little bubble and don't think about
2189250	2192210	these sort of externalities.
2192210	2193450	I don't know what the right word for it is,
2193450	2195550	but like these other ways, you know,
2195550	2196710	it's not just going to be like
2196710	2198190	fast, radiant, sign-based attacks.
2198190	2201110	It's going to be like, no, someone broke the parsing.
2201110	2203870	But then at the same time,
2203870	2205250	and this is kind of going back to maybe
2205470	2207410	a little bit of a hype panel yesterday,
2208730	2210790	having people that aren't data scientists
2211350	2213750	understand what the models can and can't do
2213750	2215710	and what kind of behavior to expect from them
2215710	2217130	so that they don't freak out
2217130	2218930	when they see something weird happen
2218930	2221710	and be like, oh, you know, whatever.
2221710	2224430	We flagged this DLL as being malware when it's not.
2224430	2226530	We're clearly under attack, right?
2226530	2228250	And you're just like, it's statistics.
2228250	2229730	Sometimes it makes mistakes.
2230590	2233470	Well, just explaining it's not magic.
2233630	2235330	It's hard, but it's not magic.
2235330	2236570	Yeah, it's hard. It's useful.
2236570	2239210	It does cool stuff, but it's a tool.
2246530	2248370	Jager's talking about implementing
2248570	2249770	a security development lifecycle
2250350	2252630	and a secure software development lifecycle
2252630	2254590	into model creation and upkeep.
2254750	2256770	I'm going to make a data scientist cry.
2258570	2259410	Yeah.
2260030	2263650	I had issues with just being like, cool,
2263650	2265910	we need to just write a few unit tests
2265910	2268330	to make sure that the functions
2268330	2270730	all kind of do what they're supposed to do.
2271390	2272950	There's a bit of pushback
2272950	2277670	from our more academically minded folk
2277670	2281290	of just a little bit of basics like that,
2281920	2285430	which may have solved some of these open CV issues.
2289550	2292630	Interesting that if you basically treat
2293160	2295330	the lifecycle of a data project
2295330	2297890	as more of an internal product for a company
2297890	2300730	and treat it as a product which serves a purpose,
2300730	2303410	provides value, and then is integrated
2303410	2305770	with other ways of developing software products,
2305770	2307950	then data science and machine learning,
2307950	2310030	they just become tools in the engineering toolbox,
2310030	2311990	but then everything else falls within methods
2311990	2314950	of continuous integration and testing and delivery.
2314950	2316630	So then perhaps the risk,
2316630	2318810	rather than the unicorn, depends on data scientists.
2318810	2319370	Yeah.
2319770	2320690	That's what we're doing.
2320690	2325250	We're doing things like hypothesis-based development,
2325250	2326750	which is like the next thing up
2326750	2328830	from behavior-based development,
2328830	2332690	which is next thing up for test-driven development.
2332910	2335890	So you very much work out what the universe
2335890	2339330	you'd like to see is, go do the maths on it,
2339330	2340650	build the systems in it.
2340970	2343090	So there are ways of working.
2345090	2346350	Yeah, so maybe looping...
2346990	2347850	Sorry, go ahead, Sven.
2347850	2348790	Go ahead.
2349690	2351370	Yeah, so I was going to loop back
2351370	2352450	to the paper one more time
2353130	2355570	and talk about the disclosure experience
2355570	2358410	that they mentioned towards the end of the article.
2358830	2361790	So basically, three of the vulnerabilities
2361790	2365570	got new CVEs because they enabled
2365570	2366950	arbitrary code execution
2366950	2368570	or denial-of-service attacks, right?
2368570	2369730	So those are both very firmly
2369730	2371770	within the wheelhouse of what people...
2371770	2372550	Page seven.
2372550	2373490	Page seven, thank you.
2374270	2375510	Very much within the wheelhouse
2375510	2376910	of what security people think of as,
2376910	2378930	oh yeah, that's a bug, that's an issue,
2378930	2380930	that's a security gap that we need to fix.
2380930	2382690	But there were a couple of other ones
2382690	2385870	that they found where they could impact
2385870	2387350	or manipulate the prediction.
2387350	2388850	And then they call out in particular
2388850	2391070	the mouthier memory corruption they found,
2391070	2392730	where basically they can rewrite
2392730	2394450	the feature vector,
2394450	2397470	which allows you to basically
2397470	2398750	make the model give you
2398750	2400930	any output you want, essentially.
2401150	2403110	Doesn't have to have anything to do
2403110	2404510	with what the actual file
2404510	2406150	that you were analyzing was.
2406250	2408050	And those specifically are the ones
2408050	2412510	that didn't get a CVE,
2412510	2413310	didn't get called out,
2413310	2415730	and mostly got labeled as won't fix.
2415730	2417030	Or as won't fix.
2417030	2418270	So that's sort of like,
2418270	2419410	we were kind of bashing
2419410	2420870	the data scientists earlier saying,
2420870	2422350	oh, we got to think of this
2422350	2423650	as part of a security system.
2423650	2425770	But then it goes back the other way, right?
2425770	2427570	We need to be able to communicate to people
2427570	2428870	that as data scientists,
2428870	2429990	as machine learning experts,
2429990	2432570	look, this is bad, right?
2432570	2434510	And how do we convince people that
2435270	2436310	something that allows
2436310	2437790	arbitrary misclassification
2437790	2438790	or arbitrary predictions
2438790	2443250	to be produced is as serious as,
2443250	2444770	or maybe not as serious,
2445550	2447650	but has some degree of severity,
2447650	2448930	just like a remote code execution
2448930	2450710	or a denial of service.
2450710	2452390	I like the example that Eric brought up
2452390	2454190	in his talk of a turtle rifle.
2454370	2456110	Like that is a very effective way
2456110	2457930	of communicating potential problems.
2458430	2460550	Whereas for those who haven't seen it,
2460550	2463810	like a turtle 3D printed object
2463810	2465550	could be applied to a turtle
2465550	2468510	to make it misclassified as a rifle.
2468570	2469650	You can imagine like systems
2469650	2471690	that track security
2471690	2476210	and stuff to do certain events.
2477230	2478050	I mean, it's like
2478050	2479290	any other ethics discussion.
2479290	2480610	We have to talk about consequences
2481190	2482490	and then track it back
2482490	2484350	and just keep pushing
2484350	2485710	and pushing examples,
2485710	2487390	real examples if we've got them.
2488830	2490270	It's hard.
2491450	2494490	Yeah, I mean, I guess like that's,
2494490	2496410	you have a joke,
2496410	2497590	but we're not really a joke,
2497590	2498870	that like airline regulations
2498870	2500990	are written in blood, right?
2500990	2501790	And so when you've got
2501790	2503210	all of these safety requirements,
2503210	2504950	it's because a lot of people got hurt
2504950	2505890	before someone was like,
2505890	2507710	we ought to do something about that.
2507750	2510050	And it would be nice to think
2510050	2512690	that we could find a way
2512690	2516870	to not have like ML disclosure requirements
2516870	2518390	written in blood, right?
2518390	2519610	Like maybe, you know,
2520610	2521830	we're at a place now
2521830	2522810	where we could maybe like
2522810	2524390	jump ahead of the game a little bit
2524390	2527710	and get an understanding
2527710	2528330	of this out there
2528330	2530030	so that we don't have to see people hurt
2530030	2530910	before people are like,
2530910	2531970	yeah, we should maybe
2531970	2533290	take that seriously.
2533290	2534710	I actually have a historical example
2534710	2537010	for that too, for being old.
2537010	2539550	So I ran one of the
2539550	2541190	unmanned air vehicle safety teams
2541190	2543900	back in the day before UAVs were sexy.
2545230	2548030	And we didn't have the
2550870	2552230	privilege, I guess,
2552230	2554810	that the rest of the airspace industry had.
2554810	2556630	So aerospace regulations
2556630	2558090	are literally written in blood.
2558090	2559930	So an aircraft would crash
2559930	2560930	into something else
2560930	2562570	or hit the ground
2562570	2564690	and you would change the regulations.
2564710	2565510	And that's literally
2565510	2566990	how the regulations are written.
2566990	2568210	Same with fire safety.
2568550	2571270	Fire happens, change the regulations.
2573690	2578010	And we were told explicitly
2578010	2579230	and understood explicitly
2579230	2582010	that we couldn't just put UAVs
2582010	2583030	into manned airspace,
2583030	2584090	have them crash into stuff
2584090	2585570	and then rewrite the rules.
2586050	2587470	We literally had to think
2587470	2588290	all the scenarios
2589590	2591350	and write the rules
2591350	2593090	for how to do it safely.
2593610	2595090	And it's whole space safety,
2595090	2596950	not just individual aircraft safety
2596950	2598190	before we were allowed
2598190	2600050	to fly in the same spaces.
2601330	2603330	So there's some precedent.
2603350	2604670	I mean, there's maybe
2604670	2606610	some looking in those spaces
2606610	2608490	to see how it was done well.
2608830	2610330	That's a good way to look.
2611670	2613770	But a lot of it's just getting the will.
2614970	2615670	Now, nothing in focus
2615670	2616230	gets your attention
2616230	2617330	like a whole bunch of airliners
2617330	2618270	about to crash.
2618290	2622370	But yeah, we could maybe
2622370	2623770	get there before that point.
2626650	2629230	So one more question from Discord,
2629230	2630230	which is good
2630230	2631950	because that's where I wanted to go next.
2632430	2633750	So Yaga has a question.
2633750	2635350	How do we force open source software
2635350	2636230	to fix itself?
2636230	2637210	We don't fix it ourselves.
2637210	2638530	That's why it's open source.
2638570	2640170	And the authors actually make a comment
2640170	2641390	in the future work
2642290	2644970	where they say that it's unclear
2644970	2646050	who should be responsible
2646050	2647390	for fixing them as well.
2647390	2649730	So when they found a bug
2649730	2651630	in the Malheur feature processing,
2651630	2653310	that was because Malheur relied
2653310	2655170	on libarchive.
2655610	2657130	And the bug was actually
2657130	2658370	in libarchive.
2658370	2659190	And again, this gets back
2659190	2660510	to this notion of like,
2660510	2662330	you have all these different components
2662330	2664090	that feed into the system.
2665130	2666470	So even though the bug
2666470	2667550	was in libarchive,
2667550	2669150	it affected Malheur.
2669150	2671510	So where do you put
2671510	2673090	the responsibility for fixing it?
2673090	2674950	And how do you convince,
2674950	2676430	you know, like whoever's
2676430	2677490	maintaining libarchive,
2677490	2679690	look, this is serious enough that,
2679690	2680630	you know, yes, I don't think
2680630	2682070	it doesn't affect you directly,
2682070	2683710	but it does affect
2683710	2684970	this other system
2684970	2686210	and it could be a critical impact.
2686210	2687530	Like, how do we,
2687530	2688270	how can we navigate
2688270	2690590	that sort of uncertainty?
2692170	2695030	How does Linux and Ubuntu
2695030	2696750	and other open source
2696750	2699150	software ecosystems do it?
2702130	2703330	Do they have liability?
2703870	2705990	I mean, I know for a lot of systems,
2705990	2707310	the end users,
2707310	2708570	especially if they're big end users,
2708570	2709130	just get involved
2709130	2710230	with the open source communities
2710870	2712710	and work in there.
2712710	2715410	But if you're using
2716870	2718330	even like Python
2718330	2721050	and your physical system crashes,
2721590	2722730	who's responsible?
2726030	2728790	Yeah. I mean, a lot of licensing argument,
2729010	2730110	a lot of licensing agreements
2730110	2731350	basically say like,
2731350	2733330	yeah, you agree that you're using this
2733330	2734990	at your own risk too.
2735030	2737470	So, you know, basically
2737470	2738650	everyone's going around
2738650	2742290	not me, which, you know, it's fair, right?
2742290	2743850	And people have, like, they have,
2743850	2744830	they can't assume liability
2744830	2746510	for everything that they do.
2746510	2749590	But, you know,
2749590	2750770	they can't agree to fix
2750770	2752630	every single thing.
2752910	2754030	But at some point, right,
2754030	2755110	there's a balance, right?
2755110	2756430	If something's widely adopted,
2756430	2757270	it's widely used,
2757270	2758190	and there's like a bug
2758190	2759410	three steps up the chain
2759410	2761510	that's causing it to behave weirdly.
2761910	2764350	It feels like somebody ought to,
2764350	2765190	there ought to be like
2765190	2767170	some impetus to fix it somehow.
2767170	2768770	The low tail of companies like Fortune 500
2768770	2769730	is more bothered to use
2769730	2770730	these open source libraries
2770730	2772450	than in theory, you know, this is AI,
2772450	2773530	so you have a lot of, like,
2773530	2775410	boost in productivity,
2775410	2776370	like, and there's a lot of value
2776370	2777090	that's being derived,
2777090	2778070	and especially if they've started
2778070	2779570	to use it for critical applications
2779570	2780830	or decisions.
2780890	2782570	So I think the onus and responsibility
2782570	2783910	also needs to fall on the end users
2783910	2785410	of, like, widely adopted packages,
2785410	2786950	and then somehow making,
2786950	2788290	making it very clear to decision makers
2788290	2789230	in those companies that,
2789230	2790370	hey, if you're using this to
2791010	2792310	predict insurance rates or something
2792310	2793430	and it fails,
2793430	2795030	that's what you could potentially face,
2795030	2796530	and then somehow channeling that
2796530	2797830	into the projects themselves.
2798010	2799810	Are you proposing that large companies
2799810	2800730	that make a lot of money
2800730	2801830	using open source tools
2801830	2802730	should actually give back
2802730	2804270	to the open source community?
2808330	2810730	Smells like communism to me.
2811850	2813250	It's kind of terrifying
2813250	2814190	that we all just laughed
2814190	2815550	when you said that.
2817390	2818270	Yep.
2818330	2819550	Sorry, I got to get my,
2819550	2820770	got to get my commentary
2820770	2822070	in once in a while.
2823110	2824290	These are complex systems,
2824290	2825130	and, like, just communicating
2825130	2827190	the impact of some small,
2827190	2827890	like, overflow bug
2827890	2829430	to somebody who only cares about,
2829430	2831390	like, the bottom line is a problem.
2831390	2832930	Maybe there needs to be, like,
2832930	2834130	awesome ML failures,
2834130	2834870	GitHub repository,
2834870	2835490	or something like that,
2835490	2836630	where you can find it.
2837070	2838810	We haven't had this.
2838950	2840270	Sounds like, actually,
2840390	2841150	a pretty good weekend project
2841150	2841990	for someone.
2841990	2842870	Yeah, I mean, like, you know,
2842870	2843490	when we started up
2843490	2844830	the ethics stuff years ago,
2844830	2846050	it just, just takes
2846050	2846910	some people determined
2846910	2848590	to make this thing a thing,
2848790	2849810	a visible thing.
2849950	2850550	Yeah.
2850610	2851630	Hey, community.
2852030	2853550	Anyone want to take this one on?
2855130	2856370	So, how many, like,
2856370	2858210	solid examples of ML failure
2858210	2859510	do we have?
2861470	2862130	If we want to...
2862130	2863230	I mean, a lot of it is.
2863310	2864710	Compile those, yeah.
2864710	2866410	How much do we actually know?
2866670	2870490	Yeah, that's, I want to say,
2870490	2873030	Andrew Davis over
2873030	2874170	in different Discord
2874170	2876370	had a pretty good comment on it,
2876370	2877630	which is that a lot of time
2877630	2879230	when ML fails,
2879230	2881170	it kind of fails silently, right?
2881170	2883170	It just, you don't flag something,
2883170	2884730	and it comes back to
2885870	2886910	a lot of the earlier discussion
2886910	2888730	about is it a bug,
2888730	2890210	or is it, like, normal,
2890210	2890870	just sort of, like,
2890870	2892990	statistical failure, right?
2892990	2896070	A 1% failure rate means that
2896070	2898430	if it's not failing 1% of the time,
2898430	2900770	you should be vaguely surprised, right?
2900830	2903430	So, yeah, like,
2903430	2904230	I think it's possible
2904230	2906010	that there's a lot of ML failures
2906010	2908030	that we just kind of assume
2908030	2909570	are in the statistical noise.
2909570	2911590	A lot of them are probably kept quiet
2911590	2912870	because, you know,
2912870	2914270	there's potentially, like,
2914270	2916550	PR risks or PR damage, right?
2916550	2917990	Or, you know, like,
2917990	2919250	like, actual, like, gross,
2919250	2922370	you know, liability attached to it.
2923150	2926090	Yeah, I mean, it's an open question
2926090	2928490	how much is actually sort of
2928490	2930730	flying beneath the radar out there.
2930790	2932610	I mentioned a rental company saying,
2932610	2934090	or a background check company
2934090	2936130	who uses ML going,
2936130	2938510	so we didn't give a lot of people
2938510	2940630	their approved, you know,
2940630	2941710	their credit check approval
2942790	2944410	because of a mistake.
2944410	2945350	And we want to publicly
2945350	2946390	apologize for that
2947050	2948870	when it's an ML mistake.
2948870	2949670	And they can just be like,
2949670	2951310	well, the model just was,
2951310	2952610	yeah, sweep it under the rug,
2952610	2954830	fix it quietly, never mention it.
2957390	2959650	Like, that would be an ethical nightmare.
2961050	2964490	Yeah, I mean, there's lots of startups
2965510	2967590	that have proposed to do things.
2967590	2968910	There was one,
2968910	2970410	what was it, about two, three years ago,
2970750	2973830	oh, we'll use ML to pre-screen your babysitters
2973830	2977750	by digging through their social media profiles and stuff.
2978050	2980630	And I have my own opinions
2980630	2982710	on how likely that was to succeed
2982710	2985070	versus how likely it was
2985070	2989090	to simply just recommend white people.
2989090	2993610	But how do you even measure
2993610	2997230	or quantify failures in that case?
2997230	2998930	Because it's all going to be completely internal
2998930	3004470	and all you can do is essentially try
3004470	3006630	and get a proxy model to be like,
3006630	3008010	aha, here are the features, right?
3008010	3009590	I've submitted a thousand different profiles
3009590	3012610	and here are the features that seem to be triggering, right?
3012610	3016410	And you have to affirmatively dig for it, right?
3016410	3018050	Those failures aren't going to be obvious
3018050	3020170	unless you're actually doing something like that.
3021170	3023910	A recent streak of sort of facial recognition failures
3023910	3026470	that lead to characteristics
3026470	3030290	that elicits a visceral gut reaction like,
3030290	3031330	yes, this is wrong.
3031730	3033590	So I think in some cases,
3033590	3036170	just seeing the outcomes, like Claire said,
3036170	3039450	helps understand whether it's a failure or not.
3039670	3043130	This gets back to the bias question.
3043130	3044670	Like, how do you tell if something's biased
3044670	3049070	or just like for this one instance, it was wrong?
3049070	3049890	Oh, man.
3051530	3052890	Yeah, it kind of works.
3052890	3054710	We're going to have to clean that one out, Tommy.
3054770	3055170	Yeah.
3055170	3058210	In images is, oh, God.
3058910	3061690	How much longer do we have for this conversation?
3062530	3064550	Yeah, we can solve it in eight minutes.
3064550	3065070	We've got eight minutes.
3065070	3066130	Yeah, yeah, no problem.
3066170	3070050	Yeah, I mean, the problem in a lot of the bias stuff, right,
3070050	3073690	is like the terms aren't even defined sometimes.
3073690	3077210	So like the example I keep coming back to
3077210	3082030	is like recidivism and like predicting re-arrest rates, right?
3082030	3083290	And someone might point out,
3083290	3086670	hey, look, this is biased, right?
3086670	3088430	It's clearly predicting re-arrest rates
3088430	3090790	that are much higher for this group, right?
3090790	3094090	And one argument would be, well, that's clearly wrong, right?
3095150	3100130	It's biased because it's trained on biased data.
3100130	3101810	And the counter argument would be like,
3101810	3104270	look, it's reproducing the data, right?
3104270	3106090	We don't live in a perfect society.
3106090	3107570	The data is what it is.
3107570	3111170	And within that box, the ML has done as much
3111170	3112530	as you can ask it to do.
3113290	3119090	And so it always boils down to these normative questions.
3119090	3122330	And this is my own totally personal opinion
3122330	3127910	that talking about unbiased systems,
3127910	3132070	unbiased ML is almost inherently impossible, right?
3132070	3136850	It's a question of how is it producing
3136850	3139210	the kinds of outcomes that you want
3139210	3144690	and how is the ML system moving power around, right?
3144690	3147080	Who is it empowering? Who is it disenfranchising?
3147890	3151470	And who is it helping and who is it hurting?
3151470	3157890	And chasing after some vague notion of fairness,
3158170	3159810	a fair system or an unbiased system
3159810	3162170	is kind of like a distraction
3162170	3165550	from more fundamental questions of who's being helped,
3165550	3169330	who's being hurt, what's being reinforced
3169330	3171730	or whose concerns are being downplayed.
3172010	3173250	And who's accountable, I guess,
3173250	3175710	also because of the reference recent Twitter story
3175710	3177450	between Timothy Kimbrough and Yann LeCun.
3177450	3179170	Basically, is it the ML engineer
3179170	3180550	or is it actually the data scientist
3180550	3182210	or is it throughout the entire chain?
3182410	3182950	Yep.
3184630	3185590	Yeah, and I mean...
3186370	3187830	Sorry, go ahead. I'm talking too much.
3187830	3189970	I'm seeing over in the chat
3189970	3192470	that we already have awesome ML failures
3192470	3194270	in the AI village as a repo.
3194270	3199030	And Stellar Athena seems to be getting in there already.
3199710	3200690	Outstanding.
3200690	3202350	Suggestions for implementation failures,
3202350	3205710	ethical failures, HCI failures and security failures.
3206010	3207730	You got that Stellar?
3208130	3208930	Yeah.
3208930	3209790	Yeah, I think...
3210250	3212170	Do we have a category for...
3212170	3213530	Because we were playing around with
3213530	3216570	this cat does not exist a couple days ago.
3216570	3218270	I think we maybe need a category.
3218270	3220210	This ML should not exist.
3220370	3221810	Yeah, nightmare fuel.
3222810	3225090	Yeah, well, I mean, I'm thinking of things like...
3225090	3225590	Yeah, you're right.
3225590	3227290	There are a whole bunch of machine learning systems
3227290	3229070	that just should not be on this planet.
3229190	3229730	Yeah.
3229790	3231990	And anyone asked to contribute to it
3231990	3234650	should have been like, perhaps not.
3236370	3238810	Yeah, as far as the responsibility...
3238810	3239950	And we would never have had Facebook
3239950	3241790	if anyone had that call.
3241910	3242590	Anyway.
3242590	3243810	I mean, hindsight.
3243810	3248550	But you can look at really short-term stuff
3248550	3251870	like even just like genderify, right?
3251870	3253530	It was should have been pretty obvious
3253530	3255430	to everyone involved that this was just
3255430	3258130	like a losing proposition from the start.
3258930	3261090	Yeah, and the recidivism through face images
3261090	3262350	stuff from China.
3262350	3263070	Oh, God.
3263070	3264470	Yeah. Okay, so...
3264470	3266070	Way to replicate somebody's bias.
3266070	3270590	So we do have a workshop coming up
3270590	3271190	in a couple of minutes.
3271190	3272430	So I think I want to kind of like
3273470	3274550	put a pin in this.
3274550	3276650	I think if everyone's enjoying this conversation,
3276650	3279870	having an awesome ethics in AI panel,
3279870	3282550	which is coming up in a couple of hours.
3282550	3284850	I'm not sure someone fill me in.
3285030	3286510	Two o'clock, two o'clock.
3286590	3287530	Thank you.
3287530	3290170	So everyone should absolutely turn,
3290170	3290890	tune in for that
3290890	3292750	if they're enjoying this discussion.
3293210	3295810	But yeah, I guess I want to just like go around
3295810	3297470	just to wrap this up
3298130	3300290	and see if we have any like last minute,
3300290	3302190	like big idea takeaways from this paper
3302190	3304150	that people thought were really, really cool.
3304990	3306630	I love the way they did it.
3306650	3308470	The idea of attacking the system
3308470	3309850	and looking at taking it apart
3309850	3311250	is just kind of my thing.
3311250	3314850	And I like the idea of using it to attack systems
3314850	3317510	that shouldn't exist.
3318310	3320390	Yep. Sven?
3321050	3323270	I really like the idea of attacking
3323270	3324670	the feature extractors.
3324670	3326710	And I think we should also add
3326710	3328670	like bypassing the feature extractors
3328670	3329670	by just doing something
3329670	3334030	you know the ML system is going to miss.
3335110	3337370	Because you can understand it well
3337950	3340010	to this whole thing.
3341030	3342250	Cool. Martin?
3343310	3345070	Yeah, that also really enjoyed the paper
3345070	3345970	and discussion.
3346190	3349170	I think to me what was clear
3349170	3350670	is that it can occur at many levels
3350670	3353010	of the basically stack.
3353010	3355150	So just instrumentation and stuff
3355150	3356230	become very, very important
3356230	3357390	to monitoring what happens.
3357390	3358570	And I would really like to see
3358570	3360510	follow on work that applies this
3360510	3361870	to TensorFlow, PyTorch,
3361870	3362910	basically the learning libraries
3362910	3364450	and seeing how you can play
3364450	3366310	with intermediate representations
3366310	3367390	and things like that too.
3368410	3369630	Yeah, for sure.
3369630	3370850	Yeah, the reminder that like
3370850	3373570	these ML things, ML projects
3373570	3376430	don't exist in some abstract theoretical space
3376430	3377990	and are implemented in real code
3377990	3379450	and real systems that come
3379450	3380770	with their own flaws.
3381190	3382890	Yeah, like you know it
3382890	3384150	but being reminded of it
3384150	3385170	sort of viscerally by,
3385170	3386350	look, we have CVEs
3386350	3388250	is always a great thing.
3388530	3389970	So cool.
3390050	3391050	So I think this is
3391050	3392550	we probably want to leave a few minutes
3392550	3393970	for people to fight with the audio
3393970	3396150	as we transition over to the workshop.
3396170	3397670	So I think we want to
3397670	3400230	call that a panel or a discussion.
3400230	3401890	So awesome. Thanks, everyone.
3401890	3402330	Thanks, Rich.
3402330	3403170	One second.
3403450	3404910	Before we jump off,
3404910	3406550	we have one on Wednesday
3406550	3409030	at 5 p.m. Pacific time.
3410510	3413310	And we have the, I'm guessing,
3413310	3415310	I think we should go with your paper, Rich.
3416310	3417510	The slide deck on...
3418770	3420470	Yeah, the anomaly detection
3421030	3422030	outside the closed world
3422030	3423010	on using machine learning
3423010	3424290	for network confusion detection
3424290	3427470	by Robin Subban and Vern Paxson.
3428050	3431230	So we'll post that in the Defcon Discord
3431230	3433710	and also be posted to our Discord and Twitter.
3433710	3435470	So if you want to participate,
3435470	3437930	you should find your way over to our Discord.
3437930	3441130	We can't post a link to that in the Defcon Discord,
3441130	3442170	but the links are places
3442170	3444530	and you can just DM us for an invite.
3444770	3445930	And we'll be discussing that
3445930	3449610	for probably an hour and a half on Wednesday.
3449610	3451310	And we might be able to get one of the authors in,
3451310	3453650	even though this is a 10-year-old paper.
3456050	3457750	How far we have come.
3459130	3460010	All right.
3460010	3460730	Thanks, everyone.
3460730	3461390	Catch you later.