start	end	text
1520	14320	Now we have some audio. Okay, so everyone somewhat speak?
22950	25910	Okay, now try again. Yeah, now we have audio.
26150	28810	Hello, good. Can you hear me, Sven?
29310	30990	I can hear you. Hey, Irem.
31150	35530	Oh my gosh, it was just a simple Windows reboot fixes everything.
36910	41130	Hey, I unfortunately, I actually have a meeting right now. So I'm going to drop and join again.
41130	44510	What time do I need to be on for speaker time?
45010	46510	In 30 minutes.
47510	50630	But only in Discord or also in Zoom?
51110	52330	Only in Discord.
53310	54190	Okay.
56070	60810	Yep, so we've got people in the Slack chat or in the Twitch chat saying they can hear us.
60810	61950	So I think we're good to go.
62330	63110	Sweet.
64070	67310	Well, hi, everyone. Sorry about the audio issues.
67310	70430	That seems to be the start of every stream is figuring out the audio.
71790	74910	The only people I've seen get it right are like Critical Role.
77730	80290	They pay a lot of people to get it right.
80490	86910	Yeah. Welcome to AI Village at DEF CON 28 safe mode.
88010	89990	We're all doing this from home.
92750	96930	I hope that this can be as fulfilling to you guys as it is to us.
98690	106430	And we just want to start off the village with some like general questions about the
106430	113730	state of the field as it is right now at the end, halfway through 2020.
114430	119710	So Jag has got a bunch of questions prepared and the rest of us are going to try to answer them.
121090	122010	Okay.
122890	124370	Welcome, everyone. Thanks.
124370	125890	Thank you for being here.
125890	129510	The first question that I have listed is...
129510	133430	Let's just knock this one out of the way real first.
133670	137610	Facial recognition is a security concern for us.
138110	140490	In the ML field, of course.
141090	144070	And before you answer, please introduce yourself.
144750	147950	Yeah. We should probably do a who is who.
149530	151990	Sven, do you want to kick us off?
153050	153990	Sure.
153990	155210	So I'm Sven.
156110	159850	I am a senior data scientist at Elastic.
159850	166870	Before that, I was a topologist who accidentally got into machine learning and security.
167850	172370	And I kind of helped run the AI village.
175010	177030	John, do you want to go next?
177230	178070	Yeah, sure.
178070	178790	Hey, everyone.
178790	179770	DeltaZero here.
179770	182770	I do data science for intrusion detection.
182770	187770	Yeah, I've been in the field for maybe, I don't know, four years now.
187770	189890	But pretty awesome.
189890	191110	Glad to be here.
194110	195270	Rich?
196170	197910	Hi, I'm Rich Huerang.
197970	201690	I'm a director of data science research at Sophos.
201690	208310	I've been working at the intersection of machine learning, security, and privacy for probably
208310	210250	about 10 years now.
210250	212250	Got my PhD from UCSB.
212250	217750	Spent some time with the US Army Research Laboratory, and I've been at Invencion Sophos
217750	219050	ever since.
224460	225600	Yaga?
226040	227740	Hi, I'm Yaga.
227740	232280	I am a professor at a small university in Virginia.
232520	242080	I've got about 25 years of experience in security, but relatively new to machine learning.
242080	250180	My research is focused on securing machine learning, and I like playing video games too.
250180	253940	So there's something interesting about it.
254200	254930	Anita?
255640	257080	Hi, I'm Anita.
257080	259360	I'm a research scientist at University of Illinois.
259360	261500	Started life as a cryptographer.
261500	264440	Been in security for almost 30 years.
268200	271120	Okay, and Will's the last guy.
272360	274400	Certainly not least.
274400	275720	Yeah.
276560	279280	Sorry, my morning's just starting.
279380	290800	Will, I am a network operator, and I guess attack machine learning models and use AI
290800	293260	for offensive purposes.
293260	296600	Phishing, pathfinding, and stuff.
298360	299760	Yeah, Will's being modest.
299760	305860	He actually has the first CVE ever assigned against a machine learning system.
305860	311240	If you attend my lab later, there'll be another one.
313900	316960	I think that's a challenge for everyone.
318540	322000	Can you get one file before Will drops number two?
325560	328460	Okay, thank you everyone for the introductions.
328460	330820	Sorry, I should have thought about that.
331120	335740	Please be sure to ask any questions over in the Discord server.
335740	338320	There is a talks channel.
340800	343420	Yes, there's a channel for talks.
343420	348000	You can ask in there or in the general, and we'll try to catch it.
348100	350420	So back to the first question.
350620	356100	This was one that was both easy and hard to throw out there.
356100	366100	But is facial recognition a security concern when we're talking about privacy?
366800	371000	For sure we know about, but in the general machine learning terms,
371000	373500	is facial recognition a security issue?
374880	378960	So I think just to make the point in our village, one of the things we're trying to do
379540	384320	is expand from just thinking of this as a technical kind of algorithm problem.
384320	388860	If you look at the broader scope of facial recognition, there are issues not just with
388860	393040	security, with data sets, with data integrity, with these kind of choices.
393440	396480	So I think my answer would be it is a security issue.
396480	397540	It's a machine learning issue.
397540	398200	It's a data issue.
398200	399480	It's a human issue.
400280	405000	And for us, we need to think about all those things as technology people.
407180	411980	Yeah, and just to add to that, a lot of the issues around facial recognition
412980	418520	are less to do with the technology and more to do with the processes that that technology is
418520	420480	embedded in.
420480	426220	So if you have a facial recognition system that works, you have issues around privacy, right?
426220	427840	Who gets access to that system?
427840	430560	Who gets to know who was seen where on which camera?
430700	433740	And then when it doesn't work, you have bias issues that sneak in.
433740	436360	So how do you control for false positives, right?
436360	442140	Someone gets reported as being, you know, facially recognized as a bank robber or something
442940	444920	when it's just like an average person.
444920	448000	And this actually extends beyond issues of just facial recognition.
448000	450880	Just computer vision in general has problems like this.
450880	456020	There was a story recently about a license plate recognition system, which led to someone
456020	460460	being pulled over and detained, you know, face down on the side of the road for hours
460460	464180	before they figured out that it was the wrong license plate number, the wrong make of car,
464180	467000	the wrong model year, et cetera, et cetera, et cetera.
467000	473360	So yeah, it's definitely a case of not only does the ML work, but do you have the right
473360	477940	controls around the ML to make it actually usable?
477940	482180	And have you, you know, assessed what happens when it actually fails so that it can fail
482180	484020	gracefully without hurting people?
485560	486840	Anyone else?
487320	491540	I think the last thing, not to dominate this, but the last thing to throw in is I feel like
491540	497560	in our village, part of our job is to also translate all this technobabble sometimes
497560	500800	to policy speak and to real world speak.
500800	502800	And that could be very challenging.
502800	504820	We know the technical parts.
504820	509000	What ends up on the other end, we need to bridge that gap.
509680	515920	I think that's important to understand that security isn't just the technical side.
515920	519220	There's a human side to it too, and that extends to policy.
519220	521140	So definitely something to think about.
521140	521520	Thank you.
522200	528740	Staying along the same lines, one of the things that I wanted to ask about is the FOX program.
528740	530620	Have you heard about it?
530620	544180	And how does it affect our ability for the end user to take back some of that control
544940	546440	over their data?
548560	554140	It's technically, again, this is, it's the gap between technical aspects of it and
554140	556380	implementation and social aspects of it.
556380	558700	Technically, it's a brilliant idea.
558700	561760	And, you know, it does what it says on the tin.
561760	567780	In practice, there are so many ways that people get registered into facial recognition systems
567780	574320	from, you know, collecting ID cards to photos that they uploaded before FOX was a big thing
574320	583520	to photos that other people take of them and maybe tag them in later, that you really can't
583520	589480	just find a quick technical band-aid for security problems that are really a combination of
589480	591260	social and technical problems.
591260	595240	The root cause here isn't like, oh, we just need better technology to fix this.
595240	601880	The root cause is society and sort of laws and norms haven't kept up with the technology.
601880	605200	And that's the angle that we need to address it from.
605200	609060	Technical band-aids are only ever going to be available to people that know about them,
609060	613820	know how to apply them, and can get over that barrier to entry.
613920	617360	Really, we need to be thinking broader about impact to society as a whole.
618160	623360	It makes me think a little bit of this field of, you know, usable privacy and, you know,
623360	624980	privacy by design.
624980	632700	See, that's analogous here because people don't understand that long-term impact of
633300	634540	having a photo op.
634540	636880	So, enabling people to use this tool is great.
636880	641100	I think the average person isn't going to even think about it because it doesn't affect
641100	642460	their day-to-day life.
642460	648980	So, again, I think the issue is, how do you express to them the long-term potential damage
648980	651280	of having their pictures out there?
652140	657460	Okay, we're going to move into a different arena.
658320	666680	What do you say the latest advances that your typical security team could use
668280	670360	in terms of machine learning?
671380	676040	You know, a lot of vendors are out there saying that this is going to be the panacea for them
676040	677880	to be able to fix things.
677880	684600	Some of the actual products that will help them or techniques that will help them.
686440	691940	I just want to take this opportunity to plug the panel that we have coming up on Saturday
691940	695540	on whether or not ML and InfoSec lives up to the hype.
695540	699640	So, we're going to be diving, hopefully, pretty deep into those questions.
699640	702680	But I'll kick it over to someone else since I've been talking a lot.
707220	712500	I really don't know what the boots on the ground could be using.
713320	717860	But there are some promising things coming down the pipe from researchers.
719560	720680	But I...
722540	725960	So, I'll jump in real quick. Sorry to dominate again.
725960	728600	And I'll let Will speak because he's more an operator.
728720	733600	For me, federated learning, federated and privacy-preserving machine learning is very
733600	735520	important for a bunch of reasons.
736020	740440	Not the least of which is being able to learn across edge devices that are in multiple disparate
740440	742740	places and being able to do it more efficiently.
742760	747840	And being able to use new types of data because there's privacy-preserving ways to do it.
747840	753320	So, I think I see federated learning as something that's promising.
754780	760040	Yeah, I'd say from a product standpoint, it's just like the simple stuff.
760040	769780	So, you know, on a network, clients will still miss password spraying internally.
770020	774520	And that's just something that shouldn't be happening with the logs that are available.
774520	782420	So, I think in terms of just instant response and stuff on networks, it's the simple stuff
782420	786540	that is going to give you the most bang for the buck.
789120	794740	Obviously, there's a lot of competition in the machine learning space, or it would seem.
797440	804140	And half of me, at least 40% of me, thinks in two years, we're going to be moving on
804140	808160	from machine learning as a solution in networks.
808160	812700	And we'll probably replace it with like zero trust or something like that.
814080	819120	But there's also the other 10% that thinks like it will always have a place.
820020	827980	But just like with the facial recognition stuff and the bias, if we don't put it out
830680	835480	in a way that's useful, and in a way that people can understand,
835480	841100	the minute it breaks, they're likely to just throw it out the window and say it doesn't work.
842700	849820	Even though, you know, it is a perfectly viable solution. But yeah, I'd say simple stuff.
850320	851580	Yeah. Okay.
851820	855600	To me, that's the more important side of the question is like,
855600	860180	practitioners on the ground, we don't need any new math or anything new interesting.
860180	864660	But there's some basics that they can get their hands on that will probably help them out when
864660	870100	they have to deal with a deluge of alerts from some sim that they have to deal with.
870100	872680	A little bit of clustering could go a long way in that situation.
874020	876440	Yeah, I think that's perfect.
876660	881220	That's perfect for what I was going to say is, I didn't mean to say products,
881220	887800	I probably wrote that down wrong. But like, what can we tell our audience here that maybe
887800	893860	practitioners that they can do on their own? So I think something like our workshops,
893860	896420	if you take a look at them, they'll be able to help you
896420	900840	see that you may not need to go get these expensive products. There's some simple things
900840	904520	you can do. Yeah.
904620	909100	Yeah, and I just add to that, but from my own experience as someone who's
909700	915180	used ML to tackle a lot of security problems, very often you don't even need like the big
915180	920920	complicated deep neural networks or gradient boosted decision trees or pick your flavor of
920920	927200	the day. Very often, it's enough to have a good idea of the problem you're trying to solve and
927280	931760	a good set of representative data and then simple methods like, you know, linear logistic
931760	936740	regression will often be enough to tackle it for you. So, you know, you don't need to think that
936740	943020	you've got to have tons of like Amazon EC2 credits to run a big beefy GPU machine.
943020	948020	A lot of times you can get really like 80% solutions, really good solutions just on your
948020	951820	laptop. Okay.
954940	960380	For most security to be able to defend yourself, you have to be able to identify
961000	968920	the issue or identify the attack. We discuss, you know, adversarial attacks quite a bit
969520	977320	in the academic realm. How could someone tell that they're seeing an adversarial attack in practice
978320	986870	today? You mean on the AI? I'm throwing out some hard ones. You mean on the AI portion?
989390	996430	Well, how could you just notice an attack? Because to defend against something,
996430	1000330	you typically have to be able to define what that attack looks like, right?
1000930	1004970	At least to some degree. Can we do that today?
1007350	1012570	It's kind of too bad Hiram's not here because I feel like he'd have a lot to say on this topic.
1015410	1016770	Let's hear it.
1017510	1024690	So for the traditional adversarial attacks, we basically are trying to modify an image
1024690	1029330	or something slightly to bypass the machine learning system. That doesn't really work
1029330	1034150	with security because our data is much more structured. You can't like do a gradient descent
1034150	1040570	on a binary and then hope it still compiles or runs because you're going to get byte codes that
1040570	1048570	are just wrong. So I don't think like adversarial attacks as academics that you read up in a big
1048570	1053230	year paper are going to really apply. There are some modifications to that that might apply that
1053230	1060370	might be very helpful. But what we do see already and have been seeing for like a decade is people
1060370	1066090	just trying a bunch of stuff and then they get past it. And the more they get informed they are
1066090	1071710	about how the machine learning model works, the more effective and more like gradient descent it
1071710	1078610	is. You know, like the more effective they are. Like for like the spam filters from back in 2001
1078610	1086290	when people were packing Wikipedia articles at the bottom of their spam messages to bypass the
1086290	1090050	naive Bayesian classifiers, they understood that it wasn't a naive Bayesian classifier. They
1090050	1096270	understood that they needed to pack the thing. And then they did that. And that's going to happen
1096270	1107590	with next-gen AV products too. It may already be happening. I mean, that was what we saw in
1107590	1115430	the silence bypass was effectively a stuffing attack. Yeah. That was researchers being nice to
1115430	1126730	us and telling us about the problem that's happening. So this one is more directed at Will,
1126730	1131650	but definitely want everyone's opinion. And it gets back to something we talked about in a journal
1131650	1137710	club, which for those of you that might not have seen it, we may be replaying it during the weekend.
1139230	1147470	Will received a CVE for the vulnerability he discovered.
1147470	1157110	But do we need actual CVEs specifically for machine learning or should we keep them more general?
1159090	1165990	So I guess first I'd like to say it was our, so Nick Landers and I did it.
1166550	1174550	I just talk about it more so. So the CVE system is designed to communicate risk.
1175310	1183330	And I think anywhere that risk is present, then a CVE is appropriate.
1186150	1191510	You know, I just, I submitted it. So I'm not the one that gets to say whether or not it's a CVE.
1192270	1199170	So I guess that the question would be for the NIST. And I know they've come out with a paper
1199170	1205910	that kind of highlights machine learning attacks or there's a taxonomy. It's like 15 taxonomies
1205910	1215470	right now. But yeah, I think in any way there's risk that is at least measurable, a CVE would
1215470	1221310	be appropriate or any standardized score. I know there's operators like...
1222130	1229630	And I'm sorry, I don't mean should it be a CVE, but more should there be a class just for machine
1229630	1235810	learning, you know, because they break it down. Yeah, yeah, I think so. It's fundamentally
1235810	1243390	different. You know, so for example, like with a traditional exploit, you know, there's like
1244170	1251750	a patch and then you can push it out. But with, for example, like with Proofpoint's model,
1251750	1258270	there's some residual, there's some like lingering theft going on. So
1258990	1265230	once you have that copycat model, you're never going to get 100% of their model. But
1266590	1273250	for some, there's some decay into the future, but you'll have some portion of that model.
1273250	1280910	Your copycat model will be right, you know, up to some future time until they've retrained enough.
1280910	1288930	But in their response, you know, unless they do things to fix it or prevent, you know,
1288930	1296610	people from stealing outputs, then you can always create a new model. So I think that
1297210	1305050	piece is important. But yeah, it's a little, it's fundamentally different beast than
1306610	1310990	than like an exploit. Yeah, I don't know, you guys probably have a better idea of like,
1310990	1316550	how you're supposed to patch a machine learning model or you take it out. I don't know.
1317510	1320250	Long and expensive retraining process.
1321130	1327210	Do you guys agree, though, that it's good to have that separate machine learning
1327210	1333550	kind of category to highlight the fact that this is a vulnerability, a specific type of vulnerability?
1335270	1342450	Yeah, I think most, so there are some exceptions. And, you know, again, I'm going to sort of like
1342450	1348890	promote the Journal Club on the Summoning Demons paper later this weekend. But most machine learning
1348890	1355250	exploits don't translate into exploits as we usually think of them in InfoSec. They are much
1355250	1362450	more either model stealing or model bypass, or maybe like a denial of service type thing. So
1363570	1368170	in my mind, keep in mind, I'm coming at this from the ML side of things and much less from the
1368170	1373130	InfoSec side, I do think it makes a lot of sense to have separate categories that reflect the
1373130	1379430	different kinds of risks that we currently think of ML models as being exposed to.
1379870	1387110	Okay. So that's kind of all the questions I had. The next was, I was going to turn it back over to
1387110	1392510	Sven to kind of give you an overview of the weekend. And thanks for being here, everyone.
1392510	1398510	And thank you for your questions. Or answers. Yeah, cool. I'm really looking forward to the
1398510	1407310	talks. This is going to be a great DEF CON. Cool. I'm out. Later, folks.
1407790	1415010	Have fun. So about this weekend, the way we're structuring it is everyone, all the talks are
1415690	1420630	recorded already and on my desktop. So the speakers won't be talking live. The reason why we're doing
1420630	1427090	that is so that the speakers can focus on answering your questions in Discord. And so during the talk,
1427090	1432110	speakers will be in the Discord and hopefully afterwards. So any questions you have about the
1432110	1437850	thing, you can just ask it in the Discord channel, and we'll answer it. The speaker will answer it,
1437850	1442710	or maybe someone else who's knowledgeable on the subject will help try to answer your question.
1445090	1452250	So we're going to, in a few minutes, launch into the first one, which is we've got Zoltan here.
1456010	1464330	Zahak, who's in Hyrum. They'll be going over to the Discord to answer any questions.
1465830	1471230	And then, yeah, then we've got Eriks and a few others. Oh,
1471810	1474670	Sarah, do you want to introduce yourself, too?
1476710	1484770	Oh, god. SJTurk, ethics chair, so I guess the buck stops here.
1486530	1490050	I guess data nerd who's working on disinformation.
1497600	1502100	I think that's it for the intro stuff.
1502100	1507120	Yeah, that's it. We were trying to leave some time to make sure the transition to Hyrum.
1507650	1512080	Yeah, I've got that all ready to hit play.
1512620	1519360	Okay, yeah. Oh, workshops. Yeah, so there's a few workshops.
1520630	1527550	They're mostly focused on sort of bypasses and interesting techniques on...
1528270	1539110	The hard part for us is text data. So, GtKondike, Gavin Kondike, well, has a lovely workshop on
1542990	1554750	spam builders. And we have... Will here is talking about a new, basically, CVE2.
1557590	1559650	Model stealing stuff.
1562930	1572030	And the way those work are they'll be streamed on Twitch live. They're not recorded. And then
1572030	1578790	there'll be a few people who are TAing who will help the workshop presenter help relay questions
1578790	1585170	from Discord over to the workshop presenter and help resolve any minor issues you guys might have.
1586330	1591270	There are Jupyter notebooks and stuff posted on the website.
1593070	1601250	So, hopefully this all works out. It's a pretty normal machine learning workshop structure.
1601570	1607570	Just online versus in person so that you can't have someone spy over your shoulder to tell you
1607570	1618050	your typo is. Well, with that, let's start Hyrum's...
1622410	1628170	Okay. I'm gonna go ahead and drop off. Hi, everyone. Have a great DevCon.
1628290	1632190	We'll do. Have fun. Awesome. Thanks all.