start	end	text
3070	18060	Okay. Sorry. Sorry. I just figured it out. The audio should now be working.
18920	20920	Seriously. Okay. Are we...
22080	26980	So, on Twitch chat, I am so sorry. We should have this all sorted now.
26980	31280	I found out what caused the feedback loop.
31280	40520	Okay. So, on Twitch chat, can you please give us a thumbs up?
40540	45920	Okay. Thumbs up if the audio is not working for everyone. So, you can hear Will,
45920	48460	and it's not doing the weird Hall of Mirrors thing.
49080	50040	Yay.
51660	54140	I'm terrified to open Twitch at this point.
54140	62700	Yeah. Cool. I'm muting myself now. The audio is now working. I am adding everyone's names.
62700	65600	So, I am so sorry for everyone.
66880	70660	This time, if it doesn't work, I'll off myself with a sword in the background.
71760	72720	Okay.
73000	74780	Okay. So, have at it, Will.
74780	80940	It's what the people want. Rich, whoever I am this week. Okay.
81780	88680	Well, last time. Last time through. So, again, welcome, everyone, to the Hype in AI panel,
88680	92980	where I'm going to talk about does AI in InfoSec actually live up to the hype?
92980	96440	I'm not going to pitch the topic again, because I'm tired of it.
96440	99820	So, Ariel, please introduce yourself one last time.
99820	104640	All right. Hello. My name is Ariel Hirt-Voss. I'm a research scientist at OpenAI,
104640	108620	where I work on misuse and abuse of the AI technologies that we develop there.
108620	112360	I'm also a PhD student at Harvard University, and I've been working on a book for No Starch Press
112360	118080	on how to do red teaming for AI systems, specifically. And that's most of what I do.
119120	126760	Cool. My name is Rich Huerang. I am a data scientist at Sophos. I have been working
126760	131280	at the intersection of security, machine learning, and privacy for about 10 years.
131280	135660	I have a PhD from the University of California at Santa Barbara. Will?
138850	144710	My name is Will. I am a network operator, and I work on building tools,
145390	149950	AI tools for red teaming and attacking machine learning systems.
150810	152370	Very cool. Masuda?
153470	160130	My name is Masuda. I'm a data scientist at Rapid7. I think a lot about incident detection,
160130	165250	anomaly detection, and I'm a relative newcomer to security. I have a PhD in physics and
165250	168690	spent some time in quantum computing research before formally switching over
168690	174190	to data science and now security. Awesome. Rob?
175270	180550	I'm Rob Brandon. Been working in the tech and security field for probably about 25 years now.
180550	184210	Currently working for Booz Allen Hamilton as a researcher and threat hunter.
184410	189830	And about eight years ago, I was doing some sock work. I got a little bit of a exposure to this
189830	194450	machine learning stuff and went back to school and got a PhD from the University of Maryland,
194990	198850	and been working in the intersection ever since.
199870	203390	Awesome. And last, but certainly not least, John.
203990	209530	Hey, everyone. I'm John Seymour, aka DeltaZero. I work as a data scientist at Salesforce,
209530	215150	where I do intrusion detection on the DNR team. I also like to just kind of put my hands in
215150	221710	everything, so I've touched a lot of, you know, offensive security and data science and all sorts
221710	226550	of things. But yeah, mostly currently do intrusion detection.
228230	233650	Awesome. So again, thanks to everyone for being here and taking part in this discussion.
233710	239550	I kind of want to kick it off with a very sort of like high-level first question.
239910	244210	So machine learning has actually been used in a wide range of in-physic problems now. So
244210	249130	we've seen it in malware detection, where it's probably sort of the biggest success story.
249130	253010	But then spam blocking is probably the oldest field it's been used in.
253010	258690	And then other stuff like network traffic classification, analyst alert ranking, and triage.
258770	264830	So what, if anything, do we get as sort of a value-add above and beyond sort of like the
264830	272410	old-school tried-and-true signature-based or rule-based techniques for security? And so what
272410	277910	I'm kind of hoping we can dig into here is like, what are the strengths and weaknesses of AI or ML
277910	284050	or whatever you want to call it? And are there classes of problems that we should always be
284050	289670	using ML for or maybe like should just completely give up and never waste time and money? And are
289670	293970	there specific areas where if a vendor says, oh yes, we can solve this problem with AI, right,
293970	297270	that immediately sets off a red flag in your head that they might not know what they're talking
297270	304690	about? So open question, please dive in. Or I can call on somebody.
306310	310890	So for intrusion detection, because this is what I spent the most time thinking about,
310890	317290	I think the class of problems really depends strongly on what the data makes available.
317690	321850	And just as an example for how I've approached the problem, when I first started,
321850	325890	I thought, you know, we're awash in data, everything's digital, records are there for
325890	331330	everything. And I viewed my role as analogous to the people that study volcanoes by looking at
331330	335450	very small earthquake signatures, and have gotten really good at predicting these eruptions.
335450	339050	And I thought, yeah, that's what I'm going to do. Can anything be more glamorous? I'm going to
339050	343970	really understand these signatures and use them to predict, you know, incoming attacks.
343970	350530	I very quickly realized it's hard to get a labeled data set of incidents. People don't want to share
350530	356010	when they've been compromised. And so that's a problem. But even if I were to get a library of
356010	362770	these attacks, attackers behave very differently. So I'd have to get a representative set of
362770	368510	different attacker behavior. And also, they're opportunistic. So it really depends on the very
368510	372710	specifics of network configurations and how different companies, you know, set up their
372710	379110	assets. So really, now I'm talking about simulating different types of incidents over many different
379110	384850	types of attackers over tons of different environments. So that very quickly put me off
384850	392730	the classification use case, let's say. But there's still a lot of data. And so I think
392730	398470	anomaly detection is great. We can baseline a lot of things. The volume-based baselining is
398470	403030	maybe straightforward and not that interesting. But I think there's an art in defining what are
403030	408490	the attributes that we expect to look really regular in reference to particular intrusions.
408490	413790	So I think that's a really useful way to go. I'm very suspicious if someone were to tell me
413790	419070	I've got a method to, you know, precisely detect a compromise for you.
419070	423430	So you're mostly thinking about network traffic in that case?
423990	429070	Network traffic, let's say endpoint data, you know, logs from endpoint data.
430250	437010	Cool. Anyone else? I guess, Will, I'd really be interested to hear your input on this,
437010	440730	since you work on the other side of the problem a lot of the time.
440730	447510	Yeah, yeah. I don't come at it from a machine learning area. I'd say the nice bit is scale
447510	455770	and building context around an event. I think that's really helpful. You know, I think the
455770	461050	people who have been doing incident response for a long time, you know, they have the knowledge.
461050	469810	They just don't necessarily have all the pieces to find, you know, a direction. So, you know,
469810	476280	we always, I think as data scientists, or as the data is coming across, it's often incomplete,
476990	482130	where you don't know what you need to solve the problem. So you only have, you know,
482130	487710	half of the information that you actually need, and the rest is kind of fuzzy. But you know,
487710	492170	maybe a direction that you're trying to go with it. So I think machine learning can help pull out
492170	498710	those insights. But you know, I think it requires that still like a human, a human touch, a human
498710	505290	direction. I think that is probably pretty limited as we go forward in time. But currently, where
505290	516990	it's at it, you know, it's still pretty ineffective in networks for detections and things. And you
517630	525470	potentially the other use cases. So proactive, secure configuration, it's like, out of this
525470	531210	machine learning thing, the only thing we've got is just more alerts. I think that that may remain,
531210	536370	you know, and people can't necessarily trust them. But I do think it's going forward in time,
536370	541350	you know, it's going to be a very, very different use case, especially as you get data scientists
541350	547450	with data sets that, you know, they start to understand. And, you know, we're all on the same
547450	553390	playing field. So, you know, once we're on a Windows host, for example, everybody has access
553390	557510	to the same information, more or less. And it's really just about how you're viewing that
557510	565310	information inside of a given context, I think. Yeah, so one, one thing that I think kind of
565310	571610	popped out from both those answers was, there's a lot of like human tuning and human feature
571610	578030	extraction that's going on in there. So the, you do see a lot of marketing literature about, oh,
578030	583190	just, you know, dump raw bytes in and detections, magical detections come out the other end. So
583190	588350	any, any thoughts on claims like that? Do you really need a, like, is the human element,
588350	593950	like mandatory still at this point? Or is there any path? No, I don't think it's mandatory.
594370	601910	So for example, any given user is going to operate on a host, you know, within some range.
602230	606650	So I think the anomaly detection is really nice. So let's say you have 1000 users,
608410	612230	998 of those users are going to come in, log in at 8am every day, they're going to
612230	618110	visit the same things, you know, then you have that 2% that are, you know, doing something that
618110	622910	they're not supposed to be doing or whatever it is. So I don't think you necessarily need
624530	631030	to label data, but you, you need to baseline your network. And I think the issue for machine
631030	636070	learning is when you try and scale it across, let's say 10 different companies, that's going
636070	643110	to be really difficult. So but I think machine learning is such that you could, you could
644510	651670	build a network for each individual network, it's just going to take time. But from a vendor's
651670	656590	perspective, like that doesn't scale as well. And so I think it gets a little more difficult
657450	658510	in that way.
660770	668910	I kind of disagree with the anomaly detection approach, just because every network is just
668910	673650	full of anomalies when you start going to look for anomalies. And while it may just be 2%,
673650	678230	that 2% is still going to be the source of, you know, to, you know, millions and millions of
678230	679070	alerts.
680290	686690	Yeah, yeah, right. I should say, in specific cases, like, let's say log on, log on times,
686690	693450	so, or if, you know, host where users are logging on. So yeah, you're absolutely right. I think
693450	698910	anomalies are everywhere. But in terms of malicious events, or potentially malicious
698910	704210	events, I think you could cut it down to a little bit where you can get some high fidelity.
704230	708010	You can narrow it down, but there's still going to be a huge number of false positives,
708010	712770	especially when you look at so many networks, the admins are doing things that look like
712770	717750	malware, just because they have this tool that somebody put together 20 years ago,
717750	722090	and nobody knows how the whole thing works anymore. And I think that's one of the interesting
722090	726270	things about looking at network security in this space is when you get down to it,
726270	733610	most networks aren't really planned and designed. They're more like a organism that's kind of grown
733610	738790	and developed over the past 20 years. And I think that that's the type of thing that machine
738790	748890	learning is really useful for. If you approach your environment as a space to be analyzed,
748890	753270	rather than something that's designed, then machine learning techniques can be really
753270	758410	useful for helping you weed out what's interesting and what's not.
759230	764930	I totally agree that anomaly detection is rife with false positives, but I think that's
764930	770730	where the human interaction becomes really important, because what the machine learning
770730	774810	algorithm can do is find things that are mathematically anomalous. And then we have
774810	780410	to go back to someone who has some knowledge of the network and say, oh, but these are network
780410	785190	backups that happen, you know, at 10am every Tuesday, throw those out, and like all of these
785190	792850	things that add contextual information, I think those can help denoise quite significantly.
793090	799810	And there are ways to automate them as well, but those are add-ons to the actual algorithm.
800310	807270	Okay, but I think that gets right into where one of the things I see as a big problem in this space,
807270	814510	at least from the analyst perspective. I've seen countless cases where you'll have a researcher
814510	818750	come in and say, yeah, I've got this great anomaly detection method, you know, let's go ahead and
819710	825990	get some of the domain experts to look at the results. And the domain experts are already
825990	830690	overworked, you know, nobody wants to spend their time tuning your anomaly detection system.
830950	837410	And I think that's one of the key downsides and reasons that those systems just aren't practical
837410	844310	from an applied perspective, you know. You don't have the people to give you the feedback on that.
844510	849350	And I think that's, you know, and I hate to say it, but I think in a lot of cases that's why
849990	855270	the only way to really do machine learning and AI successfully in this field is to
855950	858550	spend the time acquiring both skill sets.
860030	866290	I actually really strongly agree with you on that point of actually acquiring both skill sets being
866290	872440	important. But one of the pieces that I do actually slightly disagree with you on is,
872440	880480	um, well, I guess more I see just the ability to sort of identify costs between false positives
880480	886740	and false negatives and be able to tweak that really easily is a huge powerful tool I see for
886740	894240	at least statistical thinking about alerts and events, right. And, like, I do see, you know,
894240	900580	like Suricata or whatever rule set you have, like, we see a ton of, you know, false positives,
900580	906440	even in rule-based alerts, right, too. Because, as you say, like, networks are noisy.
907760	914400	So I think one of the benefits that you can sort of think of is more data science is like a layer
914400	921680	on top to sort of help control, sort of, you know, even filter down further, like, as you're saying,
921680	930500	like, you know, filter out the benign alerts, the non-anomalous sort of alerts that you get
930500	936460	for rule-based, at least as a starting point. Yeah, I mean, I think Brian Kovar had a really
936460	942020	good point at a Canvas a couple years ago where he was saying a lot of the benefit you get out
942020	947240	of machine learning isn't in detecting things, it's in throwing away the things that you don't
947240	956190	want to look at. I'll also say, like, one of the things that we've generally had success with
956190	963210	in our place is sort of, so, like, we have a lot of different rules for detecting, for example,
963210	969290	C2 toolkits like Empire, things like that, right? And one of the things that, you know,
969290	976610	when we went ahead and actually, like, tested out Empire was that it's very config-driven, right?
976610	982070	You can change things like, you know, easily change things like how often you beacon out to the C2
982070	986890	host or things like that. And a lot of these things are sort of things that you would build
986890	992730	into a rule, right? So, if, for example, you're taking a Let's Encrypt certificate as you're,
992730	998070	like, you know, one of your main things that you build into your rules, like, you can easily change
998070	1002790	things like the certificate subject to evade, right? Some of the things that we've actually
1002790	1009370	found a lot of use for is taking these rules and sort of trying to generalize them just a bit more
1009370	1015790	to where they'll capture, like, a large number of variants of, you know, Empire something. So,
1015790	1022370	not just specifically targeted to this particular certificate subject or structure, but, like,
1022890	1028550	other variants as well for things that are easily modifiable by the attacker.
1031890	1037630	Cool. I think a good point, kind of, about that is that you need to have a plan for how you're
1037630	1040310	going to deal with the alerts. And that's kind of a key thing for
1041370	1046010	anything, you know, you have to have a plan for what are you going to actually do with this model.
1046390	1048350	Yeah, 100% agree with that.
1049530	1055570	Yeah, I want to jump in really quick and give Ariel a chance to chime in if you've got anything.
1055790	1061950	I don't have anything significantly different. I firmly agree with the necessity to have both
1061950	1065890	skill sets of understanding security and also understanding the data science,
1066530	1070510	especially when you, if you don't understand the underlying data of what it is you're working with,
1070510	1073350	you're going to have a really hard time having any sort of alerts or whatever that you're trying
1073350	1076930	to build that actually tells you anything useful. That's all I would add.
1080550	1087390	Yeah, for sure. I think, so, we've kind of been maybe a little bit in the weeds of, like,
1087390	1093590	different, yeah, like, you need to be really, like, focused on sort of the technical details of,
1093590	1098030	how do I tune my anomaly detection system, or what do I know about sort of the security rules
1098030	1105550	that I'm building behind this? I guess maybe part of what contributes to all of the hype is the fact
1105550	1110670	that, like, this is, like, a really technical, really specialized discipline that not a lot of
1110670	1117590	people understand. And so that sort of, A, makes it look like magic to people that maybe don't
1117590	1122450	really understand it as well. And B, makes it really easy to oversell and to hype up because
1122870	1126450	whoever's doing the marketing and the hyping can be pretty sure that the other person doesn't
1126450	1131750	understand that. So, as people who are sort of, like, at the coalface, right, working, like,
1131750	1139730	on these specific problems, what can we do to sort of, like, help communicate what the capabilities
1139730	1144290	of these machine learning systems are and how can we sort of, like, demystify them and cut through
1144290	1149790	the hype and make it give people, like, a good idea of, yes, they actually do do really good
1149790	1154130	things and you can do really powerful, useful things with them, but no, they're not magic.
1154690	1157910	Like, how do we sort of, like, bridge that communications gap?
1160390	1167250	I'm going to say it's, I mean, InfoSec is ripe for it. Like, every year, like, the black hat
1167250	1171510	vendor, you know, two years ago, it was all about application whitelisting. And now you don't hear
1171510	1176050	about it anymore. All you hear about is machine learning. Maybe two years, it'll be something
1176050	1182970	else. But I think it's, it's, and marketing is just, is just that. It's, you know, it's designed
1182970	1188250	to make everything look better than it is. But I think in terms of practitioners, or, you know,
1188250	1194250	if you're working with a company or at a company, it's, you know, saying, yeah, the hype is real,
1194250	1199510	because this is, it can be really powerful, but it's, you're right, it's not going to do
1199510	1206010	your dishwasher. It, you know, it can assist in the things you have, but it's the ultimate garbage
1206010	1212430	in garbage out. So, you know, like, it's like the first, the first thing you find out when you
1212430	1216130	install a SIM in your network is not anything security related, it's how much stuff is
1216130	1222050	misconfigured. And I think the same is true for, for machine learning. It's like, once you put it
1222050	1228630	in, it's going to take a little bit. But if you continue to work and feed and spend that time
1228630	1233850	up front, like some of the best organizations, from the security perspective that we work with,
1235070	1238970	are just ones that have just put the legwork in, you know, they've gone through and they've
1240050	1244410	handpicked the DLLs that are allowed to run on every host, you know, and they haven't just
1244410	1250950	plugged in a box, pat themselves on the back and, you know, gone got a beer. So I think it's like,
1250950	1256550	doesn't matter what you're installing, I think the legwork needs to be done with it is obviously
1258370	1262890	misconstrued that AI will solve your problem. Maybe we'll get there one day. But I think,
1262890	1269070	you know, for now, it's still going to take some legwork, which is to be expected.
1271390	1277510	I think some of the hype also comes from, I mean, we've interacted with the results of AI
1277510	1286190	in different contexts, Netflix recommendations, Amazon recommendations. And as we think about how
1286190	1289630	to do that for security, it's natural to say, you know what, just give me a dashboard with
1289630	1294330	recommendations for what to do next. And it'll be AI powered. And it'll be just as disruptive as,
1294330	1302150	you know, Netflix. And I think security professionals are much more skeptical of
1302150	1306310	recommendations that they would get from that kind of dashboard than I as a consumer would be about
1306310	1311670	what recommendations Amazon gives me for my clothing purchases. So I think one way I'm
1311670	1317310	trying to think about this hype or talk about this hype is to just think about, like, our users
1317310	1324770	are using the output of this AI very, very differently. I've heard it referred to as
1324770	1331110	high stakes versus low stakes AI. In low stakes, you know, if my email is spam, I click it away.
1331110	1336710	It's not a big deal. But in security, if our predictions are wrong, someone can be compromised
1336710	1341190	or we could be taking someone's attention greatly away from the things they ought to be looking at
1341190	1347330	instead. So they're intrinsically interacting with our work very differently than if we were,
1347330	1352730	you know, correspondingly in e-commerce or something. And I feel like that's a conversation
1352730	1357050	I'd like to have with folks in marketing or product management as they're trying to brainstorm
1358370	1365290	how to talk about this publicly. I feel like the way that users use this is
1367090	1369410	more specific to security, let's say.
1370910	1375990	Actually, what's funny is there's two companies I can think of who their marketing departments
1375990	1380750	talk about how they don't use ML in their security products. So in terms of, like,
1380750	1385950	the skeptical and the skepticism that exists, these companies actually market that they don't use ML.
1387690	1390190	How well does that work for them? Do you know?
1390850	1392430	I guess they're still in business, so...
1392430	1393230	Well, that's good.
1393230	1401850	Okay. I do know one of them, and they are really, really solid. They are a solid company.
1402950	1403590	That's cool.
1403590	1405070	Or a solid product, I should say.
1405650	1409170	Yeah, I do wonder when we're going to see companies going back to good old-fashioned
1409590	1415450	AI. You could argue that signatures are just good old-fashioned AI, right? Rule-based.
1419630	1424970	John, any thoughts on bridging the hype gap or bringing hype back to reality?
1425770	1428750	Um, well, yeah, like...
1431050	1437730	I don't really have much to add. I actually agree with both Will and Vasudha's comments
1437730	1448750	that they've made. Like, it's... yeah. Marketing is marketing. It's really good to focus on sort
1448750	1454090	of what we're actually able to do well. And I think one of the things that we data scientists
1454090	1461210	can actually do a lot better on is communicating failures. And I have started to see, you know,
1462130	1468450	conferences like Kamla's, as well as others, sort of welcome, sort of, hey,
1468450	1473610	we tried this thing. It didn't work. Here's what we did. Don't do what we did, basically,
1474210	1480010	sort of talks. And I think sort of focusing on, even as data scientists, being able to very clearly
1481810	1486010	articulate what we were able to do, what we weren't able to do.
1486390	1490250	I think that's actually very important for bridging the gap long term.
1495190	1502430	I definitely agree with that. I think that given that a lot of the data science that we see in
1502430	1506630	InfoSec kind of comes from academia, it sort of ends up with a lot of the same incentives that
1506630	1511050	academia has for publishing stuff that you want it to look good. You don't really want to share
1511050	1516470	your failures because you don't really get any sort of, you don't get anything out of that as an
1516470	1522710	academic, unfortunately. And I think sort of taking a lot of this security data science out of that
1522710	1526950	context and putting it more in, like, here's some practical stuff. Let's talk about what works and
1526950	1531030	what doesn't work is exactly kind of the direction we should be taking things.
1534890	1539630	Cool. Yeah, I want to loop back a little bit to something Vasudha said a second ago,
1539630	1547330	where she referred to high stakes and low stakes decisions that are made. So yeah, I think it's
1547330	1552090	obvious to me at least that a lot of the decisions that get made when you're talking about using
1552090	1558990	ML or artificial intelligence or whatever you want to call it this week for InfoSec and security in
1558990	1565190	general, those tend to be higher stakes decisions. And one of the things that I've seen people push
1565270	1572470	a lot more for in recent past maybe two years or so is when you do have high stakes decisions,
1572470	1577810	having some notion of explainability or interpretability that comes along with those
1577810	1585790	models. So I kind of wonder, and this is partly driven by my own experience using ML to detect
1585790	1593850	malware, those models tend to be almost impossible to explain. So how big of a, how important is this
1593850	1598750	do you think to be able to explain the decisions that a model makes, be it in a detection,
1598850	1604890	a classification, or an anomaly detection context? And how can we get from where we are now,
1604890	1609510	where everything's about gradient boosted decision trees, or big complicated neural networks
1609510	1616150	that are almost impenetrable, to something that's maybe more explicable and more directly, something
1616150	1624140	you can directly interrogate? I think it's everything. I think it's worth sacrificing a
1624140	1631860	little bit of accuracy even, to get a simpler model that pulls out future importances, or that
1631860	1638620	you can easily explain what feature in your data led to this prediction, or led to the
1638620	1647420	the presentation of this particular record for review. I also think, this is my opinion only,
1647420	1651760	I think human-in-the-loop is super important, and that if it's a high stakes decision,
1654260	1661220	it's much better received as an augmentation that an analyst can take into account, rather than
1661220	1666060	being something that would be fully automated and trusted to make a decision on its own.
1668670	1675170	Yeah, I 100% agree with that. Like a lot, so especially like if you're sort of starting
1676010	1680350	a data science modeling, like a branch in detection or response or something,
1680350	1686970	your analysts aren't going to trust your data science models at all, right? So really,
1687550	1694330	like starting with something that's very clear and obvious, like, you know, this machine has,
1694330	1698810	is supposed to be fired walled off from the rest of the network, and it's uploading hundreds of
1698810	1703410	thousands of gigabytes, you know, to the outside world, like those sorts of things are going to
1703410	1710730	start, you know, building the trust as you go. I'm also a really, really big proponent of the
1710730	1716870	Google rules of ML that they do. And one of the things that they actually really, really recommend
1716870	1720530	is starting with, you know, whenever you're building like a product or a machine learning
1720530	1726010	based system, starting with something that's easily baselineable. And then that allows you
1726010	1731770	to actually improve upon it and actually be able to measure like how much you've improved. When you
1731770	1737290	have sort of these black box models, you lose a lot in sort of as you're trying to improve the
1737290	1742410	model. You spend a lot of effort trying to actually figure out whether you've actually
1742410	1750190	improved or how you can actually improve it further. I think organizationally, like buy-in is
1750190	1758110	pretty huge. So as explainable as you can make it. And you know, like analysts, security analysts,
1758110	1762110	you know, routiners are like, they're naturally curious. So if you're showing them something that
1762110	1767450	is, how did you get this, like, they're gonna, they're gonna go and look at it and figure out
1767450	1772490	and they, you know, even if you're right, if it's not as accurate, or it's not as effective, they'll
1772490	1779690	figure out how to, to make it that way, or they'll at least figure out, you know, what inputs, what,
1779690	1784890	what inputs require what outputs, and then you can start making it a little more
1786490	1791730	complicated or sophisticated or whatever. I think we often, you know, it's like we often
1792330	1798590	are misguided in the fact that we think complexity is sophistication. When in reality,
1798590	1804430	I think a lot of times simplicity is, is actually sophistication. It's much cleaner.
1805630	1810590	So I think we're kind of falling into the trap that people tend to fall into here, where we're
1810590	1816990	thinking that there has to be one silver bullet system that generates everything you need to know
1816990	1822350	to make a decision. In a lot of cases, I don't think explainability is necessarily that important,
1823570	1827770	as long as you can go grab more information around whatever you're looking at
1827770	1832050	to do a deeper analytic and provide more information to the analyst. Like if you've
1832050	1837170	got a binary, there's no reason that you have to rely solely on the machine learning model to make
1837170	1842470	the decision of whether it's good and bad. But you can use that as a tipper to do some more
1842470	1846230	in-depth analytics on the binary. You've got, you know, start running, throwing it through some
1846830	1852770	sandboxes, you know, do more expensive computation that you maybe couldn't, you can't do if you,
1852770	1856270	you may not be able to do it for every single binary, but you know, you could do it for a few
1856270	1862110	hundred an hour. You know, you know, in most other cases, there's additional context that
1862110	1867470	you could automatically collect and provide to the analyst to make their decision on whether
1867470	1871250	they should do something with the alert or not, rather than just relying on whatever the output
1871250	1876170	of the model is. And that's just a good point. I think, like, I know Microsoft, the Windows Defender
1876170	1881030	does that. So anything overtly malicious gets blocked on the host and anything that they're
1881030	1889500	unsure about gets sent to a cloud. I think that sometimes explainability can be a little bit of
1889500	1893740	red herring. In general, I think that it's good. I mean, for all the reasons stated above about
1893740	1900480	building trust and, and having baselines for understanding how you can improve, but when you
1900480	1905320	have, when you have like a human judge making a decision about whether or not someone's guilty,
1905320	1909080	you're, there's no part of the process whereby you go back to the judge and you say, what are
1909080	1914480	the pieces of evidence that convinced you that this person is guilty? And sometimes they can
1914480	1918620	explain why and sometimes they can't, but humans are very good about trying to come up with stories
1918620	1923620	to explain what the decisions that we make. So to some extent, I kind of view it as a little bit
1923740	1934260	I don't know, naive to try to come up with like purely explainable reasons for why things are the
1934260	1937820	way they are. Because oftentimes you can't really come up with like a true reason for why something
1937820	1951420	is the way that it is. Cool. So I have kind of a twofer that I want to, I guess, point at specifically
1951420	1960160	at Ariel and Will for initial responses. So the first bit is when we implement ML into these
1960160	1966140	security systems, right? We bake it in, we make it a big, sometimes a critical part. And this is
1966140	1973340	for Ariel, are we kind of opening up new vulnerabilities or new risks that we need
1973340	1980840	to be worried about, right? Is it, anytime you add a new component, you at least change the attack
1980840	1987720	surface. So do we really know quite what we're getting into as far as introducing new elements
1987720	1993340	to an attack surface when we start putting ML into important decision-making points
1993340	1997920	in these security systems? And then I'll come back to you, Will, for the second bit in a sec.
1999280	2005540	My answer would be yes. But I think that a lot of that can be mitigated by making sure that you
2005540	2011300	still have human in the loop. I kind of view a lot of these AI tools as sort of like centaur
2011300	2017300	chess where you have an AI system that assists a human. I don't really anticipate there being a
2017300	2021400	situation which you'd want a system to fully be automated to do everything because you're always
2021400	2027220	going to run into issues. And like we discussed earlier about interpretability, if you don't have
2027220	2031140	ways of understanding what's going on or like baselines or anything, you're going to be kind of
2031140	2036060	for a lot of this stuff. That said, there has been a lot of good research around trying to
2036060	2040680	understand what these potential vulnerabilities are, but they're all, at least for the most part,
2040680	2044040	most of them have been developed inside laboratory conditions, which makes them very hard to
2044040	2050860	translate it into the real world. Yeah, yeah. There's a lot of messed up stuff that's going
2050860	2054760	to go down, but I don't think it's going to be as bad as maybe some people think it will be.
2055880	2060080	I don't know. We are kind of clear on how poorly machine learning works for a lot of things and
2060080	2064600	how well it works for other things. And I think there's a decent amount of healthy skepticism
2064600	2069620	about machine learning at this point to sort of be some level of protection for us as we go forward
2069620	2079380	adding it to more things. Okay, cool. Thanks. And then I'm going to kick the second part over to
2079380	2085000	Will, and then I'd kind of like to hear from everyone else on the same topics. So do you see
2085260	2090960	a risk when we make models interpretable or explainable and we say, okay, I think this thing
2090960	2100960	was bad because of X. Do you think that that is making them necessarily easier to evade or bypass,
2100960	2104960	or is that a controllable risk, or does it not really matter to you? That doesn't really matter
2104960	2110640	to me. So I think when you have to sell something, making it explainable is nice.
2111200	2117740	Like that's what you're trying to do. You need to make it sellable, but I don't care if I'm using a
2117740	2122240	I don't care. I don't have to explain to anybody. I'm not trying to sell it to somebody,
2122680	2127800	right? But if you want ML to take hold in an organization, you're trying to sell a product
2128260	2132680	and no one's going to trust it, then you need to make it explainable. But for everybody else,
2132680	2138840	it doesn't matter. Like it can be as black box as you want. And I think that's an important
2138840	2146480	distinction. So it's like the... it needs to be explainable if you're trying to sell it.
2147740	2153600	You know, but even internally, it can be useful. An explainable model is more useful.
2153620	2159580	I don't necessarily think it's easier to attack or what. And it would be difficult to
2160500	2165840	assume like from an attacker's perspective, like I generally always assume that it's going to be
2165940	2170200	a black box and that you're not going to get 100% of the model, but you might be able to get some
2170200	2176000	sort of, you know, some copycat, something that helps you explain the model that you're looking
2176000	2188120	at. So yeah, explainability, my opinion is more about making it palatable to deploy into an
2188120	2194240	organization. Okay. Yeah. So I find it... it's interesting because there's, I think,
2194240	2198000	two different threads on the explainability that people are picking up on. First is
2198800	2205440	building trust to be able to get people to adopt the models. And second is reducing risk by giving
2205440	2214100	analysts who are seeing these alerts more context to kind of work with them. So, I mean,
2214100	2221980	do those seem like those are... I guess what I'm saying is like pitching it as like we're
2221980	2227760	building trust almost seems it's a little bit cynical, like a sales job, whereas we need to
2227760	2231600	be able to explain it because these are sort of critical, maybe even life and death decisions,
2231600	2237900	seems like a much higher ethical bar. So where does the balance between those two fall? Or are
2237900	2246140	they necessarily even in conflict, right? I would add a third column to that. I would say
2247740	2252860	putting aside the issue of having to convince someone else to trust these results for my own
2252860	2259540	quality control, I look to explainability to convince myself that I haven't overfit really
2259540	2265800	badly or that I'm not worried about data or that I haven't had data leakage happen or something. So
2265800	2271940	just going back and evaluating why the model made the decision that it did gives me better confidence
2271940	2281060	that it's picking up on real signal and not something spurious. Yeah, I also think explainability
2281060	2286560	also depends on how much domain expertise the person that's consuming the results of the model
2286560	2295260	has. Yeah, that's true. I'm always going to come back to the malware because that's
2295260	2299160	what has the most research on it, but I mean, if you've got a malware analyst
2299680	2307280	looking at a binary that he was delivered by a neural network said that, hey, this is
2307280	2312340	malicious, that malware analyst is going to make a decision based on their
2312340	2318800	own domain expertise, regardless of what the model says it made its decision on.
2318800	2321900	I think that's the case for a lot of other things as well.
2323320	2330760	But that leads us to the problem of the people that get the most value out of the explainability
2330760	2338120	are also the ones that are least qualified to make the decision in the first place.
2340200	2345260	Okay, so I'm going to reframe that a little bit more positively and say maybe it's a way
2345260	2352200	to maybe help turn less experienced analysts and to help them perform like more experienced
2352200	2359480	analysts maybe, hopefully. Or is that too optimistic? I don't think it's optimistic,
2359480	2364460	but you still have the chicken and the egg problem of they have to understand the results
2364460	2370560	enough. And if they have that understanding of the domain, they may not derive as much value
2370560	2376600	from the explainability. Yeah, actually, I'm gonna there's a product called Artemis by Endgame.
2376600	2382380	And it's like a talk to your sim product. And it's, you know, they were like, this is everything
2382380	2387120	machine learning that you ever wanted. I don't know what their marketing is. But it's basically
2387120	2393280	they haven't, they've only made information retrieval easier, but they it still requires
2393280	2399380	an analyst that knows what they need to be looking for. So it actually, we already had a
2399380	2403940	language to retrieve information from a sim, you know, whatever, OS query, whatever it might be,
2403940	2409000	we didn't need machine learning to do that. What we needed machine learning to do is potentially
2409000	2415360	add some context around a particular data point, not to say like, hey, you know,
2415460	2417800	like a chatbot on top of your sim, we didn't need that.
2419960	2426800	Um, I, pardon me if I'm wrong, or misunderstanding, Rob, but I thought your point was more that
2426800	2433320	basically, the domain expert is going to know what features basically would be most helpful
2433320	2440260	in determining which whether this binary was malicious or benign. And so it's kind of not the
2440260	2446300	right thing to put that on the data science to try to, you know, to be the ones to actually explain
2446300	2452220	why the model chose a particular, you know, feature or something in its decision. Am I correct
2452220	2459380	on that or? Partly, what I'm saying is that the model, the models explainability isn't necessarily
2459380	2466540	going to be helpful for the people that need it. You know, it can maybe say this, this particular
2466540	2472600	segment of the binary is what I thought is bad. But then you need the expertise to know whether
2472600	2478100	that means anything, you know, it could just be a packer, it could just be, you know, this
2478100	2483760	particular thing is using get proc address to load things for non malicious reasons. And there's,
2484640	2491800	there's a lot of context around things where, if you already know it, then you don't need the AI
2491800	2497040	to tell you that this is why this is bad. And if you need the AI to tell you that this is bad,
2497040	2500540	you may not understand it well enough to know whether the AI is wrong or not.
2504070	2510150	Yeah, sensing a lot of very healthy skepticism of ML from all quarters here, which warms my
2510150	2517050	dark little heart. So one more, one more question to kind of carry us out to the top of the hour.
2517510	2522990	So we've touched on anomaly detection. I think that's people, I think there's a little bit of
2523090	2527990	a network focus here. So that's what a lot of people are sort of talking about as a main tool.
2527990	2532970	There's also binary classification for things like static file artifacts, be it malware detection or
2532970	2543090	maldoct detection. So these are sort of maybe the, among the simpler forms of ML, right? And it
2543090	2549250	doesn't get into things like, it hasn't like, we haven't seen, as far as I know, any serious,
2549250	2557210	like, successes as far as using more complicated forms like reinforcement learning or, you know,
2557210	2561810	like active, active learning with like really good human-in-the-loop integrations in the
2561810	2569430	InfoSec space yet. So just sort of among the vast menu of new, of existing techniques that
2569430	2575990	really haven't made a dent in InfoSec, where do you think, where do people think that the field
2575990	2580270	is going next? Like what's going to be the next, the next big breakthrough as far as applying
2580270	2588880	machine learning to security problems? I think that reinforcement learning is
2588880	2593100	coming. I think it's going to have its moment at some point, but I think that that's going to
2593100	2596800	depend very heavily on the kind of data that we can get into these kind of systems.
2597280	2601620	I've seen a lot of people trying to use it for adversarial simulation, but haven't really
2601620	2607400	had good data that really allows it to do a good job that's better than what we already
2607400	2612380	have with Cobalt Strike and some of those other things. But I have seen some very convincing
2612380	2620360	sort of research in the RL space around trying to use RL for like code generation kind of stuff.
2620360	2625060	And that gives me some, I guess, I don't know if it's hope because there's also like a negative
2625060	2630580	side to any sort of adversary simulation using RL. But I think it'll definitely be interesting.
2631160	2636160	Yeah. If it is a good adversarial simulation, it's also probably a decent adversary. Yeah.
2636160	2641480	I think reinforcement learning is the next, like there's a lot of attack simulation
2642200	2648060	stuff that's being talked about. I know MITRE has some talks where they're discussing
2649600	2653780	reinforcement learning for attack simulation. I think for actual attackers, the difficult part
2655460	2661380	is you can't get on a network and learn the network and then attack it, right? You need
2661380	2666260	to be learning from all of your general experience. But I think from a defensive
2666260	2669740	standpoint, reinforcement learning could be, you know, because you can iterate through your network
2669740	2678300	infinitely, but it's just matching what is useful. I think, I hate,
2678300	2687240	I shouldn't say that. Adversary simulation is a bit of a, it's a bit of a buzzword.
2687540	2694560	I think it's not a buzzword. How do I, it's like you, from the offensive side for the last three
2694560	2698380	years, they've just been dropping techniques left and right. And so it's like, in some ways
2698380	2703600	we call it adversary simulation because the techniques that we dropped are getting picked up
2703600	2711400	by adversaries. And so it's like, we're emulating ourselves via a proxy. But it's because we wanted
2711400	2717180	to do the research, but it hasn't been really helpful. But if I think of the value, you know,
2717180	2722800	that the public research has generated in the defensive space, it's insane. You know, like
2722800	2730240	there are entire products that are based off a GitHub project. So, you know, you can't say it's
2732680	2737280	terrible, but I think the adversary simulation piece is a bit overdone. I wish organizations
2737280	2743280	would spend more time in their own networks, just figuring out where their data is and doing some
2743280	2750520	of the simple stuff like network segmentation and less time focusing on, you know, whatever
2750520	2756320	APT is they think is going to come after them next. I think that's probably more useful.
2756500	2759180	But reinforcement learning would be my answer.
2760560	2766480	Yeah, I'm fairly skeptical of reinforcement learning, just because the problems in this
2766480	2772160	space don't lend themselves to easily defining a target for the learning algorithm to
2772960	2777960	test itself against. You know, from the defensive side, it's really hard to say,
2777960	2785880	you know, we can't easily define what secure means in any meaningful sense to really be able to
2785880	2791140	teach an agent, whether it's in a secure state or not, to determine, you know, what actions it
2791140	2795680	should be taking. From the attack perspective, you know, like you said, the attacker can't just
2795680	2802580	throw a agent on to go learn the network, because that's going to be super noisy. And, you know,
2802580	2807580	hopefully the defenders will notice that that's going on as it's probing everything and everything
2807580	2814500	so my answer to what's the next big thing is, I don't think there's going to be a next big thing.
2814960	2822000	I think the real positive impact or the real kind of game changing impact is going to be
2823360	2829180	smaller, you know, people applying machine learning to solve small problems tactically
2831220	2835300	in a lot more places, you know, it's going to be the sock analyst who sits down and just
2835300	2840880	makes a naive Bayes classifier to triage the AV alerts they've got, you know, it's going to be
2840880	2846060	the small projects like that, widely distributed that are going to really make the difference.
2847600	2852200	Okay, so we've got a couple votes for reinforcement learning, one vote for
2852200	2858540	mass democratization of machine learning. Vasudha, you gotta pick for us.
2858540	2864880	I'll pile on to the small projects done with ML. But I will add on to that.
2865440	2870720	I'm really excited about graph theoretic approaches for that. Because I think as we
2870720	2876760	think about very like, maybe making classifications on very specific endpoints, that that's a machine
2876760	2882300	learning problem that I think we can do well. But to go from that to bridge from that to
2882300	2887060	talking about an intrusion means understanding network effects,
2887060	2894740	adding in other data sets to endpoint logs, it's not possible to do that using local,
2894740	2900980	like single node properties. And so I'm, I'm kind of waiting to see what the
2900980	2904160	what the network graph picture will do for us.
2906160	2912280	Very cool. So I know we're kind of at the top of the hour, but we started a little bit late.
2912280	2917000	Can people stay for another couple minutes? I want to try and take a couple of questions
2917000	2923720	from the Discord, if that's okay. Awesome. Cool. I'm just gonna, so if I don't call your
2923720	2927160	question out, please don't be offended. I'm just kind of scrolling through.
2927900	2934080	So the first one that jumps out at me, FS Montenegro says, my perennial question is,
2934080	2940020	what are the questions and practices that we should use to probe claims of a vendor's ML quality?
2940020	2946780	I particularly like that one. Anyone want to take a stab? Yeah. How do you call BS?
2952250	2959390	I'm going to put... I guess, I guess I can start right. So there's, there's a lot of different,
2959390	2966390	you know, pitfalls in, you know, data science and, and performing machine learning. And one
2966390	2970250	of the things I would like to see more of from, you know, skeptics or people trying
2970250	2975230	to evaluate these models is just the simple questions like, how did you split your train
2975230	2980770	and test? Like, did you, did you, you know, consider data leakage in sort of the time
2980770	2985850	domain, right? Like, are you learning from future samples in your past samples? Those sorts of
2985850	2992250	questions. I don't have much experience on the actual buying product side though. So I probably,
2992250	2996390	that that's about as far as my, my opinion on the subject.
2998690	2999830	I'm going to go with...
3004170	3005830	Let's go with Will then Rob.
3005830	3010250	Oh, sorry. I was just going to say, you know, from an attacker's perspective,
3010250	3014510	these things are being deployed, but they haven't really been tested. So I think
3015950	3019450	attackers are ultimately going to be the ones that decide whether or not
3019450	3025970	ML is a viable solution for network defense. And we just, from an attack perspective,
3025970	3031590	there's just not many people looking at it currently. So I would say that's probably
3032710	3037150	a good way. Eventually it'll just happen, right? Attackers will have to adapt. And
3037150	3042390	so I think that's going to be a good, a good litmus test for it.
3045450	3046410	Rob?
3047390	3052210	One of the first things I look at when I'm looking at a ML product is whether the data
3052210	3060310	that they are purporting to operate on is even reasonably possible to make a decision on.
3060510	3063770	And a lot of the products out there are saying, you know, we're going to look at an email,
3063770	3067850	we're going to look at a phishing email and, you know, we'll predict what the person had
3067850	3071510	for breakfast and, you know, all kinds of other claims that just don't really follow
3071510	3076830	from the data that they have available. So I think, you know, to me, that's really one
3076830	3080610	of the big things I look at when I'm evaluating an ML system.
3084370	3086510	Cool. Good answers. Anyone else?
3088730	3092690	Plus one for Rob's answer. I think the data tells all, really.
3092910	3099790	Yeah. Okay, cool. From... I'm going to get your name wrong, I'm sorry. Scudzi,
3099790	3103510	there's a question. Is there any focus in these vendor tools that provide good
3104250	3106870	introspection into how the models are making inferences? Yeah. So
3106870	3112490	do we have explainable ML for security out there? Does anyone know of anything offhand?
3118250	3123210	I hesitate to... It kind of depends on what you mean by explainable.
3125230	3127710	Okay, that's... Yes.
3127870	3130230	If you look at the...
3133270	3137590	I hate it when I forget the name of the paper I was an author on. But
3140870	3145910	eating binary is one that I did with Ed Raff. You know, one of the big things we looked at
3145910	3153390	for the explainability was we took a look at what parts of the binary was the model keying in on,
3153390	3157310	you know, for making its decision. You know, I think there's been
3157310	3162930	some work in things like that, but not necessarily explainability in terms that
3164250	3168190	we can write an English sentence that will tell you why the model
3168190	3172810	made its decisions and the things that the model has learned.
3174490	3179930	Yeah, I mean, I know there are a number of companies that are beginning to push...
3180690	3185850	I think explainability is probably a bridge too far, but at least they're working on enriching
3185850	3191490	context, right? I contractually oblige to mention that Sophos has done a lot of research in that
3191490	3199230	area. But yeah, I agree. I don't think there's anything that is going for specific explainability.
3200930	3207730	I'm browsing through Discord. Okay. One dad pants revolution. Awesome. Would like to hear
3207730	3212890	about strategies for optimizing the relationship between rules, analysts, and ML systems operating
3212890	3217370	on the same domain. I'm going to take that and I'm going to spin that a little bit and say,
3217370	3221690	like, maybe how do we... maybe that's like an internal communication problem,
3221690	3228490	so less of a hype issue and more of a how do you go about getting analysts comfortable with
3228490	3235330	thinking about sort of the statistical way that these machine learning tools operate? So how can
3235330	3240930	we effectively skill up analysts in using these tools and understanding what they mean and not
3240930	3250480	just sort of blindly trusting the result? So I think a lot of people are already
3250480	3254720	intuitively doing that when they're doing their work. A lot of analysts are building
3254720	3261660	up their own internal models of what's likely to be good and what isn't. I think if we can show
3262880	3266420	how a model, you know, how an ML model kind of does the same thing
3267260	3272160	and show, you know, that in a lot of cases the model's learning the same things that you would
3272160	3279680	think that would be useful for an analyst, I think that goes a long way to bridging the two.
3279820	3282520	I knew we'd get you on the explainability train somehow.
3287120	3288300	Anyone else?
3288300	3291520	I think, sorry John, go ahead.
3291880	3297300	Oh yeah, I was just going to say a lot of the like basic building blocks for ML, so like things like
3297300	3305400	historical data or like intermediate representations and transformations of data are things that
3305400	3311340	analysts wish they had available to them but like can't because, you know, maybe like a spunk is too
3311340	3319680	slow or something like that, right? And so I think sort of also allowing these intermediate
3319680	3326340	representations and stuff to people so they can use them, whether it's in the form of like a
3326340	3332880	querying engine or something like that, is something we've had a lot of good feedback
3332880	3340680	it and help with and help bridge that gap. And I think it's important that data science
3340680	3345600	doesn't have to be done only in the data science team because I think a lot of the analysts,
3345600	3352640	like Rob was mentioning, have these mental models in their head already and the conversations
3352640	3357000	are fun when we can say, you know, there's a little bit of terminology we have to add,
3357000	3361400	these are called inputs, you have a prediction label, just articulate that very precisely and
3361400	3365500	then boom, that's all it is. The model is just an equation that takes you from, you know, input to
3365500	3373860	output. And I think it helps to say, you know, go off and try that on your own without the data
3373860	3381720	science team in the middle at all. And that we're also here to advise and maybe help out with better
3381720	3387960	inputs, data cleaning, that sort of stuff. But by no means does it have to be, you know, relegated
3387960	3396340	to like the data science tower. Yeah, for sure. Integrating, like,
3396340	3402960	getting everyone kind of back to what Rob was saying about the mass democratization of data
3402960	3407220	science and machine learning, getting it integrated across the organization I think is a huge win
3407780	3412420	for the organization. Sort of giving everyone a little, just like a little baseline of maybe
3412420	3418940	statistical literacy and understanding how to use these tools. Ariel, any thoughts on?
3421100	3424740	No, I don't have any thoughts on this. Sorry, not trying to put you on the spot.
3424740	3431180	It's all good. It's all good. Okay. Cool. So, that looks like the bulk
3431180	3436940	of the questions from Discord. I really am sorry if you had a question in Discord and I missed it.
3436940	3443240	Please feel free to, like, do it again and maybe at me and I'll see if we can wrangle some responses
3443240	3448880	in the Discord chat. But I want to thank all you guys for being here and doing this panel with me.
3448880	3454500	I had a lot of fun once we got past the inevitable technical problems. I hope you did, too.
3454740	3457760	And thanks so much for a great conversation. I really appreciate your time.
3458380	3460380	Thanks, Rich. Thanks, everybody. Cool. Thanks.
3460380	3463080	Thanks, Rich. Thanks, everyone. This was awesome.
3463940	3465120	Ciao. Cheers.