Okay. Sorry. Sorry. I just figured it out. The audio should now be working.
Seriously. Okay. Are we...
So, on Twitch chat, I am so sorry. We should have this all sorted now.
I found out what caused the feedback loop.
Okay. So, on Twitch chat, can you please give us a thumbs up?
Okay. Thumbs up if the audio is not working for everyone. So, you can hear Will,
and it's not doing the weird Hall of Mirrors thing.
Yay.
I'm terrified to open Twitch at this point.
Yeah. Cool. I'm muting myself now. The audio is now working. I am adding everyone's names.
So, I am so sorry for everyone.
This time, if it doesn't work, I'll off myself with a sword in the background.
Okay.
Okay. So, have at it, Will.
It's what the people want. Rich, whoever I am this week. Okay.
Well, last time. Last time through. So, again, welcome, everyone, to the Hype in AI panel,
where I'm going to talk about does AI in InfoSec actually live up to the hype?
I'm not going to pitch the topic again, because I'm tired of it.
So, Ariel, please introduce yourself one last time.
All right. Hello. My name is Ariel Hirt-Voss. I'm a research scientist at OpenAI,
where I work on misuse and abuse of the AI technologies that we develop there.
I'm also a PhD student at Harvard University, and I've been working on a book for No Starch Press
on how to do red teaming for AI systems, specifically. And that's most of what I do.
Cool. My name is Rich Huerang. I am a data scientist at Sophos. I have been working
at the intersection of security, machine learning, and privacy for about 10 years.
I have a PhD from the University of California at Santa Barbara. Will?
My name is Will. I am a network operator, and I work on building tools,
AI tools for red teaming and attacking machine learning systems.
Very cool. Masuda?
My name is Masuda. I'm a data scientist at Rapid7. I think a lot about incident detection,
anomaly detection, and I'm a relative newcomer to security. I have a PhD in physics and
spent some time in quantum computing research before formally switching over
to data science and now security. Awesome. Rob?
I'm Rob Brandon. Been working in the tech and security field for probably about 25 years now.
Currently working for Booz Allen Hamilton as a researcher and threat hunter.
And about eight years ago, I was doing some sock work. I got a little bit of a exposure to this
machine learning stuff and went back to school and got a PhD from the University of Maryland,
and been working in the intersection ever since.
Awesome. And last, but certainly not least, John.
Hey, everyone. I'm John Seymour, aka DeltaZero. I work as a data scientist at Salesforce,
where I do intrusion detection on the DNR team. I also like to just kind of put my hands in
everything, so I've touched a lot of, you know, offensive security and data science and all sorts
of things. But yeah, mostly currently do intrusion detection.
Awesome. So again, thanks to everyone for being here and taking part in this discussion.
I kind of want to kick it off with a very sort of like high-level first question.
So machine learning has actually been used in a wide range of in-physic problems now. So
we've seen it in malware detection, where it's probably sort of the biggest success story.
But then spam blocking is probably the oldest field it's been used in.
And then other stuff like network traffic classification, analyst alert ranking, and triage.
So what, if anything, do we get as sort of a value-add above and beyond sort of like the
old-school tried-and-true signature-based or rule-based techniques for security? And so what
I'm kind of hoping we can dig into here is like, what are the strengths and weaknesses of AI or ML
or whatever you want to call it? And are there classes of problems that we should always be
using ML for or maybe like should just completely give up and never waste time and money? And are
there specific areas where if a vendor says, oh yes, we can solve this problem with AI, right,
that immediately sets off a red flag in your head that they might not know what they're talking
about? So open question, please dive in. Or I can call on somebody.
So for intrusion detection, because this is what I spent the most time thinking about,
I think the class of problems really depends strongly on what the data makes available.
And just as an example for how I've approached the problem, when I first started,
I thought, you know, we're awash in data, everything's digital, records are there for
everything. And I viewed my role as analogous to the people that study volcanoes by looking at
very small earthquake signatures, and have gotten really good at predicting these eruptions.
And I thought, yeah, that's what I'm going to do. Can anything be more glamorous? I'm going to
really understand these signatures and use them to predict, you know, incoming attacks.
I very quickly realized it's hard to get a labeled data set of incidents. People don't want to share
when they've been compromised. And so that's a problem. But even if I were to get a library of
these attacks, attackers behave very differently. So I'd have to get a representative set of
different attacker behavior. And also, they're opportunistic. So it really depends on the very
specifics of network configurations and how different companies, you know, set up their
assets. So really, now I'm talking about simulating different types of incidents over many different
types of attackers over tons of different environments. So that very quickly put me off
the classification use case, let's say. But there's still a lot of data. And so I think
anomaly detection is great. We can baseline a lot of things. The volume-based baselining is
maybe straightforward and not that interesting. But I think there's an art in defining what are
the attributes that we expect to look really regular in reference to particular intrusions.
So I think that's a really useful way to go. I'm very suspicious if someone were to tell me
I've got a method to, you know, precisely detect a compromise for you.
So you're mostly thinking about network traffic in that case?
Network traffic, let's say endpoint data, you know, logs from endpoint data.
Cool. Anyone else? I guess, Will, I'd really be interested to hear your input on this,
since you work on the other side of the problem a lot of the time.
Yeah, yeah. I don't come at it from a machine learning area. I'd say the nice bit is scale
and building context around an event. I think that's really helpful. You know, I think the
people who have been doing incident response for a long time, you know, they have the knowledge.
They just don't necessarily have all the pieces to find, you know, a direction. So, you know,
we always, I think as data scientists, or as the data is coming across, it's often incomplete,
where you don't know what you need to solve the problem. So you only have, you know,
half of the information that you actually need, and the rest is kind of fuzzy. But you know,
maybe a direction that you're trying to go with it. So I think machine learning can help pull out
those insights. But you know, I think it requires that still like a human, a human touch, a human
direction. I think that is probably pretty limited as we go forward in time. But currently, where
it's at it, you know, it's still pretty ineffective in networks for detections and things. And you
potentially the other use cases. So proactive, secure configuration, it's like, out of this
machine learning thing, the only thing we've got is just more alerts. I think that that may remain,
you know, and people can't necessarily trust them. But I do think it's going forward in time,
you know, it's going to be a very, very different use case, especially as you get data scientists
with data sets that, you know, they start to understand. And, you know, we're all on the same
playing field. So, you know, once we're on a Windows host, for example, everybody has access
to the same information, more or less. And it's really just about how you're viewing that
information inside of a given context, I think. Yeah, so one, one thing that I think kind of
popped out from both those answers was, there's a lot of like human tuning and human feature
extraction that's going on in there. So the, you do see a lot of marketing literature about, oh,
just, you know, dump raw bytes in and detections, magical detections come out the other end. So
any, any thoughts on claims like that? Do you really need a, like, is the human element,
like mandatory still at this point? Or is there any path? No, I don't think it's mandatory.
So for example, any given user is going to operate on a host, you know, within some range.
So I think the anomaly detection is really nice. So let's say you have 1000 users,
998 of those users are going to come in, log in at 8am every day, they're going to
visit the same things, you know, then you have that 2% that are, you know, doing something that
they're not supposed to be doing or whatever it is. So I don't think you necessarily need
to label data, but you, you need to baseline your network. And I think the issue for machine
learning is when you try and scale it across, let's say 10 different companies, that's going
to be really difficult. So but I think machine learning is such that you could, you could
build a network for each individual network, it's just going to take time. But from a vendor's
perspective, like that doesn't scale as well. And so I think it gets a little more difficult
in that way.
I kind of disagree with the anomaly detection approach, just because every network is just
full of anomalies when you start going to look for anomalies. And while it may just be 2%,
that 2% is still going to be the source of, you know, to, you know, millions and millions of
alerts.
Yeah, yeah, right. I should say, in specific cases, like, let's say log on, log on times,
so, or if, you know, host where users are logging on. So yeah, you're absolutely right. I think
anomalies are everywhere. But in terms of malicious events, or potentially malicious
events, I think you could cut it down to a little bit where you can get some high fidelity.
You can narrow it down, but there's still going to be a huge number of false positives,
especially when you look at so many networks, the admins are doing things that look like
malware, just because they have this tool that somebody put together 20 years ago,
and nobody knows how the whole thing works anymore. And I think that's one of the interesting
things about looking at network security in this space is when you get down to it,
most networks aren't really planned and designed. They're more like a organism that's kind of grown
and developed over the past 20 years. And I think that that's the type of thing that machine
learning is really useful for. If you approach your environment as a space to be analyzed,
rather than something that's designed, then machine learning techniques can be really
useful for helping you weed out what's interesting and what's not.
I totally agree that anomaly detection is rife with false positives, but I think that's
where the human interaction becomes really important, because what the machine learning
algorithm can do is find things that are mathematically anomalous. And then we have
to go back to someone who has some knowledge of the network and say, oh, but these are network
backups that happen, you know, at 10am every Tuesday, throw those out, and like all of these
things that add contextual information, I think those can help denoise quite significantly.
And there are ways to automate them as well, but those are add-ons to the actual algorithm.
Okay, but I think that gets right into where one of the things I see as a big problem in this space,
at least from the analyst perspective. I've seen countless cases where you'll have a researcher
come in and say, yeah, I've got this great anomaly detection method, you know, let's go ahead and
get some of the domain experts to look at the results. And the domain experts are already
overworked, you know, nobody wants to spend their time tuning your anomaly detection system.
And I think that's one of the key downsides and reasons that those systems just aren't practical
from an applied perspective, you know. You don't have the people to give you the feedback on that.
And I think that's, you know, and I hate to say it, but I think in a lot of cases that's why
the only way to really do machine learning and AI successfully in this field is to
spend the time acquiring both skill sets.
I actually really strongly agree with you on that point of actually acquiring both skill sets being
important. But one of the pieces that I do actually slightly disagree with you on is,
um, well, I guess more I see just the ability to sort of identify costs between false positives
and false negatives and be able to tweak that really easily is a huge powerful tool I see for
at least statistical thinking about alerts and events, right. And, like, I do see, you know,
like Suricata or whatever rule set you have, like, we see a ton of, you know, false positives,
even in rule-based alerts, right, too. Because, as you say, like, networks are noisy.
So I think one of the benefits that you can sort of think of is more data science is like a layer
on top to sort of help control, sort of, you know, even filter down further, like, as you're saying,
like, you know, filter out the benign alerts, the non-anomalous sort of alerts that you get
for rule-based, at least as a starting point. Yeah, I mean, I think Brian Kovar had a really
good point at a Canvas a couple years ago where he was saying a lot of the benefit you get out
of machine learning isn't in detecting things, it's in throwing away the things that you don't
want to look at. I'll also say, like, one of the things that we've generally had success with
in our place is sort of, so, like, we have a lot of different rules for detecting, for example,
C2 toolkits like Empire, things like that, right? And one of the things that, you know,
when we went ahead and actually, like, tested out Empire was that it's very config-driven, right?
You can change things like, you know, easily change things like how often you beacon out to the C2
host or things like that. And a lot of these things are sort of things that you would build
into a rule, right? So, if, for example, you're taking a Let's Encrypt certificate as you're,
like, you know, one of your main things that you build into your rules, like, you can easily change
things like the certificate subject to evade, right? Some of the things that we've actually
found a lot of use for is taking these rules and sort of trying to generalize them just a bit more
to where they'll capture, like, a large number of variants of, you know, Empire something. So,
not just specifically targeted to this particular certificate subject or structure, but, like,
other variants as well for things that are easily modifiable by the attacker.
Cool. I think a good point, kind of, about that is that you need to have a plan for how you're
going to deal with the alerts. And that's kind of a key thing for
anything, you know, you have to have a plan for what are you going to actually do with this model.
Yeah, 100% agree with that.
Yeah, I want to jump in really quick and give Ariel a chance to chime in if you've got anything.
I don't have anything significantly different. I firmly agree with the necessity to have both
skill sets of understanding security and also understanding the data science,
especially when you, if you don't understand the underlying data of what it is you're working with,
you're going to have a really hard time having any sort of alerts or whatever that you're trying
to build that actually tells you anything useful. That's all I would add.
Yeah, for sure. I think, so, we've kind of been maybe a little bit in the weeds of, like,
different, yeah, like, you need to be really, like, focused on sort of the technical details of,
how do I tune my anomaly detection system, or what do I know about sort of the security rules
that I'm building behind this? I guess maybe part of what contributes to all of the hype is the fact
that, like, this is, like, a really technical, really specialized discipline that not a lot of
people understand. And so that sort of, A, makes it look like magic to people that maybe don't
really understand it as well. And B, makes it really easy to oversell and to hype up because
whoever's doing the marketing and the hyping can be pretty sure that the other person doesn't
understand that. So, as people who are sort of, like, at the coalface, right, working, like,
on these specific problems, what can we do to sort of, like, help communicate what the capabilities
of these machine learning systems are and how can we sort of, like, demystify them and cut through
the hype and make it give people, like, a good idea of, yes, they actually do do really good
things and you can do really powerful, useful things with them, but no, they're not magic.
Like, how do we sort of, like, bridge that communications gap?
I'm going to say it's, I mean, InfoSec is ripe for it. Like, every year, like, the black hat
vendor, you know, two years ago, it was all about application whitelisting. And now you don't hear
about it anymore. All you hear about is machine learning. Maybe two years, it'll be something
else. But I think it's, it's, and marketing is just, is just that. It's, you know, it's designed
to make everything look better than it is. But I think in terms of practitioners, or, you know,
if you're working with a company or at a company, it's, you know, saying, yeah, the hype is real,
because this is, it can be really powerful, but it's, you're right, it's not going to do
your dishwasher. It, you know, it can assist in the things you have, but it's the ultimate garbage
in garbage out. So, you know, like, it's like the first, the first thing you find out when you
install a SIM in your network is not anything security related, it's how much stuff is
misconfigured. And I think the same is true for, for machine learning. It's like, once you put it
in, it's going to take a little bit. But if you continue to work and feed and spend that time
up front, like some of the best organizations, from the security perspective that we work with,
are just ones that have just put the legwork in, you know, they've gone through and they've
handpicked the DLLs that are allowed to run on every host, you know, and they haven't just
plugged in a box, pat themselves on the back and, you know, gone got a beer. So I think it's like,
doesn't matter what you're installing, I think the legwork needs to be done with it is obviously
misconstrued that AI will solve your problem. Maybe we'll get there one day. But I think,
you know, for now, it's still going to take some legwork, which is to be expected.
I think some of the hype also comes from, I mean, we've interacted with the results of AI
in different contexts, Netflix recommendations, Amazon recommendations. And as we think about how
to do that for security, it's natural to say, you know what, just give me a dashboard with
recommendations for what to do next. And it'll be AI powered. And it'll be just as disruptive as,
you know, Netflix. And I think security professionals are much more skeptical of
recommendations that they would get from that kind of dashboard than I as a consumer would be about
what recommendations Amazon gives me for my clothing purchases. So I think one way I'm
trying to think about this hype or talk about this hype is to just think about, like, our users
are using the output of this AI very, very differently. I've heard it referred to as
high stakes versus low stakes AI. In low stakes, you know, if my email is spam, I click it away.
It's not a big deal. But in security, if our predictions are wrong, someone can be compromised
or we could be taking someone's attention greatly away from the things they ought to be looking at
instead. So they're intrinsically interacting with our work very differently than if we were,
you know, correspondingly in e-commerce or something. And I feel like that's a conversation
I'd like to have with folks in marketing or product management as they're trying to brainstorm
how to talk about this publicly. I feel like the way that users use this is
more specific to security, let's say.
Actually, what's funny is there's two companies I can think of who their marketing departments
talk about how they don't use ML in their security products. So in terms of, like,
the skeptical and the skepticism that exists, these companies actually market that they don't use ML.
How well does that work for them? Do you know?
I guess they're still in business, so...
Well, that's good.
Okay. I do know one of them, and they are really, really solid. They are a solid company.
That's cool.
Or a solid product, I should say.
Yeah, I do wonder when we're going to see companies going back to good old-fashioned
AI. You could argue that signatures are just good old-fashioned AI, right? Rule-based.
John, any thoughts on bridging the hype gap or bringing hype back to reality?
Um, well, yeah, like...
I don't really have much to add. I actually agree with both Will and Vasudha's comments
that they've made. Like, it's... yeah. Marketing is marketing. It's really good to focus on sort
of what we're actually able to do well. And I think one of the things that we data scientists
can actually do a lot better on is communicating failures. And I have started to see, you know,
conferences like Kamla's, as well as others, sort of welcome, sort of, hey,
we tried this thing. It didn't work. Here's what we did. Don't do what we did, basically,
sort of talks. And I think sort of focusing on, even as data scientists, being able to very clearly
articulate what we were able to do, what we weren't able to do.
I think that's actually very important for bridging the gap long term.
I definitely agree with that. I think that given that a lot of the data science that we see in
InfoSec kind of comes from academia, it sort of ends up with a lot of the same incentives that
academia has for publishing stuff that you want it to look good. You don't really want to share
your failures because you don't really get any sort of, you don't get anything out of that as an
academic, unfortunately. And I think sort of taking a lot of this security data science out of that
context and putting it more in, like, here's some practical stuff. Let's talk about what works and
what doesn't work is exactly kind of the direction we should be taking things.
Cool. Yeah, I want to loop back a little bit to something Vasudha said a second ago,
where she referred to high stakes and low stakes decisions that are made. So yeah, I think it's
obvious to me at least that a lot of the decisions that get made when you're talking about using
ML or artificial intelligence or whatever you want to call it this week for InfoSec and security in
general, those tend to be higher stakes decisions. And one of the things that I've seen people push
a lot more for in recent past maybe two years or so is when you do have high stakes decisions,
having some notion of explainability or interpretability that comes along with those
models. So I kind of wonder, and this is partly driven by my own experience using ML to detect
malware, those models tend to be almost impossible to explain. So how big of a, how important is this
do you think to be able to explain the decisions that a model makes, be it in a detection,
a classification, or an anomaly detection context? And how can we get from where we are now,
where everything's about gradient boosted decision trees, or big complicated neural networks
that are almost impenetrable, to something that's maybe more explicable and more directly, something
you can directly interrogate? I think it's everything. I think it's worth sacrificing a
little bit of accuracy even, to get a simpler model that pulls out future importances, or that
you can easily explain what feature in your data led to this prediction, or led to the
the presentation of this particular record for review. I also think, this is my opinion only,
I think human-in-the-loop is super important, and that if it's a high stakes decision,
it's much better received as an augmentation that an analyst can take into account, rather than
being something that would be fully automated and trusted to make a decision on its own.
Yeah, I 100% agree with that. Like a lot, so especially like if you're sort of starting
a data science modeling, like a branch in detection or response or something,
your analysts aren't going to trust your data science models at all, right? So really,
like starting with something that's very clear and obvious, like, you know, this machine has,
is supposed to be fired walled off from the rest of the network, and it's uploading hundreds of
thousands of gigabytes, you know, to the outside world, like those sorts of things are going to
start, you know, building the trust as you go. I'm also a really, really big proponent of the
Google rules of ML that they do. And one of the things that they actually really, really recommend
is starting with, you know, whenever you're building like a product or a machine learning
based system, starting with something that's easily baselineable. And then that allows you
to actually improve upon it and actually be able to measure like how much you've improved. When you
have sort of these black box models, you lose a lot in sort of as you're trying to improve the
model. You spend a lot of effort trying to actually figure out whether you've actually
improved or how you can actually improve it further. I think organizationally, like buy-in is
pretty huge. So as explainable as you can make it. And you know, like analysts, security analysts,
you know, routiners are like, they're naturally curious. So if you're showing them something that
is, how did you get this, like, they're gonna, they're gonna go and look at it and figure out
and they, you know, even if you're right, if it's not as accurate, or it's not as effective, they'll
figure out how to, to make it that way, or they'll at least figure out, you know, what inputs, what,
what inputs require what outputs, and then you can start making it a little more
complicated or sophisticated or whatever. I think we often, you know, it's like we often
are misguided in the fact that we think complexity is sophistication. When in reality,
I think a lot of times simplicity is, is actually sophistication. It's much cleaner.
So I think we're kind of falling into the trap that people tend to fall into here, where we're
thinking that there has to be one silver bullet system that generates everything you need to know
to make a decision. In a lot of cases, I don't think explainability is necessarily that important,
as long as you can go grab more information around whatever you're looking at
to do a deeper analytic and provide more information to the analyst. Like if you've
got a binary, there's no reason that you have to rely solely on the machine learning model to make
the decision of whether it's good and bad. But you can use that as a tipper to do some more
in-depth analytics on the binary. You've got, you know, start running, throwing it through some
sandboxes, you know, do more expensive computation that you maybe couldn't, you can't do if you,
you may not be able to do it for every single binary, but you know, you could do it for a few
hundred an hour. You know, you know, in most other cases, there's additional context that
you could automatically collect and provide to the analyst to make their decision on whether
they should do something with the alert or not, rather than just relying on whatever the output
of the model is. And that's just a good point. I think, like, I know Microsoft, the Windows Defender
does that. So anything overtly malicious gets blocked on the host and anything that they're
unsure about gets sent to a cloud. I think that sometimes explainability can be a little bit of
red herring. In general, I think that it's good. I mean, for all the reasons stated above about
building trust and, and having baselines for understanding how you can improve, but when you
have, when you have like a human judge making a decision about whether or not someone's guilty,
you're, there's no part of the process whereby you go back to the judge and you say, what are
the pieces of evidence that convinced you that this person is guilty? And sometimes they can
explain why and sometimes they can't, but humans are very good about trying to come up with stories
to explain what the decisions that we make. So to some extent, I kind of view it as a little bit
I don't know, naive to try to come up with like purely explainable reasons for why things are the
way they are. Because oftentimes you can't really come up with like a true reason for why something
is the way that it is. Cool. So I have kind of a twofer that I want to, I guess, point at specifically
at Ariel and Will for initial responses. So the first bit is when we implement ML into these
security systems, right? We bake it in, we make it a big, sometimes a critical part. And this is
for Ariel, are we kind of opening up new vulnerabilities or new risks that we need
to be worried about, right? Is it, anytime you add a new component, you at least change the attack
surface. So do we really know quite what we're getting into as far as introducing new elements
to an attack surface when we start putting ML into important decision-making points
in these security systems? And then I'll come back to you, Will, for the second bit in a sec.
My answer would be yes. But I think that a lot of that can be mitigated by making sure that you
still have human in the loop. I kind of view a lot of these AI tools as sort of like centaur
chess where you have an AI system that assists a human. I don't really anticipate there being a
situation which you'd want a system to fully be automated to do everything because you're always
going to run into issues. And like we discussed earlier about interpretability, if you don't have
ways of understanding what's going on or like baselines or anything, you're going to be kind of
for a lot of this stuff. That said, there has been a lot of good research around trying to
understand what these potential vulnerabilities are, but they're all, at least for the most part,
most of them have been developed inside laboratory conditions, which makes them very hard to
translate it into the real world. Yeah, yeah. There's a lot of messed up stuff that's going
to go down, but I don't think it's going to be as bad as maybe some people think it will be.
I don't know. We are kind of clear on how poorly machine learning works for a lot of things and
how well it works for other things. And I think there's a decent amount of healthy skepticism
about machine learning at this point to sort of be some level of protection for us as we go forward
adding it to more things. Okay, cool. Thanks. And then I'm going to kick the second part over to
Will, and then I'd kind of like to hear from everyone else on the same topics. So do you see
a risk when we make models interpretable or explainable and we say, okay, I think this thing
was bad because of X. Do you think that that is making them necessarily easier to evade or bypass,
or is that a controllable risk, or does it not really matter to you? That doesn't really matter
to me. So I think when you have to sell something, making it explainable is nice.
Like that's what you're trying to do. You need to make it sellable, but I don't care if I'm using a
I don't care. I don't have to explain to anybody. I'm not trying to sell it to somebody,
right? But if you want ML to take hold in an organization, you're trying to sell a product
and no one's going to trust it, then you need to make it explainable. But for everybody else,
it doesn't matter. Like it can be as black box as you want. And I think that's an important
distinction. So it's like the... it needs to be explainable if you're trying to sell it.
You know, but even internally, it can be useful. An explainable model is more useful.
I don't necessarily think it's easier to attack or what. And it would be difficult to
assume like from an attacker's perspective, like I generally always assume that it's going to be
a black box and that you're not going to get 100% of the model, but you might be able to get some
sort of, you know, some copycat, something that helps you explain the model that you're looking
at. So yeah, explainability, my opinion is more about making it palatable to deploy into an
organization. Okay. Yeah. So I find it... it's interesting because there's, I think,
two different threads on the explainability that people are picking up on. First is
building trust to be able to get people to adopt the models. And second is reducing risk by giving
analysts who are seeing these alerts more context to kind of work with them. So, I mean,
do those seem like those are... I guess what I'm saying is like pitching it as like we're
building trust almost seems it's a little bit cynical, like a sales job, whereas we need to
be able to explain it because these are sort of critical, maybe even life and death decisions,
seems like a much higher ethical bar. So where does the balance between those two fall? Or are
they necessarily even in conflict, right? I would add a third column to that. I would say
putting aside the issue of having to convince someone else to trust these results for my own
quality control, I look to explainability to convince myself that I haven't overfit really
badly or that I'm not worried about data or that I haven't had data leakage happen or something. So
just going back and evaluating why the model made the decision that it did gives me better confidence
that it's picking up on real signal and not something spurious. Yeah, I also think explainability
also depends on how much domain expertise the person that's consuming the results of the model
has. Yeah, that's true. I'm always going to come back to the malware because that's
what has the most research on it, but I mean, if you've got a malware analyst
looking at a binary that he was delivered by a neural network said that, hey, this is
malicious, that malware analyst is going to make a decision based on their
own domain expertise, regardless of what the model says it made its decision on.
I think that's the case for a lot of other things as well.
But that leads us to the problem of the people that get the most value out of the explainability
are also the ones that are least qualified to make the decision in the first place.
Okay, so I'm going to reframe that a little bit more positively and say maybe it's a way
to maybe help turn less experienced analysts and to help them perform like more experienced
analysts maybe, hopefully. Or is that too optimistic? I don't think it's optimistic,
but you still have the chicken and the egg problem of they have to understand the results
enough. And if they have that understanding of the domain, they may not derive as much value
from the explainability. Yeah, actually, I'm gonna there's a product called Artemis by Endgame.
And it's like a talk to your sim product. And it's, you know, they were like, this is everything
machine learning that you ever wanted. I don't know what their marketing is. But it's basically
they haven't, they've only made information retrieval easier, but they it still requires
an analyst that knows what they need to be looking for. So it actually, we already had a
language to retrieve information from a sim, you know, whatever, OS query, whatever it might be,
we didn't need machine learning to do that. What we needed machine learning to do is potentially
add some context around a particular data point, not to say like, hey, you know,
like a chatbot on top of your sim, we didn't need that.
Um, I, pardon me if I'm wrong, or misunderstanding, Rob, but I thought your point was more that
basically, the domain expert is going to know what features basically would be most helpful
in determining which whether this binary was malicious or benign. And so it's kind of not the
right thing to put that on the data science to try to, you know, to be the ones to actually explain
why the model chose a particular, you know, feature or something in its decision. Am I correct
on that or? Partly, what I'm saying is that the model, the models explainability isn't necessarily
going to be helpful for the people that need it. You know, it can maybe say this, this particular
segment of the binary is what I thought is bad. But then you need the expertise to know whether
that means anything, you know, it could just be a packer, it could just be, you know, this
particular thing is using get proc address to load things for non malicious reasons. And there's,
there's a lot of context around things where, if you already know it, then you don't need the AI
to tell you that this is why this is bad. And if you need the AI to tell you that this is bad,
you may not understand it well enough to know whether the AI is wrong or not.
Yeah, sensing a lot of very healthy skepticism of ML from all quarters here, which warms my
dark little heart. So one more, one more question to kind of carry us out to the top of the hour.
So we've touched on anomaly detection. I think that's people, I think there's a little bit of
a network focus here. So that's what a lot of people are sort of talking about as a main tool.
There's also binary classification for things like static file artifacts, be it malware detection or
maldoct detection. So these are sort of maybe the, among the simpler forms of ML, right? And it
doesn't get into things like, it hasn't like, we haven't seen, as far as I know, any serious,
like, successes as far as using more complicated forms like reinforcement learning or, you know,
like active, active learning with like really good human-in-the-loop integrations in the
InfoSec space yet. So just sort of among the vast menu of new, of existing techniques that
really haven't made a dent in InfoSec, where do you think, where do people think that the field
is going next? Like what's going to be the next, the next big breakthrough as far as applying
machine learning to security problems? I think that reinforcement learning is
coming. I think it's going to have its moment at some point, but I think that that's going to
depend very heavily on the kind of data that we can get into these kind of systems.
I've seen a lot of people trying to use it for adversarial simulation, but haven't really
had good data that really allows it to do a good job that's better than what we already
have with Cobalt Strike and some of those other things. But I have seen some very convincing
sort of research in the RL space around trying to use RL for like code generation kind of stuff.
And that gives me some, I guess, I don't know if it's hope because there's also like a negative
side to any sort of adversary simulation using RL. But I think it'll definitely be interesting.
Yeah. If it is a good adversarial simulation, it's also probably a decent adversary. Yeah.
I think reinforcement learning is the next, like there's a lot of attack simulation
stuff that's being talked about. I know MITRE has some talks where they're discussing
reinforcement learning for attack simulation. I think for actual attackers, the difficult part
is you can't get on a network and learn the network and then attack it, right? You need
to be learning from all of your general experience. But I think from a defensive
standpoint, reinforcement learning could be, you know, because you can iterate through your network
infinitely, but it's just matching what is useful. I think, I hate,
I shouldn't say that. Adversary simulation is a bit of a, it's a bit of a buzzword.
I think it's not a buzzword. How do I, it's like you, from the offensive side for the last three
years, they've just been dropping techniques left and right. And so it's like, in some ways
we call it adversary simulation because the techniques that we dropped are getting picked up
by adversaries. And so it's like, we're emulating ourselves via a proxy. But it's because we wanted
to do the research, but it hasn't been really helpful. But if I think of the value, you know,
that the public research has generated in the defensive space, it's insane. You know, like
there are entire products that are based off a GitHub project. So, you know, you can't say it's
terrible, but I think the adversary simulation piece is a bit overdone. I wish organizations
would spend more time in their own networks, just figuring out where their data is and doing some
of the simple stuff like network segmentation and less time focusing on, you know, whatever
APT is they think is going to come after them next. I think that's probably more useful.
But reinforcement learning would be my answer.
Yeah, I'm fairly skeptical of reinforcement learning, just because the problems in this
space don't lend themselves to easily defining a target for the learning algorithm to
test itself against. You know, from the defensive side, it's really hard to say,
you know, we can't easily define what secure means in any meaningful sense to really be able to
teach an agent, whether it's in a secure state or not, to determine, you know, what actions it
should be taking. From the attack perspective, you know, like you said, the attacker can't just
throw a agent on to go learn the network, because that's going to be super noisy. And, you know,
hopefully the defenders will notice that that's going on as it's probing everything and everything
so my answer to what's the next big thing is, I don't think there's going to be a next big thing.
I think the real positive impact or the real kind of game changing impact is going to be
smaller, you know, people applying machine learning to solve small problems tactically
in a lot more places, you know, it's going to be the sock analyst who sits down and just
makes a naive Bayes classifier to triage the AV alerts they've got, you know, it's going to be
the small projects like that, widely distributed that are going to really make the difference.
Okay, so we've got a couple votes for reinforcement learning, one vote for
mass democratization of machine learning. Vasudha, you gotta pick for us.
I'll pile on to the small projects done with ML. But I will add on to that.
I'm really excited about graph theoretic approaches for that. Because I think as we
think about very like, maybe making classifications on very specific endpoints, that that's a machine
learning problem that I think we can do well. But to go from that to bridge from that to
talking about an intrusion means understanding network effects,
adding in other data sets to endpoint logs, it's not possible to do that using local,
like single node properties. And so I'm, I'm kind of waiting to see what the
what the network graph picture will do for us.
Very cool. So I know we're kind of at the top of the hour, but we started a little bit late.
Can people stay for another couple minutes? I want to try and take a couple of questions
from the Discord, if that's okay. Awesome. Cool. I'm just gonna, so if I don't call your
question out, please don't be offended. I'm just kind of scrolling through.
So the first one that jumps out at me, FS Montenegro says, my perennial question is,
what are the questions and practices that we should use to probe claims of a vendor's ML quality?
I particularly like that one. Anyone want to take a stab? Yeah. How do you call BS?
I'm going to put... I guess, I guess I can start right. So there's, there's a lot of different,
you know, pitfalls in, you know, data science and, and performing machine learning. And one
of the things I would like to see more of from, you know, skeptics or people trying
to evaluate these models is just the simple questions like, how did you split your train
and test? Like, did you, did you, you know, consider data leakage in sort of the time
domain, right? Like, are you learning from future samples in your past samples? Those sorts of
questions. I don't have much experience on the actual buying product side though. So I probably,
that that's about as far as my, my opinion on the subject.
I'm going to go with...
Let's go with Will then Rob.
Oh, sorry. I was just going to say, you know, from an attacker's perspective,
these things are being deployed, but they haven't really been tested. So I think
attackers are ultimately going to be the ones that decide whether or not
ML is a viable solution for network defense. And we just, from an attack perspective,
there's just not many people looking at it currently. So I would say that's probably
a good way. Eventually it'll just happen, right? Attackers will have to adapt. And
so I think that's going to be a good, a good litmus test for it.
Rob?
One of the first things I look at when I'm looking at a ML product is whether the data
that they are purporting to operate on is even reasonably possible to make a decision on.
And a lot of the products out there are saying, you know, we're going to look at an email,
we're going to look at a phishing email and, you know, we'll predict what the person had
for breakfast and, you know, all kinds of other claims that just don't really follow
from the data that they have available. So I think, you know, to me, that's really one
of the big things I look at when I'm evaluating an ML system.
Cool. Good answers. Anyone else?
Plus one for Rob's answer. I think the data tells all, really.
Yeah. Okay, cool. From... I'm going to get your name wrong, I'm sorry. Scudzi,
there's a question. Is there any focus in these vendor tools that provide good
introspection into how the models are making inferences? Yeah. So
do we have explainable ML for security out there? Does anyone know of anything offhand?
I hesitate to... It kind of depends on what you mean by explainable.
Okay, that's... Yes.
If you look at the...
I hate it when I forget the name of the paper I was an author on. But
eating binary is one that I did with Ed Raff. You know, one of the big things we looked at
for the explainability was we took a look at what parts of the binary was the model keying in on,
you know, for making its decision. You know, I think there's been
some work in things like that, but not necessarily explainability in terms that
we can write an English sentence that will tell you why the model
made its decisions and the things that the model has learned.
Yeah, I mean, I know there are a number of companies that are beginning to push...
I think explainability is probably a bridge too far, but at least they're working on enriching
context, right? I contractually oblige to mention that Sophos has done a lot of research in that
area. But yeah, I agree. I don't think there's anything that is going for specific explainability.
I'm browsing through Discord. Okay. One dad pants revolution. Awesome. Would like to hear
about strategies for optimizing the relationship between rules, analysts, and ML systems operating
on the same domain. I'm going to take that and I'm going to spin that a little bit and say,
like, maybe how do we... maybe that's like an internal communication problem,
so less of a hype issue and more of a how do you go about getting analysts comfortable with
thinking about sort of the statistical way that these machine learning tools operate? So how can
we effectively skill up analysts in using these tools and understanding what they mean and not
just sort of blindly trusting the result? So I think a lot of people are already
intuitively doing that when they're doing their work. A lot of analysts are building
up their own internal models of what's likely to be good and what isn't. I think if we can show
how a model, you know, how an ML model kind of does the same thing
and show, you know, that in a lot of cases the model's learning the same things that you would
think that would be useful for an analyst, I think that goes a long way to bridging the two.
I knew we'd get you on the explainability train somehow.
Anyone else?
I think, sorry John, go ahead.
Oh yeah, I was just going to say a lot of the like basic building blocks for ML, so like things like
historical data or like intermediate representations and transformations of data are things that
analysts wish they had available to them but like can't because, you know, maybe like a spunk is too
slow or something like that, right? And so I think sort of also allowing these intermediate
representations and stuff to people so they can use them, whether it's in the form of like a
querying engine or something like that, is something we've had a lot of good feedback
it and help with and help bridge that gap. And I think it's important that data science
doesn't have to be done only in the data science team because I think a lot of the analysts,
like Rob was mentioning, have these mental models in their head already and the conversations
are fun when we can say, you know, there's a little bit of terminology we have to add,
these are called inputs, you have a prediction label, just articulate that very precisely and
then boom, that's all it is. The model is just an equation that takes you from, you know, input to
output. And I think it helps to say, you know, go off and try that on your own without the data
science team in the middle at all. And that we're also here to advise and maybe help out with better
inputs, data cleaning, that sort of stuff. But by no means does it have to be, you know, relegated
to like the data science tower. Yeah, for sure. Integrating, like,
getting everyone kind of back to what Rob was saying about the mass democratization of data
science and machine learning, getting it integrated across the organization I think is a huge win
for the organization. Sort of giving everyone a little, just like a little baseline of maybe
statistical literacy and understanding how to use these tools. Ariel, any thoughts on?
No, I don't have any thoughts on this. Sorry, not trying to put you on the spot.
It's all good. It's all good. Okay. Cool. So, that looks like the bulk
of the questions from Discord. I really am sorry if you had a question in Discord and I missed it.
Please feel free to, like, do it again and maybe at me and I'll see if we can wrangle some responses
in the Discord chat. But I want to thank all you guys for being here and doing this panel with me.
I had a lot of fun once we got past the inevitable technical problems. I hope you did, too.
And thanks so much for a great conversation. I really appreciate your time.
Thanks, Rich. Thanks, everybody. Cool. Thanks.
Thanks, Rich. Thanks, everyone. This was awesome.
Ciao. Cheers.