Done. Okay, is that better?
Yes.
Nice.
That's good.
Nailed it. Thank you for hanging in there.
Okay, so I want to ask, what isn't machine learning?
So, as math underpins everything we do,
I think you get to a point where everything is machine learning.
So, if you think about when you're a child and you're trying to learn to walk,
it doesn't look that unfamiliar to when someone has reinforcement learning algorithms
that are teaching something to walk.
So from an offensive standpoint, I think it's an important distinction
because where offensive folks try to live in the realm of possibility
rather than in a box that we try to define.
So, maybe if you start thinking of everything as machine learning,
you're going to find more opportunities to attack machine learning models,
use them in your ops, whatever that might be.
But for this attack in particular, we're going to do a copycat model.
And I think this kind of concept will come out as we go through it.
So, I'm kind of tired of the name copycat model,
so I'm just going to call them Pink Panther attacks from now on.
One, it's more fun. Two, it's still kind of cat-like.
And three, I know Pink Panther involves a diamond that is always trying to be stolen.
And preconditions for a successful attack, and these are pretty loose,
are a representative data set.
So, for example, in the case of an AMSI provider,
we're looking for like PowerShell, VBA code,
a big representative data set that we want to model.
And we need the ability to get feedback from our target model.
And this doesn't have to be direct output.
So, if you imagine that a model doesn't give you back a score,
but it keeps it in a Windows event log,
you still get your hard label, hard label being a 0 or a 1.
So, even if the model doesn't give you back output,
there are still, just due to the telemetry nature of our networks,
it's more likely than not that somewhere output's being recorded.
So, rather than maybe going directly to the model,
try and think of some binary test that you can perform
that is potentially outside of the realm of a machine learning model.
So, it's a pretty simple attack once you kind of get into it.
Effectively, you grab a massive...
in this case, we found a big PowerShell corpus.
So, there's like 400,000 scripts.
And the workshop, the lab piece of it, we'll talk about that.
Effectively, the scripts get fed into an AMSI integration.
And this just came off of the Windows sample.
So, all it does is load AMSI.dll and instantiate the com object
and we feed it every script in that corpus.
And that gets passed to Defender or whatever AMSI provider that might be.
And it comes back with a score.
And then we turn around and we just collect those scores offline.
And once we have known inputs and we know what the output...
Defender thinks of the output, we can model our model.
And the hardest part about, I think, adversarial machine learning
isn't the machine learning piece.
Most of us don't have to invent math.
People much smarter than myself have already done that for me.
But what is difficult is the engineering piece.
It's getting access to the right data.
It's getting data in a way that isn't going to get you caught
or you're going to distribute your traffic accordingly.
And you can kind of fly under the radar.
In the future, currently attackers kind of have free reign
and get away with quite a lot of querying and noise in the network.
That's not always going to be the case.
And I think it is helpful for us to start thinking about limiting our queries,
thinking about information density of a command,
a ping versus a process list, for example.
Ping, you're going to get if a host is up or not.
Or you're going to get TTL, which is going to tell the OS.
But a process list, you're going to get that and more.
But it probably just depends on what you're after.
And from a machine learning perspective and an offensive perspective,
anything can be modeled.
So if defenders are using machine learning to model whatever logs they're looking for,
there's nothing to say that we as offensive folks can't also use machine learning
to say model CT traffic.
Or let's say you have semantic, you have some product that is blocking you
and you can't quite pinpoint it.
Machine learning will be able to do that for you
if you can set up the right experiment.
So even though Semantic is not using machine learning necessarily to look for your CT traffic,
you can still use it to find those relationships between whatever domain
or whatever callback interval that you're using against whatever product it might be.
So I guess I just come back to the question of what isn't machine learning.
And in my mind, everything is machine learning if you kind of go back to the math,
which is kind of fun.
Okay, so in the lab, the workshop, all the code is in there for you.
So the first half, you'll step through and it'll be very explanatory.
And then as we go through, it'll be less and less explanation.
So you'll be required to kind of look more, ask more questions, etc., etc.
Break stuff, like you're not going to break it, you can always reset it.
Google it, ask us for help.
AI Village has easily, probably the highest concentration of some of the smartest people in the industry.
And there's so many PhDs and super experienced people.
So, you know, we love talking. I like talking offensive ops, but you know,
a lot of people like talking stats and topology and homology and all this other stuff.
The other day I learned that there is more than one algebra.
But yeah, so this is the place to ask, so go ask for it.
As you go through, I would challenge those of you who are more experienced to produce a better and more efficient attack.
And in there, there is also a massive data set of VBA code that I found.
It's like nine gigs of VBA code.
So if you want, go through this code, it would work the same.
And you could get your name on that leaderboard for CVs for machine learning systems.
So if you want to do that, you can go for it.
So feel free to ask questions in Discord. There are no dumb questions.
And we have three TAs that are helping me out here.
So if you have specific questions, we can ask for it.
So hopefully you have the link.
You have everything.
So effectively, this is the... what do you call this?
This is the workshop.
And there are two files, two Python notebooks.
One is the solutions, and one is the workbook.
And you should start with the workbook.
If you do get stuck, or you need code help, or you just want to grab it and go, the solutions one is going to work for you.
This amsy.h file is if you do want to compile the amsy stream.exe so you can recreate the full attack path.
You're going to need this header file.
Collect.py is what we use to gather all the information.
And then this insights.xlsx, this Excel file.
This is the version 1 of the insights that we pulled out of Defender.
So this would be like that proof point style insight.
And it is our version 1, so this is it.
So obviously you can see there's a lot of binary blobs in here.
But these would be the most malicious tokens.
And this makes sense to me.
And at the bottom you have the least giant binary blob.
And at the bottom you have the least malicious tokens.
So that's free.
So you guys can get started.
Do you guys have any questions right away?
I haven't seen any questions so far.
Perfect.
That's good.
Is anyone else getting 403?
I have to say I haven't used Defender.
Is anyone else getting 403?
We are not seeing any chat on Twitch or Discord.
I just reopened the link and it's building the container for me.
Yeah, me too.
Okay, that's good.
I have my own copy.
I am in the Jupyter Notebook.
Nice.
Yeah, so the first little bit, I'll give you guys 10-15 minutes to rip through the first little bit.
And if you guys have questions, let me know.
Remote is always a bit weird.
But, you know what always gets me in the mood?
Can you guys hear that?
What always gets you in the mood?
Tron.
Yeah, I don't think it's a good idea to stream copyrighted music on here.
Oh, right, right, right.
Twitch doesn't really have a problem with that, but the YouTube VOD at the end will be removed.
That's fair.
Yeah, I don't spend that much time on the internet.
So, Will, I can send you the Notebook link again.
The one that myself and other people are using on the chat.
That's good. I have my own.
Excellent.
Okay, so I'm just going to talk.
Everything is pretty self-explanatory in terms of the instructions.
Can you zoom in a little bit? It's kind of hard to see on the screen.
Is that better?
One more.
That should be good.
Sorry, there's a one-year-old.
I can see it very clearly.
Excellent. So, I'm just going to talk. You guys feel free to run through.
So, sort of the basis of this attack is really the AnyMalware scan interface.
And Emsi is something they introduced in, I want to say, PowerShell 5.
A little sooner than that.
But effectively, it's just a DLL that gets called into PowerShell, UAC dialogs, any Jscript, VBScript, and now .NET 4.8 and beyond.
And inside, it has a number of blacklisted functions that whenever one of those functions is called, Emsi will get called to scan content.
And Emsi is not a security boundary. It's not a security product. It's an interface.
And so, all it does is collect whatever content is being put into the buffer or the string, and it passes it back to an Emsi provider.
And the Emsi provider then has an opportunity to scan the content and make some determination.
So, obviously, this workshop is about Windows Defender.
But there are several other providers, Emsi providers, that this attack would work against.
And that's, I suppose, kind of the nice part about machine learning currently is there's a lot of attack surface.
So, once an Emsi provider scans the content and decides what it is, there's a range of scores it can give.
Currently, it only gives back 0 or 1, 32, 768.
But in Microsoft documentation, they talk about a range of scores.
So, if you think rather than a hard label as a 0 or a 1, eventually you'd get some sort of regression type continuous variable on the out.
And so, you'd get some heuristic, I suppose.
There's a great talk by, actually Microsoft, called Badly Behaving Scripts.
It's probably like an hour long.
But I think they gave it two years ago at Blue Hat IL.
And it's two engineers from the machine learning team.
And they're just discussing how Emsi works, the things that they're looking for, the kind of models that they're building, and so on.
And so, this is kind of our first indication, at least officially, that machine learning is being pushed onto the client.
And I think it would seem anyway that overtly malicious things get stopped on the client.
But anything that's kind of in between seems to go to the cloud.
One way to test this would be to have some sort of timing attack.
You could get the response times for submission to the reception of the score.
And you could maybe see if there's a significant difference between the worst scripts and the medium bad scripts.
So, moving on. You guys can obviously read.
So, the first thing we obviously needed to do to create a copycat is we needed a data set.
And there's a talk by Lee Holmes and Daniel Bohannon in 2017 called Revoke Obfuscation, Powershot Obfuscation Detection Using Science.
So, Daniel Bohannon wrote Invoke Obfuscation, which basically takes a script or any number of different languages
and it puts it through an obfuscation.
So, it breaks it up into a million different pieces.
So, it'll break those regexes, those brittle detections.
It's an awesome, awesome tool.
These guys wrote it, wrote the talk.
And part of that research, they collected like two gigs worth of PowerShell scripts.
And they labeled them both benign and malicious for us.
And so, they're already... you can actually... the link's here.
So, you can go pull it down and rip through it.
Once you have that, the rest of it is just getting target outputs from the target models.
So, if we want to create a copycat, we want to know what Defender thinks of each script that we give it.
And then, that is remodeling the model.
So, we have the inputs and we have the outputs.
We don't know what's in the middle.
We don't know that black box.
But we can infer it with our own model.
And you'll never get 100% of the model.
But you might get just enough to bypass Windows Defender for a month or just one time or whatever it might be.
But the nice part about machine learning and the struggle that machine learning has, I think, is it does introduce a probability into what used to be a static decision.
The large repository of something interesting is VBA.
So, that's just 9 gigs of Excel macros that you could do the same with.
Which would be pretty awesome.
We have... let's see, what is this?
This is the... you're not on Windows.
If this were like Lab Lab, we'd have Windows VMs and we'd do this for real.
This is just the output from the AMSI stream.
So, InvokeWMIBackdoor looks malicious, but Defender doesn't think it's malicious.
So, if you look down, you see ScanResult is 1, IsMalware is 0, IsMalware is the official feedback loop.
And even in that talk that is above, they discuss the fact that you can't trust filenames.
And that makes sense. You can't trust headers anymore.
There's some research that did a number of, like 3 years ago now, against mail filters where you would null out the first 2 bytes of a .m file.
And depending on how the mail filter decided what kind of file it was, it would either block it or let it through.
Because it can't read the magic bytes at the top.
And then, obviously the document is corrupt at that point.
But what Windows does is when you open the .m file, it would ask you to repair it.
And if the user clicked yes, I'd like to repair this document, your macro would live and you could get code execution.
Right, but the next bit is the provider display name.
So we have Microsoft Defender Antivirus.
If this is...
If Defender is turned off for whatever reason, or there is no AMZ provider, it will give you an error.
And you will not get a score back.
The next piece is just PowerView.
So PowerView is pretty well known script, and it is definitely...
Well, it's not explicitly malicious, but it is used by malicious people.
That's not fair to say.
It's used by attackers.
I'm sure there are some nice attackers.
So in this lab, there are...
Uploading 380,000 scripts to GitHub was not popular with GitHub.
So in this little bit, you only have 3,000.
But the collect.py has everything you need.
And when we were parsing, we just did...
In your data, you have data clean and dirty.
These would be your malicious scripts.
And clean scripts are actually clean.
There's like 3,000.
There are about 1,200 malicious scripts that you can look at.
Yeah, if you guys want to start ripping through this code.
As we were parsing, you can look through collect.py.
But I personally like to keep...
So there's a lot of moving parts in machine learning.
You kind of can't get away from it.
So whenever possible, I like to keep at each step.
I'll build a data structure and I'll keep the previous output.
And I think it works nicely.
Most of my datasets are pretty small anyway.
So it's manageable.
But you can kind of see the GetScreenshot.
So this is the file name, the original file name.
This is the hash, the MD5 hash of it.
And then you have the result, being its malware.
And then you have the base64 encoded text that you can look through.
I like this because if there's any weirdness,
you have everything you need right in front of you.
You don't have to start again at a particular point.
You don't have to go back to the beginning of the process.
Being able to debug all the way through your pipeline is...
It might cost you some speed.
But at least for me, it's fine.
I'm not dealing with billions of anything.
We just got discussed lists.
So if you're not familiar with a list,
which I think probably most of you are,
raise your hand if you're familiar with lists.
Nice, okay, so about 20% of you.
Can't hear any giggling.
Alright, so you have lists. Lists are really nice.
So you can just put them on any delimiter you want.
So quick question, Will.
This is from the Twitch chat.
Should he be able to view the PowerShell code?
When they click on a file with clean or dirty,
get a mess of characters but not a script.
Yes, so you...
Yeah, so in this little code block here,
you can open it.
And it is just a...
So we've already parsed it for you.
It was just going to take forever otherwise.
So what you're going to see is the script name.
So this is the original script name.
The MD5 hash of the script content.
The AMSI result, or I should say the Defender result.
So what Defender thinks of the script.
And then the Base64 encoded version of the script.
Can you turn lines on?
So this is what we're doing here.
So we're splitting a file.
So what we're doing is we're setting up...
Eventually we're going to rip through all these files
and we're going to build a big vocab.
So this is a...
I just pulled one out as an example.
So let's see.
So for example, if we want to reference...
This is just list stuff.
So obviously zero index.
So we can reference it however we want.
You can then reference with semicolon.
You know, from index one to the end.
Or from index one to the third.
Or you can even do minus.
So you can go to the end of the list.
And just get the script content.
But if we're going to look at the script content,
we kind of want to know...
It's interesting that getScreenshot
is counted as malicious.
So if we decode it,
we can have a look and maybe...
Maybe there's some knowledge that we have
that we can think about as to why it might be malicious.
So we're just using split to make it a little nicer.
And there's a typo right there.
It's a double typo now.
Alright, so we're just going to run through it.
Actually, no, it's not.
So to make it nice, actually,
this isn't the solutions one,
so you guys already know this.
So if you want to make it a little nicer,
we're just going to split it again.
And we're going to get a list of the things.
So, looking through this list,
in my mind, some of the more malicious things
would be the AddTypeAssembly.
It would be ConvertToInt32,
NewObjectSystemIOMemoryStream,
so everything like in-memory attacks,
any sort of memory streams.
And then at the bottom, you also have
two Base64 string.
I think Base64 is typically something
that gets picked up quite easily.
But if we wanted to figure out
how this was...
why this was malicious,
or why Defender thought this was malicious,
does anybody have any ideas
that they want to type out?
Is anybody typing? I can't see.
Yeah, it doesn't seem like there is
anybody typing on Discord right now.
Nice.
Yeah, so we're just going to go through the dictionary.
So, this is just what I came up with,
and it would be to try and determine
the commonality between all the malicious scripts
and see if... see what tokens...
and when I say tokens, I mean words...
words came to the top.
And to do this, we're going to use...
we're just going to do a dictionary.
And it's kept, you know,
key-value pair.
So I'll let you guys run through that.
And if you need to play around with dictionaries,
they're pretty awesome.
I like dictionaries,
but there is one other data structure
that I like even more than dictionaries.
Excellent.
So, we know what dictionaries are now.
And we're just going to decode the content,
and we're going to add it all to a dictionary.
We're only going to do one script.
And does anybody know why we might
just only want to do one for now?
Because it's good practice?
Yeah, it's good practice.
But when you are dealing with, I don't know,
thousands of scripts,
like half a million scripts,
if you have something that's going to mess up
halfway through, or you start down that path,
you're just going to waste a lot of time fixing errors.
So, if you can get it to work with one,
and then ten, and then fifty, and then a thousand,
I think that's a much better way to go about
processing large data, large data sets.
So, for this one we're just going to do one,
but we're going to split it twice,
and then we're going to just add everything
to the word index.
So, there's a fill-in-the-blank bit here.
So, if you guys want to go ahead and run that code,
and then tell me what the issue is.
And I'll give you clues.
It's on the line with all the question marks.
I didn't know that I needed to be this close to the microphone.
Or you guys are already way, way, way past this.
So, to get a proper count of the words
as we go through them, we need to just add one.
And you would have been able to tell
after you went through it,
because there's only one token listed for each.
But the script has at least more than one.
So, if we just fix that,
now we get a better representation
of what's out there.
Going one at a time, I guess I've found
I used to have a real issue with loops,
where I wouldn't break them up properly.
And so, doing it a little more slowly,
at least one at a time, has helped me.
And then we're just going to sort the dictionary
by token.
So now, this equals sign,
there's nine of them,
there's six curly brackets,
there's four new objects,
but nothing particularly malicious.
So there is a lot of punctuation in there,
but we can deal with that later.
So, now, because we're impatient,
and we just want to get to the machine learning bit,
we're going to run through the entire...
all of the malicious scripts.
And it might take a little bit on Binder.
Might even take a little bit here.
The... yeah.
I wish I... I tried to ship
all of the scripts to you.
Okay, so now we're just going to sort the words.
So this is all of the tokens
for all of the malicious scripts.
And it isn't ideal.
So there's a lot of numbers,
and this is fairly common
when you're tokenizing text.
And I would generally just say go a little slow.
But, nice bit is, we're seeing get proc address.
Does anybody know...
you know, in what operation
get proc address might be used?
Does anybody that works in an AV vendor
know what get proc address might be used for?
You can Google it if you want.
No? Nobody?
GT?
Any penetration testers?
Malware authors? Anybody?
I'm a data scientist.
That's fair.
I'll let you look that up.
But it's used typically when you're looking up functions
in other DLLs.
Most notably
I think used in
process injection.
Anyway.
Alright, so there's a lot of numbers.
It's fairly classic.
But we're seeing write bytes to memory,
memory address, a lot of malware type things.
Remote proc handle.
Obviously this is all PowerShell.
But the numbers are actually kind of annoying.
So...
Let's scroll through.
Are there any tokens in there
that are particularly interesting to anybody?
Rob, Logistic Aggression managed to answer your question.
Thanks, Rob.
Rob's the man.
What was it?
He said you get the address of a DLL function in memory.
Also heavily used in packers.
Yeah, exactly.
Okay, so the next bit.
We're just going to continue to filter down.
And I would say that's generally the case
with most of your data.
I think most of your time spent
when you're doing machine learning
isn't the math piece that you typically think of.
It's processing data,
it's filtering data,
it's making sure that
your data sets are balanced
or have a distribution that you want
or whatever it might be.
I said earlier that we're not inventing new math.
I'm not going to be the guy to invent new math.
The place that I can be most effective
is applying my domain knowledge
to what I know of machine learning
and being extra careful with my data.
Abraham Lincoln has a saying
about sharpening an ax.
I can't remember exactly what it is.
If I had 10 hours to cut down a tree
I would spend the first 8 sharpening my ax.
That sounds right, I could have made that up.
The same is true for machine learning
and I would say data science.
I don't know if
Comath, do you want to chime in there
as a data scientist?
It's true or not?
I tend to try something that works
and then go back and perfect it
to make sure I don't get little mistakes
because when you scale things up
it really screws you.
There's a bug 0.1% of the time
and now you have 100,000 things
the bug is guaranteed to happen.
It's better to clean those up
before you really...
If you're a data scientist
in the twitch chat
what do you think the percentage of cleaning
versus actual machine learning
data processing versus machine learning
I think that would be interesting
and if you're new to machine learning
I would say it's okay
to be impatient and go really fast
because you want to get the end result
but if you want to get quality results
it is better to go a little slower.
We're just going to create a new list
and the difference here is we're creating a find all
so we're going to start removing some punctuation
and numbers and then we're just going to add it
to a new list and then we're going to print
the top 100 tokens.
So all the numbers are gone
a lot of the punctuation is gone
I'd say that looks quite a bit better
than the other one.
We still have a getproc address
we're still seeing the malicious tokens
that we would expect
shellcode
so we haven't completely ruined our
I want to call them crispy bits
but we haven't ruined our
what we want to model effectively
so that's also there.
So this bit
I just said pick some tokens and go google them
and make sure that they're
see what comes up
that's kind of the fun bit
I suppose if we are live I can do that
let's see
that's what pehandle does
I'm using edge
that's embarrassing
Have you guys heard of Mimikatz?
Do you know if that one's malicious?
Yeah, I would say so.
PowerShell Type Loader?
Mimikatz is flagged as malicious, yeah.
Yeah, yeah, definitely malicious.
And I would say, as an operator, I think a lot of the industry uses a lot of the same scripts.
So, yeah.
In fact, the DLL injection, bookdllinjection, atgraver...
Yeah, so these are malicious type scripts.
Okay, so once we have that bit, you know, you could...
We could stop here, right?
So we have some malicious tokens that we know that, you know,
these are all tokens or words that were labeled by Defender as malicious.
So we could stop here, theoretically, and have malicious scripts that never, you know,
don't use any of these words.
Alternatively, you could do the same with the clean scripts
and only use, you know, words from those clean scripts.
Or, you know, you could do both.
But I think that would probably work.
But I think you would have a hard time making it repeatable.
And you wouldn't necessarily, you know, if something...
If it all of a sudden stopped working, I don't think you would have much recourse
in being able to figure out how or why.
And so I think this is kind of where machine learning shines for us.
And it's just the ability to iterate through massive amounts of data extremely quickly.
So, you know, while... I'm going to use the GPT-2 fishing analogy.
And it's like, you can spend an hour crafting, you know, one fish for five targets.
Or you could spend an hour correcting five unique fish generated out of GPT-2 for five targets.
GPT-2 language models aren't perfect, but they do help scale.
You know, so it's like you generate five of them and you can correct them,
make them seem, you know, realistic and not sound terrible.
In the same time you could to do one.
So, you know, in terms of pulling out insights from Windows Defender, I think this is also true.
All right. So we're getting to... we got to the machine learning bit.
And in this manual, you know, we referenced 380,000 scripts.
There are, I think, 410 in the whole piece.
So in that PowerShell link at the top, you know, it has that many.
We just use... I just chose the biggest folder that had them in there.
And then introduced them to Insight.
So we're going to get into some of the data representation.
I think data representation is probably my favorite part of machine learning.
It's really where you get to shape... kind of shape the output and shape...
not the model, because the model has its own architecture,
but you get to encode and embed your domain knowledge when you...
you know, I guess it would be called... it's fine if I'm wrong,
but you could call it feature engineering.
And that piece is extremely important.
So, you know, if you want to...
the outputs of your model are... the accuracy of your model is a direct representation,
correlation with the quality of your data representation and your feature engineering.
And it's... yeah, it's my favorite part.
Okay, so...
GT went through tokenization earlier today,
but we've already kind of played with tokenization,
but tokenization is effectively the process of splitting,
you know, the words into separate words.
So, you know, each...
I'm losing it.
Not into separate words.
You piece out sentences into a list.
So once we have a tokenizer, rather than...
we could write it ourself, like we did up here,
but I would imagine there's probably better developers out there,
and I trust them probably more than I trust my own code for some reason.
But if you imagine it as a corpus,
so if you imagine this corpus is actually just our big PowerShell list,
so each line is a PowerShell script,
you know, just a list of it,
and we're going to fit on text,
and this is just to create an index.
So we're just creating those data structures,
and you can see, you know, each word gets its own integer.
And, you know, in terms of sequential numbers,
like in a text prediction scenario,
like sequences are really nice,
because you can say, you know, what number comes after what,
and you can go back to your index and look it up.
Yeah, so if we can go, my cat likes mittens.
I don't have a cat, but I assume cats like mittens.
Or most cats are called mittens.
But we're just taking the first index and we're tokenizing it.
Yeah, six small cats.
Yeah, it's a word index.
It's fairly straightforward.
And there are a number of ways to represent text.
You have machine learning models that'll do it,
you have frequency models,
you have TF-IDF,
which is Term Frequency Inverse Document Frequency,
which is like a weighted frequency.
What else do you have?
What am I missing?
Obviously, one-hot encoding, that's where we're going to go next.
So one-hot encoding is, basically, it's a vector,
which is the length of your vocabulary.
And the presence of a token or a word is denoted by a 1.
And if it doesn't exist in a document, then it's a 0.
And so what you end up getting is...
Let's see how long these are...
Well, it's only 17.
So each vector is going to be 17 integers long.
And the array is going to look like this.
So this is just the first sentence.
But as we go through, you'll see as the words change,
you get different... what do you call it?
Not indications, activations?
That's probably incorrect.
You get different indications.
Hopefully that makes a little bit of sense.
And let's see...
The nice part about one-hot encoding is that you do get every token in.
Once they're well represented, you do get some sparsity.
So if you have a really big corpus or vocabulary, you get some sparsity in there.
But it's really nice if you only care about the presence of a word.
It doesn't necessarily keep the semantics or the syntax.
It's not very good for text prediction.
It is good for classification.
Which is kind of what we want to do.
Alright, so now that we have an idea of what tokenization does and the tokenization scheme that we're going to use,
we're going to get into my favorite data structure, which is named tuples.
Named tuples are immutable... well, they're tuples, but they're like immutable classes.
So you can kind of create them, you can set your variables to whatever you want,
and it's really nice because then you can turn around and reference a tuple,
or reference a variable inside of a tuple, by name,
which makes your code read a lot cleaner than if you were just going to use a tuple or a list or even a dictionary.
Dictionaries can be... actually, I don't know the speed difference between dictionaries and named tuples.
Does anybody know?
If there's any?
Maybe not.
But this is an example of a named tuple.
It's like when you find something you like and you just use them absolutely everywhere.
This is where I'm at with named tuples.
Just because they're super nice.
Alright.
So this code, we are going to kind of create our training structure.
So this is... you should be familiar with everything in this code block by now.
So we are... we have our filter.
The tokenizer has a filter attached to it.
If you want to use that one, we're going to create a primary list to hold all of our training data,
and then we're going to hold a... we're going to create a list that we're going to use for tokenizing.
And this is kind of... eventually, when you start building these small pieces,
eventually there's a point where they kind of all come together,
and you can transition from processing into learning.
And this script here is where we do that.
So, all of a sudden we can... I shouldn't say script, but...
So we've built all these tiny pieces out, and you can kind of see how they're coming together.
And there's two really important things happening in this little code block.
Obviously the tokenization is happening, and we are collecting all of the... all of the named tuples.
So I know named tuples.
So we're going to go through every malicious script.
We're going to decode it as we did before.
We're going to remove all the punctuation.
We're going to split it.
We're going to unique it.
I did this because I assume it makes tokenization faster, but I don't...
As I said, I don't have any evidence for that.
It just seemed to make sense.
Maybe you guys can try and let me know.
We are going to then rejoin the unique tokens, unique words in a sentence,
and we're just going to add them to a named tuple, and then add that named tuple to a list.
So the important piece here is the label.
For malware, the label is a 1, and then for clean, labels are a 0.
So this is where we're kind of creating our training structure.
The other piece you might notice is originally we had 380,000 scripts that we ran this on for this demonstration,
this workshop that GitHub didn't really like it.
So we only gave you 3,000, but that's still like a 2-to-1.
But when you're having 380,000 scripts versus almost 1,200 scripts, your data is very unbalanced.
So if you, for example, were to do a similar scheme as we did earlier with just the malicious scripts, we did it for both,
what might end up happening is your clean script words will start to...
The word isn't drown out.
What's the word, Eric?
Spin.
What's the technical term?
I'm going to say drown out.
Or average out, whatever it might be.
It's like if your GPA is really bad, eventually you just can't get it up.
Anyway.
Alright, so then at the end of this...
The outcome of this is we're going to get a list that has all of our named tuples that are ready to be fed into a model,
and then we're going to have all of our documents that are ready to be tokenized.
So at the end of this, we're just going to build the vocab, and we're going to have everything.
This is a little blurb.
So it takes a little bit to go through all of them.
It's because I opened this.
I don't want to tokenize that.
Also, you guys probably know this, but if you guys use Vim, it creates a swap file.
So I helped debug some issues where they were opening their data files inside of a folder and running Python out of the same folder.
And so their Python was trying to tokenize their text, and they couldn't figure out the error.
And it was just because they were trying to read a swap file.
So now we have our vocab.
And we can kind of see the tokens that came out.
These are the tokens after our filtering.
The closer you get to training, you want to be increasingly happy with what you're seeing in terms of the words or the tokens.
And I think that I'm more happy with this than when we started.
But definitely play by ear.
But it is useful to take a moment and see.
Just take a look.
Remote DLL handle.
Just take a look.
There's quite a lot of them.
So this is something I haven't dealt with, and you can see in the insights.xls.
But if you were going to deal with it, this is probably how you would do it.
So you just want to do a regex for this TVQQAAMA.
Does anybody know what that is?
I bet Rob does.
I see you're binging it.
Yeah, this is embarrassing.
Bing, the best way to Google.
But yeah, you'd want to remove those obviously.
I didn't.
It indicates that it is a PE file.
Compressed or embedded in some other medium.
Okay, so now we're at the point where we are...
Well, I already described this, but...
If you're not familiar with machine learning at all, whatsoever,
there's a little video series by...
What's his name? I don't know.
But his YouTube channel is 3Blue1Brown.
It's just like 4 videos an hour.
It is...
It's really good.
It's going to be way better than I will ever be able to explain machine learning to you.
So I recommend having a look through that.
Watch it a couple times.
And if you're really into math, he has a lot of cool stuff.
Yeah, his animations are really good.
Neural networks, we've kind of all seen this picture.
But our input layers represent our input data.
Or our one-hot encoded text.
Hidden layers represent an activation function.
And the output layer or node is the result of the network.
Or classification, prediction, whatever it might be.
There are a lot of...
Machine learning is more than this little picture that we see everywhere.
And I would say this is like the tiniest little bit of machine learning.
And even inside of this picture, the mechanics that are going on are quite extensive.
And this is just like the most vanilla machine learning implementation.
I think the nice part about Keras and the frameworks is they bring that kind of ability and power to non-mathematicians.
What do you call non-mathematicians, Sven?
I don't.
Is there an industry word for them?
Students?
What?
Students?
Yeah, something like that.
Normies.
Normies, yeah.
Honestly, we just don't think about non...
Yeah.
I don't want to dig myself into a hole here.
You've got too much math to think about.
As a non-mathematician, I take great offense to this.
Yeah, yeah.
Laypeople is right, yeah.
Rich has got it right. Laypeople.
Yeah, laypeople.
And if you are... I think the first book I read was Make Your Own Neural Network by Tariq Rashid.
And that was... it was just the most... it was like 120 pages.
It was just the most basic, most straightforward explanation of a neural network.
I think there's a bajillion tutorials out there, and I think they are okay, but they're always so quick to get into a framework without actually explaining what's going on underneath.
And that is definitely a hindrance.
You don't have to be a mathematician to use machine learning, but you should at least spend time learning the basics.
Okay.
So.
Let's see.
We have a little explanation of machine learning.
We have our training set.
We're happy with the tokenization.
To a point.
Happy enough for now, anyway.
You know, eventually...
I think you always want to be moving forward, so you always kind of want to be thinking in pipelines.
So I wouldn't spend too much time necessarily in one area, but when you build it, make it such that it is modular and can be put into a pipeline.
Let's see.
So now we need to tokenize all the documents.
And this will take a list, which I think docs is.
We'll find out.
Nice.
And then we can... let's just double check.
Yeah, that's good.
Alright.
And now we're going to create our score array.
So here we have...
So this is a named tuple that we're going to go through.
So we're referencing E, which is a terrible variable name, sorry.
But alt text is the list that's holding all of our named tuples.
And this is called list comprehension.
Another favorite of mine.
So it creates a list.
Put a function inside of square brackets and it will create you a list.
But you can see we're referencing the labels.
So we're just going to create a score matrix.
And actually you can see what that looks like.
But they're just giant arrays.
And the reason they're giant arrays is because mechanics of machine learning is kind of...
Would you say rooted in matrix multiplication?
No, because that ignores all of the decision trees and a bunch of other things.
Yeah.
So when I was talking about this picture being kind of dumb, it's the only picture we ever see.
And it's just one of the smallest pieces.
There's so much more out there that you should look at.
And I probably could have introduced you to.
So now we have our score matrix.
Actually no, I'm looking at this.
Anyway.
So these are all our labels.
So when the network is learning, it's going to calculate a loss.
So it's going to have an input.
And then at the end of that it's going to take whatever the label was and see how close it was to the label.
So a large loss is bad and a small loss is good.
And through gradient descent and back propagation, the network will update the weights such that the next time it sees or there's something labeled like that, it'll hopefully be closer.
And we'll get into that.
I like this. This is probably one of my favorites.
So TAS vs. SPACES, NANO vs. VSCODE, MESSI vs. that other guy.
Machine learning frameworks are no different.
So do you guys have preferences for machine learning frameworks?
Why do you choose the machine learning framework you do?
I use PyTorch because it tends to look more like real code than TensorFlow.
But Keras is super easy to get up and running.
So it's a good choice.
And then when you do the real web stuff, I reach into a compiled language like Rust or C.
Oh, C, that's brave.
Has anybody used ML.NET?
Did anybody know ML.NET existed?
No.
Depends on what you want to do.
I have a Win32, so DirectX.
There's a Win32 machine learning implementation.
Actually, it's just an interface for TensorFlow on X models.
Yeah, I prefer Keras.
It's simple.
If all the math is the same, then you're really just looking at what you prefer.
Keras is good.
To get off the ground, but it may limit your customization.
And I think that's exactly what an API...
It doesn't limit you, it abstracts complexity.
So I think the reason Keras is so easy to use is because it does a really good job of abstracting complexity.
But if you want that complexity you want to dig in, it's still there for you.
You just have to potentially dig a little deeper.
Versus something like PyTorch, which is...
pretty raw, I would say.
That's probably not true.
Some guy in the back is yelling about MATLAB right now.
Oh, MATLAB. That's a whole other
stack of words you'll have a little bit of PTSD about.
Yeah, don't ever say MATLAB.
Rich is saying
Keras is good for 99% of your projects, and then PyTorch
for when you're getting freaky. And I say PyTorch is good for 90% of your projects
and Rust for when you're getting freaky. There's also JAX, all sorts of stuff.
Oh yeah, JAX. Does anybody use JAX? Do they like it?
I know Jason. Is he in here?
Yeah, it's kind of the Wild West, I would say.
There's a bajillion different frameworks.
Everybody uses something different.
I like TensorFlow. It has TensorFlow
Serving. It seems to have a good ecosystem
around it. But you obviously make trade-offs, right?
But if your
verbiage and your fundamentals are good, then I think you could probably use just about any language
with enough practice.
We're going to create our model now.
It's the same as the picture
effectively, but we're just going to create it in code.
One thing we didn't talk about are activation functions.
Actually, we didn't talk about it a lot.
As your data traverses from weights
into hidden layers and hidden nodes,
weights, their job is to modulate
inputs based on
their inputs.
Such that the output is relative.
Sorry.
I just lost him in the chat.
We're just going to use Sigmoid. I think Sigmoid is super simple.
It's a good place to start.
What are some other favorites? I know Relu
has replaced Sigmoid.
Is there a good reason for this?
Does everybody go, oh, this is better, so we're going to use it all the time now?
The Relu paper
basically said
Relu gets better accuracy scores than Sigmoid.
They just compared that across a bunch of things. For images, Relu
tends to have a higher accuracy score
than Sigmoid does.
And that's basically why it won, because on images you can easily
show that it's better. I don't know whether it's concretely better
in all situations because of that.
You're showing your age with Sigmoid here.
Relu also handles the disappearing
differential.
Vanishing gradient. Where if you go too far into
the negatives or too far into the positives, you're never going to crawl yourself
back out.
And then there's Leaky Relu that tries to help that even more.
And there's Ellu who tries to say, hey guys, you could be
Relu, you could be Leaky Relu, let's just combine the two.
I think it never goes below negative one as a weight.
I think this is
actually, I like this discussion. I don't know how many
of you listening are machine learning people to begin with.
And this is why the tutorials are kind of
they're nice to get you off the ground, but I think they're limiting
in the fact that they're always using kind of the same
architectures. So they're never to say, oh this is
good for this and this is good for this, and that's true. And you might end up
there anyway like they are. Maybe there's just lessons they've learned that you haven't.
But I think it is important to explore
different architectures, different losses, just play with
numbers. I think machine learning is ultimately
about iteration and experimentation
at scale. And that includes
activation functions, that includes output nodes, that includes
any lever you can pull. You should pull it
and see what happens.
Also, as a data scientist, you
should pull the levers and see what happens. Because you might
find that on security data, sigmoid works better than relu in this
case. And if you never tried sigmoid because you just go with standard
relu, you'd never learn that.
Read everything,
try everything. Hang out
at AI village. Super smart people.
Just hang out at journal club and
be a little fly on the wall. That's what I do.
Okay, so we've got our model now. One thing we need to do is we need
to put our documents,
our vectors,
into our model. And when I was first starting
this was actually one of the harder pieces that I had to fit in my
brain. I think that's true. It's just the shape
of arrays and how they get introduced into
models. Does anyone in the chat want
to take a stab at
what these three question marks should be?
You can try it.
I'm hung up on the missing parentheses.
Hey, you're right.
Yeah, but Rich has got it right. The feature size.
Yeah, exactly. Let's see, what is that? That would be...
I don't know.
Pretty sure it's your matrix.shape
variable.
Text matrix shape.
Man, that is one thing.
What are the things that you guys google constantly?
Sorry, you guys must be
probably missing some imports.
I'll push a new version. Alright, so if you're missing those imports
you're going to want to add. Nice.
We created our model. It's super simple.
The text matrix, the input dim that we're
doing is an array of
let's see, probably like 80
86,000
tokens. So across this you're going to see
that's going to look obviously like this.
So we just have, I think, 1,200
samples times 80,000 tokens
long. And we're just going to feed it
in. Nice.
And then we're going to do a test train split.
And when you
build a model, you obviously have test data and you want to keep some test data out.
But the idea is eventually that your model will be deployed into the
real world where it's not going to be trained on, it's not going to be seeing test
data necessarily. So ideally you want to keep some data
away or out of your training set.
So when it sees real data
or when it sees data it hasn't seen before
it can make hopefully an accurate guess.
So splitting them out
for that reason.
I've heard you guys talk about training
leaking
into your training data during training.
How does that happen?
One of the ways it could happen is
you could have duplicate samples
that are, normally you just say, pick a random
set of indices or pick the last 20% of my indices
and that's my test set. Or you do a cross-validation
where you divide it up somehow.
If you had duplicate data, say the last 20% of your data was actually duplicated
with the first 20% of your data and you trained on the first
four fifths and then you tested on the last one fifth
well since it's duplicated it's included in your training set and you do really well.
So you have to make sure that you don't have that sort of issue. And then there's other little
things that you could have.
It's basically a data cleaning issue
a lot of the times. Sometimes it's a weird bug
in your code. Sometimes you can just
reverse the model itself and you'll say, oh, show me
what this looks like and then it'll just print out an example
from the training set. Nice.
So we have early stopping
twice. Early stopping
is super nice. So if you have a really long running task and your
model is not improving, early stopping will just stop your model.
Part of me doesn't like it. I just feel like
it could get better. But this is a tiny
model so we don't really mind.
Callbacks. So if you want to use TensorBoard, whatever.
Batch size. This would be the frequency
at which updates to the waits will happen.
Epochs. This is just time
that we're going to do. Sorry, the number of times that we're going to
run through.
We can train. It's obviously
seems pretty accurate. I don't know
I haven't actually tried to optimize this
at all. But what do you guys do when
a model trains really accurately at first? Do you think it's beneficial
that you would try and overfit a model
at first? Because that would at least indicate it
could learn something.
Rich and I both get very paranoid when the thing does too well.
You've got an accuracy of 100%
but something's wrong. Something must be wrong.
Yes, something is wrong.
There could be any number of things. This one says version 1
and I'm giving you
the code so you can go and recreate the model.
Do whatever you need to and I would love to see
I'd love to see
some blog post or some write up about everything I missed.
So this is a visualization of the training loss.
Obviously with the gradient going down you would like to see
this going down. If this were inverted
that would indicate that, well it could indicate a number of things
but either it's not learning or it's overfitting.
And it's stopped learning completely.
There are some evaluation metrics.
Obviously it's going to be like, yeah I did really well on this.
As we saw up here. So that's a little suspect.
But even after you have a model, I like
to pull out a couple of the best
and worst examples of a category.
So we're just taking the first
malicious document
because we
put them in a list such that they were malicious
and then non-malicious. So we're taking the first, we're taking the second
malicious document and then we're going to take the
last non-malicious and we're just going to try and predict and see what they are.
And yeah, they're
a little too accurate for my liking.
But yeah, that's kind of it.
Okay, so this is kind of where we leave you. So the first
Excel file that you're looking at here, when I did it on
all 380,000 scripts, it took me
well, and this would depend on whatever kind of potato you're running
but it took me 17 hours to do.
So we're probably not going to run it here. You might already be running it
but have them reset a few times because you saw your thing froze.
But effectively what we're doing is we're just toggling absolutely every
possible combination and then
making a base prediction and then a new prediction
and then we're keeping a cumulative sum of those. And what you
end up getting is a spread of scores
across a number of predictions
that will sort your tokens into
malicious and non-malicious. This is the same
code from the Proofpoint research. The Proofpoint research was
easier in this regard because they had a wider
range of 1
to 999, where these are hard labels and it's a 0
or a 1. As a first run through
looking at the most malicious and the least malicious tokens
without any optimization, if it were ever
going to work, I would expect to see
that at least they're being sorted
to some degree. Because now you can go back and you can start tweaking
the model or whatever it might be
to really pull out, really hone in on what's
accurate. I think a lot of times for attackers, this
first version would be just fine. But
as you go into the future
you could even do this. You could always be collecting data
and actually I remember
after my first BSides talk about machine learning
last year, Rob came up to me and was like, hey, you should
have a separate data gathering campaign
so you keep your ops and your
data collection separate, but your data collection can support
your ops and it doesn't necessarily
burn you because collecting data can be
noisy. So it's a bit over time, but do you guys have
any questions? Comments?
What I would love to see is someone to take
this Defender model and absolutely crush it.
I didn't... I mean
it's a tiny model, I'm not sure Microsoft would care that much. There's a VBA
dataset I have not touched. So if you
guys want to race to whatever, I'd love
to see what comes out of it.
I appreciate everybody. Thanks for coming.
Hit me up in the Slack or not Slack, Discord
if you have any questions or you just want to rag on my terrible code
that's fine too, but
I will end it there.
So I think this is the
last stream of the night, so we're closing out the
Twitch stream now. Tomorrow hopefully will go smoother.
We learned some things today, but I'll
see you guys all in the morning. Hopefully.
Hopefully.