I just want to double check this.
Alright, cool. I have all my stuff muted and we are good to go.
Hello and welcome to Machine Learning for Security Analysts.
This is going to be our final workshop for the AI village.
This is going to be a beginner and interactive workshop
for all those who are interested in machine learning and security
and what this whole crazy AI thing is.
So really, the major motivation for this workshop
was that we're seeing a lot of buzz in the security community
around AI and machine learning.
But for many of us, it's not really clear what that is.
So... oh, sorry.
If you want to participate and ask questions,
head over to the DevSchool channel and join the
AIVillage-General-Voice, AIV-General-Voice.
And you'll be able to ask us questions.
If there's too many people joining and it gets unorganized,
we're going to switch to Zoom.
But it's sort of more open than it was before.
Sorry, Gavin. Just needed to say that.
No worries. Last line of this statement is
I wanted to take, for the security community,
the idea of machine learning and AI out of the buzzword
and into the mainstream.
So, my name is Gavin. I go by the pseudonym GT Klondike.
I'm an independent security researcher and a security consultant.
I'm very passionate about network attack and defense.
And through that passion, I run a project called NetSecExplain,
which is a blog and a YouTube channel,
where I explain intermediate and advanced level
computer networking concepts and security concepts
in an easy-to-understand way.
So, before I get started, usually in person,
this is where I ask people,
what are your thoughts when you hear the word
machine learning? Right?
And I've gotten a range of answers in the past
that range from, oh, Skynet's going to take over the world,
to you need a PhD in multivariable calculus
and linear algebra.
And while there is a mild amount of truth
to the math background that you should have,
neither of those are really true.
Instead, machine learning is really a form of pattern matching.
It's applied statistics.
And with the increase in power behind the computer memory
and behind the CPU,
we're able to use these statistics
in a much quicker, more processed way.
So, first I want to kind of cover
this idea of machine learning,
artificial intelligence, and deep learning.
In a lot of areas, it's being used interchangeably,
and that's not really correct.
First, what we have is this large umbrella
of artificial intelligence.
And artificial intelligence is
kind of a philosophy, in my opinion.
Artificial intelligence used to be,
when the first digital calculators came out,
that was considered artificial intelligence.
A computer can do math.
As things became more advanced,
different concepts were considered artificial intelligence.
There's video game artificial intelligence,
rule-based artificial intelligence,
classical image recognition.
But today we're talking about machine learning,
which is a subset of artificial intelligence.
And it's focused a lot more on
grabbing inferences from data
and making statistical patterns.
And then inside of machine learning,
we have a concept called deep learning.
Deep learning is specifically deep neural networks.
And so neural networks is a machine learning algorithm
that's pretty popular.
But deep learning allows us
to take that a little bit further.
And so usually we see this in the form
of convolutional neural networks,
which are primarily used for image recognition,
or recurrent neural networks,
which are used for cyclical pattern recognition.
So think of things such as a heartbeat monitor,
where there's a pattern involved
that you kind of need to match up.
And if there's any sort of heart irregularities,
the pattern will diverge from predicted patterns.
So if we wanted to formally define machine learning,
I would say that it's a set of statistical techniques
that enable the process of information mining,
pattern discovery, and drawing inferences from data.
So the idea is that the algorithms learn
from past data to predict future outcomes,
instead of creating a different algorithm
for a different scenario.
And we'll see that as we go through the workbooks,
how we're able to create multiple different types
of algorithms around the same set of data,
or how we can use the same algorithms
with different types of data.
So some key examples,
especially in the security sphere,
is domain generation algorithms.
Domain generation algorithms are heavily used
by botnets so that the bot herder
can stay in contact with the botnet itself,
while government agencies or ISPs
are taking down the domains
that they use to communicate.
We also see this with web application firewalls.
We can perform anomaly detection
to identify normal web traffic,
normal parameters that are sent by a web client,
versus potentially malicious parameters.
In this example here,
it's showing a possible SQL injection,
and it's calling that out.
And then, of course, network anomaly detection,
where what we can do is predict
how the network will operate over a period of time,
and then keep track of how it is actually operating.
And if it's beyond a certain delta or a difference,
then we can flag that and send it off
to a security analyst.
So why is this important for security analysts,
and also important for hackers?
Well, today, over a quarter of security products
for detection have some form of machine learning.
So from an analyst's perspective,
what you want to do is understand
how these machines work,
so that you can ensure that they operate efficiently
and are working effectively.
Machine learning isn't this magical black box
that just solves all of our problems.
And I want to share a story with you,
and this is what really made the idea of machine learning
and the value of machine learning really stick out to me.
So a number of years ago,
a friend of mine was approached by a non-profit.
And what this non-profit did was track
cheetah population out in Africa.
So they set up a bunch of different cameras,
and they would take a picture at different intervals,
and it would be sent off to a human analyst
to identify whether or not there's a cheetah in the picture.
Not a job I would want to do,
but it needs done, and it's really important.
So they approached my friend,
and they were saying, hey, is there a way
that we can increase our efficiency?
Something that we can use to kind of automate
some of the manual labor that we're doing.
And so what he did was spend about a week,
and he used a very simple machine learning model
to identify images.
And what he told this model was,
hey, if you see an image,
and you are at least 5% confident
that there's a cheetah in the picture,
send it off to an analyst.
If you're less than 5% confident,
just throw it away.
Now 5%, that doesn't really sound like a lot,
and to be honest, it's not.
But what this non-profit was able to do
with this simple model was
what would normally take them a year of labor
they were able to complete in one month.
That is a 1200% increase in productivity
by just implementing a little bit of machine learning
and that fuzzy logic into their process
to kind of help automate it.
So this is what I want to do
to kind of push for the security analysts
and for the hacker community
to get a little bit more involved
in AI and machine learning
so that we can see and identify ways
that we can improve our own processes,
especially ones that require
a little bit of fuzzy logic,
things that traditional signatures
or scripting aren't sufficient in.
So the 7-step machine learning process
generally looks like this.
And this isn't a list
so much as a cycle, right?
So first we're going to start
with gathering the data.
We want to identify what it is
we want our machine learning algorithm
to classify.
So first we're going to gather the data
and then we're going to prepare the data
in such a way that is easy
for machine learning models to understand.
And in this case, it's usually turning it
into some form of number
or some form of signal.
Most of our time is spent gathering the data
and preparing the data.
The rest of this tends to be a little bit quicker.
So first, based on the data that we have
and the problem set that were presented,
we choose a model.
So some models perform better than others
in certain tasks.
And sometimes you just got to kind of
figure out maybe neural nets isn't
the best model for you.
Maybe logistic regression or SVM
or random forest classifier is better for you.
But once we've identified the model
or models, then we go ahead and train it.
And so what we'll usually do is perform
what's called a training testing split.
And in this case, we have a series
or a lot of clean labeled data.
And we take that data,
we separate 80% of it and 20% of it.
80% of it is what we use to train the model.
And then the other 20%
is what is used to evaluate the model
to see how well the model is performing.
And so it's kind of like how we teach
students in elementary school, right?
We train them on the bulk of information
and then we test them to see how well
they're understanding the material.
And so some of our models
will perform very well.
Some of our models may not.
And so if we have a model
that performs fairly well,
but we think it could do better,
then we hop into what is called
hyperparameter tuning.
And we're going to see what a hyperparameter
looks like in a little bit.
But hyperparameter tuning,
think of it as little knobs and levers
and switches that we can use
to kind of tweak our model
and tune it in such a way
that it is better at a certain type of problem.
And then once we've completed all this,
we're ready to deploy.
Now the reason why this is cyclical, right?
We want to gather more data,
prepare the data,
and use the same model
that we're already using after deploy
is because machine learning
stops learning as soon as it's trained.
So to kind of give you an idea,
I know that certain spam filters
like Facebook's Facebook spam filter,
they retrain it and rebuild it
every one to three days
because that's how quickly spam
is being processed through their pipelines
and through their networks.
And so it's a way for them
to kind of stay on top of that.
So we're going to hop into
understanding how machine learning works,
especially in the security space.
And the easiest example, in my opinion,
is building a spam filter.
Now spam filters,
we already know how they operate conceptually.
So it's a lower barrier of entry.
We're going to see specifically
how the machine learning fits in.
But before I hop into that,
are there any questions
on how machine learning,
AI, and deep learning
kind of intermingle
and why it's important
to understand machine learning
from a security standpoint
or in the hacking community?
I see on the Twitch chat,
reinforcement learning,
reinforcement learning, self-learning,
contextual regret minimization,
what differences between them?
I'm not familiar with counterfactual,
not familiar with counterfactual
regret minimization,
but typically you'll see
three types of machine learning.
You'll see classification
or supervised learning,
which is what we're going to be covering.
You'll see unsupervised learning,
which is very good for data exploration.
And unsupervised learning
uses a whole different set of algorithms.
And then there's reinforcement learning,
where we embed in a model
different rewards and punishments
based on how it's performing.
And so these are all different ways
of training the model,
and then we can make decisions off of those.
Hey, Sven, are there any questions
in the Discord chat before I move on?
Okay, I'm going to assume not.
So before we look at a full spam filter,
we want to kind of look at a smaller example.
So in this case,
we have a series of sentences
talking about sports or not sports,
which are really elections.
So here we have a couple sentences.
A, great games.
Sorry. Yes.
Well, I think you got this one.
But there was a question from Norwin.
I fell asleep on the job.
No worries.
It has been a long weekend.
So what was the question?
Norwin kindly repeated Glucorio's question.
So you already answered it.
Okay, okay.
So kind of going back,
we have a great game,
which is talking about sports.
We have the election was over,
which is talking about not sports.
Very clean match, talking about sports.
Clean but forgettable game,
talking about sports.
So if we wanted to take this idea
and specify a sentence
that we've never seen before
about sports and not sports,
how would we do this?
We would want to look at keywords.
So in this case,
we can see that it's talking about games.
We know it's probably about sports.
If it's talking about elections,
it's probably not talking about sports.
And so what we want to really hone in on
are keywords.
But to put this into a machine learning model,
we need to find a way
to turn this into numbers.
So the way that we're going to do this
is Bayes' Theorem.
Bayes' Theorem is...
it may look complex at first,
but it's actually a really easy
to understand probability theorem.
And in this case,
we see the probability of A given B
equals probability of B given A
times the probability of A
divided by the probability of B.
So in this case,
we want to see what is the probability
that this sentence is talking about sports
given that the sentence is
A very close game.
So we don't know the probability of sports
given A very close game,
but we can easily plug that into this formula
and say, well, we know the probability of sports.
We saw three out of our five sentences
were talking about sports.
And we can figure out the probability
of A very close game given sports
and the probability of A very close game.
But how do we do that?
How do we get a sentence like this?
Well, with Bayes theorem,
we can actually, or with probability,
actually, we can split this up.
So instead of A very close game
as a whole sentence,
we can look at the probability of A
times the probability of very
times close times game.
And so what this turns into
in our A very close game given sports
is probability of A in sports,
probability of very in sports,
and so on.
And of course, we're going to want to do
is say, okay,
whichever is highest,
sports or not sports,
that's the one that we're going to classify
this sentence as being part of.
So at this point,
it becomes a simple counting game, right?
So how many times does the word A
show up in sports?
Well, we see that it shows up twice,
once in the first sentence
and once in the fourth sentence.
And then how many times does it show up
in not sports?
Once, right there in the fifth sentence.
So A does not show up in sports.
It only shows up in not sports.
And then we see the word game
show up twice in sports.
So we just take those numbers
and we plug them in and we say,
okay, A showed up in sports twice,
very showed up once,
close showed up zero times,
and game showed up twice.
And if we counted the number of unique words,
we would have had 11 unique words
across all of our sentences
talking about sports.
But we run into a problem
and the problem is that
what we want our machine learning model to do
is identify and classify sentences
that it has never seen before.
Unfortunately, it's never seen close
and so that winds up being a zero,
zero times everything is zero.
And we see that that easily cascades down
and just ruins our model.
So it's because we've never seen the word close,
there's a zero probability
that this exists in our sentence.
So that's a problem,
but we can get around that.
There's another algorithm
called multinomial Naive Bayes.
And so this is just a small modification
and what it does is add
what is called a smoothing filter.
And we can see this alpha here.
It's added to the numerator and the denominator.
And usually this alpha is one,
but this is going to be our hyperparameter
for Naive Bayesian,
multinomial Naive Bayesian
that we can tweak, right?
We can set it to one, we can set it to two,
we can set it to 0.1.
And so this is a thing that we can tweak
to see if we can get our model
to work for us.
So we take the same idea.
We do the word counts.
We saw that a showed up in sports twice,
very showed up once, close to zero.
But now we have this plus one.
So the probability of close given sports
is never going to reach zero.
And then of course, as I mentioned earlier,
we need to do the same thing for not sports.
And the total number of unique words
in not sports is nine.
The total number of unique words in sports is 11.
And the total number of unique words
across both classes is 14.
So we plug in our answers
and we do our calculations
and we get 0.0000461 for sports
and 0.0000143 for non-sports.
Classifying the sentence,
a very close game being about sports.
So that was a lot that I just kind of laid on you.
If you have any questions,
type them in the chat.
While you're doing that,
I'm going to show you how we can
take our raw emails
and transform them into
these types of counts
and these types of numbers
in a very clean and effective way.
So the five things we need to keep track of
are the total number of unique words,
the total number of words in spam,
the total number of words in ham,
or regular emails,
the count of each word in spam,
and the count of each word in ham.
So let's take a look at one of our emails.
We see here,
East Asian fonts in Lenny.
Thanks for your support.
Installing Unifonts did it well for me.
Now we want to take a sentence like this
and find a way to classify it
as being spam or ham.
But before we do that,
we want to kind of point out a couple things.
So we see here kind of towards the bottom,
we have the word unsubscribe,
all capitalized,
and then unsubscribe lowercase.
What we want to do is look at keywords.
And so in context,
it doesn't matter if unsubscribe is capitalized
or the first letter is capitalized
or it's all lowercase.
It means the same thing.
And so to kind of reduce our problem set,
what we're going to do is set everything to lowercase.
The other thing we want to do
is remove what are called stop words.
Stop words are words like
the, in, of, on, or...
They don't add any context to the sentence.
And we're going to do the same thing for punctuation.
So we're going to remove those.
We're going to keep our keywords.
And then one more thing that we're going to do
is called stemming.
So if we have something like the word
congrats or congratulations,
in the English language,
they mean the same thing.
So if we're looking at keywords,
we want to treat them the same.
And so what stemming does
is shrink those words down to their base form
or the root form.
And then we can count them the same.
Another really good example is the word thanks,
thank you,
where it's a just singular thank.
And I thanked somebody today
with an ed at the end.
As a keyword, they all mean the same thing.
And so stemming will reduce them all down to thank.
And so instead of three separate words,
it's going to be counted as one word.
And so that's going to drastically reduce
our dimension space.
So after our preprocessing,
which is what this is known as,
we're going to see East Asian fonts in Lenny,
or East Asian fonts Lenny,
thanks support installing Unifonts,
well me.
And then what isn't shown here
is what the stemming looks like.
So as we go through the workbook,
we'll take this exact same email
and see how it's transformed
at every step of our preprocessing.
So are there any questions so far?
I don't think so.
There's none on Twitch chat,
and there's none in the Discord.
Perfect.
So what we're going to have
are a series of workbooks.
We're going to use MyBender.
MyBender is really awesome.
TAs, if you don't mind sharing this link.
What it will do is spin up a Docker container
with the workbook linked to my GitHub
so that you can kind of code with me
as we walk through
and build our own machine learning classifiers.
While this is starting up,
what we're going to be using is
instead of coding everything raw
and doing the raw numbers like we did
on our pen and paper,
we're going to be using
the abstraction library Scikit-learn.
Scikit-learn is very useful
and it allows us to treat things
kind of like Legos,
where, OK, maybe I don't want
multinomial Naive Bayesian,
maybe I want logistic regression
or random forest.
We can just swap that out.
So once it's loaded,
and I'm going to give you a moment
to get this loaded up.
If it takes a little bit longer
than it should,
then you'll easily be able
to catch up to me
and I'll show you how.
So we're going to click on workbooks
spam filter sklearn.
If you're not very familiar
with Python, you can still follow along.
We have a completed workbook,
which is the workbook complete
spam filter sklearn.
And these are Jupyter notebooks.
If you've never used Jupyter notebooks,
I think they're the best thing
since sliced bread,
especially when it comes to data science.
But it allows us to interact
with code blocks
and visualize things in line.
And I think they work fantastic
together.
So let's go ahead, scroll down.
We're going to run this first code block.
And so what this is going to do
is install a couple of libraries
that we need,
as well as set our data directory
to where the data directory is
in the GitHub project.
So to run these, you can click run
up at the top,
or you can press shift enter,
which is what I'm pressing now.
And then you can see that it's running.
A star in the upper corner
means that it's currently running.
And then a number in the upper corner
means that that was the order
that code block was ran in.
So while that is going,
I'm going to
let it go until it stops
scrolling my thing.
So we're just going to import
a couple libraries.
We're going to import numpy,
matplotlib for graphs,
regular expressions we'll be using
and NLTK.
NLTK stands for
Natural Language Toolkit.
This will allow us to identify
and collect a list of stop words
to remove those things.
And it will also allow us
to use a stemmer function
to stem the words
and reduce our problem space.
Train test split,
I don't believe we're actually using that in here.
The data directories I have
are already split into
80% training,
20% testing.
And then we're going to use our vectorizers.
And we'll explain what the vectorizers do,
but we're going to take a look at
both of the TF-IDF vectorizer
and the count vectorizer
because they operate
slightly different.
Count vectorizer just takes a straight word count
like we did with our pen and paper example.
And then TF-IDF actually takes
all of the words into account
and gives them a weight.
Scikit-learn,
we're going to use logistic regression
and multinomial naivetation.
And the reason why we're going to use both of these
is because we don't actually know
which one's going to perform best.
And so like good data scientists,
we're going to try a couple different models
and take the one that performs best
with the problem set that we have.
And then for a couple metric
and visualizations,
we're going to use the Scikit-learn
confusion matrix and classification report.
So I'm going to go ahead
and press Shift-Enter
and that's going to run.
You will see libraries imported
once that's complete.
And we can move on to the next thing.
So this is our test email.
This is the exact same email
that we saw in the slides.
So as I mentioned earlier,
we're going to use this email
to see how it's transformed
at every step of preprocessing.
So I'm going to go ahead and run that.
And then this is our tokenizer function.
Tokenizer functions will allow us
to pull out the keywords
in the way that we would like them
to be extracted.
So in this case, we're going to grab
all the punctuation, all the stop words,
and we're going to create a
stemmer constructor.
We're going to use these to
remove the stop words from the email.
We're going to use these to remove
punctuation from the email,
and then we're going to stem the words.
We're going to move all the words
to lowercase so that they are
treated exactly the same.
So we're just going to go ahead
and run this, shift enter,
and our tokenizer is defined.
And this is where we run into
our first task, which is to
tokenize an email.
So our task says that we want
to print the full email, test email,
and then we're going to print the
results of the tokenized version
of that email.
So here I'm just going to do
print tokenizer test email.
And we can go ahead and run that,
shift enter. We can see the
original email that we have,
East Asian Fonts and Lenny,
thanks for your support.
We can see, just like in our
slides, we have East Asian Font
Lenny, slightly changed,
thank instead of thanks,
support install, because we can
have installing installed with
an ed at the end, or I
will install this software,
which is just a standard install.
We see unsubscribe has been
changed. And two of the things
that I want to point out is
every single word in here is
unique, except for the word
unsubscribe, which shows up
twice, and list.deviant.org,
which shows up twice.
So we're going to take a look
at our training data. What this
is going to do is identify the
directories where our spam
files are, our ham files are,
and our test files are. We're
going to get a count, because
that's one of the five things
that we needed to keep track
of, was the count of, actually
we want to keep track of the
count of words, we don't care
how many emails we have. So
I'm going to go ahead and run
this, and what we're going to
do is put those emails into
what is known as a corpus.
So we're going to create two
list arrays, so corpus and
labels. Corpus is going to
hold all of our emails for
spam and ham, and then the
labels is going to be each
label for our data. So the
first thing we're going to do
is corpus equals a new list,
labels equals a new list,
and then we're going to do
the same thing for spam and
ham. We're going to load the
email bodies from the ham
directories and the spam
directory into the corpus
array, and then we're going
to load the labels for each
email into the labels array.
So what that's going to
look like is we're going to
use a for loop here, so
we can say for each in
os.listdir, and then we
have our data directory,
which is defined above.
We're going to look at our
ham directory for this
first round. With each
open file, we're going to
look at our data directory
plus ham plus os.listdir.
We're going to look at
each, because each is the
name of our file, and
we're going to want to
read those as the file
descriptor f. So for
our corpus, we can do
an append and then f.read.
So this will read each
email into our corpus
and then labels.append
ham, because we know
that all of the emails
in the ham directory are
part of the ham data set.
And so we can easily do
the same thing for spam
which is we need to be
careful on how we change
this. So instead of
slash ham, it's going to
be slash spam, and then
of course our label is
also going to be spam.
So I'm going to let this
sit for a minute and let
you guys go ahead and
make sure that you copy
this down correctly.
This bit is pretty easy
if you're familiar with
Python. It might be a
little confusing if you
are not, but that's okay.
If you ever get lost at
any point, just remember
that there are a
completed solutions manual
in the workbooks directory.
So I'm going to go ahead
and run this. So shift
enter to run.
And I spelled labels
wrong. Labels, shift
enter to run. Loading
ham, loading spam, done.
So what this is going to
do is graphically load
or graphically visualize
how many ham emails we
have versus how many
spam emails we have.
And this is something we
want to generally do with
our data just to see if
there's any sort of
imbalanced data sets.
This one we have about
50% spam per ham.
It's like 2 to 1
ratio of ham to spam.
And so this I'm not
going to consider an
imbalanced data set for
this example, but if we
have something such as
like credit card fraud
where it happens less
than a fraction of a
percent of the time,
that's something that we
really want to dive into
simply because if we try
and use this exact
example on credit card
fraud, it's not going to
work because what we
will do is create a
model that is 99%
accurate because it just
classifies all credit
card transactions as
legitimate. Now if we
want to identify credit
card fraud, that's not
very helpful for us.
So this just allows us
to kind of explore the
data and see how much
of what we have.
So in this case we have
2,359 ham emails and
about 1,100 spam emails.
So task 2A, we're going
to view our training
data. So this one's just
to kind of see what our
data looks like. We
generally have an idea,
but we just kind of want
to play around with it.
And this is another thing
that you would want to
do in a data discovery
process is just kind of
see, okay, what is it
that we're actually
working with? So in this
case, I'm going to pick
an email ID, just email
ID 5. You can pick 5 or
you can pick any other.
And then I'm going to
grab the email, which
is part of the corpus
array. Corpus and then
our email ID. And then
I'm going to print that
email, and then I'm
going to do the same
thing and print the
tokenized version using
a tokenizer function of
that email. So we can
see how it's transformed.
So just so I don't go
too fast, I want to make
sure that you're able to
follow along and copy
this down. It's fairly
simple. So I will wait
just a few seconds. How
are we doing on
questions?
No one has been
posting some helpful
links, but hasn't had
too many questions so
far. So thank you very
much. No winner.
Okay. So when we run
that, what we have is
the email body. We see
product review, vert
tools, dev, 2.0 URL, and
then this URL. We see
product review, vert
tool, dev, 2.0 URL as
one word, HTTP, and then
the full URL. So this
is how we can identify
and kind of track our
data as it's being
transformed over time.
So task 3 is where
we're going to start
using our vectorizers.
What we want to do is
get a count of each
unique word, and we want
to get a count of each
unique word per class so
that we can use that in
our classification. So
there's two ways we can
do that. We can use the
straight count vectorizer,
which will get the count
of our words, and we can
use a TF-IDF vectorizer,
which will give us a
weight of each count.
So no winner says he's
behind. He's catching up
and he'll catch up
later and follow along.
I can try to figure out
a system of helping the
winner. I can send you a
Zoom link or something
and look over your
shoulder metaphorically.
Also, in the workbooks,
you can just open up
the... that's the wrong
one... you can just open
up the completed spam
filter SK Learn, and
it's completely filled out
for you. So if you ever
do fall behind and you
would like to catch up,
you can look at the
answers there.
There is a way to
follow along and make
sure your code works.
Yeah, absolutely. And I
like using the completed
workbook. You can just
run everything and see
how things are supposed
to be at each step of
the process. So we're
going to go ahead and
train our vectorizers.
And the instructions tell
us exactly how to do
this. So we're going to
use the vectorizer.
We're going to call it
cvec for the count
vectorizer, tvec for the
tfidf vectorizer, and
we're going to perform
fit transform. Fit in
scikit-learn world is
code for train. So this
is going to, quote
unquote, train our
vectorizers so they can
get the count of each
words. And then we're
going to save the
results as count x and
tfidf x. So what that
looks like is cvec
equals count
vectorizer. And then we
want to use our
tokenizer that we
defined above. So the
tokenizer that we're
going to use is called
tokenizer. I know, not
confusing at all. But
we're going to use our
count x. And pay
attention to your
capitals. Scikit-learn
is very picky and so is
my workbook. We're going
to use cvec.fit
transform. And then
we're going to train
transform or count the
words for our corpus.
And then we're going
to do the same thing
with our tfidf
vectorizer. tvec
equals tfidf
vectorizer. And we're
going to use our
tokenizer called
tokenizer. And then
our tfidf x
equals tvec.fit
transform. And it's
also going to be on
our corpus. So I'm
going to let this sit
for a minute and let
you guys be able to
catch up. And this
takes a little bit.
This takes about two
minutes to run because
what it's doing is it's
reading every single
email and it is
counting the words in
those emails in order
to get the number of
unique words. So I'm
going to go ahead and
run this because it's
going to take a little
bit. But in case you
need, here is the
code while this is
operating. And so we
can see that it is
currently training the
count vectorizer, kind
of down here at the
bottom. And then once
the count vectorizer is
complete, it will start
training the tfidf
vectorizer. And then it
will let us know when
that is complete. So
while this is going,
I'm going to go ahead
and move on to the
next code block. We
can write the code
while this is still
training, but we won't
be able to run it
because it can run
only one code block at
a time. So in this
case, we have task
3a, which is to count
the test email tokens.
So we're going to
manually count out
each of the tokens.
So what we're going to
do is say for i in
and this is a bit of
a trick so that we can
keep the count as well
as the words in order.
So we're going to use
a list of a dictionary
and we're going to
want to grab the
tokenized version of
our email. So let's
call this tokenized
email equals tokenizer
test email. And then
our from keys we're
going to have tokenized
email which we saw
above is going to be a
list. So we'll just
call that tokenized
email and then we're
going to count the
words. So print, do
some Python format,
And then we're going to look at our tokenized email.count of each word, and then the word itself.
So we see our vectorizing is complete, so we are ready to run this.
I'm going to let this sit for about 30 more seconds and let you guys catch up.
And then, of course, after this workshop, I have a whole series of workbooks, as well as their solutions.
So if you're interested in other ways to apply machine learning and security to different problems,
and using different algorithms, don't forget to take a look at those.
So I'm going to go ahead and run this, Shift-Enter, and we can see the counts.
So we have East Asian font, Lenny, thanks, support.
Each of these, we have our unique words, they only show up once in our email.
The only two that are outliers is unsubscribe, or unsubscrib, after our stemmer.
And this list.debian.org, they show up twice in our email.
So this is if we manually counted these, but what if we used our vectorizers?
So what we're going to do here in Task 3B is create new vectorizers,
the count vectorizer and TF-IDF vectorizer,
and then we're going to print out the counts that each of them represent.
So in the first one here, I'll just call this example-cvec,
and then we're going to define a new vectorizer.
And I'll just call this example-x equals example-cvec.fit-transform.
And then it expects a list, our email is a string,
so we're going to wrap it in these square brackets.
And we're just going to use our raw test email.
And then we're going to want to print our example-x.
And we can easily just copy and paste, do the same thing for our TF-IDF code.
We'll just call this example-tvec, we'll use the TF-IDF vectorizer,
tokenizer is tokenizer, we change this to tvec,
and then we're already done with our example-x,
so we don't need to change the name, we can just leave it as is.
Example-tvec, fit-transform, and then square brackets, test-email,
so that it's a list.
Yes.
Oh, it sounded like there was possibly a question.
So Voto on the stream is asking, where are those workbooks again?
Can one of the TAs post in the stream?
There you go, perfect.
Machine learning for a security analyst.
Okay, so I'm going to go ahead and run this.
Scroll up, let's take a look at the count vectorizer.
So instead of seeing the words, we see these numbers,
and that's exactly what we want.
We've created what are called tokens.
So the words themselves, the tokenizers will be able,
or the vectorizers will be able to match them together.
So all of them are one in their count, all of them are unique,
except for token 16 and token 9.
We don't know which one, either unsubscribe or list.wn.org,
maps to 16 or 9, but what we do know is that both of them show up twice.
So given that, list.wn.org is either 16 or 9,
and then the other one belongs to unsubscribe, or the word unsubscribe.
So this looks exactly the same as if we had done our raw count.
If we look at our TF-IDF vectorizer, though,
we see something a little differently.
TF-IDF, what it does is use term frequency
to give each term or token a weight.
And so in this case, we see all of the ones that showed up once
have a weight of 0.204, token 9 and token 16
have a weight of 0.408.
So you don't really need to understand the underlying math,
you just need to know that there are different ways to vectorize your data.
So there's TF-IDF, count vectorizer, word2vec,
and a couple others, bag of words.
And so there's different ones that we can try and see which one works better for us.
So let's hop down and start to load our testing data.
So loading our testing data, what this is going to do is
just show us the names of each of our files in our testing directory.
And we have train email.txt,
and then at the end we have the class that it belongs to.
So we have ham, we have spam.
So what we're going to want to do as part of loading our testing data
is we're going to want to strip off these labels.
So we're going to read the full email into our corpus,
and we're going to strip off these labels,
and those are going to be the new labels for our testing data.
So that hops us into task 4 to load the testing data.
Just like loading the training data, we're going to create two list arrays,
our corpus, we're going to call this test corpus, and our test labels.
And then we're going to load each email body,
and then strip off the label and throw that into the test labels array.
So what that's going to look like is...
So we're going to look at... what was the name of that?
Test corpus.
And we're just going to set that as a blank list,
and then test labels as a blank list.
For loading each of the emails, we're going to do just like we did before,
filename in ls.listdir,
and then our data directory plus the test directory.
So we're going to look at each filename,
we're going to open each file,
so data.dir plus test plus filename,
and then we want to read those as the file descriptor F.
So our test corpus, we're going to append F.read,
which is going to read that email into our test corpus array,
and then we want to grab the label.
And in this case, I'm just going to use a simple regular expression.
We're going to use txt, and then this is a backslash period,
and filename 1.
I'm going to scroll up real quick, because the reason why I want to do that is
what this is saying is, where we see txt.period, we're going to split,
so we have this full train txt.period, and then we have ham.
And then the bracket 1, what that is going to do is say,
this is bracket 0 once it's split, and then this is bracket 1,
so we're just going to keep our labels, which are at the end of that split.
And then with that label, or each label that we grab from the filename,
we're going to append it to our labels directory, or our labels array.
So this is what our code looks like.
Double checking for errors, looks good to me.
And I'm going to let this sit for a little bit,
so you guys can go ahead and copy that down.
How are we doing?
No questions in Twitch stream, are there any questions in the Discord channel?
Okay, I'm going to go ahead, shift, enter, and run.
So this is going to quickly load the emails.
And then let's run this next code block.
So what this is going to do is allow us to visualize our data, just like we did before.
So we see in our testing set, we have 590 ham emails, 276 spam emails,
and the graph itself looks nearly identical to the one above,
and that's because we did a random 80-20 split for our training and testing.
And that was already done beforehand.
So the next thing we want to do is use the vectorizers that we created,
the cvec and the tvec vectorizer,
and take a look at the test corpus,
and get our counts of each word in the new emails.
So let's go ahead, testCountX equals, and we're already going to use cvec,
we're not going to use fitTransform, instead we're going to use transform,
because we're not trying to train it, we're just trying to save the words themselves.
We'll do testCorpus, and then testTFIDFX equals tvec.transform testCorpus.
And that's all we need to do when we're using a piece of testing data.
So I'm going to let that sit for a second,
actually I'm going to run it because this is going to take a little bit as well.
Slightly shorter than when we were training it,
but it still needs to go through all the emails and apply the counts,
and then the TFIDF, it needs to apply the weights.
We already know what the counts are, we already know what the weights are,
we just need to apply them to each token.
So vectorizing is complete.
And now we're finally on to testing and evaluating our model.
So you remember how I said data collection and preprocessing
is what's going to take the most amount of time?
Well, that was about 80% of this workbook.
If not, maybe a little more, maybe 85%.
But yeah, data science in a nutshell.
So here we have a very helpful report generator function.
It's not important to know what's going on in this,
it's just going to be a helper function,
but we do need to know that it's going to take in our confusion matrix,
our score, which is our accuracy, and the classification report,
and then it's going to print out all the stats that we need.
So that hops us into task 6A,
which is going to be training and evaluating our first model,
which is going to be multinomial Naive Bayesian using the TFIDF vectorizer.
So there's a lot of steps to this, but we're going to walk through it step by step.
First, what we're going to do is create a multinomial Naive Bayesian constructor,
so that's going to look like mnb, tf, idf,
and the names of these variables are very important
because that's what's being used in the generator function.
So I'm going to call this Numial Naive Bayesian,
and then we're going to train the multinomial Naive Bayesian,
as I already misspelled something, mnb, tf, idf, .fit,
which is scikit-learn's code for train,
and then we're going to use our TFIDF, our vectorized corpus,
with the TFIDF array.
So now we're going to get a couple stats,
so we're going to get our score,
mnb, tf, idf, equals mnb, tf, idf, .score.
This is going to give us our accuracy,
and we want to compare this against our test, tf, idf, x,
and our test labels.
And then we want to take a look at the raw predictions,
so we're going to have our predictions, mnb, tf, idf,
equals mnb, tf, idf, .predict, using the predict function,
on our test, tf, idf, x.
Cool.
And with those results, what we are going to do
is generate a classification, a confusion matrix,
so we can see our true positives, false positives,
true negatives, false negatives.
So cmatrix, mnb, tf, idf, equals our confusion matrix,
and then we're going to use our test labels
and our predictions, mnb, tf, idf, which is our raw predictions.
We're going to do the same thing for our classification report,
so we'll see report, mnb, tf, idf,
classification report, and in the same order,
test labels and our predictions, mnb, tf, idf.
So I'm going to let this sit for a second,
give you a chance to copy all of that down.
So what's really helpful here is seeing how
at each of these tasks, 6a, 6b, 6c, and 6d,
we're only going to need to change very few things.
We're going to need to change, instead of tf, idf,
we're going to change this to count to do our train and evaluate
the multinomial Naive Bayesian with the count model.
And then we're going to change this from
the multinomial Naive Bayesian constructor
to the logistic regression constructor.
But other than that, the rest of our code is going to be identical.
The rest of our preprocessing is going to be identical.
And so this will allow us to train a total of four algorithms
using two separate machine learning algorithms
as well as two separate vectorizers.
So this is going to be very helpful for us.
So I'm going to go ahead and run this.
So shift-enter to run.
It looks like I misspelled something.
Multinomial.
There we go.
Required argument y.
That one's new.
Give me one second.
Oh, that is what I forgot.
So we want to train our multinomial Naive Bayesian
with our data as well as our labels
so that it knows what belongs to spam,
what belongs to ham, just kind of in our pen and paper example,
what belonged to sports and what belonged to not sports.
So now I can run that.
And I see where we ran into that problem yesterday.
Okay.
I need to swap.
This is the problem that I had with scikit-learn earlier.
The test labels and the predictions.
We start with the predictions and then we follow up with the test labels
instead of the other way around.
So I go ahead and run that.
There we go.
So we know that we had 590 ham emails.
We know that we had, if we add these two numbers together,
276 spam emails.
So we can see the actual number of ham emails, the actual number of spam emails
in those columns.
But let me scroll up real quick.
So this is our classification report.
Classification report we're not really going to use in this example,
but it's good to know.
There are other ways to measure how good a model is performing,
beyond accuracy.
Precision, recall, and F1 score
are all used in their own ways.
I'm not going to go into that, but it's definitely worth understanding.
Instead, we're just going to look at the straight accuracy.
For this multinomial with TF-IDF,
we have a .875
or a 87.5% accuracy.
If we look at our true positives and our true negatives,
it correctly classified or predicted all ham emails
as being part of the ham class.
So these are all of our true positives for what is ham.
And then our true negatives, what is spam,
it only classified 168 out of the 276 that we have.
The other half of the spam emails
it also thought were ham or regular emails.
So our predictor is letting some spam through,
but the good news is it's not blocking any of our
legitimate emails.
Moving on to the next example, we're going to do the same thing.
We're going to use multinomial naivete
with count vectorizer.
This is going to be the same, and then be count.
I am actually a little lazy,
so what I'm going to do
is just copy this
and I can show you all you need to do
is just change a few things around and you have a working model.
So instead of multinomial tfidf,
I'm going to be using multinomial count.
We're going to use the same constructor.
So in this case it's called count,
and instead of using the tfidfx, we're going to use the countx.
score,
count,
mnb,
count,
and our test labels stay the same because we didn't vectorize those.
There's no reason for us to change this.
I'm just going to go ahead and ctrl-c and ctrl-v some stuff.
So all I'm doing is just changing the word tfidf
to the word count.
Everything else is staying the same.
We have our predictions,
mnb, count,
our test labels, that stays the same, we don't modify it.
mnb, count,
and then we change our predictions.
So I'm going to go ahead
and let that sit for a moment.
I'll let you guys either manually type that in
or do the copy-and-paste track that I did,
which I highly recommend.
It just saves quite a bit of typing.
And then we'll type out the logistic regression
in a moment because that's the next model that we'll be making.
How are we doing on questions?
I think we're all doing well.
Everything's quiet on the Discord chat, too.
Awesome.
So I'm going to go ahead and run this.
I'm going to do Shift-Enter.
And if we look at our accuracy, our multinomial
with the count vectorizer, we have a 94% accuracy.
So quite a decent bump
compared to our tfidf vectorizer
using the same multinomial algorithm.
What we see is about 26 of our emails...
not about, it's exactly 26 of our emails
in our test set got blocked.
They were considered spam.
These are the regular ham emails.
But we have significantly fewer spam emails
that are getting through our predictor.
And so more of them are being classified as spam correctly.
And so this is what we want to see is a dark blue over in this corner.
And a darker blue
or this lighter blue given the count
over in the bottom right corner.
So this kind of hops into a conversation
about false positives and false negatives.
Which would you like to prioritize?
And so it really depends on the model.
In security, one of the things that we're concerned about
is impacting the users.
So if you look at something like an antivirus,
it looks for things that are probably
talking out to the internet, trying to take over your webcam,
uploading and downloading files.
Well, what else does that?
Chrome, Internet Explorer, Firefox.
So when you're building
an antivirus,
it's better to have
what they consider false negatives.
Which is positive is
it is malware and negative is it is not malware.
It's benign. So it's better to allow
some viruses to run instead of
blocking legitimate applications.
Because you don't want to impact the user experience.
Because if you impact the user experience, it's just going to uninstall your antivirus software.
And that leaves them unprotected and that's bad business for you
when people are uninstalling your software.
So even for us, we want to consider
and weigh the factors of
is it better for us to allow some spam to get through
as long as we're not blocking any of the legitimate emails?
Or do we want to make sure that absolutely no spam gets through
and if we block legitimate emails, well that's just collateral damage.
So these are considerations that you need to make
when you're building models like these and as a data scientist.
So for the multinomial naïve Bayesian
with the count vectorizer, we have 94% accuracy.
So let's try a different model.
This one we're going to use the logistic regression.
So if we look at the instructions, we're going to use the
logistic regression constructor, but we're going to use this
LBFGS solver. And this is one of the
hyperparameters, or at least I would consider it a hyperparameter
for logistic regression itself.
There's a couple different solvers that you can use
but in this case we're going to use the LBFGS.
So what this is going to look like is
LGS TF IDF equals
logistic regression
and then we're going to set our solver
equals LBFGS
and then of course, just like we did with the multinomial
naïve Bayesian, we're going to fit
our train, our classifier
we're going to use the TF IDF
X, which is our test data, and then we're going to
use our training data, sorry, and then we're going to use
our labels.
Next we're going to hop in and do the score predictions
classification matrix and, sorry, confusion matrix
and classification reports. We're going to use
score LGS TF IDF
LGS TF IDF.score
We're just going to look at our
test TF IDF X and our
test labels. We're going to have our
predictions
LGS TF IDF equals
LGS TF IDF.predict
That's going to give us our
raw predictions
and then I'm going to
stop saying exactly what I'm typing
but we're going to do our confusion matrix
and our
classification report
Awesome. So I'm going to let this sit
here for a little bit, let you guys
kind of copy and follow along
Hopefully we don't run into
the same problems that we did last time
Bender is fantastic
I love it, but it will time out after about
five minutes and once it
times out you've got to kind of restart the whole thing
or at least you've got to restart the notebook
which is kind of annoying
Another thing you will notice is the logistic regression
will take a little longer
to train and give us our predictions
and it's just the way that logistic regression operates
it does a little bit more mathematical
or complicated mathematical formulas
than the multinomial Naive Bayesian
so it just takes a little bit longer to train
but not too much longer, so that's good
One of them is matrix multiplication
which is a whole array of floats
that you have to process versus
Naive Bayes is literally counting and some logs
when you're doing the math correctly
which is very quick
It's just counting
Yeah, it's just counting
and logs, the latest x86 stuff
have built-in processes
for doing integer logs and stuff like that
because it's so common
to use logs for numerical accuracy
in machine learning
You'll also notice the same thing with neural networks
because neural networks do a lot of matrix multiplication
which is why GPUs work really well for neural networks
because you can parallelize matrix multiplications like that
So I'm going to go ahead and run this
Shift-Enter
And it looks like it's running. Hopefully it doesn't take too long.
Another comparison to make is a neural network versus a gradient-boosted tree.
A gradient-boosted tree is all just if-then statements.
Versus a neural network is a huge pain in the ass to multiply everything.
So a GBT could be run in milliseconds
versus a neural network taking seconds in 100% of your CPU.
Actually, speaking of neural networks, I'm kind of curious about this.
How well this would be received at DEF CON.
Would anybody be interested? And please vote in the Discord channel.
If you would be interested in learning how to build neural networks from scratch
or how to build deep learning security tools
I think both would be really interesting.
Neural networks from scratch is actually a lot easier than you would imagine.
You can do it in about 11 lines of Python code.
Today we are able to simplify and easily explain the math
whereas a lot of these algorithms were invented in the 50s
where the people who would do this kind of work were called statisticians
or people who would apply these in business scenarios were called business intelligence.
But today with computer science and these abstraction libraries
it's a lot easier to understand.
So let me know in the Discord chat if there's any interest in either of those
as potentially future workshops for future DEF CON events.
Or even just AI Village events.
So KDAMinMax has a question on Twitch chat.
I'll just read it here.
I have a possibly stupid question about input structures.
If you're learning, there's no real stupid questions.
An email predictor acts on strings or a set of strings.
Are there any machine learning approaches that act reasonably well on trees?
For example, abstract syntax tree representations of applications.
For that, you might want to use a graph convolutional network
or, honestly, you can interpret language as an abstract syntax tree.
So you might want to reach in and use some of the...
There's a huge backlog of literature that people largely ignore
because of the success of deep learning
of how to do language processing using abstract syntax trees.
So you might want to reach into some old school machine learning
from before 2010 when neural networks destroyed the world.
I mean, built our jobs.
That was a good answer. Thank you.
So what we're looking at is logistic regression with TF-IDF vectorizer.
We have a slightly better...
It's a small bump, but it's slightly better
than the multinomial with the count vectorizer.
We have a 95.9% accuracy.
We see that we have a little bit more than the 26 ham emails
that were incorrectly classified as spam.
We have significantly fewer of the spam emails being classified as ham.
So this would be a perfect instance of moving towards
having absolutely zero spam get through our filter
at the risk of losing some of our regular emails.
But what I want to show is how the accuracy improves
just changing the algorithm itself.
And then looking at multinomial Naive Bayesian
compared to logistic regression both using TF-IDF.
We had 87.5% for the multinomial
and then we have 95.9% for the logistic regression.
So far, logistic regression is kind of beating out multinomial
in the TF-IDF and the count metrics.
So we're going to do our last one which is going to be
logistic regression with count vectorizer.
Count vectorizer worked well for multinomial
so let's see if it works well for logistic regression.
So we're going to do LGS count.
And I'm just going to type this out
but you can use the same copy-paste trick that I used for the multinomial
with the count vectorizer.
I'm just doing this as a way to kind of slow myself down
so that you all have an opportunity to copy the code.
So LBFGS.
We're going to fit our labels.
And we got to do our scores, predictions,
confusion matrix, and classification report.
Yeah, one of the things
that you kind of reminded me of, Sven,
when you've been talking about the different trees and algorithms
that have been overtaken by
neural networks and deep neural networks
is random forests. Random forests operate very well
on a lot of different scenarios.
But because
random forests are a cluster of decision trees
people go, oh, they're just if-then statements.
This is dumb. This isn't machine learning.
Yeah, but for us, we have
a highly shared data where,
for example, on the Ember malware dataset,
the lightweight,
incredibly cheap
gradient decision tree built with LightGBM
outperformed any neural network you can throw at it.
So the LightGBM gets 99.8%
accuracy or something like that, and the neural network gets
98.7% accuracy, which is
significantly worse.
For some tasks,
a gradient-based decision tree or something like that
is the correct thing mathematically
because neural networks have biases
because it's easier for them to learn some things than others
and GBTs have biases where
it's easier for them to learn some things than others.
And one of the great secrets
of machine learning is
match up your problem with both
the requirements, how powerful your predictor
needs to be, and the inherent biases
of those predictors.
Yeah.
Yeah, that's pretty interesting. I didn't know that.
But that also reminds me of another thing
with neural networks. Neural networks are really good at learning superstitions.
I don't know if that's a machine learning as a whole thing
or specifically in neural networks
but I remember there was a dataset that they were using to identify
skin cancer cells. They used a convolutional
neural network and based off of that it was learning
that if there was a... it was performing too well.
It got nearly 100% and they
kind of dug into what it was doing
and it said, OK, well if I see a ruler in the picture
which the scientists, how they were collecting the images, they had a ruler
next to the skin cancer cells. If I see a ruler in the picture
that's how I know it's cancer. Not the little blob
next to it.
So I don't know if that's a whole machine learning problem
or specifically a neural net problem
but it's just kind of an interesting anecdote.
Be careful how you collect and preprocess and use your data.
A friend of mine did his PhD.
His PhD thesis was inducing the
bias that's sort of like the inherent
model bias that's in convolutional
neural networks into other models
and it slightly improved.
By inducing a bias?
Yeah, you can induce a pixel
locality bias like a great green
tree or a support vector machine
and it improves
that's one of the assumptions that people make
about convolutional neural networks is it allows you to make
small pixel local decisions
and then those can get glued together
in the right way.
And so he went and built little modifications
to a dozen other algorithms
that have sort of fallen out of favor
and all of them improved slightly on images
when you induced a pixel local bias.
Okay, interesting.
Yeah, that's
that's actually
really interesting.
Alright, so
hopefully I gave you guys enough time to make sure that this got
copied down. I'm going to go ahead and run it
and shift enter. This is going to take a little bit because it's going through
its matrix multiplications
but in the meantime, we're about to hop into
our final task
and what we have is a real spam email
I pulled this out of my inbox or my spam folder
last May
but we see your latest issue is available now
if you don't want issue notifications, click here to unsubscribe
Hi George. My name is not George
but we're going to look at this email
and we're going to try it against all four of our spam classifiers
and see how well they do. But what's also good about this exercise
is it doesn't really hold your hand. This is usually something
that I give plenty of time in a classroom
setting. This allows people to ask questions
as I go through. But this last task is going to say
OK, we have a new email we've never seen before
run it against our classifiers
that we trained. And so you have to go through
the whole process of using the
vectorizers to vectorize the tokens
or the words in the email and then putting them through the predict
function and then printing out the prediction
that it gives. So before
we hop into that, I really want this to finish
I want it to just be on its own time
but
we're going to
compare how well these models did
once this last one is complete
and we're going to see if or how much
that comparison has an influence on our test email
So we're just going to give this another
Probably be another 30 seconds or so.
So it's not too long.
How's the Discord chat?
Discord chat's doing well.
I really appreciate how much discussion we have in this version of the workshop.
Also, and this is something that I want to note for people who are seeing this for the first time.
So the last time I did my workshop on Friday,
unfortunately my webcam froze in a very unflattering position,
so I just had my mouth gaping open.
So for this one I just put a static image of myself today.
That way you can still see my beautiful face,
but you don't get a very unflattering image of it.
Alright, so this one is complete.
We are now at the best model that we have of the ones that we've trained.
We have a 97.69% accuracy.
We can round that up to a 97.7% accuracy.
We see that only 8 out of our 582 ham emails were misclassified as spam,
and then only 12 out of our 264 spam emails were misclassified as ham,
making this the best model that we have out of the 4 that we've trained.
So that's pretty solid.
So moving forward, what I would do is take this as kind of the better of the models,
and I would go ahead and deploy this in production,
or add this model into an application that's designed to kind of wrap around the idea of spam and ham.
So if you are simply an email provider, you can use this to filter email,
or if you are a spam company, or not a spam company, but like a spam research company,
you would use this to grab the spam messages and then collect that as data,
and then you can use that to be training the next version of your spam filter.
So let's take a look at our test spam email.
We have our final task, which is to use the vectorizer and the models that we created
to perform predictions on our test email and our test spam email.
So to make things easier, what I'm going to do is use a variable, I'll call this working email,
and this will allow us to switch pretty quickly.
So I have test email, and then we're going to want to vectorize our email,
so we're going to use our test email tfidf,
yes, tvec.transform,
and this is going to be, remember it needs to be in a list,
so we've got to use these square brackets, our working email,
and then we'll do the same thing with our test email count, tvec.transform,
working email.
Now we need to get the raw predictions for every single one of them.
The raw predictions, we already have the variable names here, kind of at the bottom,
so I'm just going to go ahead and write them out.
So we have test email mnb with tfidf,
which is going to be our mnbtfidf.predict,
on our test email with the tfidf vectorized info,
we're going to do the same thing for count,
using our count.predict,
with our count vectorized,
test email lgs, tfidf.predict,
tfidf.data, and our last one,
test email lgs.count,
count.
And then, just for fun and games,
we can print out our working email.
So this is going to print our test email.
Our test email was the one that we saw,
EastAsianFonts and Lenny, thanks for your support.
So you can go ahead and copy this down.
I'm going to go ahead and run this,
because we are running close to time.
So this is going to be our test email,
and ideally, this should say that
it is a regular email, not spam.
It's going to label all four of these as ham.
So let's run it.
Working is not defined,
because I forgot to call this working email.
So we're going to run it.
Here's our email.
And all four of our classifiers
correctly classified this as ham.
I'm going to scroll back up,
and we saved our email as our test spam email.
So I'm going to grab that,
and I'll run it.
Oh, I forgot to run the code block
that actually defined our test spam email.
And now we can run this.
And we see something interesting here.
Our worst performing model,
our MNB TF-IDF,
which was 87.5% accurate,
misclassified this email as being legitimate,
as a ham email.
But the other three of ours
classified this correctly as a spam email.
So there's a couple things we can do with this.
If we were so inclined,
we can use what's called ensemble learning,
which we use all four algorithms.
We take the average of them,
or in other words,
they give a vote on what they think the class is.
And then in this case,
three out of the four say it's spam,
so it would be spam.
Or we can use this as an example and say,
hey, the MNB TF-IDF,
we probably shouldn't use that
simply because it's misclassifying
known spam emails
at a high rate.
87% means that we have
about a 12.5%
chance of misclassifying spam emails.
So this is kind of
pretty close to the conclusion of the workshop.
This shows how you can take
raw emails, raw data,
transform it in a way
that can be used in
machine learning models.
And then we build out our machine learning models,
we evaluate them,
and we can test them.
And this last task shows you
how we can properly deploy them.
So we can build a piece of software
or an application around it
and say every time we get an email,
just swap out
this working email,
and then you can leave everything else the same.
Now, before I let you go,
there's one more thing that I want to show you.
I mentioned hyperparameter tuning.
So let's take a look and see
what hyperparameter tuning can do.
So when we were talking about
the multinomial Naive Bayesian,
we saw that we had that smoothing function,
the alpha, right?
And it was normally 1.
So we can add that in here.
So we have alpha equals 1.
We see that an 87.5% accuracy.
We go ahead and run that.
I'm just doing this to show you
that there are no tricks up my sleeve.
Alpha is by default 1 for this model.
But what if we change that
to maybe something a little smaller, right?
Maybe a 0.1.
Let's reduce the impact of that smoothing filter.
So I'm going to go ahead
and press Shift-Enter and run that.
And we see something interesting.
So this went from our worst model,
87.5% accuracy,
to actually our second best model,
96.5% accuracy.
And so it slightly misclassified
some of the legitimate ham emails as spam.
But it drastically reduced the,
I think it was 108 spam emails
being misclassified down to just 22.
So this is the power of hyperparameter tuning.
And if we go all the way back down
to our final task, task 7,
and we run this,
where the multinomial manipulation with TF-IDF
incorrectly classified that as ham,
we go ahead and rerun it.
Now it's correctly classifying it as spam
because its accuracy is much better.
And the only thing that we did to change it
was that hyperparameter tuning.
So that's all I have for you guys today.
I have a couple good resources,
but the biggest one that's going to be helpful
is if you have any questions,
feel free to reach out to me.
I am gtklondike on Gmail.
I'm also gtklondike on Twitter.
I'm not as active on Twitter,
but I will see your direct messages
or any at mentions.
And then the GitHub code
for all of these workbooks,
this and all the workbooks,
are on github.com slash netsecexplained.
So that is all I have for you guys.
It looks like we just barely made it.
1231.
And I will let you go.
I will stay in the chat channel
for a little bit
in case any of you would like to continue to discuss.
And I'll also stay in the voice channel
for a little bit for the same reason.
So I will see you there.
Have a good day.