Are we ready to go?
We are.
Perfect.
Hello and welcome to Machine Learning for Security Analysts.
This is our first workshop here at the AI village, DEF CON Safe Mode.
So the goal here is to kind of take machine learning out of the buzzword and into the mainstream.
So buzzwords like machine learning and AI have been in the industry for a couple of years now,
but for most people, it's not really well understood what these actually mean.
So my goal is to kind of introduce you to machine learning,
and we're going to do this by kind of going over what machine learning is,
how it fits in the broader concept of AI,
and why it's important for security analysts and hackers to understand machine learning,
as well as what the machine learning process looks like.
And then we're going to take all of that and put that into building our first machine learning model,
which will be something very familiar.
So let's go ahead and get started.
So who am I? My name is Gavin. I go by the pseudonym GT Klondike.
I'm an independent security researcher and a security consultant.
I have a passion for network security, both attack and defense.
And so through that passion, I run a project called NetSec Explained,
which is a blog and a YouTube channel where I use intermediate and advanced-level concepts
and try to explain them in a very easy-to-understand way.
So it's usually at this point where, in person, I would kind of ask the audience,
when you hear the word machine learning, what comes to mind?
Some of the common answers that I get are,
oh, you need to have a PhD in multivariable calculus and linear algebra,
or Skynet's going to take over the world, or pattern matching.
So these are all pretty interesting answers,
but what we're going to walk through in this workshop is kind of show you that
not all of that is necessary.
You don't need a PhD in multivariable calculus or linear algebra,
especially with some of the abstraction libraries like Scikit-learn, Keras, TensorFlow, and things like that.
And then as far as machines taking over the world, not quite.
So machine learning and how it fits in the broader idea of artificial intelligence.
So artificial intelligence is this big umbrella term that kind of is a catch-all
for different types of ways that machines operate.
So back in the past, when they first invented digital calculators,
they thought that was machine intelligence or artificial intelligence.
And then it came to video game AI.
Today we also see things like rules-based artificial intelligence,
or classic image recognition.
So there's a bunch of different artificial intelligence technologies
that aren't necessarily machine learning.
Machine learning itself is more about applied statistics,
and a merge between what used to be considered business intelligence,
and statisticians would do this job,
and computer science.
With the improvement of computing resources such as memory and CPU,
a lot more power can be put behind some of these algorithms.
And then deep learning is specifically talking about deep neural networks.
Neural networks are a type of machine learning algorithm,
and then deep learning allows us to stack layers on top of these.
So deep learning, usually these are referred to for convolutional neural networks,
which are used for image recognition, primarily.
And then recurrent neural networks, which are used for cyclical pattern recognition,
such as heartbeat monitors.
But if we were to find all of this,
I would say machine learning is a set of statistical techniques
that enables the process of information mining,
pattern discovery, and drawing inferences from data.
The idea is that machines learn from the data,
instead of us having to program signatures, or heuristics,
or invent new algorithms for different problems.
For example, deep learning, we have convolutional neural networks
where deciding between whether something is a bike or a bus
is the exact same algorithm and exact same problem
as deciding whether or not something is a tiger or a cheetah.
So a couple examples, especially in the security space,
we have the ability to classify domain generation algorithms.
These are primarily used by botnets as a way for the bot herder
to stay in contact with their networks.
Web application firewalls, these we will usually see
with being able to perform some anomaly detection
and see, okay, this is what normal user traffic looks like
compared to abnormal user traffic.
So in this case, the machine learning algorithm
identified a potential SQL injection in the parameter
because it's very abnormal from what it's used to seeing.
And then, of course, network anomaly detection.
This is one of the big ones, especially in the security side,
the defender side of things, where we can predict
what we think the network traffic will look like over a period of time
and then compare that to what the actual is.
Once it's beyond a certain delta, then that's something
that we should pass off to a human to kind of dig in a little deeper
and kind of figure out what's going on.
So why is this important?
Well, today we have over a quarter of security products
that are used for detection that use some form of machine learning.
And we see machine learning being applied to more and more things every day.
We see facial recognition, we see advanced spam filtering,
data heuristics and persona collection.
And so in order for security analysts to properly deploy
and manage these machine learning products,
they need to have a better understanding of how they operate
so that they can ensure that they work efficiently.
So the machine learning process, it follows seven steps.
This is not a step one, step two, step three,
but this is really a cycle.
So you start at step one, you go all the way down to step seven,
and then you hop back up to step one.
So you start off with gathering the data.
So first you want to identify if I'm going to build
a network anomaly detection system,
if I'm going to build some sort of hacking tool.
I need to gather data that applies to that,
and then I need to prepare the data.
So I can't just take raw information,
I need to look at things like metadata and pre-process that.
This step is also known as pre-processing.
We need to choose the model, and the model is going to depend...
There are some heuristics.
Some models work better for different problem sets,
but generally we're going to try a couple different models
and see what works best.
We're going to train the model, we're going to evaluate the model,
and then if we think that the model can perform a little bit better
than it's actually performing right now,
we can do what's called hyperparameter tuning.
And so this lets us make little tweaks here and there.
And then of course we deploy it.
And as I said earlier, this is a bit of a cyclical process,
because machine learning algorithms stop learning
as soon as you stop training them.
And so you want to continue to gather new data
and train new models and build them into newer and better generations.
So before we get started, what we're going to do is hop in
and learn how to build a basic spam filter.
Spam filters are fantastic first examples,
especially for a lot of the newbies,
because we already know how spam filters work.
If it says something that looks spammy, like buy Viagra now,
then that's going to be automatically thrown away into our spam filter.
If we have something that seems legitimate,
then that's going to be passed on to the user.
So before we get into that, are there any questions
on any of the stuff that I've gone over so far?
Okay.
One more thing.
I totally forgot this, but this is a bit of a story that really...
Yes.
Okay.
This is a story that really made things click for me.
So a friend of mine, this was a few years ago,
was approached by a non-profit organization.
And what this non-profit does is they have cameras set up all over Africa
and they take pictures to see if there's a cheetah.
And they're tracking cheetah population.
So they'll take pictures, it'll get sent off to a human analyst,
and the human will look at it and say,
okay, is there a cheetah in the picture or not?
And so they approached my friend and they said,
hey, is there a way that we can improve this process?
What's something that we can do here?
And so he said, okay, let me try a couple things.
So what he did was use a very common image library
and said, okay, let's train a machine learning algorithm
to identify images, if there's a cheetah in the picture or not.
If it reached above a 5% confidence,
says, okay, I'm 5% confident or 6% confident
that there's a cheetah in this image, it got sent over to an analyst.
If it was less than 5% confident, it would just throw it away.
Now, 5%, that doesn't really sound like a lot, and it's not.
But the ROI of what this model was able to do is pretty astounding.
So what normally would have taken them a year of labor to perform,
they were able to do in one month.
That is a 1,200% increase in productivity.
So that's pretty powerful.
And that's just a very simple model.
So as we can develop more and more advanced things,
these are ways that we can really improve our processes
to move at machine speed and scale.
GT?
Yeah.
We have a question, and it's,
any links to information pertaining to creating your own models
rather than training existing ones?
Creating your own models rather than training existing ones?
Yeah.
I think that you're going to be showing them how to create their own model.
We are going to create our own model.
So there are existing models that you can use,
what's called transfer learning.
So you take a problem, for example, ImageNet, which is a challenge.
It can recognize hundreds of different categories of images,
and you can take that idea, train your own model
so that it can recognize whether or not there's a cheetah in an image.
But what we're going to do is actually build a model from scratch.
There's no previously trained information,
and I'm going to walk you through how to do that,
and we're going to see a workbook in a little bit.
Thank you.
Hopefully that helps answer the question.
Okay.
So before we get started, machine learning, it uses statistics,
and so what we need to do is grab information and turn them into numbers.
But if we wanted to have the idea of a spam email
versus a regular email or a ham email,
we need to kind of understand how we're going to interpret words into numbers.
So here's a couple sentences that refer to sports and not sports,
or realistically, elections.
So we see a great game, and that talks about sports.
And then we see the election was over, and that talks about not sports.
So the idea is how do we take a sentence that we haven't seen before,
because that's what machine learning is supposed to do.
We were trained on certain information,
but we want to predict on information that we haven't seen before.
How do we take a new sentence and then try and make a decision
whether or not it's talking about sports or not sports?
So to do that, we're going to use what's called Bayes' theorem.
Bayes' theorem is a very common probability theorem that basically says,
okay, so if we have a sentence that we already know,
and given the probability that this sentence is talking about sports,
versus here's a new sentence,
what is the probability that it's talking about sports given that new sentence?
So the probability of A given B equals probability of B given A, blah, blah, blah.
We don't need to know too much about this,
but it's important to get a good foundation of this understanding.
So the idea is, what is the probability that the sentence,
a very close game, is talking about sports?
We don't know the answer to that.
We've never come across the sentence of a very close game,
but we would need to fill it into this algorithm here.
So instead, what we can do is actually break up a very close game
into their individual words, a, very, close, and game.
And then we can turn that probability into,
what's the probability that A is talking about sports?
What's the probability that very is talking about sports?
What's the probability of close talking about sports,
and game talking about sports?
So this winds up becoming a bit of a counting game.
How many times do we see A, and it's talking about sports?
Twice.
And of course we need to do the same thing for not sports,
because we need to find the probability of A happening,
the probability of B happening,
and we take whichever is higher.
So the probability that A is talking about sports,
we see it twice in two of our sentences,
or twice across all of our sports sentences.
Got to be precise.
We don't see A in not sports at all,
so that's going to be zero across all of our not sports sentences.
Very shows up in sports once, does not show up in not sports.
Close shows up in not sports once,
does not show up in sports,
and then game shows up in sports twice,
and it doesn't show up in not sports.
So we take those numbers and we plug them in.
We saw for sports, A showed up twice,
Very showed up once,
Close didn't show up at all,
and Game showed up twice.
And then if we count the number of words that we have
across all of the sports sentences,
we have a total of 11.
So it just becomes a simple 2 divided by 11,
1 divided by 11, and so on.
But we can't use straight, naive Bayesian.
And the reason why is because we have these zeros.
Now, close is a word that we've never seen in sports,
so we get zero, and that winds up being
zero divided by 11, which is zero.
Zero times everything is zero,
and so unfortunately the probability of
this sentence being about sports,
if we just use straight Bayesian,
is zero.
So that doesn't really work for us.
So instead, what we're going to use is called
multinomial naive Bayesian.
It's a very similar formula, but we have
what's called a smoothing filter,
and that's this alpha.
We can set this alpha to whatever we want.
Usually we just set it to 1.
But we can modify that, and this will be
a hyperparameter that we'll take a look at
later in our workbook.
So with the smoothing filter,
we do the exact same word counting.
We see A showed up twice in sports.
Very showed up once in sports.
Close, zero.
Game showed up twice in sports.
And then we do the same thing for not sports.
But you see here where it says close for sports?
That smoothing filter makes it so that
that number never becomes zero.
It's always greater than zero.
So our probabilities will always fit.
And then this is just simple addition
and then division.
And if we took out a calculator,
did all our math, we can see that
AVeryCloseGame has a probability of .0000461
that it's talking about sports.
And then AVeryCloseGame has a probability of
0000143 that it's talking about not sports.
So we take whichever is higher,
in this case sports, and so our classifier
has decided that AVeryCloseGame
is talking about sports.
So I know that was a lot of information.
Let me know if there's any questions.
I'm going to kind of hop forward a little bit,
but TAs, feel free to chime in
if there's any questions on this.
I know this bit's a little confusing sometimes,
but it's not something we need to harp on too much.
It's just an underlying foundation
that's good to know.
So the five things that we need to keep track of
in order to do our pen and pencil math
that we did earlier, is we need to keep track
of the total number of unique words,
the total number of words in spam,
the total number of words in ham,
and these two were what we've put in the denominator.
The count of each word in spam
and the count of each word in ham.
So this is where we had to count the word A,
count the word very, count close, count game.
And so these are the five things
that we need to pay attention to.
Are there any questions so far?
It looks like we're clear on questions right now.
Awesome.
Okay, so how do we take this idea
and expand it to a full email?
Well, here's a spam email.
Yeah, no, this is not a spam email.
This is a regular email.
And we're going to use this as a common example
throughout the rest of our workbook.
So in this example email,
we have re, re, East Asian fonts in Lenny.
Thanks for your support.
Installing Unifonts did it well for me.
Now, we could take this entire email
and look at it,
but there are some things that we want to
probably remove,
things that don't really make a lot of sense,
especially in the English language.
So punctuation, punctuation doesn't really make sense.
It doesn't add any context to a sentence.
Small words, which are called stop words,
such as in, of, on, for, the,
they also don't add any context.
So we can just go ahead and remove that,
and that'll reduce the amount of words
that we're looking at in this email,
because we want to look at keywords.
Things like unsubscribe and unsubscribed,
or thanks, or thank you.
They mean the same thing.
A really good example, especially in English,
is congrats and congratulations.
They mean the same thing.
And if we want to narrow this down to keywords,
we should do what's called stemming.
And so what stemming does
is it shortens the word down
so that you have I thanked somebody today,
as in past tense, thank-ed,
thanks, thank with an S,
and thank you, where it's just plain thank.
We want to count those all as the same word.
And then, of course,
we want to set everything to lowercase,
because capital words such as unsubscribe,
all caps, is the exact same as unsubscribe, lowercase.
So this is our preprocessing step.
And what we did was we removed all the stop words.
We removed re, in, for, your, did, it, for.
And then we're going to stem the words.
I didn't show the stemming in here,
and we'll see in the workbook as we go along
what the stemming actually looks like.
But we can see that we kind of shrunk the email a little bit,
and this is going to be a little easier
for us to compare emails to each other.
So I think this is the point
where we can go ahead and open up our workbook.
So I'm going to escape out of there, open this up.
So for our workshops,
we're going to do this for this workshop
and for the other workshops.
We're going to use a system called MyBender.
MyBender is fantastic.
It uses Jupyter notebooks in a Docker container.
So I'm going to go ahead and copy this
and send it off into Discord somehow.
Actually, is there a way...
I'm going to kind of hide my screen for a second.
Yeah, just post it to me,
and I'll put it in all the places for you.
Yeah, absolutely.
Okay.
All right, perfect.
Got it.
Display capture.
I'm still getting used to OBS and Twitch streaming.
I haven't done this before, but I'm really excited.
So MyBender, once you get the link,
you can just grab that link, paste it in here,
or you can click launch,
and you can do this with any GitHub repository.
So this is going to launch some Jupyter notebooks.
It takes about a minute or two to spin up.
So I'm going to let you guys get that link from the TA,
from the workshops channel,
and then I'll let this spin up for me.
I'll wait a little bit longer so that it spins up for you,
and then we can get moving.
While this is happening, are there any questions?
Feel free to just write them in the channel,
and the TAs will relay them.
And once yours spins up the way that I have this laid out,
go into the workbooks directory.
If you would like to follow along in the live coding session,
we're going to use the workbook spam filter sklearn document.
If you are not that familiar with Python,
and you just want to follow along,
you can go ahead and use the completed workbook.
I would give it just another couple of minutes to make sure theirs are spun up.
Yes, absolutely. Absolutely.
So I just want to check something real quick on my other screen.
And these workbooks outside of this workshop,
you can play with them on your own.
I have a couple others if you want to take a look.
But yeah, for the purpose of this workshop,
we're going to be using the spam filter sklearn,
or the completed spam filter sklearn.
Okay.
So I'm going to go ahead and start.
The first couple things that I'm going to go over are just next, next, next run.
If you're still waiting for your instance to spin up, don't feel bad.
You'll be able to catch up very quickly.
So I'm going to go ahead and open up my spam filter sklearn.
And in this one, we are going to use scikit-learn,
which is a wonderful abstraction library.
Great for beginners.
So we're going to go ahead and install a couple tools.
So for these blocks, in order to run them,
you can click on them and press shift enter,
or you can press run button up here.
You can see that it's running all the Python code.
In this one, I'm just installing the libraries
and I am configuring the data directory
where all of our spam and our ham emails will be located.
So that's going to take a little bit while that is going.
Just kind of a quick overview of this first block.
We have our imports.
So in this case, we're going to use matplotlib.
Matplotlib is going to let us print some nice graphs.
NumPy and Pandas is going to let us
grab some of the information that we need.
We're going to use a couple of the others
as some support libraries.
For example, OS, we're going to use that later
to kind of grab all of the files.
We want to use NLTK.
NLTK stands for Natural Language Toolkit.
And so this will allow us to identify our stop words
and perform our stemming functions.
From scikit-learn, we're going to use a train-test split.
The way that you typically want to train a machine learning model
is you have a bunch of clean labeled data
and you're going to take probably about 80 or 70% of that
and use it to train the model.
But then you want to see how well the model is doing.
So you're going to use the other 20 or 30% as your testing
and you're going to use this as evaluation.
It's kind of the same way we teach students in elementary school.
Here's the bulk of the information and now we're going to test you
to see how well you understand it.
And then our TF-IDF and count vectorizers,
we will get into that in a bit.
These are going to be used for transforming our words
into numbers that we can work with.
And then the two models that we're actually going to build out
are going to be the multinomial Naive Bayesian,
which we went over in the slides,
and then one that is completely unfamiliar, logistic regression.
And this is to show how incredible machine learning can be
where you can just swap out algorithms.
So we can use SVM, random forest, neural networks,
things of that nature without massively changing the way
that our data is represented.
And then to kind of represent how our models are doing,
we're going to use Seaborn as a graphic representation,
and we're going to use a confusion matrix
and a classification report to identify exactly
where our model is doing well, where it's not doing well,
and how mild changes are going to affect it.
So I'm going to go ahead, shift-enter, run that.
And that's just going to do the import,
and we'll see libraries imported.
And then here's our test email from the slides.
This is the East Asian fonts in Lenny.
We're going to see every step of the way how this email is manipulated
and transformed into workable data for our machine learning models.
So I'm going to hit shift-enter and run that.
TAs, are there any questions so far?
Is everybody able to get their stuff set up?
No questions so far. Looks like everything is going well.
Perfect.
So the tokenizer, this is the first step where we want to define
a function that will pre-process our information.
So, as we saw in the slides, we want to make everything lowercase,
we want to remove the stop words, and we want to stem the words.
So this function is already defined for you,
but we can see that we grab all the punctuation,
we grab all the stop words using nl2k,
and then we define the porter stemmer, which is going to stem our words.
And then we're going to create a list of tokens
by taking all of our words, setting them to lowercase,
and using the word tokenizer,
which will just take a sentence, split it up by its individual words,
we're going to strip out the punctuations,
and then we are going to merge everything together.
So we're going to go ahead and run that.
It's going to define our function,
and we're going to hop into our first task.
So our first task is really easy.
It's just print the full test email,
and then we're going to print the tokenized version of the test email.
So in Python we can just do print test email,
and then we can do print tokenizer test email.
And then shift-enter, going to go ahead and run that.
So we see the original email, East Asian fonts in Leni.
Then we can see the stop words have been removed.
So we're looking at East Asian font Leni.
Thanks has been turned into thank.
Support install. Install is missing an L because we see installing.
You have I installed something, or I am going to install something.
And so we're going to stem that so that all of those words are counted the same.
We see unsubscribe. Unsubscribe shows up twice.
And then we see list.debian.org, which also shows up twice.
So this is the way that our tokenizer modifies the email as we're going through.
And again, just for those that are trying to catch up,
all we did was print the test email and then print the tokenizer version of the test email.
There's a question from Twitch chat.
So why are we assuming the punctuation is not relevant?
In our case, we're looking at spam emails.
And so the punctuation, you'll look at commas, hyphens, question marks.
And so in the context, if we're looking at keywords,
the punctuation doesn't really matter.
So if you see buy a Viagra now, period,
it doesn't make any difference between buy Viagra now, exclamation point,
or buy Viagra now with a question mark at the end.
So in the context of the sentence,
that one sentence would likely be classified as spam
because it's talking about Viagra and it's very unlikely.
That's a very common spam word we'll see in a lot of spam messages.
So punctuation is just not relevant
because it doesn't add any context to the message.
And so that's something that we can remove.
If you would like, you can follow through on this workbook
and then try it without removing the punctuation
and see if you notice any differences.
But I don't think there would be any differences
just because of that context.
There are better things to weigh than periods, question marks, commas,
things of that nature.
And then as you see, emails or websites,
such as lists.debian.org, they still have their punctuation
because that's not the word lists, debian, and org.
That is a URL, lists.debian.org.
And so that's kind of the difference there.
If we had hyphenated words, we would see them included in here as well.
So we're going to go on and load the training data.
This one is just going to give us the counts that we need.
This is just done for you.
Remember, one of those five things that we needed to keep track
was all of the word counts in the ham emails,
all the word counts in the spam emails.
But what this is going to do is count the number of emails
that we have in ham, count the number of emails that we have in spam,
and then count the total number of emails
that we have in our testing set.
So for here, our testing and training split is already defined.
It's 80-20, and so we'll see that in a moment.
And we go ahead and hop into our second task,
which is to load the training data.
So task two, it asks us to create two lists,
and we're going to have corpus.
And it's important to get these words identical
because of how we use them later on.
So we're going to have a list called corpus.
We're going to have a list called labels.
And then the next bit is we want to load each of the email bodies
from the data directory slash ham and data directory slash spam
into the corpus array.
So what we're going to do is for each in os.listdir,
data.dir, this was defined above, slash ham.
And then we're going to say with each open file,
ham slash plus each,
because that's what we defined in our for loop.
We're going to read them as a file named f,
corpus.append, f.read,
and then labels.append the word ham.
And the reason why is because this third task
is load the labels for each email into the labels array.
So in this first code block, we're going to read
every single ham email in the ham directory.
We're going to add the bodies to the corpus array,
and then we're going to add the label ham into the labels array.
Then we can go ahead and do the exact same thing for spam.
We're just going to have to change each thing to spam,
spam, and the labels to spam.
And this will go through and read all of the emails.
So I'm going to leave this up for a moment,
let you guys copy that down.
And then I'm going to press shift enter and run it.
So this was really quick, just read all the ham emails,
all the spam emails, and now it's loaded into our arrays.
If we want to get an idea of what our class count looks like,
we have a total of 2,359 ham emails,
1,100 spam emails, and then just as a graph we can see here.
And I think it's important to graph these counts
because sometimes you will have what are considered
unbalanced data sets, and so you need to
perform other changes here and there
just to make sure that the data sets are balanced.
Especially for things like credit card fraud,
where that happens less than a percent of the time.
You can build a model that's 99% accurate
by just saying everything is a legitimate transaction,
but unfortunately that doesn't really help
when you're trying to stop credit card fraud.
So when you have imbalanced data sets,
there's additional things that you do.
In this case, it looks a little unbalanced,
but we'll be able to work with it no problems.
So we are on task 2A.
In this one we want to see the full text body
of a random email in our corpus array,
and then we want to see the tokenized result of that same email.
So this is going to be very similar to what we did for task 1,
which was printing out the test email,
and then printing out the tokenized version of the test email.
So we'll just go ahead and grab a random email.
So email equals corpus.
Actually, let me go ahead and do this.
Let's have email ID equals just the number 5.
Why not?
And then we're going to grab email ID.
So this is going to be our random email.
We're going to...
probably going to print a...
No, I already have a divider in there. Cool.
So let's just print that email.
And then, same thing, we're going to do print tokenizer,
and then that same email.
So this line and this line is exactly the same that we did in task 1.
This is the only thing that changes so that we grab a random email.
And then you don't have to put 5, you can put any number,
but I'm just picking 5.
So go ahead, Shift-Enter.
And this one's pretty nice.
So we see the body of the email, product review, virttools, dev, URL,
product review, virttool, dev, 2.0 URL.
We see the full path for the URL here.
Let's see if there's anything interesting.
Maybe features, feature is shortened to feature,
because you can say featured in a motion picture with an ed at the end,
features as they have here, or we will feature with just an e at the end.
So this is the way that our tokenizer operates,
but we still need to turn these words into numbers,
and this is where our vectorizers come in.
So our vectorizers, we're going to use two kinds of vectorizer.
We're going to use the count vectorizer, which just takes the straight word count.
It's exactly what we did in the slides.
Then we're going to use a TF-IDF vectorizer that applies a mathematical weight to each of the words.
And they work slightly different, but let's go ahead and define them,
and we'll show you what the difference looks like.
So we're going to name this CVEC as our count vectorizer.
Count vectorizer, and the capitalizations are important.
Then we're going to use our tokenizer function above as our tokenizer function here.
So we're going to have tokenizer equals tokenizer.
Awesome.
And then we want to vectorize our words, so this is going to be saved as countX.
So we're going to have count underscore capital X equals...
Oops, did not mean to open that.
CVEC, our vectorizer, and then we're going to use fit transform.
Fit is Scikit-learn's way of saying train.
So in this instance, we're going to train our vectorizer.
So fit transform, and then our purpose, which is just our list of words.
And then we're going to do the same thing for our TF-IDF vectorizer.
So we're going to take TVEC equals TF-IDF.
Vectorizer and capitalization is important.
Tokenizer equals tokenizer.
And then our TF-IDF X capital X equals TVEC dot fit transform corpus.
Cool.
So I'm going to let that sit there for a second while you guys are typing in.
And then I'm going to go ahead and run it.
So this takes about a minute because what it's doing is it is counting all of the words across all of the emails, the ham emails and the spam emails.
And so as it's counting each of these words, it's applying a word count for the count vectorizer, and then it applies a weight for the TF-IDF vectorizer.
And the weight takes all the words into account.
So it becomes a whole process.
But this takes about 60 to 120 seconds.
So we'll go ahead and let that run.
While that's running, we're just going to get started on task 3A, which is just to print each token from the test email.
Now this one can be a little tricky.
So there's a bit of a trick to identify the unique words and the counts.
So what we're going to do is create a list of dictionary keys.
So what that's going to look like is for i in list dict from keys.
Tokenized email was defined above, and what that is is the tokenized version of our test email.
So we're going to go ahead and print.
I'm just using the Python format.
Tokenized email dot count of each word, and then the word in the tokenized email.
So I'm going to verify vectorizing is complete.
And I'm going to let you guys run your vectorizers, let that sit for a little bit.
And then we're going to run this, because we need...
Oh, I don't believe we do need the previous one to run this.
So if your vectorizer is still going, your code here will not run right away,
but it will run as soon as the vectorizer completes.
So I'm going to go ahead and run mine.
So shift-enter, and tokenizer email is not correct.
I think it's tokenized email, that makes a little bit more sense.
I'm going to run that. Tokenized email is not defined.
Okay, give me a second. I'm going to scroll up to our tokenizer.
I guess I don't have that defined.
Where is that?
Ah, I was supposed to save it up above. That's okay.
So we'll go ahead and create it here.
Tokenized email equals tokenizer test email.
So we're going to go ahead and save that.
And here we have the word counts of our test email.
We see the familiar East Asian font, Lenny.
We know unsubscribe showed up twice, and we know list.debian.org showed up twice.
Everything else in here only showed up once.
I know I went very quickly, but this was the line that I needed to add.
Tokenized email equals tokenizer test email.
And then we grab the list of the dictionary from that list of the tokenized email, and we print out the word counts.
So going on to 3b, what we want to see is, okay, we know what the word count looks like if we just did a straight word count.
How does the count vectorizer and the tf.idf vectorizer transform our test email as we're going through?
So we're going to go ahead and use our example CVEC.
And the reason why we're using a separate example version is because we want to redefine a count vectorizer.
Don't forget our tokenizer equals tokenizer.
And then we're going to have our exampleX using our exampleCVEC.fitTransform.
And then it expects a list.
Our test email is a simple string, so we're going to have to wrap it in these brackets so that it becomes a list.
And then we're going to have our test email.
Cool.
And then we're going to go ahead and print our example email.
Oops.
ExampleX.
And then here, we're going to do the same thing.
We're going to have exampleTVEC equals tf.idf vectorizer.
Capitals are important.
Tokenizer equals tokenizer.
And then our example, tf.idf.
I'll just call it exampleX because it doesn't matter.
Once we print out our original exampleX, we don't need it anymore.
ExampleTVEC.fitTransform.
And then the brackets, test email.
And then we can print our exampleX.
So I'm going to let that sit for a moment.
Let you guys copy that down.
I'm going to just look ahead real quick.
We are doing great.
Okay.
So I'm going to go ahead and run this now.
I pressed Shift-Enter.
And if we look at our count vectorizer of test email, we see these different numbers.
And what these numbers represent are tokens.
So, for example, a very nice game.
A would be token 1.
Very would be token 2.
Nice would be token 3.
Game would be token 4.
So in this case, Scikit-Learn actually knows the best way that it wants to tokenize things.
So the letters, for example, 0, this does not necessarily represent East.
And so, basically, Scikit-Learn knows how these tokens are used.
We don't know how these tokens are used.
But something that we can point out is we see token 16 shows up twice and token 9 shows up twice.
So based on what we saw up above, we can guarantee that either token 16 or token 9 is unsubscribe.
And then the other one is list.debian.org.
And so these show up twice.
So this is exactly the same as if we had done the word count ourselves.
The TF-IDF vectorizer, however, uses a bit of a weight algorithm.
And so everything that was set as 1 has this 0.204 weight.
The 2, which is token 9 and token 16, have this 0.408 weight.
And this doesn't necessarily happen in every instance.
But in this instance, you can see the 0.408 shows up twice as much as the 0.204 exactly.
And so this is the way that count vectorizer operates and TF-IDF vectorizer operates.
The underlying, what happens underneath the hood, we don't really need to worry about.
We just need to know how these work and the differences between the two.
So moving on, we're going to load the testing data.
This is already written for us.
So we're just going to grab the first 10 files in the testing directory and print those out.
So we see we just have all these test emails.
And at the end of the file name, we have the label that it's supposed to belong to.
So we have ham, ham, spam, spam, spam, ham.
And so what we're going to do is read each of these emails.
And then we're going to pull off the last little bit of the file name and use that as our label.
And we want to have the testing separate from the training so that we can evaluate how well our model is doing.
So we're going to load the testing data.
And this one's a little complicated, so we have a couple tasks to run through.
So we're going to create two list arrays, kind of like we did earlier.
We're going to have test corpus, and we're going to set that as an empty list.
And we're going to have test labels, which is also going to be an empty list.
And then we are going to load all of the emails from the test directory.
So we're going to say, for filename in os.listdir, our data directory, plus test, which is where it's located, with each file that is open.
And we need to tell it which file test plus filename are.
For read, as filenamed f, we are going to do test corpus.append, f.read.
And then we are going to do something a little weird here, so bear with me.
I'm just going to use re.split, just to split this up.
I'm looking for txt.
And then we're just going to grab the last bit in that split.
So this is going to split the file trainemail.txt.
So it's going to split it to this section, and then to this section.
So this would be considered index 0, and this last little bit would be considered index 1, which is why we have this 1 here at the end.
And then we're going to append the labels.
So test.labels.appendLabel
And so I'm going to give you guys a little bit to go ahead and get this all written up.
Be careful, there is a backslash before this period, because this is a regular expression.
And everything should be good.
So I'm just going to let that sit for a little bit, let you guys do your thing.
In the meantime, I'm just going to look ahead real quick.
The rest of this gets so much easier.
It's pretty crazy how easy it gets.
In the meantime, TAs, are there any questions in the chat?
We appear to be good so far.
Awesome.
So I'm going to go ahead and move on.
I'm going to press Shift-Enter.
It's going to load all of those emails.
So this is just reading all of the test emails into our corpus array and grabbing those labels and putting them into the end of our test labels array.
And then just like last time, what this is going to do is print out the number of test emails.
We have 590 ham emails, 276 spam emails.
And here's just a nice pretty graph representing what that looks like.
And as I said earlier, this information or this data has already been split up in an 80-20 split.
So 80% of it is being used for our training and then 20% of it is being used for our testing.
And then this one, it might sound complicated, but this is actually really easy.
We have already trained our CVEC and our TVEC.
And now all we need to do is collect our testing values.
We need to tokenize our testing values using these already trained vectorizers.
So this is super easy.
testCountX equals CVEC.transformTestCorpus.
And that's it. That's all we've got to do for that.
We can just copy and paste here.
Say our testTFIDFX is our TVEC.transformTestCorpus.
The reason this works is if you remember in the slides how we had the sentences that we already knew about,
and then we had to perform a counting game for a very close game.
We had to count how many times a showed up in the sentences that we were already aware of,
how many times very showed up, how many times close showed up, how many times game showed up.
So this transform allows us to do just that.
So what this will do is grab each email and it will say,
okay, here's the tokenized words of the email that I have.
How many times does it show up in the count vectorizer?
And then what is the weight that it shows up in the TFIDF vectorizer?
So I'm going to go ahead, Shift-Enter, run that.
And just like last time, this is going to take a little bit.
It's counting all the words and associating them with the different tokens.
This is going to be significantly faster than training it.
See, it's almost done already.
And so we're going to go ahead and move on.
So this bit, we're going to get ready to test and evaluate our models.
So this is just a helper function that's going to show us a bunch of really neat statistics.
The details of this helper function you can look into on your own.
But for now, what we need to use as an input is our confusion matrix,
our score, and our classification report.
So we're just going to go ahead, Shift-Enter, run that.
Definition defined.
And then here's where we get into tasks 6a, 6b, 6c, and 6d,
which is going to be us training and evaluating our different models.
So we're going to start with multinomial Naive Bayesian with a TF-IDF vectorizer,
with a count vectorizer, and then we're going to do the same for logistic regression with TF-IDF
and logistic regression with a count vectorizer.
So we're going to start out just one thing at a time.
We're going to define our constructor.
So this is going to be MNB TF-IDF equals multinomial NB.
Watch your capitalization.
We're going to define that, and then we're going to train it.
MNB TF-IDF dot fit, which is train in scikit-learn world.
And we're going to use our TF-IDF X, which is our preprocessed information,
and then we're going to compare it to our labels.
So that's going to train our model.
Now we're going to grab a couple really neat statistics.
So the first one we're going to get is our score, and this wants us to call it score.
Score MNB TF-IDF predictions using our predict function.
We're going to develop a confusion matrix called C matrix, MNB TF-IDF,
and then classification report, C report, MNB TF-IDF.
So we're going to start with our score.
Score MNB TF-IDF equals MNB TF-IDF dot score.
And then we're going to use our test values, because we've trained it on our training data,
and we want to test it with our testing.
So we're going to do test TF-IDF X,
because that's the way that our system was vectorizing these.
So we don't want to use count vectorizer with a TF-IDF train,
because that's not going to work.
And then we're going to use test labels.
That's going to give us our score.
So we're going to get our predictions.
Predictions MNB TF-IDF equals MNB TF-IDF dot predict.
And this is going to give us our raw predictions.
This is going to tell us what it thinks each email is,
whereas our score is going to see how accurate it is.
And then this is going to be C matrix.
MNB TF-IDF equals confusion matrix.
We're going to use our test labels and our predictions.
MNB TF-IDF.
So this is going to tell us what it thinks the value is,
if it's a spam or ham email,
and then what it actually is.
So it's going to give us our true positives, our false positives,
our true negatives, and our false negatives.
And then the C report.
We're not going to do too much with this,
but what a classification report will do is show us some more metrics,
such as the recall and the precision,
which we don't need to worry about too much.
We're just going to focus on the accuracy for now.
But for data science, it's important to know what the recall
and the precision are as we go through things.
Report, test labels, and then we're going to use our predictions again.
And this is just like our C matrix MNB TF-IDF,
in the order that we put things.
We have test labels, test labels, predictions, predictions.
And that is it.
And then we're going to use our helper function generate report.
So I know that's a lot of typing.
I'm going to go ahead and sit here as you guys kind of follow through.
So we have our MNB TF-IDF constructor.
We are going to train it using the fit function.
We're going to train it on our testing data,
which is stored as TF-IDFX,
and our testing labels, which is just stored as labels.
We are going to use score, MNB TF-IDF.score,
to evaluate our accuracy.
We're going to use our predictions to get the raw predictions.
And then with the raw predictions,
we are going to develop a confusion matrix and a classification report,
which will tell us how well our model is doing visually.
So I'm going to let that sit for one more minute.
TA, are there any questions?
No, I think... well, hang on, let me go back one here.
Nope, still got it. So I think we're good.
Awesome. Awesome. Fantastic.
So I am going to go ahead and move on.
I'm going to press Shift-Enter. It's going to run.
And now we have our... up here at the top is our classification report.
And so what this is going to do is show us our precision
and our recall. We don't need to worry too much about this one.
Instead, I want you to focus on the lower chart.
So here we have an accuracy of 87.5%.
We can see that all of the actual ham emails were predicted as ham.
So these are our true positives.
A little more than half of our spam emails were correctly predicted as spam.
So these are our true negatives.
And then, unfortunately, it misclassified...
Oh, my mistake. I have that backwards.
So it says 108 of our ham emails it thinks are spam emails,
whereas all of the spam emails, so the actual spam emails,
were correctly classified as spam.
And then all of our ham emails, what it thought were ham emails, were ham.
So we're going to go ahead and do the same thing for 6b, 6c, and 6d.
One of the beautiful things, and I'm going to show you this
just as a super quick and easy way of going about and doing things,
is I'm just going to copy the code that I'd already written.
This is why machine learning is awesome,
and very different from programming normal algorithms,
is all we need to do is just change these tf.ids to count,
and then everything else is the same.
So we use mnbcount to define our multinomial Naive Bayesian,
as we have been asked of us here in 6b.
mnbcount.fit, and we're going to use our countX
because that's how it's been vectorized.
We're going to change our score mnbcount using mnbcount.score.
We're going to change our tf.idsx to testCountX,
and then likewise kind of all throughout here.
TestLabels is going to stay the same,
predictions is going to change because it is no longer predictions.tf.ids,
it is predictionsCount,
and then we have our confusion matrix and our classification report.
So just to go over it one more time,
we define our multinomial Naive Bayesian constructor,
we train it using the fit function and the count vectorizer,
our testing data, our training data.
We develop our score off of our testCount data using our testLabels,
and then we have our predictions,
we have our confusion matrix,
and our classification report.
So I'm going to let this sit for just a little bit
as you guys go through and copy that down.
What's really cool is in the GitHub repository,
there are other workbooks.
So for example, we use the exact same methodology
to build a malicious URL predictor,
so things that look like it's possibly part of a botnet.
All of this code is exactly the same.
Once we vectorize our information,
we just copy and paste this code in,
and it does the exact same thing.
And so that's the benefit of machine learning,
is it learns from the data itself
instead of us having to manually program,
okay, this solves problem A,
but now we have to rewrite this to solve problem B.
It just all does the same thing.
So I am going to go ahead and move on.
I'm going to press Shift-Enter.
And we see that we got a little bit better, right?
We are at a 94% accuracy.
It thinks that some of our ham emails are still spam,
but significantly less.
We went from 108 down to 23,
but now it thinks that some of our spam emails
are actually ham.
And so that's a problem.
So we have a few more emails going through our filter
than we would like,
but we have significantly fewer of our ham emails
from being blocked.
GT, we've got a question from Twitch chat.
Absolutely.
So Chris...
I'm not going to try and pronounce the name.
Did you intentionally avoid having spam emails classified as ham
when you created the model,
or is it simply a try and error kind of thing
when training the model?
Intentionally avoid having spam emails classified as ham?
Is that what they said?
Yeah.
Yeah, we wouldn't want spam emails to be classified as ham.
We want to draw a line and separate
so that you stop getting emails about Viagra
and you start getting the regular emails from your school or your work.
And so this is just kind of the way that spam filters operate.
So we could build a model that would classify more spam emails
as being part of ham or regular emails,
or we can build a model that says,
okay, we want absolutely zero spam emails.
If we wind up losing regular emails
and classifying those as spam as a mistake,
we're okay with that.
A really good example, actually I'm glad you brought this up,
is antivirus.
So the way that antivirus companies work
is they can err on the side of,
oh, I think this is a virus, therefore it must be a virus,
or this could be benign,
or I kind of think this is a virus,
but instead of blocking it, I'm just going to let it go through.
And so you think about the way that malware works.
You have malware that will make calls out to the internet,
it'll probably take over your webcam,
it'll try and write files,
it'll probably take over your microphone.
What else does that?
Chrome, IE, Firefox, browsers.
And so the idea is instead of blocking
what could be legitimate applications,
and so that would really interrupt the user experience,
they kind of err on the side of,
we don't want to block the user experience
because then you'll uninstall the antivirus.
So they are more willing to let viruses go through
than blocking legitimate applications.
And so in this case,
these are trade-offs that we have to make as data scientists
of if we are okay with letting a few spam emails through
versus blocking all of our regular emails.
So he's specifically asking about task 6A.
So if you scroll up a little bit.
Task 6A?
Yeah, I think the answer is a bit of that
and a bit of the question of when you build these models,
the training process literally punishes the model
for getting it wrong every single time.
And in this case...
Yeah, so Chris, you're asking about a false negative.
A false negative is when you've got something that's malicious,
that's misclassified as benign,
which is what the point of the model is,
but what GT brought up is probably more important
is when you have something that's benign
that you classify as malicious,
that is much more common
because you get more benign things on users' devices,
as GT brought up.
So in this case, I don't think this was intentional.
No, this wasn't intentional.
This is just the way that the model learned.
And then as we go through the other models,
you'll see how these shift.
Yeah, but this is honestly the way it looks like.
You've got the ham stuff that are classified as spam
is very low so that all the user experience
isn't impacted that badly.
And the spam stuff that's classified as ham
is relatively high,
but you've got a trade-off
between the false positives and the false negatives.
Yeah, exactly.
And I hope that answered your question, Christopher.
I think going through the other models
will kind of help answer it a little bit more too.
So accuracy, 87%.
87.5%.
94.3%.
In this one, we're going to use
a model that we haven't used before,
logistic regression.
And so with Scikit-learn, it's very easy.
It's the exact same thing.
The difference is we need to
use a LBFGS solver.
So what that's going to look like is
LGSTFIDF
equals
logistic
regression.
Then we're going to set our solver
LBFGS.
And this is just a thing with
logistic regression specifically.
We saw in the slides
that multinomial naivetation
has a variable called alpha.
We can tweak that.
That's one of our hyperparameters.
This is considered a hyperparameter.
It's the way that logistic regression
interprets the information.
But for this case, we're just going to use LBFGS.
We're going to do the same thing.
We're going to fit LGSTFIDF
dot
fit
TFIDFX
and then
it enables.
Let me just hop down here.
We're going to do score equals
LG
S
TFIDF equals
LGSTFIDF dot
score
testTFIDFX
Pay attention to your capitals.
Test labels.
Then we're going to do
predictions
shuns
LGSTFIDF
equals
LGSTFIDF
dot predict
and then
our test data
Typing TFIDF so much
it's very confusing for my hands.
T matrix
we're going to use our
confusion matrix
test labels
and predictions
LGSTFIDF
and then our
C report
LGSTFIDF
equals
location
report
test labels
predictions
TFIDF
That should be good.
Just checking for errors.
That looks right.
So, go ahead
copy this down.
I will sit and
wait for a bit, but you'll notice
a lot of similarities to the multinomial
Naive Bayesian. In fact, we could have just copied and
pasted and just changed it from
LGS to
MNB or from MNB
to LGS and then
change our constructor to be a logistic regression
with the LBFGS solver.
We have our logistic regression constructor.
We train it using the fit function
using our TFIDF vectorized
data and our labels.
We use our test data
and our test labels
to generate a score and to
generate predictions.
And then, off of those predictions, we generate
a confusion matrix and a
classification report.
And then, of course, our nice
little helper function at the bottom that's going to print us out
some helpful stats.
So, I'm just going to let this
sit for a little bit longer.
Kometh,
did Chris ask any more
questions, follow-up, or did we take care
of them?
A bunch of people
piled on and I helped answer the question
because of what he asked.
Honestly, one of the most important things
is the false positive
versus false negative trade-off.
Yeah.
Yeah,
a lot of people have strong opinions about that
because it's a lot of...
It's important. It's definitely
important, especially in data science.
This, I'm...
That might
have been a good talk.
So, here's the interesting thing, is when I
first started out, this
I thought was amazing.
Now, as I've learned
more over time, I kind of feel like, oh, no, this
is a little clunky and a little ham-fisted.
I'm ignoring this, I'm ignoring this, I'm ignoring this,
I'm ignoring this. But this is a great
way for beginners to understand machine
learning. And then from there,
you can kind of hop into
a little bit more of the data
science considerations, like
imbalanced data sets, true
positives, false positives,
and some of those trade-offs,
as well as more hyperparameter tuning.
If you want to look at
a malware, you keep bringing it up,
you can look at the
Google Endgame Ember.
I'll drop a link in the
Twitch chat in a sec. But there's
a malware model that you can look
into, like the open-source code,
and the final thing at the end of that is
it sets the false-positive rate to
0.05%
or 0.5%
at the end, because that's
what you should do in production,
because customers care more about
false positives, because
false positives lead to
alert fatigue, and alert fatigue leads to
just ignoring every single
alert, and then all the false negatives get
through anyway. All the true negatives get through
anyway.
It is something...
Yeah, and even that
model is a little clunky and hampers.
Well, and
it kind of leads to
with the alert fatigue, it kind of leads
to the common thing that we
hear, especially in the technical
community from non-technical people,
of why does my thing not work?
Okay.
So that kind of helps address it.
But yeah, these are all data science considerations.
So I'm
going to go ahead and run this.
Shift-Enter.
And this is going to take a little bit longer than the multinomial
Naive Bayesian, because logistic
regression interprets
the data a little differently, and so it needs to
go through and kind of count things properly.
This
takes, I think, about
30 seconds?
Yeah, it's
not too bad. It's
30 to 60 seconds.
So I'm just going to go ahead and run this.
And then for the logistic
regression count, we're going to do the same thing we did
with the multinomial Naive Bayesian count
and just copy and change words
because I'm lazy.
But yeah, I'm really excited to see that
Twitch chat after I hop off to this
workshop and see just how
passionate people are
about the false positive,
true positive, and XYZ.
Oh, we're going to erase it before
you get off.
Ruin my day.
Yeah, it's taking a little bit longer than I'm used to.
But that's okay. While this is going,
I'm just going to go ahead
and hop into the next bit
and
paste this. All we need to do
is trade this
TF-IDF with COUNT.
So I'm going to go ahead
and copy that. We're going to use the same
constructor and the same solver.
We're going to use COUNTEX.
We're going to use COUNTFIT.
COUNTCOUNT
COUNTCOUNT
COUNTCOUNT
Test labels is the same.
These are
predictions.
And just one
quick look over.
Yep, I have everything in.
That's exactly how it's supposed to be.
Scroll up.
Oh, this is still going.
Look at that.
Weird.
I'm going to let that sit for a bit then.
So this one, I'm going to let you guys write down.
This is task 6D.
So go ahead
and make sure that you have these written down
while the other one is still going.
I'm going to run this and then take a look at
a
logistic regression with the
TF-IDF.
We're actually doing really good
on time. We have one more task
after this. It is
going to be the most challenging task.
But it is going to
take
everything from A to Z
and show us what we
learned.
So I just want to check one last
time. And that is
still going.
So I'm going to go ahead and run
this.
This was 6D that I
just clicked to run.
It says the kernel is idle.
That's not good.
Let me see if I can
pause this.
Let me just interrupt it.
Interrupt the kernel.
So I'm going to run this
because that took a second.
Uh oh, I hope I didn't
time out.
That would be very bad.
Let's try and run it again.
Yeah,
that's not good.
Restart and clear output.
So this is going to go all the way to the beginning.
Let's see if I can get this to run.
Okay, so that runs now.
So I ran into a bit of a technical difficulty.
Instead of going through
and filling
everything out, I'm just going to quickly
grab the completed workbook.
And I'm
going to
just
go ahead, run all.
This is going to run
everything for me.
So
while I'm waiting for some of
this stuff to run,
let's see that packages are
being downloaded, tokenization is defined,
everything's good,
everything's good, everything's good.
Where is it stuck now?
It is stuck...
Looks like it's stuck reading the data.
Training spam, training ham.
That one's done, done.
Yeah, when you see
numbers here, that means that that was the order
it was ran in. When you see a star,
that means that it's still running.
I'm not sure why the logistic regression
was causing problems,
but let's take
a quick look at...
Oh.
Looks like there's some stuff missing.
Well, that's unfortunate.
So I'm just going to go
ahead and create new code
blocks. I'm just going
to copy the code we had
from task 6D
and 6C.
I apologize for scrolling
so fast.
So let's go ahead
and grab all of that,
paste it in, run it.
Logistic regression
is not defined.
That's weird.
Oh, it's not even in here.
That is something I'm going to have to fix.
I'm not sure... Oh.
Because that's
a whole different workbook.
I need complete spam filter
sklearn. That's the one that I need.
My mistake.
So kernel, restart,
run all.
So this is going to run
all of them.
Let's see where it
gets stuck.
Prints out the emails.
It's training the vectorizers.
I remember that took about a minute or so.
But what we want
to see
after task 6D,
we'll look at 6C and 6D,
is at the end
of this, what we're going to do is
grab a spam
email. This one is actually one that I grabbed
directly from my spam box
last year.
And I like it because we can use
our spam email predictor
to identify whether or not
it's spam. And then we can look and
utilize
any of the models
to really identify that.
So this was supposed to be
a fill-in-the-blank
kind of thing like we've been
going through.
But this last task, it just asks us
use the vectorizers and the models created to perform
predictions on the test email
that we created up at the top
and the test spam email
which is the real
world email that I pulled from a spam
inbox.
And so what we want to do
is grab that as
a working email.
And this will let us change this to
test email and test spam
email.
Remember, we want to
use the tf.idf
and the count vectorizers
in order to use
the working email.
We need to use that to preprocess
and because it's
a string, we just wrap it in these
square brackets in order to classify
it as a list because the transform function
expects a list.
We have that as test email,
tf.idf, and test email
count.
And then we use each of our models
to form a prediction. So we have
the multinomial
Naive Bayesian with tf.idf, multinomial
with count, logistic regression
tf.idf, and logistic regression
with count. They all use
the same predict function. They're all using
either the tf.idf
vectorized version
of the email or the count
vectorized version of the email
based on
whether or not it's the tf.idf
or the count version of the model.
And then
we make our predictions,
we print
the email itself, and then we can
print what
each model thinks
it is.
So if you're
still on the fill in the blank bit,
I encourage you to
fill this out.
If you are using
the completed workbook, like
I am, it's already filled out
and we're just going to go ahead and
let everything else
run in the background until we get
to this point.
So, sorry about that.
In the meantime, are there any
questions, any thoughts,
concerns, ideas?
I'd love to hear from you.
All right.
Psychdude
says, on the completed workbook
6a, show the
top right and bottom left
numbers swapped in 6a.
Wait a second.
But the one
you were working on earlier
had those flipped.
The fill in the blank and
the completed, were
they different somehow?
Yeah, so I'm
thinking the reason why they're swapped
is because
there was
a way that
Scikit-learn
documented how they did
their confusion matrix versus how they
actually did their confusion matrix,
and it was swapped from what you would normally see.
And so, following
what you would normally see is what I did,
and that's where the mistake came in.
And I can actually show you that.
I was hoping nobody noticed, but
good eye.
So if you look at 6,
actually before we go there, let's take a look
and see the count.
So we have 590 ham emails in our
test dataset, 276
spam emails in our
test dataset. And so if we
look at 6A,
we see 590 ham
emails, but
as far as actual label, we
should see in the column for spam
a total of the
276. We don't see
that. And the reason why
is because this is supposed to be
the actual, see
108 plus 168 is 276,
and then
this was supposed to be the predicted.
The problem is
the way that scikit-learn
swapped them,
and I guess I did not swap
them back.
To get the
correct result, you just need to switch the
test labels and the predictions
MNB on both sides
for the confusion matrix
and the classification report.
So, yeah.
That's why they seem swapped.
The one on the completed workbook
is correct, because I believe
the completed workbook
shows in the actual label
column the spam emails, and if you total
the two numbers, you get 276.
So, good eye on that.
Go ahead and check this.
Perfect.
Before we ran into the problems,
we're going to look at
6C and 6D with a logistic
regression.
Remember, we had 0.85
or 0.87
0.5% accuracy
for the TF-IDF
multinomial Naive Bayesian.
For the count multinomial Naive Bayesian,
we have 0.94.
For the logistic regression,
we get a little bit better.
We're at 0.959
as our accuracy.
We have fewer
ham emails being classified as
spam and fewer
spam emails being classified as ham.
We're getting better.
With the
task 6D, this is logistic
regression with count vectorizer.
This is going to be our
best model at 0.976.
Here we only have 8 of the
ham emails being classified as spam
and 12 of the spam emails being
classified as ham.
Of the four models that we
trained, this
is the better one
in terms of accuracy.
This is why we want to try
different algorithms, different machine
learning algorithms, as well as different ways
that we preprocess or vectorize our data.
That way we can create
different models, see which ones perform
the best, and then we can gravitate
towards that one.
As far
as the
test spam email,
this is a brand new email.
It comes in on our email server.
We want to use our models to
classify it.
Let's go ahead and make a prediction.
As I went over earlier, we're going to have
our test spam email, which is called working email,
so we can swap this out pretty
seamlessly. We're going to use our
vectorizers that we already defined
to transform our working
email. And then of course, being a
string, we need to put it in a list.
That's what the square brackets are for.
Then we use all four of our models,
the multinomial and the logistic
regression with the tf and the count
vectorizers. We use the predict function
on the vectorized
emails.
And then we just go
ahead and print out the working email itself.
Your latest issue is available
now. If you don't want issue notifications,
click here to unsubscribe.
And then
we print out the values
that are predicted
from each of our models.
If we look at it, we're going to see something interesting.
Our best
performing model, which is the
logistic regression with the count vectorizer,
correctly classified this as a
spam email. But our worst
performing model, even though it still had an
87.5% accuracy,
our multinomial Naive Bayesian
with tf.idf incorrectly classified
this as a ham email.
And so this is also
why we want to
try building different models and get things that
predict much better.
Let me know if there's any
questions on that.
We are short on time. We're actually running
a little over, but before I let you go, there's one more
thing I want to show you. And that is
the hyperparameter
tuning. So if we
jump up to that worst performing model,
that 0.85%,
the multinomial with tf.idf,
sorry,
0.875%.
And we use that
alpha that we saw in our
formulas on the slides.
Alpha is set to 1.
We can go ahead and run that.
And we see that we're at
0.875.
But what if we shrink that alpha
to 0.1?
This is hyperparameter tuning. We can change
it.
Now our accuracy changed from
one of the worst models that we have
to one of the better models that we have.
In fact, this is second place.
So we have 0.965%
accuracy.
And so you can
see how hyperparameter tuning can
really help out in tweaking
the way that our models operate.
Before I let you go,
here's some references
for more information.
And if you want the slides to this
and more information from me
and the GitHub
project that this is all part of,
check out github.com.netsec
explain and you can shoot me an email with any
questions, thoughts, comments
at gtklondike
at gmail.com
So that's all I have for you guys.
If there's any discussion,
TAs, do we have a separate room for the workshops
or are we just going to use this
Discord?
Nope, there is a separate room for
the workshops here. Let me
post that in Twitch.
Um...
There you go, folks. If you have any
other questions or want to continue talking
about this workshop, you can find us
at that channel
in the AI Village
in the DC Discord.
Perfect.
And then I'm going to go ahead
and end the stream. I'm going to hop into
that channel and I will see you guys there.
Thanks a lot, Gavin. Absolutely.
Thank you.