WEBVTT

00:00.000 --> 00:05.620
I just want to double check this.

00:06.280 --> 00:07.420
Alright, cool.

00:07.420 --> 00:10.840
I have all my stuff muted and we are good to go.

00:10.960 --> 00:15.020
Hello and welcome to Machine Learning for Security Analysts.

00:15.020 --> 00:18.640
This is going to be our final workshop for the AI village.

00:19.020 --> 00:28.700
This is going to be a beginner and interactive workshop for all those who are interested in machine learning and security and what this whole crazy AI thing is.

00:28.700 --> 00:38.360
So really, the major motivation for this workshop was that we're seeing a lot of buzz in the security community around AI and machine learning.

00:38.360 --> 00:42.640
But for many of us, it's not really clear what that is.

00:43.460 --> 00:46.620
So... oh, sorry.

00:46.620 --> 00:55.780
If you want to participate and ask questions, head over to the DevSchool channel and join the AIVillage-General-Voice, AIV-General-Voice.

00:55.960 --> 00:58.680
And you'll be able to ask us questions.

00:59.280 --> 01:03.900
If there's too many people joining and it gets unorganized, we're going to switch to Zoom.

01:03.900 --> 01:08.140
But it's sort of more open than it was before.

01:09.000 --> 01:10.160
Sorry, Gavin.

01:10.160 --> 01:11.920
Just needed to say that.

01:12.260 --> 01:12.920
No worries.

01:12.920 --> 01:20.720
Last line of this statement is I wanted to take, for the security community, the idea of machine learning and AI out of the buzzword and into the mainstream.

01:22.040 --> 01:23.480
So, my name is Gavin.

01:23.480 --> 01:25.520
I go by the pseudonym GT Klondike.

01:25.520 --> 01:28.680
I'm an independent security researcher and a security consultant.

01:28.700 --> 01:32.720
I'm very passionate about network attack and defense.

01:32.720 --> 01:45.800
And through that passion, I run a project called NetSecExplain, which is a blog and a YouTube channel, where I explain intermediate and advanced level computer networking concepts and security concepts in an easy-to-understand way.

01:47.420 --> 01:54.660
So, before I get started, usually in person, this is where I ask people, what are your thoughts when you hear the word machine learning?

01:54.760 --> 01:55.380
Right?

01:55.380 --> 02:06.620
And I've gotten a range of answers in the past that range from, oh, Skynet's going to take over the world, to you need a PhD in multivariable calculus and linear algebra.

02:07.700 --> 02:16.520
And while there is a mild amount of truth to the math background that you should have, neither of those are really true.

02:17.920 --> 02:21.220
Instead, machine learning is really a form of pattern matching.

02:21.220 --> 02:22.740
It's applied statistics.

02:22.740 --> 02:34.380
And with the increase in power behind the computer memory and behind the CPU, we're able to use these statistics in a much quicker, more processed way.

02:35.980 --> 02:41.920
So, first I want to kind of cover this idea of machine learning, artificial intelligence, and deep learning.

02:41.920 --> 02:46.960
In a lot of areas, it's being used interchangeably, and that's not really correct.

02:47.240 --> 02:51.620
First, what we have is this large umbrella of artificial intelligence.

02:51.620 --> 02:58.340
And artificial intelligence is kind of a philosophy, in my opinion.

02:58.340 --> 03:05.540
Artificial intelligence used to be, when the first digital calculators came out, that was considered artificial intelligence.

03:05.540 --> 03:07.380
A computer can do math.

03:07.400 --> 03:13.040
As things became more advanced, different concepts were considered artificial intelligence.

03:13.040 --> 03:18.760
There's video game artificial intelligence, rule-based artificial intelligence, classical image recognition.

03:20.200 --> 03:24.840
But today we're talking about machine learning, which is a subset of artificial intelligence.

03:24.840 --> 03:32.320
And it's focused a lot more on grabbing inferences from data and making statistical patterns.

03:33.680 --> 03:38.020
And then inside of machine learning, we have a concept called deep learning.

03:38.020 --> 03:41.520
Deep learning is specifically deep neural networks.

03:41.960 --> 03:47.180
And so neural networks is a machine learning algorithm that's pretty popular.

03:47.180 --> 03:50.200
But deep learning allows us to take that a little bit further.

03:50.200 --> 04:01.400
And so usually we see this in the form of convolutional neural networks, which are primarily used for image recognition, or recurrent neural networks, which are used for cyclical pattern recognition.

04:01.400 --> 04:10.020
So think of things such as a heartbeat monitor, where there's a pattern involved that you kind of need to match up.

04:10.020 --> 04:16.320
And if there's any sort of heart irregularities, the pattern will diverge from predicted patterns.

04:17.960 --> 04:29.480
So if we wanted to formally define machine learning, I would say that it's a set of statistical techniques that enable the process of information mining, pattern discovery, and drawing inferences from data.

04:29.540 --> 04:40.080
So the idea is that the algorithms learn from past data to predict future outcomes, instead of creating a different algorithm for a different scenario.

04:40.400 --> 04:53.640
And we'll see that as we go through the workbooks, how we're able to create multiple different types of algorithms around the same set of data, or how we can use the same algorithms with different types of data.

04:54.440 --> 04:59.920
So some key examples, especially in the security sphere, is domain generation algorithms.

04:59.920 --> 05:14.580
Domain generation algorithms are heavily used by botnets so that the bot herder can stay in contact with the botnet itself, while government agencies or ISPs are taking down the domains that they use to communicate.

05:14.940 --> 05:17.680
We also see this with web application firewalls.

05:17.680 --> 05:30.260
We can perform anomaly detection to identify normal web traffic, normal parameters that are sent by a web client, versus potentially malicious parameters.

05:30.260 --> 05:36.940
In this example here, it's showing a possible SQL injection, and it's calling that out.

05:37.220 --> 05:50.780
And then, of course, network anomaly detection, where what we can do is predict how the network will operate over a period of time, and then keep track of how it is actually operating.

05:50.780 --> 05:56.740
And if it's beyond a certain delta or a difference, then we can flag that and send it off to a security analyst.

05:58.040 --> 06:02.200
So why is this important for security analysts, and also important for hackers?

06:02.700 --> 06:08.640
Well, today, over a quarter of security products for detection have some form of machine learning.

06:08.800 --> 06:18.280
So from an analyst's perspective, what you want to do is understand how these machines work, so that you can ensure that they operate efficiently and are working effectively.

06:19.100 --> 06:23.180
Machine learning isn't this magical black box that just solves all of our problems.

06:23.920 --> 06:31.020
And I want to share a story with you, and this is what really made the idea of machine learning and the value of machine learning really stick out to me.

06:31.480 --> 06:35.420
So a number of years ago, a friend of mine was approached by a non-profit.

06:35.420 --> 06:39.180
And what this non-profit did was track cheetah population out in Africa.

06:39.180 --> 06:49.900
So they set up a bunch of different cameras, and they would take a picture at different intervals, and it would be sent off to a human analyst to identify whether or not there's a cheetah in the picture.

06:50.420 --> 06:54.880
Not a job I would want to do, but it needs done, and it's really important.

06:56.100 --> 07:00.260
So they approached my friend, and they were saying, hey, is there a way that we can increase our efficiency?

07:00.260 --> 07:04.780
Something that we can use to kind of automate some of the manual labor that we're doing.

07:05.180 --> 07:13.140
And so what he did was spend about a week, and he used a very simple machine learning model to identify images.

07:13.200 --> 07:23.740
And what he told this model was, hey, if you see an image, and you are at least 5% confident that there's a cheetah in the picture, send it off to an analyst.

07:23.740 --> 07:26.920
If you're less than 5% confident, just throw it away.

07:27.640 --> 07:31.160
Now 5%, that doesn't really sound like a lot, and to be honest, it's not.

07:31.320 --> 07:41.980
But what this non-profit was able to do with this simple model was what would normally take them a year of labor they were able to complete in one month.

07:42.620 --> 07:52.320
That is a 1200% increase in productivity by just implementing a little bit of machine learning and that fuzzy logic into their process to kind of help automate it.

07:52.860 --> 08:09.060
So this is what I want to do to kind of push for the security analysts and for the hacker community to get a little bit more involved in AI and machine learning so that we can see and identify ways that we can improve our own processes, especially ones that require a little bit of fuzzy logic,

08:09.060 --> 08:14.160
things that traditional signatures or scripting aren't sufficient in.

08:16.380 --> 08:20.500
So the 7-step machine learning process generally looks like this.

08:20.660 --> 08:25.560
And this isn't a list so much as a cycle, right?

08:26.560 --> 08:28.980
So first we're going to start with gathering the data.

08:28.980 --> 08:35.660
We want to identify what it is we want our machine learning algorithm to classify.

08:35.960 --> 08:43.100
So first we're going to gather the data and then we're going to prepare the data in such a way that is easy for machine learning models to understand.

08:43.300 --> 08:47.440
And in this case, it's usually turning it into some form of number or some form of signal.

08:48.780 --> 08:51.860
Most of our time is spent gathering the data and preparing the data.

08:51.860 --> 08:54.280
The rest of this tends to be a little bit quicker.

08:55.160 --> 09:00.260
So first, based on the data that we have and the problem set that were presented, we choose a model.

09:00.260 --> 09:03.420
So some models perform better than others in certain tasks.

09:03.420 --> 09:04.720
And sometimes you just got to kind of

09:07.880 --> 09:10.120
figure out maybe neural nets isn't the best model for you.

09:10.120 --> 09:15.220
Maybe logistic regression or SVM or random forest classifier is better for you.

09:15.680 --> 09:19.560
But once we've identified the model or models, then we go ahead and train it.

09:19.560 --> 09:23.600
And so what we'll usually do is perform what's called a training testing split.

09:23.600 --> 09:29.320
And in this case, we have a series or a lot of clean labeled data.

09:29.480 --> 09:33.440
And we take that data, we separate 80% of it and 20% of it.

09:33.440 --> 09:35.880
80% of it is what we use to train the model.

09:35.880 --> 09:41.940
And then the other 20% is what is used to evaluate the model to see how well the model is performing.

09:42.840 --> 09:47.160
And so it's kind of like how we teach students in elementary school, right?

09:47.160 --> 09:53.140
We train them on the bulk of information and then we test them to see how well they're understanding the material.

09:54.040 --> 09:57.740
And so some of our models will perform very well.

09:57.740 --> 10:00.640
Some of our models may not.

10:00.640 --> 10:08.260
And so if we have a model that performs fairly well, but we think it could do better, then we hop into what is called hyperparameter tuning.

10:08.500 --> 10:11.600
And we're going to see what a hyperparameter looks like in a little bit.

10:11.880 --> 10:24.160
But hyperparameter tuning, think of it as little knobs and levers and switches that we can use to kind of tweak our model and tune it in such a way that it is better at a certain type of problem.

10:24.980 --> 10:27.680
And then once we've completed all this, we're ready to deploy.

10:28.000 --> 10:30.040
Now the reason why this is cyclical, right?

10:30.040 --> 10:42.860
We want to gather more data, prepare the data, and use the same model that we're already using after deploy is because machine learning stops learning as soon as it's trained.

10:43.300 --> 10:58.260
So to kind of give you an idea, I know that certain spam filters like Facebook's Facebook spam filter, they retrain it and rebuild it every one to three days because that's how quickly spam is being processed through their pipelines and through their networks.

10:58.320 --> 11:01.420
And so it's a way for them to kind of stay on top of that.

11:04.280 --> 11:10.240
So we're going to hop into understanding how machine learning works, especially in the security space.

11:10.240 --> 11:14.040
And the easiest example, in my opinion, is building a spam filter.

11:14.600 --> 11:17.900
Now spam filters, we already know how they operate conceptually.

11:18.180 --> 11:20.660
So it's a lower barrier of entry.

11:20.660 --> 11:24.580
We're going to see specifically how the machine learning fits in.

11:24.760 --> 11:39.220
But before I hop into that, are there any questions on how machine learning, AI, and deep learning kind of intermingle and why it's important to understand machine learning from a security standpoint or in the hacking community?

11:46.490 --> 11:54.110
I see on the Twitch chat, reinforcement learning, reinforcement learning, self-learning, contextual regret minimization, what differences between them?

11:54.410 --> 12:07.970
I'm not familiar with counterfactual, not familiar with counterfactual regret minimization, but typically you'll see three types of machine learning.

12:07.970 --> 12:12.490
You'll see classification or supervised learning, which is what we're going to be covering.

12:12.490 --> 12:17.430
You'll see unsupervised learning, which is very good for data exploration.

12:18.870 --> 12:21.990
And unsupervised learning uses a whole different set of algorithms.

12:21.990 --> 12:32.410
And then there's reinforcement learning, where we embed in a model different rewards and punishments based on how it's performing.

12:32.690 --> 12:37.590
And so these are all different ways of training the model, and then we can make decisions off of those.

12:39.830 --> 12:45.070
Hey, Sven, are there any questions in the Discord chat before I move on?

12:55.840 --> 12:57.780
Okay, I'm going to assume not.

12:59.480 --> 13:04.740
So before we look at a full spam filter, we want to kind of look at a smaller example.

13:04.880 --> 13:11.800
So in this case, we have a series of sentences talking about sports or not sports, which are really elections.

13:12.260 --> 13:13.980
So here we have a couple sentences.

13:13.980 --> 13:14.920
A, great games.

13:15.100 --> 13:15.980
Sorry.

13:16.060 --> 13:16.760
Yes.

13:18.260 --> 13:20.420
Well, I think you got this one.

13:21.340 --> 13:23.200
But there was a question from Norwin.

13:23.200 --> 13:24.680
I fell asleep on the job.

13:26.340 --> 13:27.660
No worries.

13:27.780 --> 13:29.280
It has been a long weekend.

13:30.260 --> 13:31.780
So what was the question?

13:34.340 --> 13:38.680
Norwin kindly repeated Glucorio's question.

13:40.200 --> 13:42.060
So you already answered it.

13:42.060 --> 13:43.320
Okay, okay.

13:45.000 --> 13:49.260
So kind of going back, we have a great game, which is talking about sports.

13:49.260 --> 13:53.200
We have the election was over, which is talking about not sports.

13:53.240 --> 13:55.680
Very clean match, talking about sports.

13:55.680 --> 13:58.400
Clean but forgettable game, talking about sports.

13:58.400 --> 14:08.500
So if we wanted to take this idea and specify a sentence that we've never seen before about sports and not sports, how would we do this?

14:08.700 --> 14:11.640
We would want to look at keywords.

14:11.900 --> 14:16.380
So in this case, we can see that it's talking about games.

14:16.380 --> 14:18.060
We know it's probably about sports.

14:18.060 --> 14:21.720
If it's talking about elections, it's probably not talking about sports.

14:22.500 --> 14:26.640
And so what we want to really hone in on are keywords.

14:26.940 --> 14:33.500
But to put this into a machine learning model, we need to find a way to turn this into numbers.

14:34.660 --> 14:37.920
So the way that we're going to do this is Bayes' Theorem.

14:37.920 --> 14:39.380
Bayes' Theorem is...

14:40.030 --> 14:44.800
it may look complex at first, but it's actually a really easy to understand probability theorem.

14:45.480 --> 14:53.540
And in this case, we see the probability of A given B equals probability of B given A times the probability of A divided by the probability of B.

14:53.540 --> 15:02.800
So in this case, we want to see what is the probability that this sentence is talking about sports given that the sentence is A very close game.

15:03.280 --> 15:11.780
So we don't know the probability of sports given A very close game, but we can easily plug that into this formula and say, well, we know the probability of sports.

15:11.780 --> 15:15.120
We saw three out of our five sentences were talking about sports.

15:15.120 --> 15:21.500
And we can figure out the probability of A very close game given sports and the probability of A very close game.

15:22.200 --> 15:23.140
But how do we do that?

15:24.220 --> 15:25.540
How do we get a sentence like this?

15:26.000 --> 15:31.020
Well, with Bayes theorem, we can actually, or with probability, actually, we can split this up.

15:31.020 --> 15:39.160
So instead of A very close game as a whole sentence, we can look at the probability of A times the probability of very times close times game.

15:39.640 --> 15:49.840
And so what this turns into in our A very close game given sports is probability of A in sports, probability of very in sports, and so on.

15:49.840 --> 15:50.920
And of course, we're going to want to do

15:54.220 --> 16:02.140
is say, okay, whichever is highest, sports or not sports, that's the one that we're going to classify this sentence as being part of.

16:03.380 --> 16:06.260
So at this point, it becomes a simple counting game, right?

16:06.260 --> 16:10.160
So how many times does the word A show up in sports?

16:10.400 --> 16:14.520
Well, we see that it shows up twice, once in the first sentence and once in the fourth sentence.

16:14.520 --> 16:17.340
And then how many times does it show up in not sports?

16:17.540 --> 16:19.600
Once, right there in the fifth sentence.

16:24.220 --> 16:25.360
So A does not show up in sports.

16:25.360 --> 16:26.920
It only shows up in not sports.

16:26.920 --> 16:30.040
And then we see the word game show up twice in sports.

16:31.740 --> 16:41.180
So we just take those numbers and we plug them in and we say, okay, A showed up in sports twice, very showed up once, close showed up zero times, and game showed up twice.

16:41.280 --> 16:48.020
And if we counted the number of unique words, we would have had 11 unique words across all of our sentences talking about sports.

16:48.880 --> 16:50.240
But we run into a problem

16:54.620 --> 17:02.400
and the problem is that what we want our machine learning model to do is identify and classify sentences that it has never seen before.

17:02.960 --> 17:08.380
Unfortunately, it's never seen close and so that winds up being a zero, zero times everything is zero.

17:08.380 --> 17:11.380
And we see that that easily cascades down and just ruins our model.

17:11.380 --> 17:17.180
So it's because we've never seen the word close, there's a zero probability that this exists in our sentence.

17:18.080 --> 17:21.480
So that's a problem, but we can get around that.

17:22.700 --> 17:26.060
There's another algorithm called multinomial Naive Bayes.

17:26.180 --> 17:31.140
And so this is just a small modification and what it does is add what is called a smoothing filter.

17:31.140 --> 17:33.080
And we can see this alpha here.

17:33.080 --> 17:35.740
It's added to the numerator and the denominator.

17:35.800 --> 17:44.540
And usually this alpha is one, but this is going to be our hyperparameter for Naive Bayesian, multinomial Naive Bayesian that we can tweak, right?

17:44.540 --> 17:47.900
We can set it to one, we can set it to two, we can set it to 0.1.

17:48.040 --> 17:54.660
And so this is a thing that we can tweak to see if we can get our model to work for us .

17:55.440 --> 17:56.780
So we take the same idea.

17:56.780 --> 17:57.680
We do the word counts.

17:57.680 --> 18:01.360
We saw that a showed up in sports twice, very showed up once, close to zero.

18:01.360 --> 18:03.180
But now we have this plus one.

18:03.580 --> 18:08.480
So the probability of close given sports is never going to reach zero.

18:09.180 --> 18:13.060
And then of course, as I mentioned earlier, we need to do the same thing for not sports.

18:13.060 --> 18:16.280
And the total number of unique words in not sports is nine.

18:16.280 --> 18:18.600
The total number of unique words in sports is 11.

18:18.600 --> 18:23.320
And the total number of unique words across both classes is 14.

18:26.080 --> 18:37.880
So we plug in our answers and we do our calculations and we get 0.0000461 for sports and 0.0000143 for non-sports.

18:38.200 --> 18:42.800
Classifying the sentence, a very close game being about sports.

18:45.480 --> 18:48.320
So that was a lot that I just kind of laid on you.

18:49.080 --> 18:51.660
If you have any questions, type them in the chat.

18:52.020 --> 19:06.500
While you're doing that, I'm going to show you how we can take our raw emails and transform them into these types of counts and these types of numbers in a very clean and effective way.

19:06.960 --> 19:22.700
So the five things we need to keep track of are the total number of unique words, the total number of words in spam, the total number of words in ham, or regular emails, the count of each word in spam, and the count of each word in ham.

19:25.960 --> 19:28.000
So let's take a look at one of our emails.

19:28.720 --> 19:31.620
We see here, East Asian fonts in Lenny.

19:31.620 --> 19:32.640
Thanks for your support.

19:32.640 --> 19:34.760
Installing Unifonts did it well for me.

19:35.340 --> 19:42.240
Now we want to take a sentence like this and find a way to classify it as being spam or ham.

19:42.700 --> 19:45.840
But before we do that, we want to kind of point out a couple things.

19:45.840 --> 19:52.700
So we see here kind of towards the bottom, we have the word unsubscribe, all capitalized, and then unsubscribe lowercase.

19:52.900 --> 19:55.460
What we want to do is look at keywords.

19:55.500 --> 20:04.180
And so in context, it doesn't matter if unsubscribe is capitalized or the first letter is capitalized or it's all lowercase.

20:04.280 --> 20:06.300
It means the same thing.

20:06.320 --> 20:13.420
And so to kind of reduce our problem set, what we're going to do is set everything to lowercase.

20:13.900 --> 20:17.180
The other thing we want to do is remove what are called stop words.

20:17.180 --> 20:21.700
Stop words are words like the, in, of, on, or...

20:21.700 --> 20:24.420
They don't add any context to the sentence.

20:24.780 --> 20:27.100
And we're going to do the same thing for punctuation.

20:27.200 --> 20:28.580
So we're going to remove those.

20:28.580 --> 20:30.300
We're going to keep our keywords.

20:30.300 --> 20:33.300
And then one more thing that we're going to do is called stemming.

20:33.580 --> 20:41.620
So if we have something like the word congrats or congratulations, in the English language, they mean the same thing.

20:41.620 --> 20:45.820
So if we're looking at keywords, we want to treat them the same.

20:45.820 --> 20:50.960
And so what stemming does is shrink those words down to their base form or the root form.

20:50.960 --> 20:52.860
And then we can count them the same.

20:52.940 --> 20:59.840
Another really good example is the word thanks, thank you, where it's a just singular thank.

20:59.840 --> 21:03.380
And I thanked somebody today with an ed at the end.

21:04.040 --> 21:05.940
As a keyword, they all mean the same thing.

21:05.940 --> 21:08.360
And so stemming will reduce them all down to thank.

21:08.540 --> 21:11.760
And so instead of three separate words, it's going to be counted as one word.

21:11.760 --> 21:17.200
And so that's going to drastically reduce our dimension space.

21:18.300 --> 21:30.080
So after our preprocessing, which is what this is known as, we're going to see East Asian fonts in Lenny, or East Asian fonts Lenny, thanks support installing Unifonts, well me.

21:31.400 --> 21:36.060
And then what isn't shown here is what the stemming looks like.

21:36.060 --> 21:45.840
So as we go through the workbook, we'll take this exact same email and see how it's transformed at every step of our preprocessing.

21:46.480 --> 21:49.260
So are there any questions so far?

21:55.420 --> 21:56.680
I don't think so.

21:56.680 --> 21:59.240
There's none on Twitch chat, and there's none in the Discord.

21:59.280 --> 22:00.320
Perfect.

22:02.320 --> 22:05.860
So what we're going to have are a series of workbooks.

22:05.860 --> 22:07.820
We're going to use MyBender.

22:07.820 --> 22:09.740
MyBender is really awesome.

22:10.700 --> 22:13.620
TAs, if you don't mind sharing this link.

22:13.880 --> 22:26.740
What it will do is spin up a Docker container with the workbook linked to my GitHub so that you can kind of code with me as we walk through and build our own machine learning classifiers.

22:27.980 --> 22:39.600
While this is starting up, what we're going to be using is instead of coding everything raw and doing the raw numbers like we did on our pen and paper, we're going to be using the abstraction library Scikit-learn.

22:39.600 --> 22:50.700
Scikit-learn is very useful and it allows us to treat things kind of like Legos, where, OK, maybe I don't want multinomial Naive Bayesian, maybe I want logistic regression or random forest.

22:50.700 --> 22:52.680
We can just swap that out.

22:54.780 --> 23:00.780
So once it's loaded, and I'm going to give you a moment to get this loaded up.

23:01.380 --> 23:09.600
If it takes a little bit longer than it should, then you'll easily be able to catch up to me and I'll show you how.

23:10.160 --> 23:15.860
So we're going to click on workbooks spam filter sklearn.

23:15.860 --> 23:19.940
If you're not very familiar with Python, you can still follow along.

23:19.940 --> 23:24.780
We have a completed workbook, which is the workbook complete spam filter sklearn.

23:25.720 --> 23:26.800
And these are Jupyter notebooks.

23:26.800 --> 23:32.380
If you've never used Jupyter notebooks, I think they're the best thing since sliced bread, especially when it comes to data science.

23:32.820 --> 23:38.540
But it allows us to interact with code blocks and visualize things in line.

23:38.540 --> 23:39.740
And I think they work fantastic

23:43.520 --> 23:44.040
together.

23:45.080 --> 23:47.800
So let's go ahead, scroll down.

23:47.800 --> 23:50.560
We're going to run this first code block.

23:51.100 --> 24:00.960
And so what this is going to do is install a couple of libraries that we need, as well as set our data directory to where the data directory is in the GitHub project.

24:00.960 --> 24:07.900
So to run these, you can click run up at the top, or you can press shift enter, which is what I'm pressing now.

24:07.900 --> 24:10.560
And then you can see that it's running.

24:10.560 --> 24:13.780
A star in the upper corner means that it's currently running.

24:13.780 --> 24:19.840
And then a number in the upper corner means that that was the order that code block was ran in.

24:22.560 --> 24:29.880
So while that is going, I'm going to let it go until it stops scrolling my thing.

24:30.900 --> 24:33.180
So we're just going to import a couple libraries.

24:33.180 --> 24:41.360
We're going to import numpy, matplotlib for graphs, regular expressions we'll be using and NLTK.

24:41.360 --> 24:44.380
NLTK stands for Natural Language Toolkit.

24:44.380 --> 24:51.560
This will allow us to identify and collect a list of stop words to remove those things.

24:51.560 --> 24:59.780
And it will also allow us to use a stemmer function to stem the words and reduce our problem space.

25:00.800 --> 25:04.320
Train test split, I don't believe we're actually using that in here.

25:04.320 --> 25:11.060
The data directories I have are already split into 80% training, 20% testing.

25:11.300 --> 25:13.320
And then we're going to use our vectorizers.

25:13.320 --> 25:24.140
And we'll explain what the vectorizers do, but we're going to take a look at both of the TF-IDF vectorizer and the count vectorizer because they operate slightly different.

25:24.140 --> 25:29.300
Count vectorizer just takes a straight word count like we did with our pen and paper example.

25:29.300 --> 25:34.680
And then TF-IDF actually takes all of the words into account and gives them a weight.

25:35.800 --> 25:41.060
Scikit-learn, we're going to use logistic regression and multinomial naivetation.

25:41.060 --> 25:46.140
And the reason why we're going to use both of these is because we don't actually know which one's going to perform best.

25:46.180 --> 25:52.960
And so like good data scientists, we're going to try a couple different models and take the one that performs best with the problem set that we have.

25:53.560 --> 26:01.640
And then for a couple metric and visualizations, we're going to use the Scikit-learn confusion matrix and classification report.

26:03.040 --> 26:06.240
So I'm going to go ahead and press Shift-Enter and that's going to run.

26:06.240 --> 26:09.960
You will see libraries imported once that's complete.

26:11.520 --> 26:13.300
And we can move on to the next thing.

26:13.620 --> 26:14.700
So this is our test email.

26:14.700 --> 26:17.760
This is the exact same email that we saw in the slides.

26:17.980 --> 26:25.040
So as I mentioned earlier, we're going to use this email to see how it's transformed at every step of preprocessing.

26:25.040 --> 26:26.540
So I'm going to go ahead and run that.

26:27.980 --> 26:30.000
And then this is our tokenizer function.

26:30.000 --> 26:37.060
Tokenizer functions will allow us to pull out the keywords in the way that we would like them to be extracted.

26:37.180 --> 26:43.500
So in this case, we're going to grab all the punctuation, all the stop words, and we're going to create a stemmer constructor.

26:44.560 --> 26:49.100
We're going to use these to remove the stop words from the email.

26:49.100 --> 26:54.120
We're going to use these to remove punctuation from the email, and then we're going to stem the words.

26:56.540 --> 27:00.100
We're going to move all the words to lowercase so that they are treated exactly the same.

27:00.980 --> 27:05.400
So we're just going to go ahead and run this, shift enter, and our tokenizer is defined.

27:05.960 --> 27:10.420
And this is where we run into our first task, which is to tokenize an email.

27:10.780 --> 27:18.920
So our task says that we want to print the full email, test email, and then we're going to print the results of the tokenized version of that email.

27:19.580 --> 27:21.140
So here I'm just going to do

27:27.880 --> 27:31.040
print tokenizer test email.

27:31.520 --> 27:33.860
And we can go ahead and run that, shift enter.

27:34.020 --> 27:39.660
We can see the original email that we have, East Asian Fonts and Lenny, thanks for your support.

27:39.740 --> 28:00.140
We can see, just like in our slides, we have East Asian Font Lenny, slightly changed, thank instead of thanks, support install, because we can have installing installed with an ed at the end, or I will install this software, which is just a standard install.

28:01.340 --> 28:03.920
We see unsubscribe has been changed.

28:04.020 --> 28:13.940
And two of the things that I want to point out is every single word in here is unique, except for the word unsubscribe, which shows up twice, and list.deviant.org, which shows up twice.

28:16.360 --> 28:18.440
So we're going to take a look at our training data.

28:18.440 --> 28:28.980
What this is going to do is identify the directories where our spam files are, our ham files are, and our test files are.

28:28.980 --> 28:40.380
We're going to get a count, because that's one of the five things that we needed to keep track of, was the count of, actually we want to keep track of the count of words, we don't care how many emails we have.

28:40.480 --> 28:48.900
So I'm going to go ahead and run this, and what we're going to do is put those emails into what is known as a corpus.

28:48.900 --> 28:52.180
So we're going to create two list arrays, so corpus and labels.

28:52.180 --> 29:01.620
Corpus is going to hold all of our emails for spam and ham, and then the labels is going to be each label for our data.

29:01.680 --> 29:10.780
So the first thing we're going to do is corpus equals a new list, labels equals a new list, and then we're going to do the same thing for spam and ham.

29:10.780 --> 29:20.380
We're going to load the email bodies from the ham directories and the spam directory into the corpus array, and then we're going to load the labels for each email into the labels array.

29:20.980 --> 29:31.400
So what that's going to look like is we're going to use a for loop here, so we can say for each in os.listdir, and then we have our data directory, which is defined above.

29:31.900 --> 29:36.080
We're going to look at our ham directory for this first round.

29:36.760 --> 29:44.920
With each open file, we're going to look at our data directory plus ham plus os.listdir.

29:44.940 --> 29:52.240
We're going to look at each , because each is the name of our file, and we're going to want to read those as the file descriptor f.

29:52.420 --> 29:59.320
So for our corpus, we can do an append and then f.read.

29:59.320 --> 30:12.380
So this will read each email into our corpus and then labels.append ham, because we know that all of the emails in the ham directory are part of the ham data set.

30:12.460 --> 30:17.780
And so we can easily do the same thing for spam which is we need to be careful on how we change this.

30:17.780 --> 30:24.140
So instead of slash ham, it's going to be slash spam, and then of course our label is also going to be spam.

30:26.060 --> 30:32.000
So I'm going to let this sit for a minute and let you guys go ahead and make sure that you copy this down correctly.

30:33.520 --> 30:36.920
This bit is pretty easy if you're familiar with Python.

30:36.920 --> 30:40.580
It might be a little confusing if you are not, but that's okay.

30:40.580 --> 30:47.990
If you ever get lost at any point, just remember that there are a completed solutions manual in the workbooks directory.

30:50.580 --> 30:54.820
So I'm going to go ahead and run this.

30:54.820 --> 30:56.930
So shift enter to run.

30:57.740 --> 31:00.980
And I spelled labels wrong.

31:02.680 --> 31:05.660
Labels, shift enter to run.

31:05.660 --> 31:07.810
Loading ham, loading spam, done.

31:08.940 --> 31:18.240
So what this is going to do is graphically load or graphically visualize how many ham emails we have versus how many spam emails we have.

31:18.240 --> 31:23.280
And this is something we want to generally do with our data just to see if there's any sort of imbalanced data sets.

31:23.380 --> 31:28.600
This one we have about 50% spam per ham.

31:29.280 --> 31:32.560
It's like 2 to 1 ratio of ham to spam.

31:32.560 --> 31:51.580
And so this I'm not going to consider an imbalanced data set for this example, but if we have something such as like credit card fraud where it happens less than a fraction of a percent of the time, that's something that we really want to dive into simply because if we try and use this exact example on credit card fraud,

31:51.580 --> 32:01.540
it's not going to work because what we will do is create a model that is 99% accurate because it just classifies all credit card transactions as legitimate.

32:01.820 --> 32:05.340
Now if we want to identify credit card fraud , that's not very helpful for us.

32:05.340 --> 32:11.060
So this just allows us to kind of explore the data and see how much of what we have.

32:11.060 --> 32:17.780
So in this case we have 2,359 ham emails and about 1,100 spam emails.

32:19.700 --> 32:23.020
So task 2A, we're going to view our training data.

32:23.020 --> 32:27.260
So this one's just to kind of see what our data looks like.

32:27.860 --> 32:30.880
We generally have an idea, but we just kind of want to play around with it.

32:30.880 --> 32:37.200
And this is another thing that you would want to do in a data discovery process is just kind of see, okay, what is it that we're actually working with?

32:37.600 --> 32:44.360
So in this case, I'm going to pick an email ID, just email ID 5.

32:44.360 --> 32:46.640
You can pick 5 or you can pick any other.

32:47.520 --> 32:52.560
And then I'm going to grab the email, which is part of the corpus array.

32:53.000 --> 32:55.920
Corpus and then our email ID.

32:56.880 --> 33:09.920
And then I'm going to print that email, and then I'm going to do the same thing and print the tokenized version using a tokenizer function of that email.

33:09.920 --> 33:11.700
So we can see how it's transformed.

33:14.180 --> 33:20.760
So just so I don't go too fast, I want to make sure that you're able to follow along and copy this down.

33:21.280 --> 33:22.900
It's fairly simple.

33:23.120 --> 33:26.040
So I will wait just a few seconds.

33:27.060 --> 33:29.460
How are we doing on questions?

33:31.180 --> 33:40.460
No one has been posting some helpful links, but hasn't had too many questions so far.

33:40.500 --> 33:42.500
So thank you very much.

33:42.500 --> 33:43.420
No winner.

33:44.540 --> 33:45.300
Okay.

33:45.300 --> 33:48.780
So when we run that, what we have is the email body.

33:48.780 --> 33:53.900
We see product review, vert tools, dev, 2.0 URL, and then this URL.

33:54.280 --> 34:03.920
We see product review, vert tool, dev, 2.0 URL as one word, HTTP, and then the full URL.

34:03.920 --> 34:09.620
So this is how we can identify and kind of track our data as it's being transformed over time.

34:11.980 --> 34:14.700
So task 3 is where we're going to start using our vectorizers.

34:14.700 --> 34:25.880
What we want to do is get a count of each unique word, and we want to get a count of each unique word per class so that we can use that in our classification.

34:25.980 --> 34:27.180
So there's two ways we can do that.

34:27.180 --> 34:36.920
We can use the straight count vectorizer, which will get the count of our words, and we can use a TF-IDF vectorizer, which will give us a weight of each count.

34:39.480 --> 34:42.980
So no winner says he's behind.

34:43.520 --> 34:46.760
He's catching up and he'll catch up later and follow along.

34:49.500 --> 34:54.620
I can try to figure out a system of helping the winner.

34:54.700 --> 35:00.680
I can send you a Zoom link or something and look over your shoulder metaphorically.

35:01.100 --> 35:04.980
Also , in the workbooks, you can just open up the...

35:04.980 --> 35:06.000
that's the wrong one...

35:06.000 --> 35:12.560
you can just open up the completed spam filter SK Learn, and it's completely filled out for you.

35:12.560 --> 35:16.960
So if you ever do fall behind and you would like to catch up, you can look at the answers there.

35:18.820 --> 35:23.000
There is a way to follow along and make sure your code works.

35:23.880 --> 35:25.140
Yeah, absolutely.

35:25.140 --> 35:26.780
And I like using the completed workbook.

35:26.780 --> 35:32.040
You can just run everything and see how things are supposed to be at each step of the process.

35:33.540 --> 35:35.540
So we're going to go ahead and train our vectorizers.

35:35.540 --> 35:37.620
And the instructions tell us exactly how to do this.

35:37.620 --> 35:39.520
So we're going to use the vectorizer.

35:39.520 --> 35:46.020
We're going to call it cvec for the count vectorizer, tvec for the tfidf vectorizer, and we're going to perform fit transform.

35:46.780 --> 35:50.840
Fit in scikit-learn world is code for train.

35:50.880 --> 35:55.160
So this is going to, quote unquote, train our vectorizers so they can get the count of each words.

35:55.160 --> 35:59.460
And then we're going to save the results as count x and tfidf x.

36:00.400 --> 36:06.160
So what that looks like is cvec equals count vectorizer.

36:06.160 --> 36:09.560
And then we want to use our tokenizer that we defined above.

36:09.560 --> 36:14.540
So the tokenizer that we're going to use is called tokenizer.

36:14.540 --> 36:16.740
I know, not confusing at all.

36:16.740 --> 36:23.140
But we're going to use our count x.

36:23.140 --> 36:25.020
And pay attention to your capitals.

36:25.040 --> 36:28.200
Scikit-learn is very picky and so is my workbook.

36:28.500 --> 36:33.260
We're going to use cvec.fit transform.

36:33.260 --> 36:39.400
And then we're going to train transform or count the words for our corpus.

36:40.160 --> 36:43.620
And then we're going to do the same thing with our tfidf vectorizer.

36:43.820 --> 36:48.100
tvec equals tfidf vectorizer.

36:48.100 --> 36:52.220
And we're going to use our tokenizer called tokenizer.

36:52.220 --> 37:01.080
And then our tfidf x equals tvec.fit transform.

37:01.080 --> 37:04.160
And it's also going to be on our corpus.

37:04.160 --> 37:09.100
So I'm going to let this sit for a minute and let you guys be able to catch up.

37:12.720 --> 37:13.820
And this takes a little bit.

37:13.820 --> 37:27.440
This takes about two minutes to run because what it's doing is it's reading every single email and it is counting the words in those emails in order to get the number of unique words.

37:30.060 --> 37:34.620
So I'm going to go ahead and run this because it's going to take a little bit.

37:35.020 --> 37:39.020
But in case you need, here is the code while this is operating.

37:39.020 --> 37:43.920
And so we can see that it is currently training the count vectorizer, kind of down here at the bottom.

37:44.380 --> 37:48.620
And then once the count vectorizer is complete, it will start training the tfidf vectorizer.

37:48.720 --> 37:51.740
And then it will let us know when that is complete.

38:02.120 --> 38:06.840
So while this is going, I'm going to go ahead and move on to the next code block.

38:06.840 --> 38:12.980
We can write the code while this is still training, but we won't be able to run it because it can run only one code block at a time.

38:13.500 --> 38:18.060
So in this case, we have task 3a, which is to count the test email tokens.

38:18.060 --> 38:22.420
So we're going to manually count out each of the tokens.

38:22.420 --> 38:33.040
So what we're going to do is say for i in and this is a bit of a trick so that we can keep the count as well as the words in order.

38:33.040 --> 38:35.620
So we're going to use a list of a dictionary

38:38.820 --> 38:44.880
and we're going to want to grab the tokenized version of our email.

38:44.880 --> 38:54.580
So let's call this tokenized email equals tokenizer test email.

38:54.720 --> 38:59.900
And then our from keys we're going to have tokenized email which we saw above is going to be a list .

38:59.900 --> 39:06.860
So we'll just call that tokenized email and then we're going to count the words .

39:06.860 --> 39:23.000
So print, do some Python format, And then we're going to look at our tokenized email.count of each word, and then the word itself.

39:23.440 --> 39:29.660
So we see our vectorizing is complete, so we are ready to run this.

39:29.660 --> 39:36.060
I'm going to let this sit for about 30 more seconds and let you guys catch up.

39:36.060 --> 39:36.960
And

39:43.710 --> 39:50.970
then, of course, after this workshop, I have a whole series of workbooks, as well as their solutions.

39:50.970 --> 40:00.050
So if you're interested in other ways to apply machine learning and security to different problems, and using different algorithms, don't forget to take a look at those.

40:02.630 --> 40:08.350
So I'm going to go ahead and run this, Shift-Enter, and we can see the counts.

40:08.350 --> 40:11.490
So we have East Asian font, Lenny, thanks, support.

40:11.830 --> 40:17.710
Each of these, we have our unique words, they only show up once in our email.

40:17.710 --> 40:23.830
The only two that are outliers is unsubscribe, or unsubscrib, after our stemmer.

40:23.830 --> 40:28.890
And this list.debian.org, they show up twice in our email.

40:28.890 --> 40:33.030
So this is if we manually counted these, but what if we used our vectorizers?

40:33.770 --> 40:47.430
So what we're going to do here in Task 3B is create new vectorizers, the count vectorizer and TF-IDF vectorizer, and then we're going to print out the counts that each of them represent.

40:47.670 --> 40:57.150
So in the first one here, I'll just call this example-cvec, and then we're going to define a new vectorizer.

41:02.220 --> 41:13.760
And I'll just call this example-x equals example-cvec.fit-transform.

41:14.340 --> 41:20.700
And then it expects a list, our email is a string, so we're going to wrap it in these square brackets.

41:21.000 --> 41:23.600
And we're just going to use our raw test email.

41:26.080 --> 41:30.960
And then we're going to want to print our example-x.

41:33.140 --> 41:38.180
And we can easily just copy and paste, do the same thing for our TF-IDF code.

41:38.180 --> 41:52.840
We'll just call this example-tvec, we'll use the TF-IDF vectorizer, tokenizer is tokenizer, we change this to tvec, and then we're already done with our example-x, so we don't need to change the name, we can just leave it as is.

41:53.980 --> 41:59.940
Example-tvec, fit-transform, and then square brackets, test-email, so that it's a list.

42:00.860 --> 42:01.760
Yes.

42:04.900 --> 42:07.800
Oh, it sounded like there was possibly a question.

42:13.190 --> 42:18.510
So Voto on the stream is asking, where are those workbooks again?

42:24.050 --> 42:28.070
Can one of the TAs post in the stream?

42:28.070 --> 42:29.750
There you go, perfect.

42:30.270 --> 42:32.170
Machine learning for a security analyst.

42:33.530 --> 42:37.830
Okay, so I'm going to go ahead and run this.

42:38.570 --> 42:41.030
Scroll up, let's take a look at the count vectorizer.

42:41.130 --> 42:44.890
So instead of seeing the words, we see these numbers, and that's exactly what we want.

42:44.890 --> 42:47.090
We've created what are called tokens.

42:47.330 --> 42:53.750
So the words themselves, the tokenizers will be able, or the vectorizers will be able to match them together.

42:55.090 --> 43:03.150
So all of them are one in their count, all of them are unique, except for token 16 and token 9.

43:03.150 --> 43:12.910
We don't know which one, either unsubscribe or list.wn.org, maps to 16 or 9, but what we do know is that both of them show up twice.

43:12.910 --> 43:21.990
So given that, list.wn.org is either 16 or 9, and then the other one belongs to unsubscribe, or the word unsubscribe.

43:21.990 --> 43:27.010
So this looks exactly the same as if we had done our raw count.

43:27.650 --> 43:32.470
If we look at our TF-IDF vectorizer, though, we see something a little differently.

43:32.470 --> 43:39.950
TF-IDF, what it does is use term frequency to give each term or token a weight.

43:40.030 --> 43:53.790
And so in this case, we see all of the ones that showed up once have a weight of 0.204, token 9 and token 16 have a weight of 0.408.

43:54.270 --> 44:00.930
So you don't really need to understand the underlying math, you just need to know that there are different ways to vectorize your data.

44:01.590 --> 44:08.130
So there's TF-IDF, count vectorizer, word2vec, and a couple others, bag of words.

44:08.870 --> 44:12.470
And so there's different ones that we can try and see which one works better for us.

44:14.170 --> 44:19.090
So let's hop down and start to load our testing data.

44:19.090 --> 44:30.450
So loading our testing data, what this is going to do is just show us the names of each of our files in our testing directory.

44:31.430 --> 44:36.650
And we have train email.txt, and then at the end we have the class that it belongs to.

44:36.650 --> 44:39.030
So we have ham, we have spam.

44:39.750 --> 44:45.310
So what we're going to want to do as part of loading our testing data is we're going to want to strip off these labels.

44:45.910 --> 44:55.010
So we're going to read the full email into our corpus, and we're going to strip off these labels, and those are going to be the new labels for our testing data.

44:56.410 --> 45:00.110
So that hops us into task 4 to load the testing data.

45:00.110 --> 45:06.810
Just like loading the training data, we're going to create two list arrays, our corpus, we're going to call this test corpus, and our test labels.

45:06.970 --> 45:14.310
And then we're going to load each email body, and then strip off the label and throw that into the test labels array.

45:16.450 --> 45:20.810
So what that's going to look like is...

45:25.550 --> 45:28.530
So we're going to look at... what was the name of that?

45:28.730 --> 45:30.770
Test corpus.

45:30.950 --> 45:37.270
And we're just going to set that as a blank list, and then test labels as a blank list.

45:37.990 --> 45:50.130
For loading each of the emails, we're going to do just like we did before, filename in ls.listdir, and then our data directory plus the test directory.

45:50.130 --> 46:08.230
So we're going to look at each filename, we're going to open each file, so data.dir plus test plus filename, and then we want to read those as the file descriptor F.

46:09.190 --> 46:19.830
So our test corpus, we're going to append F.read, which is going to read that email into our test corpus array, and then we want to grab the label.

46:19.830 --> 46:23.950
And in this case, I'm just going to use a simple regular expression.

46:25.390 --> 46:36.710
We're going to use txt, and then this is a backslash period, and filename 1.

46:36.710 --> 46:49.690
I'm going to scroll up real quick, because the reason why I want to do that is what this is saying is, where we see txt.period, we're going to split, so we have this full train txt.period, and then we have ham.

46:49.910 --> 47:02.610
And then the bracket 1, what that is going to do is say, this is bracket 0 once it's split, and then this is bracket 1, so we're just going to keep our labels, which are at the end of that split.

47:05.230 --> 47:17.350
And then with that label, or each label that we grab from the filename, we're going to append it to our labels directory, or our labels array.

47:19.210 --> 47:21.450
So this is what our code looks like.

47:21.450 --> 47:25.990
Double checking for errors, looks good to me.

47:26.390 --> 47:31.630
And I'm going to let this sit for a little bit, so you guys can go ahead and copy that down.

47:39.290 --> 47:41.170
How are we doing?

47:45.460 --> 47:50.780
No questions in Twitch stream, are there any questions in the Discord channel?

47:57.980 --> 48:01.420
Okay, I'm going to go ahead, shift, enter, and run.

48:01.420 --> 48:04.040
So this is going to quickly load the emails.

48:06.240 --> 48:09.400
And then let's run this next code block.

48:09.400 --> 48:13.060
So what this is going to do is allow us to visualize our data, just like we did before.

48:13.060 --> 48:29.160
So we see in our testing set, we have 590 ham emails, 276 spam emails, and the graph itself looks nearly identical to the one above, and that's because we did a random 80-20 split for our training and testing.

48:30.300 --> 48:33.020
And that was already done beforehand.

48:34.460 --> 48:51.780
So the next thing we want to do is use the vectorizers that we created, the cvec and the tvec vectorizer, and take a look at the test corpus, and get our counts of each word in the new emails.

48:52.560 --> 49:06.420
So let's go ahead, testCountX equals, and we're already going to use cvec, we're not going to use fitTransform, instead we're going to use transform, because we're not trying to train it, we're just trying to save the words themselves.

49:07.680 --> 49:20.040
We'll do testCorpus, and then testTFIDFX equals tvec.transform testCorpus.

49:20.280 --> 49:25.940
And that's all we need to do when we're using a piece of testing data.

49:27.700 --> 49:33.620
So I'm going to let that sit for a second, actually I'm going to run it because this is going to take a little bit as well.

49:36.620 --> 49:47.460
Slightly shorter than when we were training it, but it still needs to go through all the emails and apply the counts, and then the TFIDF, it needs to apply the weights.

49:47.460 --> 49:51.640
We already know what the counts are, we already know what the weights are, we just need to apply them to each token.

49:52.220 --> 49:53.840
So vectorizing is complete.

49:54.520 --> 49:59.220
And now we're finally on to testing and evaluating our model.

49:59.640 --> 50:05.460
So you remember how I said data collection and preprocessing is what's going to take the most amount of time?

50:05.460 --> 50:07.640
Well, that was about 80% of this workbook.

50:08.140 --> 50:11.340
If not, maybe a little more, maybe 85%.

50:12.320 --> 50:14.740
But yeah, data science in a nutshell.

50:15.680 --> 50:20.620
So here we have a very helpful report generator function.

50:20.920 --> 50:35.120
It's not important to know what's going on in this, it's just going to be a helper function, but we do need to know that it's going to take in our confusion matrix, our score, which is our accuracy, and the classification report, and then it's going to print out all the stats that we need.

50:36.520 --> 50:48.340
So that hops us into task 6A, which is going to be training and evaluating our first model, which is going to be multinomial Naive Bayesian using the TFIDF vectorizer.

50:49.980 --> 50:53.940
So there's a lot of steps to this, but we're going to walk through it step by step.

50:54.200 --> 51:08.000
First, what we're going to do is create a multinomial Naive Bayesian constructor, so that's going to look like mnb, tf, idf, and the names of these variables are very important because that's what's being used in the generator function.

51:08.980 --> 51:31.300
So I'm going to call this Numial Naive Bayesian, and then we're going to train the multinomial Naive Bayesian, as I already misspelled something, mnb, tf, idf, .fit, which is scikit-learn's code for train, and then we're going to use our TFIDF, our vectorized corpus,

51:31.300 --> 51:33.440
with the TFIDF array.

51:35.360 --> 51:48.740
So now we're going to get a couple stats, so we're going to get our score, mnb, tf, idf, equals mnb, tf, idf, .score.

51:48.740 --> 51:57.380
This is going to give us our accuracy, and we want to compare this against our test, tf, idf, x, and our test labels.

51:58.260 --> 52:14.760
And then we want to take a look at the raw predictions, so we're going to have our predictions, mnb, tf, idf, equals mnb, tf, idf, .predict, using the predict function, on our test, tf, idf, x.

52:15.540 --> 52:16.400
Cool.

52:16.400 --> 52:27.540
And with those results, what we are going to do is generate a classification, a confusion matrix, so we can see our true positives, false positives, true negatives, false negatives.

52:27.580 --> 52:50.800
So cmatrix, mnb, tf, idf, equals our confusion matrix, and then we're going to use our test labels and our predictions, mnb, tf, idf, which is our raw predictions.

52:50.800 --> 53:09.320
We're going to do the same thing for our classification report, so we'll see report, mnb, tf, idf, classification report, and in the same order, test labels and our predictions, mnb, tf, idf.

53:10.680 --> 53:17.520
So I'm going to let this sit for a second, give you a chance to copy all of that down.

53:19.580 --> 53:34.240
So what's really helpful here is seeing how at each of these tasks, 6a, 6b, 6c, and 6d, we're only going to need to change very few things.

53:34.240 --> 53:43.500
We're going to need to change, instead of tf, idf, we're going to change this to count to do our train and evaluate the multinomial Naive Bayesian with the count model.

53:43.720 --> 53:51.280
And then we're going to change this from the multinomial Naive Bayesian constructor to the logistic regression constructor.

53:51.400 --> 53:53.840
But other than that, the rest of our code is going to be identical.

53:53.840 --> 53:56.280
The rest of our preprocessing is going to be identical.

53:56.480 --> 54:05.620
And so this will allow us to train a total of four algorithms using two separate machine learning algorithms as well as two separate vectorizers.

54:06.340 --> 54:09.100
So this is going to be very helpful for us.

54:10.400 --> 54:14.040
So I'm going to go ahead and run this.

54:14.040 --> 54:16.260
So shift-enter to run.

54:16.260 --> 54:20.240
It looks like I misspelled something.

54:23.120 --> 54:24.560
Multinomial.

54:24.840 --> 54:26.660
There we go.

54:27.280 --> 54:29.440
Required argument y.

54:29.900 --> 54:31.400
That one's new.

54:32.980 --> 54:36.580
Give me one second.

54:36.900 --> 54:39.320
Oh, that is what I forgot.

54:39.360 --> 54:56.560
So we want to train our multinomial Naive Bayesian with our data as well as our labels so that it knows what belongs to spam, what belongs to ham, just kind of in our pen and paper example, what belonged to sports and what belonged to not sports.

54:57.020 --> 54:58.940
So now I can run that.

55:00.460 --> 55:04.360
And I see where we ran into that problem yesterday.

55:04.580 --> 55:05.260
Okay.

55:06.600 --> 55:08.740
I need to swap.

55:08.740 --> 55:11.800
This is the problem that I had with scikit-learn earlier.

55:13.140 --> 55:15.560
The test labels and the predictions.

55:15.560 --> 55:20.500
We start with the predictions and then we follow up with the test labels instead of the other way around.

55:21.240 --> 55:23.100
So I go ahead and run that.

55:23.800 --> 55:24.840
There we go.

55:24.840 --> 55:29.200
So we know that we had 590 ham emails.

55:29.200 --> 55:34.920
We know that we had, if we add these two numbers together, 276 spam emails.

55:34.920 --> 55:39.960
So we can see the actual number of ham emails, the actual number of spam emails in those columns.

55:40.800 --> 55:42.640
But let me scroll up real quick.

55:42.760 --> 55:45.660
So this is our classification report.

55:45.700 --> 55:51.500
Classification report we're not really going to use in this example, but it's good to know.

55:51.500 --> 55:56.800
There are other ways to measure how good a model is performing, beyond accuracy.

55:57.500 --> 56:03.780
Precision, recall, and F1 score are all used in their own ways.

56:03.820 --> 56:07.700
I'm not going to go into that, but it's definitely worth understanding.

56:08.040 --> 56:10.780
Instead, we're just going to look at the straight accuracy.

56:10.780 --> 56:19.860
For this multinomial with TF-IDF, we have a .875 or a 87.5% accuracy.

56:19.860 --> 56:28.920
If we look at our true positives and our true negatives, it correctly classified or predicted all ham emails as being part of the ham class.

56:28.960 --> 56:32.460
So these are all of our true positives for what is ham.

56:32.540 --> 56:40.280
And then our true negatives, what is spam, it only classified 168 out of the 276 that we have.

56:40.720 --> 56:46.660
The other half of the spam emails it also thought were ham or regular emails.

56:46.660 --> 56:54.920
So our predictor is letting some spam through, but the good news is it's not blocking any of our legitimate emails.

56:58.820 --> 57:02.660
Moving on to the next example, we're going to do the same thing.

57:02.660 --> 57:07.700
We're going to use multinomial naivete with count vectorizer.

57:07.700 --> 57:11.020
This is going to be the same, and then be count.

57:13.020 --> 57:27.020
I am actually a little lazy, so what I'm going to do is just copy this and I can show you all you need to do is just change a few things around and you have a working model.

57:27.020 --> 57:31.980
So instead of multinomial tfidf, I'm going to be using multinomial count.

57:31.980 --> 57:33.800
We're going to use the same constructor.

57:34.780 --> 57:41.580
So in this case it's called count, and instead of using the tfidfx, we're going to use the countx.

57:44.320 --> 57:56.680
score, count, mnb, count, and our test labels stay the same because we didn't vectorize those.

57:56.680 --> 57:58.920
There's no reason for us to change this.

58:00.120 --> 58:04.160
I'm just going to go ahead and ctrl-c and ctrl-v some stuff.

58:05.900 --> 58:11.100
So all I'm doing is just changing the word tfidf to the word count.

58:11.100 --> 58:14.300
Everything else is staying the same.

58:15.040 --> 58:22.840
We have our predictions, mnb, count, our test labels, that stays the same, we don't modify it.

58:22.900 --> 58:29.760
mnb, count, and then we change our predictions.

58:30.600 --> 58:33.680
So I'm going to go ahead and let that sit for a moment.

58:33.680 --> 58:42.700
I'll let you guys either manually type that in or do the copy-and-paste track that I did, which I highly recommend.

58:42.700 --> 58:44.980
It just saves quite a bit of typing.

58:46.280 --> 58:51.860
And then we'll type out the logistic regression in a moment because that's the next model that we'll be making.

58:54.780 --> 58:57.600
How are we doing on questions?

59:01.160 --> 59:03.520
I think we're all doing well.

59:04.300 --> 59:06.980
Everything's quiet on the Discord chat, too.

59:11.580 --> 59:12.380
Awesome.

59:12.380 --> 59:14.040
So I'm going to go ahead and run this.

59:14.040 --> 59:16.340
I'm going to do Shift-Enter.

59:17.640 --> 59:25.220
And if we look at our accuracy, our multinomial with the count vectorizer, we have a 94% accuracy.

59:25.380 --> 59:32.700
So quite a decent bump compared to our tfidf vectorizer using the same multinomial algorithm.

59:32.700 --> 59:36.760
What we see is about 26 of our emails...

59:36.760 --> 59:41.900
not about, it's exactly 26 of our emails in our test set got blocked.

59:41.900 --> 59:43.880
They were considered spam.

59:44.480 --> 59:46.940
These are the regular ham emails.

59:46.940 --> 59:53.900
But we have significantly fewer spam emails that are getting through our predictor.

59:53.900 --> 59:57.340
And so more of them are being classified as spam correctly.

59:57.340 --> 01:00:00.040
And so this is what we want to see is a dark blue over in this corner.

01:00:00.040 --> 01:00:09.080
And a darker blue or this lighter blue given the count over in the bottom right corner.

01:00:09.560 --> 01:00:15.840
So this kind of hops into a conversation about false positives and false negatives.

01:00:15.840 --> 01:00:19.000
Which would you like to prioritize?

01:00:19.980 --> 01:00:22.340
And so it really depends on the model.

01:00:22.340 --> 01:00:26.860
In security, one of the things that we're concerned about is impacting the users.

01:00:26.860 --> 01:00:38.660
So if you look at something like an antivirus, it looks for things that are probably talking out to the internet, trying to take over your webcam, uploading and downloading files.

01:00:38.720 --> 01:00:40.820
Well, what else does that?

01:00:41.400 --> 01:00:44.980
Chrome, Internet Explorer, Firefox.

01:00:45.760 --> 01:00:56.660
So when you're building an antivirus, it's better to have what they consider false negatives.

01:00:57.480 --> 01:01:03.180
Which is positive is it is malware and negative is it is not malware.

01:01:03.180 --> 01:01:04.200
It's benign.

01:01:04.220 --> 01:01:11.560
So it's better to allow some viruses to run instead of blocking legitimate applications.

01:01:11.560 --> 01:01:14.140
Because you don't want to impact the user experience.

01:01:14.140 --> 01:01:17.980
Because if you impact the user experience, it's just going to uninstall your antivirus software.

01:01:17.980 --> 01:01:23.320
And that leaves them unprotected and that's bad business for you when people are uninstalling your software.

01:01:24.960 --> 01:01:35.000
So even for us, we want to consider and weigh the factors of is it better for us to allow some spam to get through as long as we're not blocking any of the legitimate emails?

01:01:35.000 --> 01:01:42.480
Or do we want to make sure that absolutely no spam gets through and if we block legitimate emails, well that's just collateral damage.

01:01:42.560 --> 01:01:47.600
So these are considerations that you need to make when you're building models like these and as a data scientist.

01:01:49.180 --> 01:01:55.360
So for the multinomial naïve Bayesian with the count vectorizer, we have 94% accuracy.

01:01:55.740 --> 01:01:58.640
So let's try a different model.

01:01:58.640 --> 01:02:00.840
This one we're going to use the logistic regression.

01:02:01.120 --> 01:02:08.960
So if we look at the instructions, we're going to use the logistic regression constructor, but we're going to use this LBFGS solver.

01:02:08.960 --> 01:02:16.360
And this is one of the hyperparameters, or at least I would consider it a hyperparameter for logistic regression itself.

01:02:16.980 --> 01:02:22.020
There's a couple different solvers that you can use but in this case we're going to use the LBFGS.

01:02:22.360 --> 01:02:26.700
So what this is going to look like is LGS TF IDF equals

01:02:30.340 --> 01:02:53.640
logistic regression and then we're going to set our solver equals LBFGS and then of course, just like we did with the multinomial naïve Bayesian, we're going to fit our train, our classifier we're going to use the TF IDF X, which is our test data, and then we're going to use our training data,

01:02:53.640 --> 01:02:55.980
sorry, and then we're going to use our labels.

01:02:57.620 --> 01:03:05.260
Next we're going to hop in and do the score predictions classification matrix and, sorry, confusion matrix and classification reports.

01:03:05.260 --> 01:03:19.460
We're going to use score LGS TF IDF LGS TF IDF.score We're just going to look at our test TF IDF X and our test labels.

01:03:19.460 --> 01:03:44.360
We're going to have our predictions LGS TF IDF equals LGS TF IDF.predict That's going to give us our raw predictions and then I'm going to stop saying exactly what I'm typing but we're going to do our confusion matrix

01:03:59.660 --> 01:04:02.820
and our classification report

01:04:17.280 --> 01:04:18.540
Awesome.

01:04:18.540 --> 01:05:14.620
So I'm going to let this sit here for a little bit, let you guys kind of copy and follow along Hopefully we don't run into the same problems that we did last time Bender is fantastic I love it, but it will time out after about five minutes and once it times out you've got to kind of restart the whole thing or at least you've got to restart the notebook which is kind of annoying Another thing you will notice is the logistic regression will take a little longer to train and give us our predictions and it's just the way that logistic regression operates it does a little bit more mathematical or complicated mathematical formulas than the multinomial Naive Bayesian so it just takes a little bit longer to train but not too much longer,

01:05:14.620 --> 01:05:39.200
so that's good One of them is matrix multiplication which is a whole array of floats that you have to process versus Naive Bayes is literally counting and some logs when you're doing the math correctly which is very quick It's just counting Yeah, it's just counting and logs,

01:05:39.200 --> 01:06:15.100
the latest x86 stuff have built-in processes for doing integer logs and stuff like that because it's so common to use logs for numerical accuracy in machine learning You'll also notice the same thing with neural networks because neural networks do a lot of matrix multiplication which is why GPUs work really well for neural networks because you can parallelize matrix multiplications like that So I'm going to go ahead and run this Shift-Enter And it looks like it's running.

01:06:15.100 --> 01:06:17.220
Hopefully it doesn't take too long.

01:06:19.260 --> 01:06:23.920
Another comparison to make is a neural network versus a gradient-boosted tree.

01:06:23.920 --> 01:06:28.580
A gradient-boosted tree is all just if-then statements.

01:06:29.460 --> 01:06:34.000
Versus a neural network is a huge pain in the ass to multiply everything.

01:06:34.000 --> 01:06:43.060
So a GBT could be run in milliseconds versus a neural network taking seconds in 100% of your CPU.

01:06:45.000 --> 01:06:48.120
Actually, speaking of neural networks, I'm kind of curious about this.

01:06:48.120 --> 01:06:50.440
How well this would be received at DEF CON.

01:06:50.440 --> 01:06:51.940
Would anybody be interested?

01:06:51.940 --> 01:06:54.220
And please vote in the Discord channel.

01:06:54.240 --> 01:07:05.920
If you would be interested in learning how to build neural networks from scratch or how to build deep learning security tools I think both would be really interesting.

01:07:05.920 --> 01:07:09.120
Neural networks from scratch is actually a lot easier than you would imagine.

01:07:09.120 --> 01:07:11.480
You can do it in about 11 lines of Python code.

01:07:14.640 --> 01:07:33.520
Today we are able to simplify and easily explain the math whereas a lot of these algorithms were invented in the 50s where the people who would do this kind of work were called statisticians or people who would apply these in business scenarios were called business intelligence.

01:07:33.960 --> 01:07:39.560
But today with computer science and these abstraction libraries it's a lot easier to understand.

01:07:39.580 --> 01:07:47.200
So let me know in the Discord chat if there's any interest in either of those as potentially future workshops for future DEF CON events.

01:07:47.220 --> 01:07:49.280
Or even just AI Village events.

01:07:49.840 --> 01:07:53.020
So KDAMinMax has a question on Twitch chat.

01:07:53.020 --> 01:07:54.980
I'll just read it here.

01:07:54.980 --> 01:07:58.380
I have a possibly stupid question about input structures.

01:07:58.380 --> 01:08:01.860
If you're learning, there's no real stupid questions.

01:08:04.720 --> 01:08:08.860
An email predictor acts on strings or a set of strings.

01:08:08.860 --> 01:08:12.600
Are there any machine learning approaches that act reasonably well on trees?

01:08:12.640 --> 01:08:16.020
For example, abstract syntax tree representations of applications.

01:08:16.600 --> 01:08:28.340
For that, you might want to use a graph convolutional network or, honestly, you can interpret language as an abstract syntax tree.

01:08:28.340 --> 01:08:31.020
So you might want to reach in and use some of the...

01:08:31.020 --> 01:08:44.680
There's a huge backlog of literature that people largely ignore because of the success of deep learning of how to do language processing using abstract syntax trees.

01:08:44.900 --> 01:08:52.660
So you might want to reach into some old school machine learning from before 2010 when neural networks destroyed the world.

01:08:53.960 --> 01:08:56.760
I mean, built our jobs.

01:08:59.300 --> 01:09:00.500
That was a good answer.

01:09:00.500 --> 01:09:01.420
Thank you.

01:09:04.600 --> 01:09:09.660
So what we're looking at is logistic regression with TF-IDF vectorizer.

01:09:09.660 --> 01:09:11.540
We have a slightly better...

01:09:11.540 --> 01:09:16.020
It's a small bump, but it's slightly better than the multinomial with the count vectorizer.

01:09:16.020 --> 01:09:20.140
We have a 95.9% accuracy.

01:09:20.400 --> 01:09:27.280
We see that we have a little bit more than the 26 ham emails that were incorrectly classified as spam.

01:09:27.280 --> 01:09:31.900
We have significantly fewer of the spam emails being classified as ham.

01:09:32.100 --> 01:09:44.840
So this would be a perfect instance of moving towards having absolutely zero spam get through our filter at the risk of losing some of our regular emails.

01:09:46.020 --> 01:09:52.880
But what I want to show is how the accuracy improves just changing the algorithm itself.

01:09:52.880 --> 01:09:59.640
And then looking at multinomial Naive Bayesian compared to logistic regression both using TF-IDF.

01:09:59.640 --> 01:10:06.720
We had 87.5% for the multinomial and then we have 95.9% for the logistic regression.

01:10:06.720 --> 01:10:14.100
So far, logistic regression is kind of beating out multinomial in the TF-IDF and the count metrics.

01:10:14.100 --> 01:10:16.680
So we're going to do our last one which is going to be

01:10:19.980 --> 01:10:23.760
logistic regression with count vectorizer.

01:10:23.760 --> 01:10:28.860
Count vectorizer worked well for multinomial so let's see if it works well for logistic regression.

01:10:28.860 --> 01:10:30.960
So we're going to do LGS count.

01:10:36.550 --> 01:10:42.810
And I'm just going to type this out but you can use the same copy-paste trick that I used for the multinomial with the count vectorizer.

01:10:44.670 --> 01:10:52.130
I'm just doing this as a way to kind of slow myself down so that you all have an opportunity to copy the code.

01:10:52.350 --> 01:10:54.150
So LBFGS.

01:10:54.570 --> 01:10:58.190
We're going to fit our

01:11:02.540 --> 01:11:03.500
labels.

01:11:03.500 --> 01:11:08.940
And we got to do our scores, predictions, confusion matrix, and classification report.

01:11:18.730 --> 01:11:33.370
Yeah, one of the things that you kind of reminded me of, Sven, when you've been talking about the different trees and algorithms that have been overtaken by neural networks and deep neural networks is random forests.

01:11:33.370 --> 01:11:39.530
Random forests operate very well on a lot of different scenarios.

01:11:40.310 --> 01:11:46.810
But because random forests are a cluster of decision trees people go, oh, they're just if-then statements.

01:11:46.810 --> 01:11:47.370
This is dumb.

01:11:47.370 --> 01:11:48.870
This isn't machine learning.

01:11:49.670 --> 01:12:11.190
Yeah, but for us, we have a highly shared data where, for example, on the Ember malware dataset, the lightweight, incredibly cheap gradient decision tree built with LightGBM outperformed any neural network you can throw at it.

01:12:11.190 --> 01:12:21.870
So the LightGBM gets 99.8% accuracy or something like that, and the neural network gets 98.7% accuracy, which is significantly worse.

01:12:24.350 --> 01:12:43.930
For some tasks, a gradient-based decision tree or something like that is the correct thing mathematically because neural networks have biases because it's easier for them to learn some things than others and GBTs have biases where it's easier for them to learn some things than others.

01:12:43.930 --> 01:13:00.810
And one of the great secrets of machine learning is match up your problem with both the requirements, how powerful your predictor needs to be, and the inherent biases of those predictors.

01:13:02.130 --> 01:13:02.910
Yeah.

01:13:06.480 --> 01:13:08.120
Yeah, that's pretty interesting.

01:13:08.120 --> 01:13:09.300
I didn't know that.

01:13:10.700 --> 01:13:14.260
But that also reminds me of another thing with neural networks.

01:13:14.260 --> 01:13:16.720
Neural networks are really good at learning superstitions.

01:13:17.060 --> 01:13:26.700
I don't know if that's a machine learning as a whole thing or specifically in neural networks but I remember there was a dataset that they were using to identify skin cancer cells.

01:13:26.940 --> 01:13:34.640
They used a convolutional neural network and based off of that it was learning that if there was a... it was performing too well.

01:13:34.640 --> 01:13:47.640
It got nearly 100% and they kind of dug into what it was doing and it said, OK, well if I see a ruler in the picture which the scientists, how they were collecting the images, they had a ruler next to the skin cancer cells.

01:13:47.640 --> 01:13:51.080
If I see a ruler in the picture that's how I know it's cancer.

01:13:51.200 --> 01:13:53.480
Not the little blob next to it.

01:13:55.100 --> 01:14:03.140
So I don't know if that's a whole machine learning problem or specifically a neural net problem but it's just kind of an interesting anecdote.

01:14:04.060 --> 01:14:07.740
Be careful how you collect and preprocess and use your data.

01:14:09.060 --> 01:14:11.640
A friend of mine did his PhD.

01:14:11.640 --> 01:14:26.140
His PhD thesis was inducing the bias that's sort of like the inherent model bias that's in convolutional neural networks into other models and it slightly improved.

01:14:29.200 --> 01:14:31.620
By inducing a bias?

01:14:31.840 --> 01:14:58.500
Yeah, you can induce a pixel locality bias like a great green tree or a support vector machine and it improves that's one of the assumptions that people make about convolutional neural networks is it allows you to make small pixel local decisions and then those can get glued together in the right way.

01:15:00.160 --> 01:15:14.500
And so he went and built little modifications to a dozen other algorithms that have sort of fallen out of favor and all of them improved slightly on images when you induced a pixel local bias.

01:15:14.980 --> 01:15:17.160
Okay, interesting.

01:15:17.160 --> 01:15:24.900
Yeah , that's that's actually really interesting.

01:15:27.480 --> 01:15:33.840
Alright, so hopefully I gave you guys enough time to make sure that this got copied down.

01:15:33.840 --> 01:15:36.200
I'm going to go ahead and run it and shift enter.

01:15:36.360 --> 01:16:00.020
This is going to take a little bit because it's going through its matrix multiplications but in the meantime, we're about to hop into our final task and what we have is a real spam email I pulled this out of my inbox or my spam folder last May but we see your latest issue is available now if you don't want issue notifications,

01:16:00.020 --> 01:16:02.860
click here to unsubscribe Hi George.

01:16:02.900 --> 01:16:11.840
My name is not George but we're going to look at this email and we're going to try it against all four of our spam classifiers and see how well they do.

01:16:12.080 --> 01:16:15.560
But what's also good about this exercise is it doesn't really hold your hand.

01:16:15.560 --> 01:16:19.980
This is usually something that I give plenty of time in a classroom setting.

01:16:19.980 --> 01:16:23.220
This allows people to ask questions as I go through.

01:16:23.220 --> 01:16:32.180
But this last task is going to say OK, we have a new email we've never seen before run it against our classifiers that we trained.

01:16:33.180 --> 01:16:47.000
And so you have to go through the whole process of using the vectorizers to vectorize the tokens or the words in the email and then putting them through the predict function and then printing out the prediction that it gives.

01:16:47.960 --> 01:16:56.380
So before we hop into that, I really want this to finish I want it to just be on its own time but

01:16:59.600 --> 01:17:21.180
we're going to compare how well these models did once this last one is complete and we're going to see if or how much that comparison has an influence on our test email So we're just going to give this another Probably be another 30 seconds or so.

01:17:21.180 --> 01:17:22.860
So it's not too long.

01:17:22.860 --> 01:17:24.000
How's the Discord chat?

01:17:27.820 --> 01:17:29.460
Discord chat's doing well.

01:17:36.560 --> 01:17:40.940
I really appreciate how much discussion we have in this version of the workshop.

01:17:41.180 --> 01:17:45.000
Also, and this is something that I want to note for people who are seeing this for the first time.

01:17:45.000 --> 01:17:53.140
So the last time I did my workshop on Friday, unfortunately my webcam froze in a very unflattering position, so I just had my mouth gaping open.

01:17:54.540 --> 01:17:59.600
So for this one I just put a static image of myself today.

01:17:59.940 --> 01:18:04.940
That way you can still see my beautiful face, but you don't get a very unflattering image of it.

01:18:05.780 --> 01:18:08.320
Alright, so this one is complete.

01:18:08.320 --> 01:18:13.780
We are now at the best model that we have of the ones that we've trained.

01:18:13.780 --> 01:18:18.200
We have a 97.69% accuracy.

01:18:18.200 --> 01:18:21.100
We can round that up to a 97.7% accuracy.

01:18:21.100 --> 01:18:39.680
We see that only 8 out of our 582 ham emails were misclassified as spam, and then only 12 out of our 264 spam emails were misclassified as ham, making this the best model that we have out of the 4 that we've trained.

01:18:41.400 --> 01:18:43.980
So that's pretty solid.

01:18:44.960 --> 01:19:00.440
So moving forward, what I would do is take this as kind of the better of the models, and I would go ahead and deploy this in production, or add this model into an application that's designed to kind of wrap around the idea of spam and ham.

01:19:00.960 --> 01:19:25.540
So if you are simply an email provider, you can use this to filter email, or if you are a spam company, or not a spam company, but like a spam research company, you would use this to grab the spam messages and then collect that as data, and then you can use that to be training the next version of your spam filter.

01:19:27.680 --> 01:19:31.580
So let's take a look at our test spam email.

01:19:32.180 --> 01:19:42.360
We have our final task, which is to use the vectorizer and the models that we created to perform predictions on our test email and our test spam email.

01:19:42.360 --> 01:19:53.420
So to make things easier, what I'm going to do is use a variable, I'll call this working email, and this will allow us to switch pretty quickly.

01:19:54.180 --> 01:20:19.620
So I have test email, and then we're going to want to vectorize our email, so we're going to use our test email tfidf, yes, tvec.transform, and this is going to be, remember it needs to be in a list, so we've got to use these square brackets, our working email,

01:20:20.280 --> 01:20:27.660
and then we'll do the same thing with our test email count, tvec.transform,

01:20:30.760 --> 01:20:32.180
working email.

01:20:33.180 --> 01:20:36.520
Now we need to get the raw predictions for every single one of them.

01:20:36.520 --> 01:20:42.960
The raw predictions, we already have the variable names here, kind of at the bottom, so I'm just going to go ahead and write them out.

01:20:42.960 --> 01:21:06.800
So we have test email mnb with tfidf, which is going to be our mnbtfidf.predict, on our test email with the tfidf vectorized info, we're going to do the same thing for count, using our count.predict,

01:21:14.490 --> 01:21:23.850
with our count vectorized, test email lgs, tfidf.predict,

01:21:32.020 --> 01:21:38.480
tfidf.data, and our last one, test email lgs.count,

01:21:44.520 --> 01:21:45.560
count.

01:21:47.040 --> 01:21:52.800
And then, just for fun and games, we can print out our working email.

01:21:53.020 --> 01:21:55.240
So this is going to print our test email.

01:21:55.240 --> 01:21:59.940
Our test email was the one that we saw, EastAsianFonts and Lenny, thanks for your support.

01:22:02.500 --> 01:22:06.640
So you can go ahead and copy this down.

01:22:06.640 --> 01:22:11.780
I'm going to go ahead and run this, because we are running close to time.

01:22:11.780 --> 01:22:18.140
So this is going to be our test email, and ideally, this should say that it is a regular email, not spam.

01:22:18.140 --> 01:22:21.100
It's going to label all four of these as ham.

01:22:21.120 --> 01:22:22.620
So let's run it.

01:22:22.920 --> 01:22:29.540
Working is not defined, because I forgot to call this working email.

01:22:30.200 --> 01:22:31.560
So we're going to run it.

01:22:32.280 --> 01:22:33.720
Here's our email.

01:22:34.380 --> 01:22:39.500
And all four of our classifiers correctly classified this as ham.

01:22:40.360 --> 01:22:48.280
I'm going to scroll back up, and we saved our email as our test spam email.

01:22:48.280 --> 01:22:52.520
So I'm going to grab that, and I'll run it.

01:22:53.260 --> 01:22:58.200
Oh, I forgot to run the code block that actually defined our test spam email.

01:22:59.120 --> 01:23:00.960
And now we can run this.

01:23:01.560 --> 01:23:03.060
And we see something interesting here.

01:23:03.060 --> 01:23:14.000
Our worst performing model, our MNB TF-IDF, which was 87.5% accurate, misclassified this email as being legitimate, as a ham email.

01:23:14.240 --> 01:23:18.100
But the other three of ours classified this correctly as a spam email.

01:23:18.740 --> 01:23:20.700
So there's a couple things we can do with this.

01:23:20.700 --> 01:23:27.380
If we were so inclined, we can use what's called ensemble learning, which we use all four algorithms.

01:23:27.380 --> 01:23:34.820
We take the average of them, or in other words, they give a vote on what they think the class is.

01:23:34.820 --> 01:23:38.740
And then in this case, three out of the four say it's spam, so it would be spam.

01:23:38.740 --> 01:23:52.340
Or we can use this as an example and say, hey, the MNB TF-IDF, we probably shouldn't use that simply because it's misclassifying known spam emails at a high rate.

01:23:52.340 --> 01:24:03.760
87% means that we have about a 12.5% chance of misclassifying spam emails.

01:24:05.020 --> 01:24:08.960
So this is kind of pretty close to the conclusion of the workshop.

01:24:08.960 --> 01:24:17.520
This shows how you can take raw emails, raw data, transform it in a way that can be used in machine learning models.

01:24:17.720 --> 01:24:22.660
And then we build out our machine learning models, we evaluate them, and we can test them.

01:24:22.660 --> 01:24:26.220
And this last task shows you how we can properly deploy them.

01:24:26.220 --> 01:24:37.580
So we can build a piece of software or an application around it and say every time we get an email, just swap out this working email, and then you can leave everything else the same.

01:24:38.800 --> 01:24:41.900
Now, before I let you go, there's one more thing that I want to show you.

01:24:42.260 --> 01:24:44.440
I mentioned hyperparameter tuning.

01:24:44.440 --> 01:24:48.620
So let's take a look and see what hyperparameter tuning can do.

01:24:48.980 --> 01:24:55.620
So when we were talking about the multinomial Naive Bayesian, we saw that we had that smoothing function, the alpha, right?

01:24:56.000 --> 01:24:58.280
And it was normally 1.

01:24:58.580 --> 01:25:00.980
So we can add that in here.

01:25:00.980 --> 01:25:03.360
So we have alpha equals 1.

01:25:03.360 --> 01:25:07.900
We see that an 87.5% accuracy.

01:25:08.280 --> 01:25:09.720
We go ahead and run that.

01:25:09.720 --> 01:25:12.260
I'm just doing this to show you that there are no tricks up my sleeve.

01:25:12.260 --> 01:25:15.460
Alpha is by default 1 for this model.

01:25:16.240 --> 01:25:20.060
But what if we change that to maybe something a little smaller, right?

01:25:20.060 --> 01:25:21.480
Maybe a 0.1.

01:25:21.480 --> 01:25:24.200
Let's reduce the impact of that smoothing filter.

01:25:24.200 --> 01:25:28.200
So I'm going to go ahead and press Shift-Enter and run that.

01:25:28.860 --> 01:25:30.300
And we see something interesting.

01:25:30.300 --> 01:25:43.060
So this went from our worst model, 87.5% accuracy, to actually our second best model, 96.5% accuracy.

01:25:43.740 --> 01:25:50.720
And so it slightly misclassified some of the legitimate ham emails as spam.

01:25:50.720 --> 01:26:00.260
But it drastically reduced the, I think it was 108 spam emails being misclassified down to just 22.

01:26:01.740 --> 01:26:04.240
So this is the power of hyperparameter tuning.

01:26:04.240 --> 01:26:16.440
And if we go all the way back down to our final task, task 7, and we run this, where the multinomial manipulation with TF-IDF incorrectly classified that as ham, we go ahead and rerun it.

01:26:16.760 --> 01:26:20.500
Now it's correctly classifying it as spam because its accuracy is much better.

01:26:20.500 --> 01:26:23.840
And the only thing that we did to change it was that hyperparameter tuning.

01:26:26.920 --> 01:26:29.060
So that's all I have for you guys today.

01:26:29.060 --> 01:26:37.720
I have a couple good resources, but the biggest one that's going to be helpful is if you have any questions, feel free to reach out to me.

01:26:37.720 --> 01:26:39.540
I am gtklondike on Gmail.

01:26:39.540 --> 01:26:41.760
I'm also gtklondike on Twitter.

01:26:42.020 --> 01:26:47.540
I'm not as active on Twitter, but I will see your direct messages or any at mentions.

01:26:48.340 --> 01:26:57.000
And then the GitHub code for all of these workbooks, this and all the workbooks, are on github.com slash netsecexplained.

01:26:59.060 --> 01:27:01.540
So that is all I have for you guys.

01:27:01.540 --> 01:27:04.660
It looks like we just barely made it.

01:27:04.780 --> 01:27:05.740
1231.

01:27:06.960 --> 01:27:09.160
And I will let you go.

01:27:09.160 --> 01:27:16.140
I will stay in the chat channel for a little bit in case any of you would like to continue to discuss.

01:27:16.140 --> 01:27:20.100
And I'll also stay in the voice channel for a little bit for the same reason.

01:27:20.220 --> 01:27:22.480
So I will see you there.

01:27:22.720 --> 01:27:23.900
Have a good day.