1
00:00:03,800 --> 00:00:06,200
Are we ready to go?

2
00:00:07,320 --> 00:00:08,580
We are.

3
00:00:08,700 --> 00:00:09,740
Perfect.

4
00:00:10,340 --> 00:00:14,280
Hello and welcome to Machine Learning for Security Analysts.

5
00:00:14,280 --> 00:00:19,200
This is our first workshop here at the AI village, DEF CON Safe Mode.

6
00:00:19,620 --> 00:00:24,520
So the goal here is to kind of take machine learning out of the buzzword and into the mainstream.

7
00:00:25,200 --> 00:00:36,620
So buzzwords like machine learning and AI have been in the industry for a couple of years now, but for most people, it's not really well understood what these actually mean.

8
00:00:36,900 --> 00:00:52,340
So my goal is to kind of introduce you to machine learning, and we're going to do this by kind of going over what machine learning is, how it fits in the broader concept of AI, and why it's important for security analysts and hackers to understand machine learning,

9
00:00:52,340 --> 00:00:56,000
as well as what the machine learning process looks like.

10
00:00:56,000 --> 00:01:03,380
And then we're going to take all of that and put that into building our first machine learning model, which will be something very familiar.

11
00:01:03,820 --> 00:01:06,020
So let's go ahead and get started.

12
00:01:07,600 --> 00:01:08,700
So who am I?

13
00:01:08,700 --> 00:01:09,520
My name is Gavin.

14
00:01:09,520 --> 00:01:11,940
I go by the pseudonym GT Klondike.

15
00:01:11,940 --> 00:01:15,220
I'm an independent security researcher and a security consultant.

16
00:01:15,220 --> 00:01:19,980
I have a passion for network security, both attack and defense.

17
00:01:20,080 --> 00:01:33,020
And so through that passion, I run a project called NetSec Explained, which is a blog and a YouTube channel where I use intermediate and advanced-level concepts and try to explain them in a very easy-to-understand way.

18
00:01:35,260 --> 00:01:43,100
So it's usually at this point where, in person, I would kind of ask the audience, when you hear the word machine learning, what comes to mind?

19
00:01:43,260 --> 00:01:58,580
Some of the common answers that I get are, oh, you need to have a PhD in multivariable calculus and linear algebra, or Skynet's going to take over the world, or pattern matching.

20
00:01:59,980 --> 00:02:08,600
So these are all pretty interesting answers, but what we're going to walk through in this workshop is kind of show you that not all of that is necessary.

21
00:02:08,600 --> 00:02:19,520
You don't need a PhD in multivariable calculus or linear algebra, especially with some of the abstraction libraries like Scikit-learn, Keras, TensorFlow, and things like that.

22
00:02:19,520 --> 00:02:24,880
And then as far as machines taking over the world, not quite.

23
00:02:27,140 --> 00:02:32,300
So machine learning and how it fits in the broader idea of artificial intelligence.

24
00:02:32,300 --> 00:02:41,360
So artificial intelligence is this big umbrella term that kind of is a catch-all for different types of ways that machines operate.

25
00:02:41,360 --> 00:02:48,680
So back in the past, when they first invented digital calculators, they thought that was machine intelligence or artificial intelligence.

26
00:02:48,680 --> 00:02:51,400
And then it came to video game AI.

27
00:02:52,420 --> 00:02:58,560
Today we also see things like rules-based artificial intelligence, or classic image recognition.

28
00:02:58,560 --> 00:03:03,320
So there's a bunch of different artificial intelligence technologies that aren't necessarily machine learning.

29
00:03:03,560 --> 00:03:17,060
Machine learning itself is more about applied statistics, and a merge between what used to be considered business intelligence, and statisticians would do this job, and computer science.

30
00:03:17,060 --> 00:03:25,880
With the improvement of computing resources such as memory and CPU, a lot more power can be put behind some of these algorithms.

31
00:03:26,640 --> 00:03:30,260
And then deep learning is specifically talking about deep neural networks.

32
00:03:30,260 --> 00:03:38,140
Neural networks are a type of machine learning algorithm, and then deep learning allows us to stack layers on top of these.

33
00:03:38,520 --> 00:03:45,900
So deep learning, usually these are referred to for convolutional neural networks, which are used for image recognition, primarily.

34
00:03:45,900 --> 00:03:51,940
And then recurrent neural networks, which are used for cyclical pattern recognition, such as heartbeat monitors.

35
00:03:54,060 --> 00:04:05,900
But if we were to find all of this, I would say machine learning is a set of statistical techniques that enables the process of information mining, pattern discovery, and drawing inferences from data.

36
00:04:05,900 --> 00:04:15,300
The idea is that machines learn from the data, instead of us having to program signatures, or heuristics, or invent new algorithms for different problems.

37
00:04:15,860 --> 00:04:29,400
For example, deep learning, we have convolutional neural networks where deciding between whether something is a bike or a bus is the exact same algorithm and exact same problem as deciding whether or not something is a tiger or a cheetah.

38
00:04:31,300 --> 00:04:37,920
So a couple examples, especially in the security space, we have the ability to classify domain generation algorithms.

39
00:04:37,920 --> 00:04:46,540
These are primarily used by botnets as a way for the bot herder to stay in contact with their networks.

40
00:04:47,360 --> 00:05:00,680
Web application firewalls, these we will usually see with being able to perform some anomaly detection and see, okay, this is what normal user traffic looks like compared to abnormal user traffic.

41
00:05:00,980 --> 00:05:11,460
So in this case, the machine learning algorithm identified a potential SQL injection in the parameter because it's very abnormal from what it's used to seeing.

42
00:05:12,400 --> 00:05:14,240
And then, of course, network anomaly detection.

43
00:05:14,240 --> 00:05:25,860
This is one of the big ones, especially in the security side, the defender side of things, where we can predict what we think the network traffic will look like over a period of time and then compare that to what the actual is.

44
00:05:25,860 --> 00:05:35,340
Once it's beyond a certain delta, then that's something that we should pass off to a human to kind of dig in a little deeper and kind of figure out what's going on.

45
00:05:36,920 --> 00:05:38,840
So why is this important?

46
00:05:39,100 --> 00:05:45,260
Well, today we have over a quarter of security products that are used for detection that use some form of machine learning.

47
00:05:45,400 --> 00:05:49,120
And we see machine learning being applied to more and more things every day.

48
00:05:49,120 --> 00:05:57,360
We see facial recognition, we see advanced spam filtering, data heuristics and persona collection.

49
00:05:58,100 --> 00:06:11,260
And so in order for security analysts to properly deploy and manage these machine learning products, they need to have a better understanding of how they operate so that they can ensure that they work efficiently.

50
00:06:15,060 --> 00:06:18,040
So the machine learning process, it follows seven steps.

51
00:06:18,240 --> 00:06:22,340
This is not a step one, step two, step three, but this is really a cycle.

52
00:06:22,460 --> 00:06:27,460
So you start at step one, you go all the way down to step seven, and then you hop back up to step one.

53
00:06:27,520 --> 00:06:30,360
So you start off with gathering the data.

54
00:06:30,360 --> 00:06:36,520
So first you want to identify if I'm going to build a network anomaly detection system, if I'm going to build some sort of hacking tool.

55
00:06:36,860 --> 00:06:40,800
I need to gather data that applies to that, and then I need to prepare the data.

56
00:06:40,800 --> 00:06:47,740
So I can't just take raw information, I need to look at things like metadata and pre-process that.

57
00:06:48,220 --> 00:06:51,220
This step is also known as pre-processing.

58
00:06:51,840 --> 00:06:56,080
We need to choose the model, and the model is going to depend...

59
00:06:56,080 --> 00:06:57,160
There are some heuristics.

60
00:06:57,160 --> 00:07:05,180
Some models work better for different problem sets, but generally we're going to try a couple different models and see what works best.

61
00:07:05,640 --> 00:07:16,580
We're going to train the model, we're going to evaluate the model, and then if we think that the model can perform a little bit better than it's actually performing right now, we can do what's called hyperparameter tuning.

62
00:07:16,680 --> 00:07:20,100
And so this lets us make little tweaks here and there.

63
00:07:20,360 --> 00:07:22,220
And then of course we deploy it.

64
00:07:22,220 --> 00:07:30,820
And as I said earlier, this is a bit of a cyclical process, because machine learning algorithms stop learning as soon as you stop training them.

65
00:07:30,820 --> 00:07:38,960
And so you want to continue to gather new data and train new models and build them into newer and better generations.

66
00:07:42,020 --> 00:07:48,620
So before we get started, what we're going to do is hop in and learn how to build a basic spam filter.

67
00:07:48,620 --> 00:07:54,420
Spam filters are fantastic first examples, especially for a lot of the newbies, because we already know how spam filters work.

68
00:07:54,620 --> 00:08:02,540
If it says something that looks spammy, like buy Viagra now, then that's going to be automatically thrown away into our spam filter.

69
00:08:02,720 --> 00:08:07,660
If we have something that seems legitimate, then that's going to be passed on to the user.

70
00:08:08,320 --> 00:08:13,740
So before we get into that, are there any questions on any of the stuff that I've gone over so far?

71
00:08:22,970 --> 00:08:23,850
Okay.

72
00:08:26,810 --> 00:08:28,030
One more thing.

73
00:08:28,190 --> 00:08:32,330
I totally forgot this, but this is a bit of a story that really...

74
00:08:32,330 --> 00:08:33,310
Yes.

75
00:08:36,290 --> 00:08:37,170
Okay.

76
00:08:37,250 --> 00:08:39,870
This is a story that really made things click for me.

77
00:08:40,490 --> 00:08:46,010
So a friend of mine, this was a few years ago, was approached by a non-profit organization.

78
00:08:46,010 --> 00:08:53,830
And what this non-profit does is they have cameras set up all over Africa and they take pictures to see if there's a cheetah.

79
00:08:54,170 --> 00:08:55,950
And they're tracking cheetah population.

80
00:08:55,990 --> 00:09:03,170
So they'll take pictures, it'll get sent off to a human analyst, and the human will look at it and say, okay, is there a cheetah in the picture or not?

81
00:09:03,810 --> 00:09:08,170
And so they approached my friend and they said, hey, is there a way that we can improve this process?

82
00:09:08,170 --> 00:09:09,690
What's something that we can do here?

83
00:09:09,690 --> 00:09:13,630
And so he said, okay, let me try a couple things.

84
00:09:13,910 --> 00:09:26,050
So what he did was use a very common image library and said, okay, let's train a machine learning algorithm to identify images, if there's a cheetah in the picture or not.

85
00:09:26,450 --> 00:09:35,550
If it reached above a 5% confidence, says, okay, I'm 5% confident or 6% confident that there's a cheetah in this image, it got sent over to an analyst.

86
00:09:35,810 --> 00:09:39,650
If it was less than 5% confident, it would just throw it away.

87
00:09:40,630 --> 00:09:45,210
Now, 5%, that doesn't really sound like a lot, and it's not.

88
00:09:46,350 --> 00:09:51,790
But the ROI of what this model was able to do is pretty astounding.

89
00:09:52,090 --> 00:09:58,270
So what normally would have taken them a year of labor to perform, they were able to do in one month.

90
00:09:58,450 --> 00:10:01,750
That is a 1,200% increase in productivity.

91
00:10:02,750 --> 00:10:04,190
So that's pretty powerful.

92
00:10:04,190 --> 00:10:05,970
And that's just a very simple model.

93
00:10:05,970 --> 00:10:14,630
So as we can develop more and more advanced things, these are ways that we can really improve our processes to move at machine speed and scale.

94
00:10:16,730 --> 00:10:17,670
GT?

95
00:10:17,670 --> 00:10:18,470
Yeah.

96
00:10:18,470 --> 00:10:29,250
We have a question, and it's, any links to information pertaining to creating your own models rather than training existing ones?

97
00:10:30,670 --> 00:10:34,590
Creating your own models rather than training existing ones?

98
00:10:34,890 --> 00:10:35,310
Yeah.

99
00:10:35,310 --> 00:10:38,550
I think that you're going to be showing them how to create their own model.

100
00:10:38,550 --> 00:10:40,370
We are going to create our own model.

101
00:10:41,610 --> 00:10:45,610
So there are existing models that you can use, what's called transfer learning.

102
00:10:45,610 --> 00:10:50,170
So you take a problem, for example, ImageNet, which is a challenge.

103
00:10:50,170 --> 00:10:59,630
It can recognize hundreds of different categories of images, and you can take that idea, train your own model so that it can recognize whether or not there's a cheetah in an image.

104
00:11:00,270 --> 00:11:03,030
But what we're going to do is actually build a model from scratch.

105
00:11:03,030 --> 00:11:10,630
There's no previously trained information, and I'm going to walk you through how to do that, and we're going to see a workbook in a little bit.

106
00:11:11,650 --> 00:11:12,330
Thank you.

107
00:11:12,330 --> 00:11:14,050
Hopefully that helps answer the question.

108
00:11:16,750 --> 00:11:17,670
Okay.

109
00:11:18,170 --> 00:11:26,790
So before we get started, machine learning, it uses statistics, and so what we need to do is grab information and turn them into numbers.

110
00:11:27,090 --> 00:11:38,410
But if we wanted to have the idea of a spam email versus a regular email or a ham email, we need to kind of understand how we're going to interpret words into numbers.

111
00:11:39,270 --> 00:11:46,690
So here's a couple sentences that refer to sports and not sports, or realistically, elections.

112
00:11:46,690 --> 00:11:51,690
So we see a great game, and that talks about sports.

113
00:11:51,950 --> 00:11:55,150
And then we see the election was over, and that talks about not sports.

114
00:11:55,790 --> 00:12:01,130
So the idea is how do we take a sentence that we haven't seen before, because that's what machine learning is supposed to do.

115
00:12:01,130 --> 00:12:06,690
We were trained on certain information, but we want to predict on information that we haven't seen before.

116
00:12:07,010 --> 00:12:13,230
How do we take a new sentence and then try and make a decision whether or not it's talking about sports or not sports?

117
00:12:14,950 --> 00:12:18,350
So to do that, we're going to use what's called Bayes' theorem.

118
00:12:18,390 --> 00:12:42,130
Bayes' theorem is a very common probability theorem that basically says, okay, so if we have a sentence that we already know, and given the probability that this sentence is talking about sports, versus here's a new sentence, what is the probability that it's talking about sports given that new sentence?

119
00:12:42,590 --> 00:12:47,990
So the probability of A given B equals probability of B given A, blah, blah, blah.

120
00:12:47,990 --> 00:12:54,250
We don't need to know too much about this, but it's important to get a good foundation of this understanding.

121
00:12:54,250 --> 00:13:00,230
So the idea is, what is the probability that the sentence, a very close game, is talking about sports?

122
00:13:01,050 --> 00:13:02,430
We don't know the answer to that.

123
00:13:02,430 --> 00:13:10,090
We've never come across the sentence of a very close game, but we would need to fill it into this algorithm here.

124
00:13:10,930 --> 00:13:19,050
So instead, what we can do is actually break up a very close game into their individual words, a, very, close, and game.

125
00:13:19,110 --> 00:13:26,510
And then we can turn that probability into, what's the probability that A is talking about sports?

126
00:13:26,510 --> 00:13:28,990
What's the probability that very is talking about sports?

127
00:13:28,990 --> 00:13:33,810
What's the probability of close talking about sports, and game talking about sports?

128
00:13:35,970 --> 00:13:39,030
So this winds up becoming a bit of a counting game.

129
00:13:39,230 --> 00:13:42,630
How many times do we see A, and it's talking about sports?

130
00:13:42,630 --> 00:13:43,550
Twice.

131
00:13:43,910 --> 00:13:52,790
And of course we need to do the same thing for not sports, because we need to find the probability of A happening, the probability of B happening, and we take whichever is higher.

132
00:13:52,910 --> 00:14:02,250
So the probability that A is talking about sports, we see it twice in two of our sentences, or twice across all of our sports sentences.

133
00:14:02,250 --> 00:14:03,470
Got to be precise.

134
00:14:03,770 --> 00:14:10,110
We don't see A in not sports at all, so that's going to be zero across all of our not sports sentences.

135
00:14:10,870 --> 00:14:15,150
Very shows up in sports once, does not show up in not sports.

136
00:14:15,150 --> 00:14:25,810
Close shows up in not sports once, does not show up in sports, and then game shows up in sports twice, and it doesn't show up in not sports.

137
00:14:27,110 --> 00:14:29,170
So we take those numbers and we plug them in.

138
00:14:29,170 --> 00:14:36,350
We saw for sports, A showed up twice, Very showed up once, Close didn't show up at all, and Game showed up twice.

139
00:14:36,450 --> 00:14:41,890
And then if we count the number of words that we have across all of the sports sentences, we have a total of 11.

140
00:14:41,890 --> 00:14:46,870
So it just becomes a simple 2 divided by 11, 1 divided by 11, and so on.

141
00:14:47,550 --> 00:14:50,370
But we can't use straight, naive Bayesian.

142
00:14:50,430 --> 00:14:53,530
And the reason why is because we have these zeros.

143
00:14:53,910 --> 00:15:01,350
Now, close is a word that we've never seen in sports, so we get zero, and that winds up being zero divided by 11, which is zero.

144
00:15:01,350 --> 00:15:12,950
Zero times everything is zero, and so unfortunately the probability of this sentence being about sports, if we just use straight Bayesian, is zero.

145
00:15:13,530 --> 00:15:15,550
So that doesn't really work for us.

146
00:15:16,590 --> 00:15:20,230
So instead, what we're going to use is called multinomial naive Bayesian.

147
00:15:20,230 --> 00:15:25,050
It's a very similar formula, but we have what's called a smoothing filter, and that's this alpha.

148
00:15:25,050 --> 00:15:27,070
We can set this alpha to whatever we want.

149
00:15:27,070 --> 00:15:28,650
Usually we just set it to 1.

150
00:15:29,830 --> 00:15:36,430
But we can modify that, and this will be a hyperparameter that we'll take a look at later in our workbook.

151
00:15:38,590 --> 00:15:42,250
So with the smoothing filter, we do the exact same word counting.

152
00:15:42,250 --> 00:15:43,750
We see A showed up twice in sports.

153
00:15:43,750 --> 00:15:45,070
Very showed up once in sports.

154
00:15:45,070 --> 00:15:46,210
Close, zero.

155
00:15:46,650 --> 00:15:48,610
Game showed up twice in sports.

156
00:15:48,650 --> 00:15:50,590
And then we do the same thing for not sports.

157
00:15:50,810 --> 00:15:53,530
But you see here where it says close for sports?

158
00:15:53,830 --> 00:15:58,710
That smoothing filter makes it so that that number never becomes zero.

159
00:15:58,810 --> 00:16:00,810
It's always greater than zero.

160
00:16:00,810 --> 00:16:03,090
So our probabilities will always fit.

161
00:16:03,530 --> 00:16:06,450
And then this is just simple addition and then division.

162
00:16:07,450 --> 00:16:18,810
And if we took out a calculator, did all our math, we can see that AVeryCloseGame has a probability of .0000461 that it's talking about sports.

163
00:16:18,810 --> 00:16:25,790
And then AVeryCloseGame has a probability of 0000143 that it's talking about not sports.

164
00:16:25,970 --> 00:16:35,970
So we take whichever is higher, in this case sports, and so our classifier has decided that AVeryCloseGame is talking about sports.

165
00:16:39,430 --> 00:16:41,670
So I know that was a lot of information.

166
00:16:41,930 --> 00:16:43,690
Let me know if there's any questions.

167
00:16:44,090 --> 00:16:51,130
I'm going to kind of hop forward a little bit, but TAs, feel free to chime in if there's any questions on this.

168
00:16:51,170 --> 00:16:56,110
I know this bit's a little confusing sometimes, but it's not something we need to harp on too much.

169
00:16:56,110 --> 00:16:58,930
It's just an underlying foundation that's good to know.

170
00:17:01,450 --> 00:17:19,870
So the five things that we need to keep track of in order to do our pen and pencil math that we did earlier, is we need to keep track of the total number of unique words, the total number of words in spam, the total number of words in ham, and these two were what we've put in the denominator.

171
00:17:20,450 --> 00:17:24,490
The count of each word in spam and the count of each word in ham.

172
00:17:24,490 --> 00:17:30,310
So this is where we had to count the word A, count the word very, count close, count game.

173
00:17:31,290 --> 00:17:34,090
And so these are the five things that we need to pay attention to.

174
00:17:35,290 --> 00:17:37,610
Are there any questions so far?

175
00:17:41,050 --> 00:17:43,730
It looks like we're clear on questions right now.

176
00:17:43,770 --> 00:17:44,650
Awesome.

177
00:17:45,770 --> 00:17:50,490
Okay, so how do we take this idea and expand it to a full email?

178
00:17:51,810 --> 00:17:54,210
Well, here's a spam email.

179
00:17:55,110 --> 00:17:56,910
Yeah, no, this is not a spam email.

180
00:17:56,910 --> 00:17:58,550
This is a regular email.

181
00:17:58,550 --> 00:18:02,310
And we're going to use this as a common example throughout the rest of our workbook.

182
00:18:02,650 --> 00:18:07,390
So in this example email, we have re, re, East Asian fonts in Lenny.

183
00:18:07,390 --> 00:18:08,410
Thanks for your support.

184
00:18:08,410 --> 00:18:10,350
Installing Unifonts did it well for me.

185
00:18:11,870 --> 00:18:24,210
Now, we could take this entire email and look at it, but there are some things that we want to probably remove, things that don't really make a lot of sense, especially in the English language.

186
00:18:24,350 --> 00:18:26,950
So punctuation, punctuation doesn't really make sense.

187
00:18:26,950 --> 00:18:29,610
It doesn't add any context to a sentence.

188
00:18:30,950 --> 00:18:39,870
Small words, which are called stop words, such as in, of, on, for, the, they also don't add any context.

189
00:18:39,870 --> 00:18:46,970
So we can just go ahead and remove that, and that'll reduce the amount of words that we're looking at in this email, because we want to look at keywords.

190
00:18:48,810 --> 00:18:54,990
Things like unsubscribe and unsubscribed, or thanks, or thank you.

191
00:18:54,990 --> 00:18:57,310
They mean the same thing.

192
00:18:57,390 --> 00:19:02,090
A really good example, especially in English, is congrats and congratulations.

193
00:19:02,470 --> 00:19:04,050
They mean the same thing.

194
00:19:04,050 --> 00:19:08,350
And if we want to narrow this down to keywords, we should do what's called stemming.

195
00:19:08,350 --> 00:19:23,290
And so what stemming does is it shortens the word down so that you have I thanked somebody today, as in past tense, thank-ed, thanks, thank with an S, and thank you, where it's just plain thank.

196
00:19:23,290 --> 00:19:25,710
We want to count those all as the same word.

197
00:19:27,630 --> 00:19:38,450
And then, of course, we want to set everything to lowercase, because capital words such as unsubscribe, all caps, is the exact same as unsubscribe, lowercase.

198
00:19:38,710 --> 00:19:41,370
So this is our preprocessing step.

199
00:19:41,370 --> 00:19:43,370
And what we did was we removed all the stop words.

200
00:19:43,370 --> 00:19:47,070
We removed re, in, for, your, did, it, for.

201
00:19:48,130 --> 00:19:50,350
And then we're going to stem the words.

202
00:19:50,350 --> 00:19:56,490
I didn't show the stemming in here, and we'll see in the workbook as we go along what the stemming actually looks like.

203
00:19:56,770 --> 00:20:05,270
But we can see that we kind of shrunk the email a little bit, and this is going to be a little easier for us to compare emails to each other.

204
00:20:06,430 --> 00:20:12,730
So I think this is the point where we can go ahead and open up our workbook.

205
00:20:13,430 --> 00:20:17,110
So I'm going to escape out of there, open this up.

206
00:20:17,410 --> 00:20:20,870
So for our workshops, we're going to do this for this workshop and for the other workshops.

207
00:20:20,870 --> 00:20:22,490
We're going to use a system called MyBender.

208
00:20:22,490 --> 00:20:24,310
MyBender is fantastic.

209
00:20:24,310 --> 00:20:27,870
It uses Jupyter notebooks in a Docker container.

210
00:20:27,870 --> 00:20:36,590
So I'm going to go ahead and copy this and send it off into Discord somehow.

211
00:20:37,810 --> 00:20:41,790
Actually, is there a way...

212
00:20:41,790 --> 00:20:44,870
I'm going to kind of hide my screen for a second.

213
00:20:45,730 --> 00:20:50,530
Yeah, just post it to me, and I'll put it in all the places for you.

214
00:20:51,950 --> 00:20:53,590
Yeah, absolutely.

215
00:20:55,690 --> 00:20:56,570
Okay.

216
00:20:56,830 --> 00:20:58,290
All right, perfect.

217
00:20:58,290 --> 00:20:59,410
Got it.

218
00:21:00,150 --> 00:21:01,450
Display capture.

219
00:21:01,490 --> 00:21:03,870
I'm still getting used to OBS and Twitch streaming.

220
00:21:03,870 --> 00:21:06,590
I haven't done this before, but I'm really excited.

221
00:21:07,990 --> 00:21:16,810
So MyBender, once you get the link, you can just grab that link, paste it in here, or you can click launch, and you can do this with any GitHub repository.

222
00:21:17,210 --> 00:21:20,850
So this is going to launch some Jupyter notebooks.

223
00:21:20,850 --> 00:21:25,150
It takes about a minute or two to spin up.

224
00:21:25,430 --> 00:21:34,030
So I'm going to let you guys get that link from the TA, from the workshops channel, and then I'll let this spin up for me.

225
00:21:34,030 --> 00:21:38,710
I'll wait a little bit longer so that it spins up for you, and then we can get moving.

226
00:21:40,130 --> 00:21:42,570
While this is happening, are there any questions?

227
00:21:42,570 --> 00:21:46,270
Feel free to just write them in the channel, and the TAs will relay them.

228
00:22:02,570 --> 00:22:08,810
And once yours spins up the way that I have this laid out, go into the workbooks directory.

229
00:22:09,070 --> 00:22:18,550
If you would like to follow along in the live coding session, we're going to use the workbook spam filter sklearn document.

230
00:22:18,550 --> 00:22:26,310
If you are not that familiar with Python, and you just want to follow along, you can go ahead and use the completed workbook.

231
00:22:27,870 --> 00:22:32,830
I would give it just another couple of minutes to make sure theirs are spun up.

232
00:22:32,830 --> 00:22:33,950
Yes, absolutely.

233
00:22:33,950 --> 00:22:34,690
Absolutely.

234
00:22:36,050 --> 00:22:39,170
So I just want to check something real quick on my other screen.

235
00:23:06,800 --> 00:23:11,480
And these workbooks outside of this workshop, you can play with them on your own.

236
00:23:11,680 --> 00:23:15,300
I have a couple others if you want to take a look.

237
00:23:16,160 --> 00:23:23,600
But yeah, for the purpose of this workshop, we're going to be using the spam filter sklearn, or the completed spam filter sklearn.

238
00:23:32,700 --> 00:23:33,620
Okay.

239
00:23:35,280 --> 00:23:38,120
So I'm going to go ahead and start.

240
00:23:39,020 --> 00:23:43,660
The first couple things that I'm going to go over are just next, next, next run.

241
00:23:43,880 --> 00:23:47,560
If you're still waiting for your instance to spin up, don't feel bad.

242
00:23:47,560 --> 00:23:49,400
You'll be able to catch up very quickly.

243
00:23:51,080 --> 00:23:54,600
So I'm going to go ahead and open up my spam filter sklearn.

244
00:23:54,600 --> 00:24:00,100
And in this one, we are going to use scikit-learn, which is a wonderful abstraction library.

245
00:24:00,240 --> 00:24:01,660
Great for beginners.

246
00:24:03,500 --> 00:24:07,740
So we're going to go ahead and install a couple tools.

247
00:24:07,900 --> 00:24:17,980
So for these blocks, in order to run them, you can click on them and press shift enter, or you can press run button up here.

248
00:24:17,980 --> 00:24:20,260
You can see that it's running all the Python code.

249
00:24:20,260 --> 00:24:30,140
In this one, I'm just installing the libraries and I am configuring the data directory where all of our spam and our ham emails will be located.

250
00:24:31,680 --> 00:24:36,900
So that's going to take a little bit while that is going.

251
00:24:37,820 --> 00:24:41,380
Just kind of a quick overview of this first block.

252
00:24:41,380 --> 00:24:42,700
We have our imports.

253
00:24:42,700 --> 00:24:44,580
So in this case, we're going to use matplotlib.

254
00:24:45,180 --> 00:24:48,820
Matplotlib is going to let us print some nice graphs.

255
00:24:48,820 --> 00:24:53,020
NumPy and Pandas is going to let us grab some of the information that we need.

256
00:24:53,380 --> 00:24:59,660
We're going to use a couple of the others as some support libraries.

257
00:24:59,660 --> 00:25:04,580
For example, OS, we're going to use that later to kind of grab all of the files.

258
00:25:05,420 --> 00:25:07,480
We want to use NLTK.

259
00:25:07,480 --> 00:25:10,580
NLTK stands for Natural Language Toolkit.

260
00:25:10,820 --> 00:25:16,960
And so this will allow us to identify our stop words and perform our stemming functions.

261
00:25:18,100 --> 00:25:21,820
From scikit-learn, we're going to use a train-test split.

262
00:25:21,820 --> 00:25:35,840
The way that you typically want to train a machine learning model is you have a bunch of clean labeled data and you're going to take probably about 80 or 70% of that and use it to train the model.

263
00:25:35,840 --> 00:25:37,700
But then you want to see how well the model is doing.

264
00:25:37,700 --> 00:25:43,060
So you're going to use the other 20 or 30% as your testing and you're going to use this as evaluation.

265
00:25:43,440 --> 00:25:45,960
It's kind of the same way we teach students in elementary school.

266
00:25:45,960 --> 00:25:50,840
Here's the bulk of the information and now we're going to test you to see how well you understand it.

267
00:25:52,620 --> 00:25:57,840
And then our TF-IDF and count vectorizers, we will get into that in a bit.

268
00:25:57,840 --> 00:26:03,880
These are going to be used for transforming our words into numbers that we can work with.

269
00:26:04,960 --> 00:26:15,980
And then the two models that we're actually going to build out are going to be the multinomial Naive Bayesian, which we went over in the slides, and then one that is completely unfamiliar, logistic regression.

270
00:26:16,020 --> 00:26:23,720
And this is to show how incredible machine learning can be where you can just swap out algorithms.

271
00:26:23,780 --> 00:26:33,180
So we can use SVM, random forest, neural networks, things of that nature without massively changing the way that our data is represented.

272
00:26:33,980 --> 00:26:49,880
And then to kind of represent how our models are doing, we're going to use Seaborn as a graphic representation, and we're going to use a confusion matrix and a classification report to identify exactly where our model is doing well, where it's not doing well,

273
00:26:49,880 --> 00:26:53,920
and how mild changes are going to affect it.

274
00:26:54,580 --> 00:26:57,540
So I'm going to go ahead, shift-enter, run that.

275
00:26:58,160 --> 00:27:02,040
And that's just going to do the import, and we'll see libraries imported.

276
00:27:02,440 --> 00:27:04,660
And then here's our test email from the slides.

277
00:27:04,660 --> 00:27:07,100
This is the East Asian fonts in Lenny.

278
00:27:07,100 --> 00:27:14,440
We're going to see every step of the way how this email is manipulated and transformed into workable data for our machine learning models.

279
00:27:14,660 --> 00:27:18,080
So I'm going to hit shift-enter and run that.

280
00:27:20,840 --> 00:27:22,840
TAs, are there any questions so far?

281
00:27:22,840 --> 00:27:25,740
Is everybody able to get their stuff set up?

282
00:27:26,660 --> 00:27:28,060
No questions so far.

283
00:27:28,060 --> 00:27:29,660
Looks like everything is going well.

284
00:27:29,660 --> 00:27:30,540
Perfect.

285
00:27:31,720 --> 00:27:39,260
So the tokenizer, this is the first step where we want to define a function that will pre-process our information.

286
00:27:39,740 --> 00:27:46,300
So, as we saw in the slides, we want to make everything lowercase, we want to remove the stop words, and we want to stem the words.

287
00:27:46,880 --> 00:27:59,280
So this function is already defined for you, but we can see that we grab all the punctuation, we grab all the stop words using nl2k, and then we define the porter stemmer, which is going to stem our words.

288
00:27:59,780 --> 00:28:19,700
And then we're going to create a list of tokens by taking all of our words, setting them to lowercase, and using the word tokenizer, which will just take a sentence, split it up by its individual words, we're going to strip out the punctuations, and then we are going to merge everything together.

289
00:28:20,480 --> 00:28:22,360
So we're going to go ahead and run that.

290
00:28:22,360 --> 00:28:28,360
It's going to define our function, and we're going to hop into our first task.

291
00:28:28,560 --> 00:28:30,180
So our first task is really easy.

292
00:28:30,180 --> 00:28:37,680
It's just print the full test email, and then we're going to print the tokenized version of the test email.

293
00:28:37,820 --> 00:28:49,880
So in Python we can just do print test email, and then we can do print tokenizer test email.

294
00:28:50,160 --> 00:28:53,040
And then shift-enter, going to go ahead and run that.

295
00:28:53,040 --> 00:28:56,220
So we see the original email, East Asian fonts in Leni.

296
00:28:56,380 --> 00:28:58,340
Then we can see the stop words have been removed.

297
00:28:58,340 --> 00:29:01,660
So we're looking at East Asian font Leni.

298
00:29:01,800 --> 00:29:04,800
Thanks has been turned into thank.

299
00:29:05,240 --> 00:29:06,780
Support install.

300
00:29:06,780 --> 00:29:09,980
Install is missing an L because we see installing.

301
00:29:09,980 --> 00:29:14,440
You have I installed something, or I am going to install something.

302
00:29:14,440 --> 00:29:18,460
And so we're going to stem that so that all of those words are counted the same.

303
00:29:20,060 --> 00:29:21,960
We see unsubscribe.

304
00:29:21,960 --> 00:29:24,540
Unsubscribe shows up twice.

305
00:29:24,540 --> 00:29:30,700
And then we see list.debian.org, which also shows up twice.

306
00:29:30,700 --> 00:29:34,960
So this is the way that our tokenizer modifies the email as we're going through.

307
00:29:36,420 --> 00:29:44,840
And again, just for those that are trying to catch up, all we did was print the test email and then print the tokenizer version of the test email.

308
00:29:46,040 --> 00:29:50,520
There's a question from Twitch chat.

309
00:29:55,220 --> 00:30:01,520
So why are we assuming the punctuation is not relevant?

310
00:30:01,800 --> 00:30:05,180
In our case, we're looking at spam emails.

311
00:30:05,180 --> 00:30:09,320
And so the punctuation, you'll look at commas, hyphens, question marks.

312
00:30:09,320 --> 00:30:15,660
And so in the context, if we're looking at keywords, the punctuation doesn't really matter.

313
00:30:15,660 --> 00:30:25,260
So if you see buy a Viagra now, period, it doesn't make any difference between buy Viagra now, exclamation point, or buy Viagra now with a question mark at the end.

314
00:30:25,780 --> 00:30:28,300
So in the context of the sentence,

315
00:30:31,940 --> 00:30:38,440
that one sentence would likely be classified as spam because it's talking about Viagra and it's very unlikely.

316
00:30:38,440 --> 00:30:41,480
That's a very common spam word we'll see in a lot of spam messages.

317
00:30:41,480 --> 00:30:47,480
So punctuation is just not relevant because it doesn't add any context to the message.

318
00:30:48,720 --> 00:30:50,560
And so that's something that we can remove.

319
00:30:50,820 --> 00:30:58,740
If you would like, you can follow through on this workbook and then try it without removing the punctuation and see if you notice any differences.

320
00:30:59,960 --> 00:31:04,460
But I don't think there would be any differences just because of that context.

321
00:31:04,640 --> 00:31:10,480
There are better things to weigh than periods, question marks, commas, things of that nature.

322
00:31:10,480 --> 00:31:21,080
And then as you see, emails or websites, such as lists.debian.org, they still have their punctuation because that's not the word lists, debian, and org.

323
00:31:21,080 --> 00:31:24,460
That is a URL, lists.debian.org.

324
00:31:24,460 --> 00:31:26,360
And so that's kind of the difference there.

325
00:31:26,360 --> 00:31:30,120
If we had hyphenated words, we would see them included in here as well.

326
00:31:33,660 --> 00:31:36,920
So we're going to go on and load the training data.

327
00:31:36,920 --> 00:31:42,820
This one is just going to give us the counts that we need.

328
00:31:42,820 --> 00:31:44,500
This is just done for you.

329
00:31:44,500 --> 00:31:54,940
Remember, one of those five things that we needed to keep track was all of the word counts in the ham emails, all the word counts in the spam emails.

330
00:31:55,200 --> 00:32:07,600
But what this is going to do is count the number of emails that we have in ham, count the number of emails that we have in spam, and then count the total number of emails that we have in our testing set.

331
00:32:07,980 --> 00:32:12,180
So for here, our testing and training split is already defined.

332
00:32:12,180 --> 00:32:16,660
It's 80-20, and so we'll see that in a moment.

333
00:32:17,440 --> 00:32:22,200
And we go ahead and hop into our second task, which is to load the training data.

334
00:32:22,600 --> 00:32:27,260
So task two, it asks us to create two lists, and we're going to have corpus.

335
00:32:28,040 --> 00:32:33,240
And it's important to get these words identical because of how we use them later on.

336
00:32:33,240 --> 00:32:35,880
So we're going to have a list called corpus.

337
00:32:35,880 --> 00:32:38,700
We're going to have a list called labels.

338
00:32:39,300 --> 00:32:49,680
And then the next bit is we want to load each of the email bodies from the data directory slash ham and data directory slash spam into the corpus array.

339
00:32:49,840 --> 00:33:02,280
So what we're going to do is for each in os.listdir, data.dir, this was defined above, slash ham.

340
00:33:03,860 --> 00:33:09,660
And then we're going to say with each open file,

341
00:33:15,290 --> 00:33:20,970
ham slash plus each, because that's what we defined in our for loop.

342
00:33:20,970 --> 00:33:36,850
We're going to read them as a file named f, corpus.append, f.read, and then labels.append the word ham.

343
00:33:37,450 --> 00:33:42,950
And the reason why is because this third task is load the labels for each email into the labels array.

344
00:33:42,950 --> 00:33:47,710
So in this first code block, we're going to read every single ham email in the ham directory.

345
00:33:47,710 --> 00:33:56,050
We're going to add the bodies to the corpus array, and then we're going to add the label ham into the labels array.

346
00:33:56,050 --> 00:34:00,790
Then we can go ahead and do the exact same thing for spam.

347
00:34:00,790 --> 00:34:09,210
We're just going to have to change each thing to spam, spam, and the labels to spam.

348
00:34:09,630 --> 00:34:12,390
And this will go through and read all of the emails.

349
00:34:12,490 --> 00:34:17,290
So I'm going to leave this up for a moment, let you guys copy that down.

350
00:34:28,440 --> 00:34:32,000
And then I'm going to press shift enter and run it.

351
00:34:32,000 --> 00:34:39,080
So this was really quick, just read all the ham emails, all the spam emails, and now it's loaded into our arrays.

352
00:34:39,820 --> 00:34:53,880
If we want to get an idea of what our class count looks like, we have a total of 2,359 ham emails, 1,100 spam emails, and then just as a graph we can see here.

353
00:34:54,340 --> 00:35:08,240
And I think it's important to graph these counts because sometimes you will have what are considered unbalanced data sets, and so you need to perform other changes here and there just to make sure that the data sets are balanced.

354
00:35:08,420 --> 00:35:15,820
Especially for things like credit card fraud, where that happens less than a percent of the time.

355
00:35:15,820 --> 00:35:25,800
You can build a model that's 99% accurate by just saying everything is a legitimate transaction, but unfortunately that doesn't really help when you're trying to stop credit card fraud.

356
00:35:26,080 --> 00:35:29,660
So when you have imbalanced data sets, there's additional things that you do.

357
00:35:29,660 --> 00:35:35,900
In this case, it looks a little unbalanced, but we'll be able to work with it no problems.

358
00:35:38,360 --> 00:35:41,120
So we are on task 2A.

359
00:35:41,460 --> 00:35:48,400
In this one we want to see the full text body of a random email in our corpus array, and then we want to see the tokenized result of that same email.

360
00:35:48,400 --> 00:35:56,380
So this is going to be very similar to what we did for task 1, which was printing out the test email, and then printing out the tokenized version of the test email.

361
00:35:56,380 --> 00:35:58,720
So we'll just go ahead and grab a random email.

362
00:35:58,720 --> 00:36:02,140
So email equals corpus.

363
00:36:03,240 --> 00:36:04,980
Actually, let me go ahead and do this.

364
00:36:04,980 --> 00:36:09,500
Let's have email ID equals just the number 5.

365
00:36:09,500 --> 00:36:10,280
Why not?

366
00:36:10,920 --> 00:36:15,060
And then we're going to grab email ID.

367
00:36:15,760 --> 00:36:17,820
So this is going to be our random email.

368
00:36:18,480 --> 00:36:20,340
We're going to...

369
00:36:20,340 --> 00:36:23,280
probably going to print a...

370
00:36:23,280 --> 00:36:24,780
No, I already have a divider in there.

371
00:36:24,780 --> 00:36:25,380
Cool.

372
00:36:25,380 --> 00:36:27,100
So let's just print that email.

373
00:36:28,180 --> 00:36:36,500
And then, same thing, we're going to do print tokenizer, and then that same email.

374
00:36:36,680 --> 00:36:41,360
So this line and this line is exactly the same that we did in task 1.

375
00:36:41,360 --> 00:36:44,260
This is the only thing that changes so that we grab a random email.

376
00:36:44,260 --> 00:36:49,000
And then you don't have to put 5, you can put any number, but I'm just picking 5.

377
00:36:49,280 --> 00:36:51,520
So go ahead, Shift-Enter.

378
00:36:52,440 --> 00:36:54,800
And this one's pretty nice.

379
00:36:54,800 --> 00:37:06,380
So we see the body of the email, product review, virttools, dev, URL, product review, virttool, dev, 2.0 URL.

380
00:37:06,600 --> 00:37:12,420
We see the full path for the URL here.

381
00:37:12,940 --> 00:37:15,600
Let's see if there's anything interesting.

382
00:37:15,780 --> 00:37:30,100
Maybe features, feature is shortened to feature, because you can say featured in a motion picture with an ed at the end, features as they have here, or we will feature with just an e at the end.

383
00:37:31,320 --> 00:37:41,440
So this is the way that our tokenizer operates, but we still need to turn these words into numbers, and this is where our vectorizers come in.

384
00:37:41,980 --> 00:37:44,440
So our vectorizers, we're going to use two kinds of vectorizer.

385
00:37:44,440 --> 00:37:47,600
We're going to use the count vectorizer, which just takes the straight word count.

386
00:37:47,600 --> 00:37:49,760
It's exactly what we did in the slides.

387
00:37:49,760 --> 00:37:56,460
Then we're going to use a TF-IDF vectorizer that applies a mathematical weight to each of the words.

388
00:37:56,680 --> 00:38:04,440
And they work slightly different, but let's go ahead and define them, and we'll show you what the difference looks like.

389
00:38:04,440 --> 00:38:08,140
So we're going to name this CVEC as our count vectorizer.

390
00:38:08,460 --> 00:38:12,580
Count vectorizer, and the capitalizations are important.

391
00:38:12,680 --> 00:38:16,500
Then we're going to use our tokenizer function above as our tokenizer function here.

392
00:38:16,500 --> 00:38:20,840
So we're going to have tokenizer equals tokenizer.

393
00:38:21,620 --> 00:38:22,660
Awesome.

394
00:38:23,180 --> 00:38:30,980
And then we want to vectorize our words, so this is going to be saved as countX.

395
00:38:30,980 --> 00:38:36,500
So we're going to have count underscore capital X equals...

396
00:38:37,020 --> 00:38:38,920
Oops, did not mean to open that.

397
00:38:42,860 --> 00:38:46,660
CVEC, our vectorizer, and then we're going to use fit transform.

398
00:38:47,900 --> 00:38:51,320
Fit is Scikit-learn's way of saying train.

399
00:38:51,360 --> 00:38:56,000
So in this instance, we're going to train our vectorizer.

400
00:38:56,320 --> 00:39:02,580
So fit transform, and then our purpose, which is just our list of words.

401
00:39:02,980 --> 00:39:06,640
And then we're going to do the same thing for our TF-IDF vectorizer.

402
00:39:06,640 --> 00:39:10,920
So we're going to take TVEC equals TF-IDF.

403
00:39:11,460 --> 00:39:14,420
Vectorizer and capitalization is important.

404
00:39:14,420 --> 00:39:17,720
Tokenizer equals tokenizer.

405
00:39:17,720 --> 00:39:31,280
And then our TF-IDF X capital X equals TVEC dot fit transform corpus.

406
00:39:32,180 --> 00:39:33,460
Cool.

407
00:39:33,460 --> 00:39:37,520
So I'm going to let that sit there for a second while you guys are typing in.

408
00:39:43,580 --> 00:39:46,000
And then I'm going to go ahead and run it.

409
00:39:46,600 --> 00:39:56,380
So this takes about a minute because what it's doing is it is counting all of the words across all of the emails, the ham emails and the spam emails.

410
00:39:56,660 --> 00:40:06,460
And so as it's counting each of these words, it's applying a word count for the count vectorizer, and then it applies a weight for the TF-IDF vectorizer.

411
00:40:06,460 --> 00:40:08,900
And the weight takes all the words into account.

412
00:40:08,900 --> 00:40:10,620
So it becomes a whole process.

413
00:40:10,620 --> 00:40:15,720
But this takes about 60 to 120 seconds.

414
00:40:15,960 --> 00:40:18,520
So we'll go ahead and let that run.

415
00:40:18,600 --> 00:40:27,320
While that's running, we're just going to get started on task 3A, which is just to print each token from the test email.

416
00:40:27,840 --> 00:40:29,460
Now this one can be a little tricky.

417
00:40:29,460 --> 00:40:34,700
So there's a bit of a trick to identify the unique words and the counts.

418
00:40:34,700 --> 00:40:39,800
So what we're going to do is create a list of dictionary keys.

419
00:40:39,800 --> 00:40:47,100
So what that's going to look like is for i in list dict from keys.

420
00:40:48,480 --> 00:40:55,440
Tokenized email was defined above, and what that is is the tokenized version of our test email.

421
00:40:55,820 --> 00:40:57,900
So we're going to go ahead and print.

422
00:40:58,480 --> 00:41:01,280
I'm just using the Python format.

423
00:41:04,220 --> 00:41:12,520
Tokenized email dot count of each word, and then the word in the tokenized email.

424
00:41:12,880 --> 00:41:16,220
So I'm going to verify vectorizing is complete.

425
00:41:17,360 --> 00:41:22,100
And I'm going to let you guys run your vectorizers, let that sit for a little bit.

426
00:41:22,100 --> 00:41:24,280
And then we're going to run this, because we need...

427
00:41:25,680 --> 00:41:28,720
Oh, I don't believe we do need the previous one to run this.

428
00:41:28,720 --> 00:41:38,200
So if your vectorizer is still going, your code here will not run right away, but it will run as soon as the vectorizer completes.

429
00:41:38,440 --> 00:41:40,540
So I'm going to go ahead and run mine.

430
00:41:40,560 --> 00:41:45,900
So shift-enter, and tokenizer email is not correct.

431
00:41:45,900 --> 00:41:50,400
I think it's tokenized email, that makes a little bit more sense.

432
00:41:50,580 --> 00:41:51,400
I'm going to run that.

433
00:41:51,440 --> 00:41:54,140
Tokenized email is not defined.

434
00:41:55,140 --> 00:41:56,660
Okay, give me a second.

435
00:41:56,660 --> 00:42:01,220
I'm going to scroll up to our tokenizer.

436
00:42:07,040 --> 00:42:09,680
I guess I don't have that defined.

437
00:42:12,820 --> 00:42:14,940
Where is that?

438
00:42:22,370 --> 00:42:24,530
Ah, I was supposed to save it up above.

439
00:42:24,530 --> 00:42:25,550
That's okay.

440
00:42:25,690 --> 00:42:28,590
So we'll go ahead and create it here.

441
00:42:29,750 --> 00:42:36,370
Tokenized email equals tokenizer test email.

442
00:42:36,370 --> 00:42:38,450
So we're going to go ahead and save that.

443
00:42:40,790 --> 00:42:43,910
And here we have the word counts of our test email.

444
00:42:43,910 --> 00:42:46,550
We see the familiar East Asian font, Lenny.

445
00:42:46,550 --> 00:42:53,110
We know unsubscribe showed up twice, and we know list.debian.org showed up twice.

446
00:42:53,110 --> 00:42:56,690
Everything else in here only showed up once.

447
00:42:56,690 --> 00:43:02,790
I know I went very quickly, but this was the line that I needed to add.

448
00:43:02,790 --> 00:43:06,130
Tokenized email equals tokenizer test email.

449
00:43:07,130 --> 00:43:15,110
And then we grab the list of the dictionary from that list of the tokenized email, and we print out the word counts.

450
00:43:18,390 --> 00:43:26,130
So going on to 3b, what we want to see is, okay, we know what the word count looks like if we just did a straight word count.

451
00:43:26,130 --> 00:43:34,130
How does the count vectorizer and the tf.idf vectorizer transform our test email as we're going through?

452
00:43:34,610 --> 00:43:41,590
So we're going to go ahead and use our example CVEC.

453
00:43:41,590 --> 00:43:48,110
And the reason why we're using a separate example version is because we want to redefine a count vectorizer.

454
00:43:50,330 --> 00:43:54,310
Don't forget our tokenizer equals tokenizer.

455
00:43:55,330 --> 00:44:08,290
And then we're going to have our exampleX using our exampleCVEC.fitTransform.

456
00:44:08,450 --> 00:44:11,510
And then it expects a list.

457
00:44:12,110 --> 00:44:20,330
Our test email is a simple string, so we're going to have to wrap it in these brackets so that it becomes a list.

458
00:44:20,330 --> 00:44:22,970
And then we're going to have our test email.

459
00:44:24,310 --> 00:44:25,410
Cool.

460
00:44:25,690 --> 00:44:31,560
And then we're going to go ahead and print our example email.

461
00:44:32,470 --> 00:44:33,510
Oops.

462
00:44:33,770 --> 00:44:35,370
ExampleX.

463
00:44:36,050 --> 00:44:40,170
And then here, we're going to do the same thing.

464
00:44:40,170 --> 00:44:47,270
We're going to have exampleTVEC equals tf.idf vectorizer.

465
00:44:47,270 --> 00:44:49,310
Capitals are important.

466
00:44:49,350 --> 00:44:51,430
Tokenizer equals tokenizer.

467
00:44:51,430 --> 00:44:57,710
And then our example, tf.idf.

468
00:44:57,710 --> 00:45:00,110
I'll just call it exampleX because it doesn't matter.

469
00:45:00,110 --> 00:45:03,790
Once we print out our original exampleX, we don't need it anymore.

470
00:45:07,830 --> 00:45:09,890
ExampleTVEC.fitTransform.

471
00:45:13,630 --> 00:45:17,890
And then the brackets, test email.

472
00:45:18,470 --> 00:45:22,750
And then we can print our exampleX.

473
00:45:23,670 --> 00:45:26,030
So I'm going to let that sit for a moment.

474
00:45:26,130 --> 00:45:28,230
Let you guys copy that down.

475
00:45:32,690 --> 00:45:35,210
I'm going to just look ahead real quick.

476
00:45:35,310 --> 00:45:36,990
We are doing great.

477
00:45:40,670 --> 00:45:41,370
Okay.

478
00:45:41,370 --> 00:45:43,310
So I'm going to go ahead and run this now.

479
00:45:43,310 --> 00:45:44,870
I pressed Shift-Enter.

480
00:45:45,910 --> 00:45:53,090
And if we look at our count vectorizer of test email, we see these different numbers.

481
00:45:53,090 --> 00:45:56,150
And what these numbers represent are tokens.

482
00:45:56,150 --> 00:45:59,390
So, for example, a very nice game.

483
00:45:59,390 --> 00:46:01,170
A would be token 1.

484
00:46:01,170 --> 00:46:02,750
Very would be token 2.

485
00:46:02,750 --> 00:46:04,130
Nice would be token 3.

486
00:46:04,130 --> 00:46:05,970
Game would be token 4.

487
00:46:06,530 --> 00:46:11,850
So in this case, Scikit-Learn actually knows the best way that it wants to tokenize things.

488
00:46:11,850 --> 00:46:17,810
So the letters, for example, 0, this does not necessarily represent East.

489
00:46:18,930 --> 00:46:23,310
And so, basically, Scikit-Learn knows how these tokens are used.

490
00:46:23,310 --> 00:46:24,950
We don't know how these tokens are used.

491
00:46:24,950 --> 00:46:32,490
But something that we can point out is we see token 16 shows up twice and token 9 shows up twice.

492
00:46:32,490 --> 00:46:41,850
So based on what we saw up above, we can guarantee that either token 16 or token 9 is unsubscribe.

493
00:46:41,850 --> 00:46:45,950
And then the other one is list.debian.org.

494
00:46:46,350 --> 00:46:47,510
And so these show up twice.

495
00:46:47,510 --> 00:46:50,730
So this is exactly the same as if we had done the word count ourselves.

496
00:46:51,390 --> 00:46:58,830
The TF-IDF vectorizer, however, uses a bit of a weight algorithm.

497
00:46:58,830 --> 00:47:03,270
And so everything that was set as 1 has this 0.204 weight.

498
00:47:03,450 --> 00:47:10,410
The 2, which is token 9 and token 16, have this 0.408 weight.

499
00:47:10,650 --> 00:47:14,810
And this doesn't necessarily happen in every instance.

500
00:47:14,810 --> 00:47:22,090
But in this instance, you can see the 0.408 shows up twice as much as the 0.204 exactly.

501
00:47:22,710 --> 00:47:28,030
And so this is the way that count vectorizer operates and TF-IDF vectorizer operates.

502
00:47:29,270 --> 00:47:33,810
The underlying, what happens underneath the hood, we don't really need to worry about.

503
00:47:33,810 --> 00:47:40,070
We just need to know how these work and the differences between the two.

504
00:47:42,950 --> 00:47:46,030
So moving on, we're going to load the testing data.

505
00:47:46,030 --> 00:47:47,390
This is already written for us.

506
00:47:47,390 --> 00:47:54,130
So we're just going to grab the first 10 files in the testing directory and print those out.

507
00:47:54,130 --> 00:47:57,930
So we see we just have all these test emails.

508
00:47:57,970 --> 00:48:02,870
And at the end of the file name, we have the label that it's supposed to belong to.

509
00:48:02,870 --> 00:48:06,110
So we have ham, ham, spam, spam, spam, ham.

510
00:48:08,830 --> 00:48:12,130
And so what we're going to do is read each of these emails.

511
00:48:12,130 --> 00:48:17,890
And then we're going to pull off the last little bit of the file name and use that as our label.

512
00:48:17,950 --> 00:48:22,950
And we want to have the testing separate from the training so that we can evaluate how well our model is doing.

513
00:48:23,450 --> 00:48:26,310
So we're going to load the testing data.

514
00:48:26,310 --> 00:48:30,630
And this one's a little complicated, so we have a couple tasks to run through.

515
00:48:30,630 --> 00:48:32,970
So we're going to create two list arrays, kind of like we did earlier.

516
00:48:32,970 --> 00:48:37,150
We're going to have test corpus, and we're going to set that as an empty list.

517
00:48:37,150 --> 00:48:42,610
And we're going to have test labels, which is also going to be an empty list.

518
00:48:43,390 --> 00:48:48,650
And then we are going to load all of the emails from the test directory.

519
00:48:48,650 --> 00:49:05,650
So we're going to say, for filename in os.listdir, our data directory, plus test, which is where it's located, with each file that is open.

520
00:49:05,650 --> 00:49:14,830
And we need to tell it which file test plus filename are.

521
00:49:15,470 --> 00:49:26,050
For read, as filenamed f, we are going to do test corpus.append, f.read.

522
00:49:26,690 --> 00:49:32,610
And then we are going to do something a little weird here, so bear with me.

523
00:49:32,610 --> 00:49:37,110
I'm just going to use re.split, just to split this up.

524
00:49:37,310 --> 00:49:39,690
I'm looking for txt.

525
00:49:41,750 --> 00:49:46,250
And then we're just going to grab the last bit in that split.

526
00:49:46,250 --> 00:49:51,990
So this is going to split the file trainemail.txt.

527
00:49:52,450 --> 00:49:57,210
So it's going to split it to this section, and then to this section.

528
00:49:58,050 --> 00:50:05,830
So this would be considered index 0, and this last little bit would be considered index 1, which is why we have this 1 here at the end.

529
00:50:07,570 --> 00:50:09,290
And then we're going to append the labels.

530
00:50:09,290 --> 00:50:21,290
So test.labels.appendLabel And so I'm going to give you guys a little bit to go ahead and get this all written up.

531
00:50:21,410 --> 00:50:27,090
Be careful, there is a backslash before this period, because this is a regular expression.

532
00:50:29,650 --> 00:50:31,590
And everything should be good.

533
00:50:32,390 --> 00:50:35,670
So I'm just going to let that sit for a little bit, let you guys do your thing.

534
00:50:35,670 --> 00:50:37,850
In the meantime, I'm just going to look ahead real quick.

535
00:50:39,950 --> 00:50:41,850
The rest of this gets so much easier.

536
00:50:42,030 --> 00:50:44,750
It's pretty crazy how easy it gets.

537
00:50:46,030 --> 00:50:48,890
In the meantime, TAs, are there any questions in the chat?

538
00:50:57,320 --> 00:50:59,620
We appear to be good so far.

539
00:51:00,060 --> 00:51:01,020
Awesome.

540
00:51:03,200 --> 00:51:06,260
So I'm going to go ahead and move on.

541
00:51:06,260 --> 00:51:07,720
I'm going to press Shift-Enter.

542
00:51:07,720 --> 00:51:10,440
It's going to load all of those emails.

543
00:51:10,440 --> 00:51:19,660
So this is just reading all of the test emails into our corpus array and grabbing those labels and putting them into the end of our test labels array.

544
00:51:20,160 --> 00:51:24,060
And then just like last time, what this is going to do is print out the number of test emails.

545
00:51:24,060 --> 00:51:27,780
We have 590 ham emails, 276 spam emails.

546
00:51:27,780 --> 00:51:31,200
And here's just a nice pretty graph representing what that looks like.

547
00:51:31,860 --> 00:51:37,900
And as I said earlier, this information or this data has already been split up in an 80-20 split.

548
00:51:37,900 --> 00:51:43,900
So 80% of it is being used for our training and then 20% of it is being used for our testing.

549
00:51:46,080 --> 00:51:51,460
And then this one, it might sound complicated, but this is actually really easy.

550
00:51:51,460 --> 00:51:55,760
We have already trained our CVEC and our TVEC.

551
00:51:55,760 --> 00:52:01,700
And now all we need to do is collect our testing values.

552
00:52:01,700 --> 00:52:06,660
We need to tokenize our testing values using these already trained vectorizers.

553
00:52:06,660 --> 00:52:08,800
So this is super easy.

554
00:52:10,800 --> 00:52:19,860
testCountX equals CVEC.transformTestCorpus.

555
00:52:20,140 --> 00:52:21,460
And that's it.

556
00:52:21,580 --> 00:52:23,060
That's all we've got to do for that.

557
00:52:23,480 --> 00:52:25,320
We can just copy and paste here.

558
00:52:25,580 --> 00:52:31,760
Say our testTFIDFX is our TVEC.transformTestCorpus.

559
00:52:32,560 --> 00:52:43,120
The reason this works is if you remember in the slides how we had the sentences that we already knew about, and then we had to perform a counting game for a very close game.

560
00:52:43,180 --> 00:52:50,820
We had to count how many times a showed up in the sentences that we were already aware of, how many times very showed up, how many times close showed up, how many times game showed up.

561
00:52:50,820 --> 00:52:54,260
So this transform allows us to do just that.

562
00:52:54,260 --> 00:53:02,040
So what this will do is grab each email and it will say, okay, here's the tokenized words of the email that I have.

563
00:53:02,040 --> 00:53:04,500
How many times does it show up in the count vectorizer?

564
00:53:04,500 --> 00:53:08,940
And then what is the weight that it shows up in the TFIDF vectorizer?

565
00:53:09,740 --> 00:53:12,640
So I'm going to go ahead, Shift-Enter, run that.

566
00:53:13,180 --> 00:53:15,260
And just like last time, this is going to take a little bit.

567
00:53:15,340 --> 00:53:18,900
It's counting all the words and associating them with the different tokens.

568
00:53:19,280 --> 00:53:21,980
This is going to be significantly faster than training it.

569
00:53:21,980 --> 00:53:23,680
See, it's almost done already.

570
00:53:25,020 --> 00:53:26,720
And so we're going to go ahead and move on.

571
00:53:28,580 --> 00:53:32,700
So this bit, we're going to get ready to test and evaluate our models.

572
00:53:32,700 --> 00:53:36,840
So this is just a helper function that's going to show us a bunch of really neat statistics.

573
00:53:37,100 --> 00:53:40,940
The details of this helper function you can look into on your own.

574
00:53:41,180 --> 00:53:50,400
But for now, what we need to use as an input is our confusion matrix, our score, and our classification report.

575
00:53:51,660 --> 00:53:54,600
So we're just going to go ahead, Shift-Enter, run that.

576
00:53:54,960 --> 00:53:57,780
Definition defined.

577
00:53:59,220 --> 00:54:09,300
And then here's where we get into tasks 6a, 6b, 6c, and 6d, which is going to be us training and evaluating our different models.

578
00:54:09,380 --> 00:54:22,040
So we're going to start with multinomial Naive Bayesian with a TF-IDF vectorizer, with a count vectorizer, and then we're going to do the same for logistic regression with TF-IDF and logistic regression with a count vectorizer.

579
00:54:24,020 --> 00:54:26,560
So we're going to start out just one thing at a time.

580
00:54:27,320 --> 00:54:30,420
We're going to define our constructor.

581
00:54:30,420 --> 00:54:36,480
So this is going to be MNB TF-IDF equals multinomial NB.

582
00:54:36,480 --> 00:54:38,660
Watch your capitalization.

583
00:54:39,580 --> 00:54:42,360
We're going to define that, and then we're going to train it.

584
00:54:43,740 --> 00:54:52,220
MNB TF-IDF dot fit, which is train in scikit-learn world.

585
00:54:52,220 --> 00:55:00,080
And we're going to use our TF-IDF X, which is our preprocessed information, and then we're going to compare it to our labels.

586
00:55:02,060 --> 00:55:03,860
So that's going to train our model.

587
00:55:03,860 --> 00:55:07,300
Now we're going to grab a couple really neat statistics.

588
00:55:07,300 --> 00:55:12,020
So the first one we're going to get is our score, and this wants us to call it score.

589
00:55:14,380 --> 00:55:18,880
Score MNB TF-IDF predictions using our predict function.

590
00:55:18,880 --> 00:55:27,440
We're going to develop a confusion matrix called C matrix, MNB TF-IDF, and then classification report, C report, MNB TF-IDF.

591
00:55:28,540 --> 00:55:30,280
So we're going to start with our score.

592
00:55:30,280 --> 00:55:37,380
Score MNB TF-IDF equals MNB TF-IDF dot score.

593
00:55:37,980 --> 00:55:43,280
And then we're going to use our test values, because we've trained it on our training data, and we want to test it with our testing.

594
00:55:43,280 --> 00:55:54,060
So we're going to do test TF-IDF X, because that's the way that our system was vectorizing these.

595
00:55:54,500 --> 00:55:58,840
So we don't want to use count vectorizer with a TF-IDF train, because that's not going to work.

596
00:55:59,140 --> 00:56:01,540
And then we're going to use test labels.

597
00:56:03,980 --> 00:56:05,200
That's going to give us our score.

598
00:56:05,200 --> 00:56:06,860
So we're going to get our predictions.

599
00:56:07,180 --> 00:56:14,760
Predictions MNB TF-IDF equals MNB TF-IDF dot predict.

600
00:56:14,900 --> 00:56:18,600
And this is going to give us our raw predictions.

601
00:56:21,360 --> 00:56:29,720
This is going to tell us what it thinks each email is, whereas our score is going to see how accurate it is.

602
00:56:29,720 --> 00:56:33,880
And then this is going to be C matrix.

603
00:56:34,880 --> 00:56:42,680
MNB TF-IDF equals confusion matrix.

604
00:56:42,980 --> 00:56:48,620
We're going to use our test labels and our predictions.

605
00:56:48,780 --> 00:56:51,280
MNB TF-IDF.

606
00:56:51,280 --> 00:57:01,300
So this is going to tell us what it thinks the value is, if it's a spam or ham email, and then what it actually is.

607
00:57:01,500 --> 00:57:05,740
So it's going to give us our true positives, our false positives, our true negatives, and our false negatives.

608
00:57:06,080 --> 00:57:07,700
And then the C report.

609
00:57:07,700 --> 00:57:18,080
We're not going to do too much with this, but what a classification report will do is show us some more metrics, such as the recall and the precision, which we don't need to worry about too much.

610
00:57:18,080 --> 00:57:20,340
We're just going to focus on the accuracy for now.

611
00:57:20,340 --> 00:57:28,740
But for data science, it's important to know what the recall and the precision are as we go through things.

612
00:57:30,340 --> 00:57:34,800
Report, test labels, and then we're going to use our predictions again.

613
00:57:36,180 --> 00:57:45,740
And this is just like our C matrix MNB TF-IDF, in the order that we put things.

614
00:57:45,740 --> 00:57:48,020
We have test labels, test labels, predictions, predictions.

615
00:57:48,820 --> 00:57:49,920
And that is it.

616
00:57:49,920 --> 00:57:52,240
And then we're going to use our helper function generate report.

617
00:57:53,560 --> 00:57:55,680
So I know that's a lot of typing.

618
00:57:55,720 --> 00:58:00,420
I'm going to go ahead and sit here as you guys kind of follow through.

619
00:58:00,420 --> 00:58:03,880
So we have our MNB TF-IDF constructor.

620
00:58:03,880 --> 00:58:07,120
We are going to train it using the fit function.

621
00:58:07,120 --> 00:58:13,980
We're going to train it on our testing data, which is stored as TF-IDFX, and our testing labels, which is just stored as labels.

622
00:58:13,980 --> 00:58:22,560
We are going to use score, MNB TF-IDF.score, to evaluate our accuracy.

623
00:58:22,880 --> 00:58:25,740
We're going to use our predictions to get the raw predictions.

624
00:58:25,740 --> 00:58:35,620
And then with the raw predictions, we are going to develop a confusion matrix and a classification report, which will tell us how well our model is doing visually.

625
00:58:38,920 --> 00:58:41,100
So I'm going to let that sit for one more minute.

626
00:58:41,100 --> 00:58:43,220
TA, are there any questions?

627
00:58:52,160 --> 00:58:56,320
No, I think... well, hang on, let me go back one here.

628
00:58:57,940 --> 00:58:59,060
Nope, still got it.

629
00:58:59,060 --> 00:59:00,300
So I think we're good.

630
00:59:00,340 --> 00:59:00,920
Awesome.

631
00:59:01,140 --> 00:59:01,680
Awesome.

632
00:59:01,680 --> 00:59:02,460
Fantastic.

633
00:59:03,500 --> 00:59:08,100
So I am going to go ahead and move on.

634
00:59:08,100 --> 00:59:09,360
I'm going to press Shift-Enter.

635
00:59:09,360 --> 00:59:10,400
It's going to run.

636
00:59:11,260 --> 00:59:16,800
And now we have our... up here at the top is our classification report.

637
00:59:17,280 --> 00:59:20,940
And so what this is going to do is show us our precision and our recall.

638
00:59:20,940 --> 00:59:23,460
We don't need to worry too much about this one.

639
00:59:23,620 --> 00:59:26,140
Instead, I want you to focus on the lower chart.

640
00:59:26,140 --> 00:59:28,680
So here we have an accuracy of 87.5%.

641
00:59:29,420 --> 00:59:35,840
We can see that all of the actual ham emails were predicted as ham.

642
00:59:35,840 --> 00:59:37,800
So these are our true positives.

643
00:59:38,520 --> 00:59:43,300
A little more than half of our spam emails were correctly predicted as spam.

644
00:59:43,300 --> 00:59:45,760
So these are our true negatives.

645
00:59:45,760 --> 00:59:51,020
And then, unfortunately, it misclassified...

646
00:59:51,760 --> 00:59:52,720
Oh, my mistake.

647
00:59:52,720 --> 00:59:53,760
I have that backwards.

648
00:59:53,760 --> 01:00:08,580
So it says 108 of our ham emails it thinks are spam emails, whereas all of the spam emails, so the actual spam emails, were correctly classified as spam.

649
01:00:08,580 --> 01:00:17,880
And then all of our ham emails, what it thought were ham emails, were ham.

650
01:00:20,760 --> 01:00:26,440
So we're going to go ahead and do the same thing for 6b, 6c, and 6d.

651
01:00:27,460 --> 01:00:39,540
One of the beautiful things, and I'm going to show you this just as a super quick and easy way of going about and doing things, is I'm just going to copy the code that I'd already written.

652
01:00:39,540 --> 01:00:51,740
This is why machine learning is awesome, and very different from programming normal algorithms, is all we need to do is just change these tf.ids to count, and then everything else is the same.

653
01:00:52,880 --> 01:01:02,140
So we use mnbcount to define our multinomial Naive Bayesian, as we have been asked of us here in 6b.

654
01:01:03,080 --> 01:01:07,560
mnbcount.fit, and we're going to use our countX because that's how it's been vectorized.

655
01:01:08,120 --> 01:01:13,980
We're going to change our score mnbcount using mnbcount.score.

656
01:01:13,980 --> 01:01:25,580
We're going to change our tf.idsx to testCountX, and then likewise kind of all throughout here.

657
01:01:29,310 --> 01:01:41,310
TestLabels is going to stay the same, predictions is going to change because it is no longer predictions.tf.ids, it is predictionsCount, and then we have our confusion matrix and our classification report.

658
01:01:41,670 --> 01:01:53,410
So just to go over it one more time, we define our multinomial Naive Bayesian constructor, we train it using the fit function and the count vectorizer, our testing data, our training data.

659
01:01:54,250 --> 01:02:10,350
We develop our score off of our testCount data using our testLabels, and then we have our predictions, we have our confusion matrix, and our classification report.

660
01:02:10,350 --> 01:02:16,910
So I'm going to let this sit for just a little bit as you guys go through and copy that down.

661
01:02:25,810 --> 01:02:32,370
What's really cool is in the GitHub repository, there are other workbooks.

662
01:02:32,370 --> 01:02:45,510
So for example, we use the exact same methodology to build a malicious URL predictor, so things that look like it's possibly part of a botnet.

663
01:02:47,510 --> 01:02:49,590
All of this code is exactly the same.

664
01:02:49,590 --> 01:02:54,990
Once we vectorize our information, we just copy and paste this code in, and it does the exact same thing.

665
01:02:55,250 --> 01:03:05,610
And so that's the benefit of machine learning, is it learns from the data itself instead of us having to manually program, okay, this solves problem A, but now we have to rewrite this to solve problem B.

666
01:03:05,610 --> 01:03:07,450
It just all does the same thing.

667
01:03:09,290 --> 01:03:13,390
So I am going to go ahead and move on.

668
01:03:13,390 --> 01:03:15,390
I'm going to press Shift-Enter.

669
01:03:16,710 --> 01:03:18,710
And we see that we got a little bit better, right?

670
01:03:18,710 --> 01:03:21,090
We are at a 94% accuracy.

671
01:03:21,630 --> 01:03:28,150
It thinks that some of our ham emails are still spam, but significantly less.

672
01:03:28,150 --> 01:03:37,750
We went from 108 down to 23, but now it thinks that some of our spam emails are actually ham.

673
01:03:37,810 --> 01:03:38,670
And so that's a problem.

674
01:03:38,670 --> 01:03:49,250
So we have a few more emails going through our filter than we would like, but we have significantly fewer of our ham emails from being blocked.

675
01:03:50,590 --> 01:03:54,110
GT, we've got a question from Twitch chat.

676
01:03:54,110 --> 01:03:54,650
Absolutely.

677
01:03:54,650 --> 01:03:56,670
So Chris...

678
01:03:56,670 --> 01:03:59,270
I'm not going to try and pronounce the name.

679
01:03:59,270 --> 01:04:08,610
Did you intentionally avoid having spam emails classified as ham when you created the model, or is it simply a try and error kind of thing when training the model?

680
01:04:09,550 --> 01:04:13,530
Intentionally avoid having spam emails classified as ham?

681
01:04:14,170 --> 01:04:16,430
Is that what they said?

682
01:04:16,430 --> 01:04:17,310
Yeah.

683
01:04:17,990 --> 01:04:22,050
Yeah, we wouldn't want spam emails to be classified as ham.

684
01:04:22,050 --> 01:04:30,990
We want to draw a line and separate so that you stop getting emails about Viagra and you start getting the regular emails from your school or your work.

685
01:04:31,130 --> 01:04:34,330
And so this is just kind of the way that spam filters operate.

686
01:04:34,870 --> 01:04:49,990
So we could build a model that would classify more spam emails as being part of ham or regular emails, or we can build a model that says, okay, we want absolutely zero spam emails.

687
01:04:49,990 --> 01:04:57,390
If we wind up losing regular emails and classifying those as spam as a mistake, we're okay with that.

688
01:04:57,770 --> 01:05:01,990
A really good example, actually I'm glad you brought this up, is antivirus.

689
01:05:02,170 --> 01:05:18,250
So the way that antivirus companies work is they can err on the side of, oh, I think this is a virus, therefore it must be a virus, or this could be benign, or I kind of think this is a virus, but instead of blocking it, I'm just going to let it go through.

690
01:05:18,870 --> 01:05:20,790
And so you think about the way that malware works.

691
01:05:23,110 --> 01:05:31,490
You have malware that will make calls out to the internet, it'll probably take over your webcam, it'll try and write files, it'll probably take over your microphone.

692
01:05:31,750 --> 01:05:33,150
What else does that?

693
01:05:33,470 --> 01:05:37,910
Chrome, IE, Firefox, browsers.

694
01:05:38,070 --> 01:05:55,870
And so the idea is instead of blocking what could be legitimate applications, and so that would really interrupt the user experience, they kind of err on the side of, we don't want to block the user experience because then you'll uninstall the antivirus.

695
01:05:55,990 --> 01:06:04,570
So they are more willing to let viruses go through than blocking legitimate applications.

696
01:06:05,450 --> 01:06:15,730
And so in this case, these are trade-offs that we have to make as data scientists of if we are okay with letting a few spam emails through versus blocking all of our regular emails.

697
01:06:16,790 --> 01:06:20,670
So he's specifically asking about task 6A.

698
01:06:20,670 --> 01:06:22,150
So if you scroll up a little bit.

699
01:06:22,150 --> 01:06:23,330
Task 6A?

700
01:06:23,710 --> 01:06:38,750
Yeah, I think the answer is a bit of that and a bit of the question of when you build these models, the training process literally punishes the model for getting it wrong every single time.

701
01:06:41,510 --> 01:06:43,610
And in this case...

702
01:06:45,730 --> 01:06:50,770
Yeah, so Chris, you're asking about a false negative.

703
01:06:54,010 --> 01:07:11,370
A false negative is when you've got something that's malicious, that's misclassified as benign, which is what the point of the model is, but what GT brought up is probably more important is when you have something that's benign that you classify as malicious,

704
01:07:11,370 --> 01:07:20,890
that is much more common because you get more benign things on users' devices, as GT brought up.

705
01:07:22,370 --> 01:07:25,450
So in this case, I don't think this was intentional.

706
01:07:26,930 --> 01:07:28,530
No, this wasn't intentional.

707
01:07:28,530 --> 01:07:31,090
This is just the way that the model learned.

708
01:07:32,270 --> 01:07:35,810
And then as we go through the other models, you'll see how these shift.

709
01:07:36,410 --> 01:07:40,130
Yeah, but this is honestly the way it looks like.

710
01:07:40,130 --> 01:07:47,470
You've got the ham stuff that are classified as spam is very low so that all the user experience isn't impacted that badly.

711
01:07:47,510 --> 01:07:55,790
And the spam stuff that's classified as ham is relatively high, but you've got a trade-off between the false positives and the false negatives.

712
01:07:55,810 --> 01:07:56,970
Yeah, exactly.

713
01:07:57,430 --> 01:08:01,010
And I hope that answered your question, Christopher.

714
01:08:01,470 --> 01:08:08,390
I think going through the other models will kind of help answer it a little bit more too.

715
01:08:10,830 --> 01:08:12,390
So accuracy, 87%.

716
01:08:12,950 --> 01:08:14,630
87.5%.

717
01:08:14,630 --> 01:08:16,390
94.3%.

718
01:08:17,870 --> 01:08:23,450
In this one, we're going to use a model that we haven't used before, logistic regression.

719
01:08:23,690 --> 01:08:25,970
And so with Scikit-learn, it's very easy.

720
01:08:25,970 --> 01:08:27,570
It's the exact same thing.

721
01:08:27,570 --> 01:08:34,990
The difference is we need to use a LBFGS solver.

722
01:08:35,650 --> 01:08:45,170
So what that's going to look like is LGSTFIDF equals logistic regression.

723
01:08:45,390 --> 01:08:50,030
Then we're going to set our solver LBFGS.

724
01:08:50,030 --> 01:08:54,110
And this is just a thing with logistic regression specifically.

725
01:08:54,430 --> 01:09:00,470
We saw in the slides that multinomial naivetation has a variable called alpha.

726
01:09:00,470 --> 01:09:01,650
We can tweak that.

727
01:09:01,650 --> 01:09:03,050
That's one of our hyperparameters.

728
01:09:03,050 --> 01:09:04,990
This is considered a hyperparameter.

729
01:09:04,990 --> 01:09:08,090
It's the way that logistic regression interprets the information.

730
01:09:08,750 --> 01:09:10,890
But for this case, we're just going to use LBFGS.

731
01:09:11,490 --> 01:09:12,630
We're going to do the same thing.

732
01:09:12,630 --> 01:09:17,770
We're going to fit LGSTFIDF dot fit

733
01:09:20,970 --> 01:09:24,110
TFIDFX and then it enables.

734
01:09:26,570 --> 01:09:28,310
Let me just hop down here.

735
01:09:28,510 --> 01:09:30,310
We're going to do score equals

736
01:09:34,090 --> 01:09:46,370
LG S TFIDF equals LGSTFIDF dot score testTFIDFX Pay attention to your capitals.

737
01:09:46,370 --> 01:09:48,870
Test labels.

738
01:09:48,870 --> 01:10:11,750
Then we're going to do predictions shuns LGSTFIDF equals LGSTFIDF dot predict and then our test data Typing TFIDF so much it's very confusing for my hands.

739
01:10:12,010 --> 01:10:12,710
T matrix

740
01:10:17,850 --> 01:10:48,070
we're going to use our confusion matrix test labels and predictions LGSTFIDF and then our C report LGSTFIDF equals location report test labels predictions TFIDF That should be good.

741
01:10:48,070 --> 01:10:51,010
Just checking for errors.

742
01:10:52,550 --> 01:10:54,710
That looks right.

743
01:10:56,370 --> 01:11:00,930
So, go ahead copy this down.

744
01:11:00,930 --> 01:11:07,630
I will sit and wait for a bit, but you'll notice a lot of similarities to the multinomial Naive Bayesian.

745
01:11:07,630 --> 01:11:21,350
In fact, we could have just copied and pasted and just changed it from LGS to MNB or from MNB to LGS and then change our constructor to be a logistic regression with the LBFGS solver.

746
01:11:24,030 --> 01:11:27,430
We have our logistic regression constructor.

747
01:11:27,430 --> 01:11:33,050
We train it using the fit function using our TFIDF vectorized data and our labels.

748
01:11:33,470 --> 01:11:40,670
We use our test data and our test labels to generate a score and to generate predictions.

749
01:11:41,150 --> 01:11:46,670
And then, off of those predictions, we generate a confusion matrix and a classification report.

750
01:11:48,070 --> 01:11:53,750
And then, of course, our nice little helper function at the bottom that's going to print us out some helpful stats.

751
01:11:54,270 --> 01:11:57,330
So , I'm just going to let this sit for a little bit longer.

752
01:11:58,930 --> 01:12:04,090
Kometh, did Chris ask any more questions, follow-up, or did we take care of them?

753
01:12:06,010 --> 01:12:11,150
A bunch of people piled on and I helped answer the question because of what he asked.

754
01:12:11,450 --> 01:12:17,650
Honestly, one of the most important things is the false positive versus false negative trade-off.

755
01:12:17,650 --> 01:12:18,170
Yeah.

756
01:12:20,710 --> 01:12:25,670
Yeah, a lot of people have strong opinions about that because it's a lot of...

757
01:12:25,670 --> 01:12:26,830
It's important.

758
01:12:27,110 --> 01:12:30,090
It's definitely important, especially in data science.

759
01:12:30,090 --> 01:12:31,670
This , I'm...

760
01:12:33,030 --> 01:12:34,970
That might have been a good talk.

761
01:12:35,350 --> 01:12:41,550
So, here's the interesting thing, is when I first started out, this I thought was amazing.

762
01:12:41,550 --> 01:12:47,570
Now, as I've learned more over time, I kind of feel like, oh, no, this is a little clunky and a little ham-fisted.

763
01:12:47,570 --> 01:12:50,770
I'm ignoring this, I'm ignoring this, I'm ignoring this, I'm ignoring this.

764
01:12:50,770 --> 01:12:54,470
But this is a great way for beginners to understand machine learning.

765
01:12:55,030 --> 01:13:10,070
And then from there, you can kind of hop into a little bit more of the data science considerations, like imbalanced data sets, true positives, false positives, and some of those trade-offs, as well as more hyperparameter tuning.

766
01:13:13,790 --> 01:13:21,970
If you want to look at a malware, you keep bringing it up, you can look at the Google Endgame Ember.

767
01:13:21,970 --> 01:13:25,470
I'll drop a link in the Twitch chat in a sec.

768
01:13:25,510 --> 01:13:45,010
But there's a malware model that you can look into, like the open-source code, and the final thing at the end of that is it sets the false-positive rate to 0.05% or 0.5% at the end, because that's what you should do in production, because customers care more about false positives,

769
01:13:45,010 --> 01:13:54,870
because false positives lead to alert fatigue, and alert fatigue leads to just ignoring every single alert, and then all the false negatives get through anyway.

770
01:13:54,870 --> 01:13:56,770
All the true negatives get through anyway.

771
01:13:57,910 --> 01:13:59,930
It is something...

772
01:14:00,750 --> 01:14:04,030
Yeah, and even that model is a little clunky and hampers.

773
01:14:05,090 --> 01:14:17,770
Well, and it kind of leads to with the alert fatigue, it kind of leads to the common thing that we hear, especially in the technical community from non-technical people, of why does my thing not work?

774
01:14:18,610 --> 01:14:19,410
Okay.

775
01:14:20,270 --> 01:14:21,990
So that kind of helps address it.

776
01:14:21,990 --> 01:14:24,410
But yeah, these are all data science considerations.

777
01:14:25,410 --> 01:14:27,150
So I'm going to go ahead and run this.

778
01:14:27,430 --> 01:14:28,650
Shift-Enter.

779
01:14:29,690 --> 01:14:40,150
And this is going to take a little bit longer than the multinomial Naive Bayesian, because logistic regression interprets the data a little differently, and so it needs to go through and kind of count things properly.

780
01:14:41,650 --> 01:14:44,810
This takes, I think, about 30 seconds?

781
01:14:46,310 --> 01:14:48,610
Yeah, it's not too bad.

782
01:14:48,610 --> 01:14:50,430
It's 30 to 60 seconds.

783
01:14:51,430 --> 01:14:53,210
So I'm just going to go ahead and run this.

784
01:14:55,210 --> 01:15:04,850
And then for the logistic regression count, we're going to do the same thing we did with the multinomial Naive Bayesian count and just copy and change words because I'm lazy.

785
01:15:09,390 --> 01:15:22,710
But yeah, I'm really excited to see that Twitch chat after I hop off to this workshop and see just how passionate people are about the false positive, true positive, and XYZ.

786
01:15:23,290 --> 01:15:25,750
Oh, we're going to erase it before you get off.

787
01:15:26,910 --> 01:15:28,250
Ruin my day.

788
01:15:33,270 --> 01:15:35,630
Yeah, it's taking a little bit longer than I'm used to.

789
01:15:37,310 --> 01:15:38,630
But that's okay.

790
01:15:38,630 --> 01:15:46,970
While this is going, I'm just going to go ahead and hop into the next bit and paste this.

791
01:15:46,970 --> 01:15:51,930
All we need to do is trade this TF-IDF with COUNT.

792
01:15:53,030 --> 01:15:54,750
So I'm going to go ahead and copy that.

793
01:15:54,750 --> 01:15:57,910
We're going to use the same constructor and the same solver.

794
01:15:57,930 --> 01:15:59,730
We're going to use COUNTEX.

795
01:15:59,730 --> 01:16:02,050
We're going to use COUNTFIT.

796
01:16:03,630 --> 01:16:07,850
COUNTCOUNT COUNTCOUNT COUNTCOUNT

797
01:16:12,880 --> 01:16:14,860
Test labels is the same.

798
01:16:18,410 --> 01:16:20,070
These are predictions.

799
01:16:21,010 --> 01:16:24,890
And just one quick look over.

800
01:16:27,010 --> 01:16:28,830
Yep, I have everything in.

801
01:16:29,430 --> 01:16:31,350
That's exactly how it's supposed to be.

802
01:16:31,910 --> 01:16:33,230
Scroll up.

803
01:16:33,870 --> 01:16:35,190
Oh, this is still going.

804
01:16:35,190 --> 01:16:35,830
Look at that.

805
01:16:38,990 --> 01:16:39,870
Weird.

806
01:16:41,250 --> 01:16:42,890
I'm going to let that sit for a bit then.

807
01:16:43,270 --> 01:16:45,090
So this one, I'm going to let you guys write down.

808
01:16:45,090 --> 01:16:47,590
This is task 6D.

809
01:16:48,310 --> 01:16:53,130
So go ahead and make sure that you have these written down while the other one is still going.

810
01:16:53,130 --> 01:16:59,970
I'm going to run this and then take a look at a logistic regression with the TF-IDF .

811
01:17:04,570 --> 01:17:06,790
We're actually doing really good on time.

812
01:17:06,790 --> 01:17:08,610
We have one more task after this.

813
01:17:08,610 --> 01:17:12,350
It is going to be the most challenging task.

814
01:17:12,630 --> 01:17:20,990
But it is going to take everything from A to Z and show us what we learned.

815
01:17:23,940 --> 01:17:27,140
So I just want to check one last time.

816
01:17:27,220 --> 01:17:28,940
And that is still going.

817
01:17:32,740 --> 01:17:35,880
So I'm going to go ahead and run this.

818
01:17:37,060 --> 01:17:40,880
This was 6D that I just clicked to run.

819
01:17:44,420 --> 01:17:46,840
It says the kernel is idle.

820
01:17:46,840 --> 01:17:48,020
That's not good.

821
01:17:52,780 --> 01:17:54,980
Let me see if I can pause this.

822
01:17:59,080 --> 01:18:00,720
Let me just interrupt it.

823
01:18:00,980 --> 01:18:01,920
Interrupt the kernel.

824
01:18:05,540 --> 01:18:07,780
So I'm going to run this because that took a second.

825
01:18:11,470 --> 01:18:13,330
Uh oh, I hope I didn't time out.

826
01:18:14,190 --> 01:18:15,650
That would be very bad.

827
01:18:30,950 --> 01:18:32,370
Let's try and run it again.

828
01:18:34,230 --> 01:18:35,690
Yeah, that's not good.

829
01:18:42,270 --> 01:18:44,150
Restart and clear output.

830
01:18:47,180 --> 01:18:49,620
So this is going to go all the way to the beginning.

831
01:18:51,140 --> 01:18:52,680
Let's see if I can get this to run.

832
01:18:52,680 --> 01:18:54,120
Okay, so that runs now.

833
01:18:55,180 --> 01:18:57,520
So I ran into a bit of a technical difficulty.

834
01:18:57,640 --> 01:19:04,860
Instead of going through and filling everything out, I'm just going to quickly grab the completed workbook.

835
01:19:06,260 --> 01:19:15,480
And I'm going to just go ahead, run all.

836
01:19:17,180 --> 01:19:21,120
This is going to run everything for me.

837
01:19:22,860 --> 01:19:34,980
So while I'm waiting for some of this stuff to run, let's see that packages are being downloaded, tokenization is defined, everything's good, everything's good, everything's good.

838
01:19:42,490 --> 01:19:44,010
Where is it stuck now?

839
01:19:44,010 --> 01:19:46,790
It is stuck...

840
01:19:50,540 --> 01:19:52,520
Looks like it's stuck reading the data.

841
01:19:53,720 --> 01:19:55,400
Training spam, training ham.

842
01:20:03,010 --> 01:20:05,270
That one's done, done.

843
01:20:08,370 --> 01:20:11,970
Yeah, when you see numbers here, that means that that was the order it was ran in.

844
01:20:11,970 --> 01:20:14,890
When you see a star, that means that it's still running.

845
01:20:15,170 --> 01:20:23,730
I'm not sure why the logistic regression was causing problems, but let's take a quick look at...

846
01:20:28,190 --> 01:20:29,430
Oh.

847
01:20:30,310 --> 01:20:32,450
Looks like there's some stuff missing.

848
01:20:37,160 --> 01:20:38,820
Well, that's unfortunate.

849
01:20:43,000 --> 01:20:47,180
So I'm just going to go ahead and create new code blocks.

850
01:20:47,700 --> 01:20:55,940
I'm just going to copy the code we had from task 6D and 6C.

851
01:20:57,080 --> 01:21:00,220
I apologize for scrolling so fast.

852
01:21:01,300 --> 01:21:07,040
So let's go ahead and grab all of that, paste it in, run it.

853
01:21:07,940 --> 01:21:10,460
Logistic regression is not defined.

854
01:21:12,000 --> 01:21:13,180
That's weird.

855
01:21:17,240 --> 01:21:18,860
Oh, it's not even in here.

856
01:21:18,860 --> 01:21:21,440
That is something I'm going to have to fix.

857
01:21:21,440 --> 01:21:23,400
I'm not sure... Oh.

858
01:21:24,640 --> 01:21:27,220
Because that's a whole different workbook.

859
01:21:27,220 --> 01:21:30,960
I need complete spam filter sklearn.

860
01:21:30,960 --> 01:21:32,640
That's the one that I need.

861
01:21:33,840 --> 01:21:35,560
My mistake.

862
01:21:36,100 --> 01:21:38,920
So kernel, restart, run all.

863
01:21:41,880 --> 01:21:44,800
So this is going to run all of them.

864
01:21:46,000 --> 01:21:49,100
Let's see where it gets stuck.

865
01:21:49,760 --> 01:21:51,480
Prints out the emails.

866
01:21:52,340 --> 01:21:54,160
It's training the vectorizers.

867
01:21:54,160 --> 01:21:56,780
I remember that took about a minute or so.

868
01:21:59,000 --> 01:22:12,560
But what we want to see after task 6D, we'll look at 6C and 6D, is at the end of this, what we're going to do is grab a spam email.

869
01:22:12,560 --> 01:22:17,680
This one is actually one that I grabbed directly from my spam box last year.

870
01:22:18,020 --> 01:22:24,340
And I like it because we can use our spam email predictor to identify whether or not it's spam.

871
01:22:24,340 --> 01:22:31,300
And then we can look and utilize any of the models to really identify that.

872
01:22:33,820 --> 01:22:40,980
So this was supposed to be a fill-in-the-blank kind of thing like we've been going through.

873
01:22:41,460 --> 01:22:56,740
But this last task, it just asks us use the vectorizers and the models created to perform predictions on the test email that we created up at the top and the test spam email which is the real world email that I pulled from a spam inbox.

874
01:22:57,620 --> 01:23:03,340
And so what we want to do is grab that as a working email.

875
01:23:03,340 --> 01:23:08,660
And this will let us change this to test email and test spam email.

876
01:23:10,200 --> 01:23:19,600
Remember, we want to use the tf.idf and the count vectorizers in order to use the working email.

877
01:23:19,600 --> 01:23:31,140
We need to use that to preprocess and because it's a string, we just wrap it in these square brackets in order to classify it as a list because the transform function expects a list.

878
01:23:31,800 --> 01:23:36,520
We have that as test email, tf.idf, and test email count.

879
01:23:37,940 --> 01:23:41,100
And then we use each of our models to form a prediction.

880
01:23:41,100 --> 01:23:50,860
So we have the multinomial Naive Bayesian with tf.idf, multinomial with count, logistic regression tf.idf, and logistic regression with count.

881
01:23:50,860 --> 01:23:53,000
They all use the same predict function.

882
01:23:53,000 --> 01:24:07,580
They're all using either the tf.idf vectorized version of the email or the count vectorized version of the email based on whether or not it's the tf.idf or the count version of the model.

883
01:24:08,820 --> 01:24:20,800
And then we make our predictions, we print the email itself, and then we can print what each model thinks it is.

884
01:24:22,660 --> 01:24:29,940
So if you're still on the fill in the blank bit, I encourage you to fill this out.

885
01:24:30,360 --> 01:24:42,940
If you are using the completed workbook, like I am, it's already filled out and we're just going to go ahead and let everything else run in the background until we get to this point.

886
01:24:45,840 --> 01:24:47,800
So, sorry about that.

887
01:24:49,360 --> 01:24:55,900
In the meantime, are there any questions, any thoughts, concerns, ideas?

888
01:24:55,900 --> 01:24:57,640
I'd love to hear from you.

889
01:25:15,770 --> 01:25:17,210
All right.

890
01:25:17,630 --> 01:25:26,510
Psychdude says, on the completed workbook 6a, show the top right and bottom left numbers swapped in 6a.

891
01:25:26,510 --> 01:25:27,790
Wait a second.

892
01:25:27,810 --> 01:25:33,530
But the one you were working on earlier had those flipped.

893
01:25:33,530 --> 01:25:40,430
The fill in the blank and the completed, were they different somehow?

894
01:25:40,430 --> 01:26:00,950
Yeah, so I'm thinking the reason why they're swapped is because there was a way that Scikit-learn documented how they did their confusion matrix versus how they actually did their confusion matrix, and it was swapped from what you would normally see.

895
01:26:01,370 --> 01:26:06,610
And so, following what you would normally see is what I did, and that's where the mistake came in.

896
01:26:06,610 --> 01:26:08,650
And I can actually show you that.

897
01:26:08,950 --> 01:26:11,410
I was hoping nobody noticed, but good eye.

898
01:26:12,930 --> 01:26:18,090
So if you look at 6, actually before we go there, let's take a look and see the count.

899
01:26:18,090 --> 01:26:25,090
So we have 590 ham emails in our test dataset, 276 spam emails in our test dataset.

900
01:26:25,290 --> 01:26:38,830
And so if we look at 6A, we see 590 ham emails, but as far as actual label, we should see in the column for spam a total of the 276.

901
01:26:38,830 --> 01:26:40,870
We don't see that.

902
01:26:40,950 --> 01:26:51,750
And the reason why is because this is supposed to be the actual, see 108 plus 168 is 276, and then this was supposed to be the predicted.

903
01:26:51,790 --> 01:27:00,750
The problem is the way that scikit-learn swapped them, and I guess I did not swap them back.

904
01:27:02,290 --> 01:27:13,630
To get the correct result, you just need to switch the test labels and the predictions MNB on both sides for the confusion matrix and the classification report.

905
01:27:14,630 --> 01:27:15,490
So, yeah.

906
01:27:15,490 --> 01:27:17,350
That's why they seem swapped.

907
01:27:17,350 --> 01:27:29,650
The one on the completed workbook is correct, because I believe the completed workbook shows in the actual label column the spam emails, and if you total the two numbers, you get 276.

908
01:27:29,670 --> 01:27:31,510
So, good eye on that.

909
01:27:34,980 --> 01:27:36,740
Go ahead and check this.

910
01:27:36,980 --> 01:27:37,760
Perfect.

911
01:27:41,220 --> 01:27:48,100
Before we ran into the problems, we're going to look at 6C and 6D with a logistic regression.

912
01:27:48,720 --> 01:27:58,440
Remember, we had 0.85 or 0.87 0.5% accuracy for the TF-IDF multinomial Naive Bayesian.

913
01:27:58,440 --> 01:28:02,580
For the count multinomial Naive Bayesian, we have 0.94.

914
01:28:03,480 --> 01:28:06,320
For the logistic regression, we get a little bit better.

915
01:28:06,320 --> 01:28:10,300
We're at 0.959 as our accuracy.

916
01:28:10,580 --> 01:28:19,260
We have fewer ham emails being classified as spam and fewer spam emails being classified as ham.

917
01:28:19,260 --> 01:28:21,040
We're getting better.

918
01:28:23,060 --> 01:28:29,320
With the task 6D, this is logistic regression with count vectorizer.

919
01:28:29,520 --> 01:28:34,040
This is going to be our best model at 0.976.

920
01:28:34,800 --> 01:28:42,420
Here we only have 8 of the ham emails being classified as spam and 12 of the spam emails being classified as ham.

921
01:28:42,420 --> 01:28:50,340
Of the four models that we trained, this is the better one in terms of accuracy.

922
01:28:51,320 --> 01:28:58,940
This is why we want to try different algorithms, different machine learning algorithms, as well as different ways that we preprocess or vectorize our data.

923
01:28:59,520 --> 01:29:05,620
That way we can create different models, see which ones perform the best, and then we can gravitate towards that one.

924
01:29:09,800 --> 01:29:15,300
As far as the test spam email, this is a brand new email.

925
01:29:15,400 --> 01:29:17,060
It comes in on our email server.

926
01:29:17,060 --> 01:29:20,380
We want to use our models to classify it.

927
01:29:22,480 --> 01:29:25,140
Let's go ahead and make a prediction.

928
01:29:26,140 --> 01:29:33,340
As I went over earlier, we're going to have our test spam email, which is called working email, so we can swap this out pretty seamlessly.

929
01:29:33,580 --> 01:29:39,200
We're going to use our vectorizers that we already defined to transform our working email.

930
01:29:39,500 --> 01:29:42,300
And then of course, being a string , we need to put it in a list.

931
01:29:42,300 --> 01:29:44,100
That's what the square brackets are for.

932
01:29:44,460 --> 01:29:51,260
Then we use all four of our models, the multinomial and the logistic regression with the tf and the count vectorizers.

933
01:29:51,260 --> 01:29:55,860
We use the predict function on the vectorized emails.

934
01:29:57,540 --> 01:30:00,940
And then we just go ahead and print out the working email itself.

935
01:30:01,120 --> 01:30:02,960
Your latest issue is available now.

936
01:30:02,960 --> 01:30:05,920
If you don't want issue notifications, click here to unsubscribe.

937
01:30:07,440 --> 01:30:13,800
And then we print out the values that are predicted from each of our models.

938
01:30:14,140 --> 01:30:16,480
If we look at it, we're going to see something interesting.

939
01:30:20,020 --> 01:30:27,560
Our best performing model, which is the logistic regression with the count vectorizer, correctly classified this as a spam email.

940
01:30:27,640 --> 01:30:38,660
But our worst performing model, even though it still had an 87.5% accuracy, our multinomial Naive Bayesian with tf.idf incorrectly classified this as a ham email.

941
01:30:39,620 --> 01:30:45,760
And so this is also why we want to try building different models and get things that predict much better.

942
01:30:51,020 --> 01:30:53,700
Let me know if there's any questions on that.

943
01:30:54,220 --> 01:30:55,920
We are short on time.

944
01:30:55,920 --> 01:30:59,840
We're actually running a little over, but before I let you go, there's one more thing I want to show you.

945
01:31:00,040 --> 01:31:03,420
And that is the hyperparameter tuning.

946
01:31:03,660 --> 01:31:13,940
So if we jump up to that worst performing model, that 0.85%, the multinomial with tf.idf, sorry, 0.875%.

947
01:31:15,160 --> 01:31:20,500
And we use that alpha that we saw in our formulas on the slides.

948
01:31:21,320 --> 01:31:23,040
Alpha is set to 1.

949
01:31:23,040 --> 01:31:24,980
We can go ahead and run that.

950
01:31:25,060 --> 01:31:27,560
And we see that we're at 0.875.

951
01:31:28,060 --> 01:31:31,740
But what if we shrink that alpha to 0.1?

952
01:31:31,740 --> 01:31:33,560
This is hyperparameter tuning.

953
01:31:33,560 --> 01:31:34,840
We can change it.

954
01:31:35,780 --> 01:31:42,320
Now our accuracy changed from one of the worst models that we have to one of the better models that we have.

955
01:31:42,640 --> 01:31:44,480
In fact, this is second place.

956
01:31:44,660 --> 01:31:48,060
So we have 0.965% accuracy.

957
01:31:49,480 --> 01:31:55,960
And so you can see how hyperparameter tuning can really help out in tweaking the way that our models operate.

958
01:32:03,130 --> 01:32:04,870
Before I let you go,

959
01:32:08,830 --> 01:32:12,550
here's some references for more information.

960
01:32:12,770 --> 01:32:34,210
And if you want the slides to this and more information from me and the GitHub project that this is all part of, check out github.com.netsec explain and you can shoot me an email with any questions, thoughts, comments at gtklondike at gmail.com So that's all I have for you guys.

961
01:32:34,210 --> 01:32:40,750
If there's any discussion, TAs, do we have a separate room for the workshops or are we just going to use this Discord?

962
01:32:41,570 --> 01:32:45,310
Nope, there is a separate room for the workshops here.

963
01:32:45,310 --> 01:32:49,010
Let me post that in Twitch.

964
01:32:51,430 --> 01:32:51,910
Um...

965
01:32:53,190 --> 01:32:54,350
There you go, folks.

966
01:32:54,350 --> 01:33:05,170
If you have any other questions or want to continue talking about this workshop, you can find us at that channel in the AI Village in the DC Discord.

967
01:33:06,070 --> 01:33:07,030
Perfect.

968
01:33:07,670 --> 01:33:10,370
And then I'm going to go ahead and end the stream.

969
01:33:10,370 --> 01:33:13,690
I'm going to hop into that channel and I will see you guys there.

970
01:33:15,110 --> 01:33:16,310
Thanks a lot, Gavin.

971
01:33:16,490 --> 01:33:17,370
Absolutely.

972
01:33:17,370 --> 01:33:18,230
Thank you.