1
00:00:00,000 --> 00:00:05,620
I just want to double check this.

2
00:00:06,280 --> 00:00:07,420
Alright, cool.

3
00:00:07,420 --> 00:00:10,840
I have all my stuff muted and we are good to go.

4
00:00:10,960 --> 00:00:15,020
Hello and welcome to Machine Learning for Security Analysts.

5
00:00:15,020 --> 00:00:18,640
This is going to be our final workshop for the AI village.

6
00:00:19,020 --> 00:00:28,700
This is going to be a beginner and interactive workshop for all those who are interested in machine learning and security and what this whole crazy AI thing is.

7
00:00:28,700 --> 00:00:38,360
So really, the major motivation for this workshop was that we're seeing a lot of buzz in the security community around AI and machine learning.

8
00:00:38,360 --> 00:00:42,640
But for many of us, it's not really clear what that is.

9
00:00:43,460 --> 00:00:46,620
So... oh, sorry.

10
00:00:46,620 --> 00:00:55,780
If you want to participate and ask questions, head over to the DevSchool channel and join the AIVillage-General-Voice, AIV-General-Voice.

11
00:00:55,960 --> 00:00:58,680
And you'll be able to ask us questions.

12
00:00:59,280 --> 00:01:03,900
If there's too many people joining and it gets unorganized, we're going to switch to Zoom.

13
00:01:03,900 --> 00:01:08,140
But it's sort of more open than it was before.

14
00:01:09,000 --> 00:01:10,160
Sorry, Gavin.

15
00:01:10,160 --> 00:01:11,920
Just needed to say that.

16
00:01:12,260 --> 00:01:12,920
No worries.

17
00:01:12,920 --> 00:01:20,720
Last line of this statement is I wanted to take, for the security community, the idea of machine learning and AI out of the buzzword and into the mainstream.

18
00:01:22,040 --> 00:01:23,480
So, my name is Gavin.

19
00:01:23,480 --> 00:01:25,520
I go by the pseudonym GT Klondike.

20
00:01:25,520 --> 00:01:28,680
I'm an independent security researcher and a security consultant.

21
00:01:28,700 --> 00:01:32,720
I'm very passionate about network attack and defense.

22
00:01:32,720 --> 00:01:45,800
And through that passion, I run a project called NetSecExplain, which is a blog and a YouTube channel, where I explain intermediate and advanced level computer networking concepts and security concepts in an easy-to-understand way.

23
00:01:47,420 --> 00:01:54,660
So, before I get started, usually in person, this is where I ask people, what are your thoughts when you hear the word machine learning?

24
00:01:54,760 --> 00:01:55,380
Right?

25
00:01:55,380 --> 00:02:06,620
And I've gotten a range of answers in the past that range from, oh, Skynet's going to take over the world, to you need a PhD in multivariable calculus and linear algebra.

26
00:02:07,700 --> 00:02:16,520
And while there is a mild amount of truth to the math background that you should have, neither of those are really true.

27
00:02:17,920 --> 00:02:21,220
Instead, machine learning is really a form of pattern matching.

28
00:02:21,220 --> 00:02:22,740
It's applied statistics.

29
00:02:22,740 --> 00:02:34,380
And with the increase in power behind the computer memory and behind the CPU, we're able to use these statistics in a much quicker, more processed way.

30
00:02:35,980 --> 00:02:41,920
So, first I want to kind of cover this idea of machine learning, artificial intelligence, and deep learning.

31
00:02:41,920 --> 00:02:46,960
In a lot of areas, it's being used interchangeably, and that's not really correct.

32
00:02:47,240 --> 00:02:51,620
First, what we have is this large umbrella of artificial intelligence.

33
00:02:51,620 --> 00:02:58,340
And artificial intelligence is kind of a philosophy, in my opinion.

34
00:02:58,340 --> 00:03:05,540
Artificial intelligence used to be, when the first digital calculators came out, that was considered artificial intelligence.

35
00:03:05,540 --> 00:03:07,380
A computer can do math.

36
00:03:07,400 --> 00:03:13,040
As things became more advanced, different concepts were considered artificial intelligence.

37
00:03:13,040 --> 00:03:18,760
There's video game artificial intelligence, rule-based artificial intelligence, classical image recognition.

38
00:03:20,200 --> 00:03:24,840
But today we're talking about machine learning, which is a subset of artificial intelligence.

39
00:03:24,840 --> 00:03:32,320
And it's focused a lot more on grabbing inferences from data and making statistical patterns.

40
00:03:33,680 --> 00:03:38,020
And then inside of machine learning, we have a concept called deep learning.

41
00:03:38,020 --> 00:03:41,520
Deep learning is specifically deep neural networks.

42
00:03:41,960 --> 00:03:47,180
And so neural networks is a machine learning algorithm that's pretty popular.

43
00:03:47,180 --> 00:03:50,200
But deep learning allows us to take that a little bit further.

44
00:03:50,200 --> 00:04:01,400
And so usually we see this in the form of convolutional neural networks, which are primarily used for image recognition, or recurrent neural networks, which are used for cyclical pattern recognition.

45
00:04:01,400 --> 00:04:10,020
So think of things such as a heartbeat monitor, where there's a pattern involved that you kind of need to match up.

46
00:04:10,020 --> 00:04:16,320
And if there's any sort of heart irregularities, the pattern will diverge from predicted patterns.

47
00:04:17,960 --> 00:04:29,480
So if we wanted to formally define machine learning, I would say that it's a set of statistical techniques that enable the process of information mining, pattern discovery, and drawing inferences from data.

48
00:04:29,540 --> 00:04:40,080
So the idea is that the algorithms learn from past data to predict future outcomes, instead of creating a different algorithm for a different scenario.

49
00:04:40,400 --> 00:04:53,640
And we'll see that as we go through the workbooks, how we're able to create multiple different types of algorithms around the same set of data, or how we can use the same algorithms with different types of data.

50
00:04:54,440 --> 00:04:59,920
So some key examples, especially in the security sphere, is domain generation algorithms.

51
00:04:59,920 --> 00:05:14,580
Domain generation algorithms are heavily used by botnets so that the bot herder can stay in contact with the botnet itself, while government agencies or ISPs are taking down the domains that they use to communicate.

52
00:05:14,940 --> 00:05:17,680
We also see this with web application firewalls.

53
00:05:17,680 --> 00:05:30,260
We can perform anomaly detection to identify normal web traffic, normal parameters that are sent by a web client, versus potentially malicious parameters.

54
00:05:30,260 --> 00:05:36,940
In this example here, it's showing a possible SQL injection, and it's calling that out.

55
00:05:37,220 --> 00:05:50,780
And then, of course, network anomaly detection, where what we can do is predict how the network will operate over a period of time, and then keep track of how it is actually operating.

56
00:05:50,780 --> 00:05:56,740
And if it's beyond a certain delta or a difference, then we can flag that and send it off to a security analyst.

57
00:05:58,040 --> 00:06:02,200
So why is this important for security analysts, and also important for hackers?

58
00:06:02,700 --> 00:06:08,640
Well, today, over a quarter of security products for detection have some form of machine learning.

59
00:06:08,800 --> 00:06:18,280
So from an analyst's perspective, what you want to do is understand how these machines work, so that you can ensure that they operate efficiently and are working effectively.

60
00:06:19,100 --> 00:06:23,180
Machine learning isn't this magical black box that just solves all of our problems.

61
00:06:23,920 --> 00:06:31,020
And I want to share a story with you, and this is what really made the idea of machine learning and the value of machine learning really stick out to me.

62
00:06:31,480 --> 00:06:35,420
So a number of years ago, a friend of mine was approached by a non-profit.

63
00:06:35,420 --> 00:06:39,180
And what this non-profit did was track cheetah population out in Africa.

64
00:06:39,180 --> 00:06:49,900
So they set up a bunch of different cameras, and they would take a picture at different intervals, and it would be sent off to a human analyst to identify whether or not there's a cheetah in the picture.

65
00:06:50,420 --> 00:06:54,880
Not a job I would want to do, but it needs done, and it's really important.

66
00:06:56,100 --> 00:07:00,260
So they approached my friend, and they were saying, hey, is there a way that we can increase our efficiency?

67
00:07:00,260 --> 00:07:04,780
Something that we can use to kind of automate some of the manual labor that we're doing.

68
00:07:05,180 --> 00:07:13,140
And so what he did was spend about a week, and he used a very simple machine learning model to identify images.

69
00:07:13,200 --> 00:07:23,740
And what he told this model was, hey, if you see an image, and you are at least 5% confident that there's a cheetah in the picture, send it off to an analyst.

70
00:07:23,740 --> 00:07:26,920
If you're less than 5% confident, just throw it away.

71
00:07:27,640 --> 00:07:31,160
Now 5%, that doesn't really sound like a lot, and to be honest, it's not.

72
00:07:31,320 --> 00:07:41,980
But what this non-profit was able to do with this simple model was what would normally take them a year of labor they were able to complete in one month.

73
00:07:42,620 --> 00:07:52,320
That is a 1200% increase in productivity by just implementing a little bit of machine learning and that fuzzy logic into their process to kind of help automate it.

74
00:07:52,860 --> 00:08:09,060
So this is what I want to do to kind of push for the security analysts and for the hacker community to get a little bit more involved in AI and machine learning so that we can see and identify ways that we can improve our own processes, especially ones that require a little bit of fuzzy logic,

75
00:08:09,060 --> 00:08:14,160
things that traditional signatures or scripting aren't sufficient in.

76
00:08:16,380 --> 00:08:20,500
So the 7-step machine learning process generally looks like this.

77
00:08:20,660 --> 00:08:25,560
And this isn't a list so much as a cycle, right?

78
00:08:26,560 --> 00:08:28,980
So first we're going to start with gathering the data.

79
00:08:28,980 --> 00:08:35,660
We want to identify what it is we want our machine learning algorithm to classify.

80
00:08:35,960 --> 00:08:43,100
So first we're going to gather the data and then we're going to prepare the data in such a way that is easy for machine learning models to understand.

81
00:08:43,300 --> 00:08:47,440
And in this case, it's usually turning it into some form of number or some form of signal.

82
00:08:48,780 --> 00:08:51,860
Most of our time is spent gathering the data and preparing the data.

83
00:08:51,860 --> 00:08:54,280
The rest of this tends to be a little bit quicker.

84
00:08:55,160 --> 00:09:00,260
So first, based on the data that we have and the problem set that were presented, we choose a model.

85
00:09:00,260 --> 00:09:03,420
So some models perform better than others in certain tasks.

86
00:09:03,420 --> 00:09:04,720
And sometimes you just got to kind of

87
00:09:07,880 --> 00:09:10,120
figure out maybe neural nets isn't the best model for you.

88
00:09:10,120 --> 00:09:15,220
Maybe logistic regression or SVM or random forest classifier is better for you.

89
00:09:15,680 --> 00:09:19,560
But once we've identified the model or models, then we go ahead and train it.

90
00:09:19,560 --> 00:09:23,600
And so what we'll usually do is perform what's called a training testing split.

91
00:09:23,600 --> 00:09:29,320
And in this case, we have a series or a lot of clean labeled data.

92
00:09:29,480 --> 00:09:33,440
And we take that data, we separate 80% of it and 20% of it.

93
00:09:33,440 --> 00:09:35,880
80% of it is what we use to train the model.

94
00:09:35,880 --> 00:09:41,940
And then the other 20% is what is used to evaluate the model to see how well the model is performing.

95
00:09:42,840 --> 00:09:47,160
And so it's kind of like how we teach students in elementary school, right?

96
00:09:47,160 --> 00:09:53,140
We train them on the bulk of information and then we test them to see how well they're understanding the material.

97
00:09:54,040 --> 00:09:57,740
And so some of our models will perform very well.

98
00:09:57,740 --> 00:10:00,640
Some of our models may not.

99
00:10:00,640 --> 00:10:08,260
And so if we have a model that performs fairly well, but we think it could do better, then we hop into what is called hyperparameter tuning.

100
00:10:08,500 --> 00:10:11,600
And we're going to see what a hyperparameter looks like in a little bit.

101
00:10:11,880 --> 00:10:24,160
But hyperparameter tuning, think of it as little knobs and levers and switches that we can use to kind of tweak our model and tune it in such a way that it is better at a certain type of problem.

102
00:10:24,980 --> 00:10:27,680
And then once we've completed all this, we're ready to deploy.

103
00:10:28,000 --> 00:10:30,040
Now the reason why this is cyclical, right?

104
00:10:30,040 --> 00:10:42,860
We want to gather more data, prepare the data, and use the same model that we're already using after deploy is because machine learning stops learning as soon as it's trained.

105
00:10:43,300 --> 00:10:58,260
So to kind of give you an idea, I know that certain spam filters like Facebook's Facebook spam filter, they retrain it and rebuild it every one to three days because that's how quickly spam is being processed through their pipelines and through their networks.

106
00:10:58,320 --> 00:11:01,420
And so it's a way for them to kind of stay on top of that.

107
00:11:04,280 --> 00:11:10,240
So we're going to hop into understanding how machine learning works, especially in the security space.

108
00:11:10,240 --> 00:11:14,040
And the easiest example, in my opinion, is building a spam filter.

109
00:11:14,600 --> 00:11:17,900
Now spam filters, we already know how they operate conceptually.

110
00:11:18,180 --> 00:11:20,660
So it's a lower barrier of entry.

111
00:11:20,660 --> 00:11:24,580
We're going to see specifically how the machine learning fits in.

112
00:11:24,760 --> 00:11:39,220
But before I hop into that, are there any questions on how machine learning, AI, and deep learning kind of intermingle and why it's important to understand machine learning from a security standpoint or in the hacking community?

113
00:11:46,490 --> 00:11:54,110
I see on the Twitch chat, reinforcement learning, reinforcement learning, self-learning, contextual regret minimization, what differences between them?

114
00:11:54,410 --> 00:12:07,970
I'm not familiar with counterfactual, not familiar with counterfactual regret minimization, but typically you'll see three types of machine learning.

115
00:12:07,970 --> 00:12:12,490
You'll see classification or supervised learning, which is what we're going to be covering.

116
00:12:12,490 --> 00:12:17,430
You'll see unsupervised learning, which is very good for data exploration.

117
00:12:18,870 --> 00:12:21,990
And unsupervised learning uses a whole different set of algorithms.

118
00:12:21,990 --> 00:12:32,410
And then there's reinforcement learning, where we embed in a model different rewards and punishments based on how it's performing.

119
00:12:32,690 --> 00:12:37,590
And so these are all different ways of training the model, and then we can make decisions off of those.

120
00:12:39,830 --> 00:12:45,070
Hey, Sven, are there any questions in the Discord chat before I move on?

121
00:12:55,840 --> 00:12:57,780
Okay, I'm going to assume not.

122
00:12:59,480 --> 00:13:04,740
So before we look at a full spam filter, we want to kind of look at a smaller example.

123
00:13:04,880 --> 00:13:11,800
So in this case, we have a series of sentences talking about sports or not sports, which are really elections.

124
00:13:12,260 --> 00:13:13,980
So here we have a couple sentences.

125
00:13:13,980 --> 00:13:14,920
A, great games.

126
00:13:15,100 --> 00:13:15,980
Sorry.

127
00:13:16,060 --> 00:13:16,760
Yes.

128
00:13:18,260 --> 00:13:20,420
Well, I think you got this one.

129
00:13:21,340 --> 00:13:23,200
But there was a question from Norwin.

130
00:13:23,200 --> 00:13:24,680
I fell asleep on the job.

131
00:13:26,340 --> 00:13:27,660
No worries.

132
00:13:27,780 --> 00:13:29,280
It has been a long weekend.

133
00:13:30,260 --> 00:13:31,780
So what was the question?

134
00:13:34,340 --> 00:13:38,680
Norwin kindly repeated Glucorio's question.

135
00:13:40,200 --> 00:13:42,060
So you already answered it.

136
00:13:42,060 --> 00:13:43,320
Okay, okay.

137
00:13:45,000 --> 00:13:49,260
So kind of going back, we have a great game, which is talking about sports.

138
00:13:49,260 --> 00:13:53,200
We have the election was over, which is talking about not sports.

139
00:13:53,240 --> 00:13:55,680
Very clean match, talking about sports.

140
00:13:55,680 --> 00:13:58,400
Clean but forgettable game, talking about sports.

141
00:13:58,400 --> 00:14:08,500
So if we wanted to take this idea and specify a sentence that we've never seen before about sports and not sports, how would we do this?

142
00:14:08,700 --> 00:14:11,640
We would want to look at keywords.

143
00:14:11,900 --> 00:14:16,380
So in this case, we can see that it's talking about games.

144
00:14:16,380 --> 00:14:18,060
We know it's probably about sports.

145
00:14:18,060 --> 00:14:21,720
If it's talking about elections, it's probably not talking about sports.

146
00:14:22,500 --> 00:14:26,640
And so what we want to really hone in on are keywords.

147
00:14:26,940 --> 00:14:33,500
But to put this into a machine learning model, we need to find a way to turn this into numbers.

148
00:14:34,660 --> 00:14:37,920
So the way that we're going to do this is Bayes' Theorem.

149
00:14:37,920 --> 00:14:39,380
Bayes' Theorem is...

150
00:14:40,030 --> 00:14:44,800
it may look complex at first, but it's actually a really easy to understand probability theorem.

151
00:14:45,480 --> 00:14:53,540
And in this case, we see the probability of A given B equals probability of B given A times the probability of A divided by the probability of B.

152
00:14:53,540 --> 00:15:02,800
So in this case, we want to see what is the probability that this sentence is talking about sports given that the sentence is A very close game.

153
00:15:03,280 --> 00:15:11,780
So we don't know the probability of sports given A very close game, but we can easily plug that into this formula and say, well, we know the probability of sports.

154
00:15:11,780 --> 00:15:15,120
We saw three out of our five sentences were talking about sports.

155
00:15:15,120 --> 00:15:21,500
And we can figure out the probability of A very close game given sports and the probability of A very close game.

156
00:15:22,200 --> 00:15:23,140
But how do we do that?

157
00:15:24,220 --> 00:15:25,540
How do we get a sentence like this?

158
00:15:26,000 --> 00:15:31,020
Well, with Bayes theorem, we can actually, or with probability, actually, we can split this up.

159
00:15:31,020 --> 00:15:39,160
So instead of A very close game as a whole sentence, we can look at the probability of A times the probability of very times close times game.

160
00:15:39,640 --> 00:15:49,840
And so what this turns into in our A very close game given sports is probability of A in sports, probability of very in sports, and so on.

161
00:15:49,840 --> 00:15:50,920
And of course, we're going to want to do

162
00:15:54,220 --> 00:16:02,140
is say, okay, whichever is highest, sports or not sports, that's the one that we're going to classify this sentence as being part of.

163
00:16:03,380 --> 00:16:06,260
So at this point, it becomes a simple counting game, right?

164
00:16:06,260 --> 00:16:10,160
So how many times does the word A show up in sports?

165
00:16:10,400 --> 00:16:14,520
Well, we see that it shows up twice, once in the first sentence and once in the fourth sentence.

166
00:16:14,520 --> 00:16:17,340
And then how many times does it show up in not sports?

167
00:16:17,540 --> 00:16:19,600
Once, right there in the fifth sentence.

168
00:16:24,220 --> 00:16:25,360
So A does not show up in sports.

169
00:16:25,360 --> 00:16:26,920
It only shows up in not sports.

170
00:16:26,920 --> 00:16:30,040
And then we see the word game show up twice in sports.

171
00:16:31,740 --> 00:16:41,180
So we just take those numbers and we plug them in and we say, okay, A showed up in sports twice, very showed up once, close showed up zero times, and game showed up twice.

172
00:16:41,280 --> 00:16:48,020
And if we counted the number of unique words, we would have had 11 unique words across all of our sentences talking about sports.

173
00:16:48,880 --> 00:16:50,240
But we run into a problem

174
00:16:54,620 --> 00:17:02,400
and the problem is that what we want our machine learning model to do is identify and classify sentences that it has never seen before.

175
00:17:02,960 --> 00:17:08,380
Unfortunately, it's never seen close and so that winds up being a zero, zero times everything is zero.

176
00:17:08,380 --> 00:17:11,380
And we see that that easily cascades down and just ruins our model.

177
00:17:11,380 --> 00:17:17,180
So it's because we've never seen the word close, there's a zero probability that this exists in our sentence.

178
00:17:18,080 --> 00:17:21,480
So that's a problem, but we can get around that.

179
00:17:22,700 --> 00:17:26,060
There's another algorithm called multinomial Naive Bayes.

180
00:17:26,180 --> 00:17:31,140
And so this is just a small modification and what it does is add what is called a smoothing filter.

181
00:17:31,140 --> 00:17:33,080
And we can see this alpha here.

182
00:17:33,080 --> 00:17:35,740
It's added to the numerator and the denominator.

183
00:17:35,800 --> 00:17:44,540
And usually this alpha is one, but this is going to be our hyperparameter for Naive Bayesian, multinomial Naive Bayesian that we can tweak, right?

184
00:17:44,540 --> 00:17:47,900
We can set it to one, we can set it to two, we can set it to 0.1.

185
00:17:48,040 --> 00:17:54,660
And so this is a thing that we can tweak to see if we can get our model to work for us .

186
00:17:55,440 --> 00:17:56,780
So we take the same idea.

187
00:17:56,780 --> 00:17:57,680
We do the word counts.

188
00:17:57,680 --> 00:18:01,360
We saw that a showed up in sports twice, very showed up once, close to zero.

189
00:18:01,360 --> 00:18:03,180
But now we have this plus one.

190
00:18:03,580 --> 00:18:08,480
So the probability of close given sports is never going to reach zero.

191
00:18:09,180 --> 00:18:13,060
And then of course, as I mentioned earlier, we need to do the same thing for not sports.

192
00:18:13,060 --> 00:18:16,280
And the total number of unique words in not sports is nine.

193
00:18:16,280 --> 00:18:18,600
The total number of unique words in sports is 11.

194
00:18:18,600 --> 00:18:23,320
And the total number of unique words across both classes is 14.

195
00:18:26,080 --> 00:18:37,880
So we plug in our answers and we do our calculations and we get 0.0000461 for sports and 0.0000143 for non-sports.

196
00:18:38,200 --> 00:18:42,800
Classifying the sentence, a very close game being about sports.

197
00:18:45,480 --> 00:18:48,320
So that was a lot that I just kind of laid on you.

198
00:18:49,080 --> 00:18:51,660
If you have any questions, type them in the chat.

199
00:18:52,020 --> 00:19:06,500
While you're doing that, I'm going to show you how we can take our raw emails and transform them into these types of counts and these types of numbers in a very clean and effective way.

200
00:19:06,960 --> 00:19:22,700
So the five things we need to keep track of are the total number of unique words, the total number of words in spam, the total number of words in ham, or regular emails, the count of each word in spam, and the count of each word in ham.

201
00:19:25,960 --> 00:19:28,000
So let's take a look at one of our emails.

202
00:19:28,720 --> 00:19:31,620
We see here, East Asian fonts in Lenny.

203
00:19:31,620 --> 00:19:32,640
Thanks for your support.

204
00:19:32,640 --> 00:19:34,760
Installing Unifonts did it well for me.

205
00:19:35,340 --> 00:19:42,240
Now we want to take a sentence like this and find a way to classify it as being spam or ham.

206
00:19:42,700 --> 00:19:45,840
But before we do that, we want to kind of point out a couple things.

207
00:19:45,840 --> 00:19:52,700
So we see here kind of towards the bottom, we have the word unsubscribe, all capitalized, and then unsubscribe lowercase.

208
00:19:52,900 --> 00:19:55,460
What we want to do is look at keywords.

209
00:19:55,500 --> 00:20:04,180
And so in context, it doesn't matter if unsubscribe is capitalized or the first letter is capitalized or it's all lowercase.

210
00:20:04,280 --> 00:20:06,300
It means the same thing.

211
00:20:06,320 --> 00:20:13,420
And so to kind of reduce our problem set, what we're going to do is set everything to lowercase.

212
00:20:13,900 --> 00:20:17,180
The other thing we want to do is remove what are called stop words.

213
00:20:17,180 --> 00:20:21,700
Stop words are words like the, in, of, on, or...

214
00:20:21,700 --> 00:20:24,420
They don't add any context to the sentence.

215
00:20:24,780 --> 00:20:27,100
And we're going to do the same thing for punctuation.

216
00:20:27,200 --> 00:20:28,580
So we're going to remove those.

217
00:20:28,580 --> 00:20:30,300
We're going to keep our keywords.

218
00:20:30,300 --> 00:20:33,300
And then one more thing that we're going to do is called stemming.

219
00:20:33,580 --> 00:20:41,620
So if we have something like the word congrats or congratulations, in the English language, they mean the same thing.

220
00:20:41,620 --> 00:20:45,820
So if we're looking at keywords, we want to treat them the same.

221
00:20:45,820 --> 00:20:50,960
And so what stemming does is shrink those words down to their base form or the root form.

222
00:20:50,960 --> 00:20:52,860
And then we can count them the same.

223
00:20:52,940 --> 00:20:59,840
Another really good example is the word thanks, thank you, where it's a just singular thank.

224
00:20:59,840 --> 00:21:03,380
And I thanked somebody today with an ed at the end.

225
00:21:04,040 --> 00:21:05,940
As a keyword, they all mean the same thing.

226
00:21:05,940 --> 00:21:08,360
And so stemming will reduce them all down to thank.

227
00:21:08,540 --> 00:21:11,760
And so instead of three separate words, it's going to be counted as one word.

228
00:21:11,760 --> 00:21:17,200
And so that's going to drastically reduce our dimension space.

229
00:21:18,300 --> 00:21:30,080
So after our preprocessing, which is what this is known as, we're going to see East Asian fonts in Lenny, or East Asian fonts Lenny, thanks support installing Unifonts, well me.

230
00:21:31,400 --> 00:21:36,060
And then what isn't shown here is what the stemming looks like.

231
00:21:36,060 --> 00:21:45,840
So as we go through the workbook, we'll take this exact same email and see how it's transformed at every step of our preprocessing.

232
00:21:46,480 --> 00:21:49,260
So are there any questions so far?

233
00:21:55,420 --> 00:21:56,680
I don't think so.

234
00:21:56,680 --> 00:21:59,240
There's none on Twitch chat, and there's none in the Discord.

235
00:21:59,280 --> 00:22:00,320
Perfect.

236
00:22:02,320 --> 00:22:05,860
So what we're going to have are a series of workbooks.

237
00:22:05,860 --> 00:22:07,820
We're going to use MyBender.

238
00:22:07,820 --> 00:22:09,740
MyBender is really awesome.

239
00:22:10,700 --> 00:22:13,620
TAs, if you don't mind sharing this link.

240
00:22:13,880 --> 00:22:26,740
What it will do is spin up a Docker container with the workbook linked to my GitHub so that you can kind of code with me as we walk through and build our own machine learning classifiers.

241
00:22:27,980 --> 00:22:39,600
While this is starting up, what we're going to be using is instead of coding everything raw and doing the raw numbers like we did on our pen and paper, we're going to be using the abstraction library Scikit-learn.

242
00:22:39,600 --> 00:22:50,700
Scikit-learn is very useful and it allows us to treat things kind of like Legos, where, OK, maybe I don't want multinomial Naive Bayesian, maybe I want logistic regression or random forest.

243
00:22:50,700 --> 00:22:52,680
We can just swap that out.

244
00:22:54,780 --> 00:23:00,780
So once it's loaded, and I'm going to give you a moment to get this loaded up.

245
00:23:01,380 --> 00:23:09,600
If it takes a little bit longer than it should, then you'll easily be able to catch up to me and I'll show you how.

246
00:23:10,160 --> 00:23:15,860
So we're going to click on workbooks spam filter sklearn.

247
00:23:15,860 --> 00:23:19,940
If you're not very familiar with Python, you can still follow along.

248
00:23:19,940 --> 00:23:24,780
We have a completed workbook, which is the workbook complete spam filter sklearn.

249
00:23:25,720 --> 00:23:26,800
And these are Jupyter notebooks.

250
00:23:26,800 --> 00:23:32,380
If you've never used Jupyter notebooks, I think they're the best thing since sliced bread, especially when it comes to data science.

251
00:23:32,820 --> 00:23:38,540
But it allows us to interact with code blocks and visualize things in line.

252
00:23:38,540 --> 00:23:39,740
And I think they work fantastic

253
00:23:43,520 --> 00:23:44,040
together.

254
00:23:45,080 --> 00:23:47,800
So let's go ahead, scroll down.

255
00:23:47,800 --> 00:23:50,560
We're going to run this first code block.

256
00:23:51,100 --> 00:24:00,960
And so what this is going to do is install a couple of libraries that we need, as well as set our data directory to where the data directory is in the GitHub project.

257
00:24:00,960 --> 00:24:07,900
So to run these, you can click run up at the top, or you can press shift enter, which is what I'm pressing now.

258
00:24:07,900 --> 00:24:10,560
And then you can see that it's running.

259
00:24:10,560 --> 00:24:13,780
A star in the upper corner means that it's currently running.

260
00:24:13,780 --> 00:24:19,840
And then a number in the upper corner means that that was the order that code block was ran in.

261
00:24:22,560 --> 00:24:29,880
So while that is going, I'm going to let it go until it stops scrolling my thing.

262
00:24:30,900 --> 00:24:33,180
So we're just going to import a couple libraries.

263
00:24:33,180 --> 00:24:41,360
We're going to import numpy, matplotlib for graphs, regular expressions we'll be using and NLTK.

264
00:24:41,360 --> 00:24:44,380
NLTK stands for Natural Language Toolkit.

265
00:24:44,380 --> 00:24:51,560
This will allow us to identify and collect a list of stop words to remove those things.

266
00:24:51,560 --> 00:24:59,780
And it will also allow us to use a stemmer function to stem the words and reduce our problem space.

267
00:25:00,800 --> 00:25:04,320
Train test split, I don't believe we're actually using that in here.

268
00:25:04,320 --> 00:25:11,060
The data directories I have are already split into 80% training, 20% testing.

269
00:25:11,300 --> 00:25:13,320
And then we're going to use our vectorizers.

270
00:25:13,320 --> 00:25:24,140
And we'll explain what the vectorizers do, but we're going to take a look at both of the TF-IDF vectorizer and the count vectorizer because they operate slightly different.

271
00:25:24,140 --> 00:25:29,300
Count vectorizer just takes a straight word count like we did with our pen and paper example.

272
00:25:29,300 --> 00:25:34,680
And then TF-IDF actually takes all of the words into account and gives them a weight.

273
00:25:35,800 --> 00:25:41,060
Scikit-learn, we're going to use logistic regression and multinomial naivetation.

274
00:25:41,060 --> 00:25:46,140
And the reason why we're going to use both of these is because we don't actually know which one's going to perform best.

275
00:25:46,180 --> 00:25:52,960
And so like good data scientists, we're going to try a couple different models and take the one that performs best with the problem set that we have.

276
00:25:53,560 --> 00:26:01,640
And then for a couple metric and visualizations, we're going to use the Scikit-learn confusion matrix and classification report.

277
00:26:03,040 --> 00:26:06,240
So I'm going to go ahead and press Shift-Enter and that's going to run.

278
00:26:06,240 --> 00:26:09,960
You will see libraries imported once that's complete.

279
00:26:11,520 --> 00:26:13,300
And we can move on to the next thing.

280
00:26:13,620 --> 00:26:14,700
So this is our test email.

281
00:26:14,700 --> 00:26:17,760
This is the exact same email that we saw in the slides.

282
00:26:17,980 --> 00:26:25,040
So as I mentioned earlier, we're going to use this email to see how it's transformed at every step of preprocessing.

283
00:26:25,040 --> 00:26:26,540
So I'm going to go ahead and run that.

284
00:26:27,980 --> 00:26:30,000
And then this is our tokenizer function.

285
00:26:30,000 --> 00:26:37,060
Tokenizer functions will allow us to pull out the keywords in the way that we would like them to be extracted.

286
00:26:37,180 --> 00:26:43,500
So in this case, we're going to grab all the punctuation, all the stop words, and we're going to create a stemmer constructor.

287
00:26:44,560 --> 00:26:49,100
We're going to use these to remove the stop words from the email.

288
00:26:49,100 --> 00:26:54,120
We're going to use these to remove punctuation from the email, and then we're going to stem the words.

289
00:26:56,540 --> 00:27:00,100
We're going to move all the words to lowercase so that they are treated exactly the same.

290
00:27:00,980 --> 00:27:05,400
So we're just going to go ahead and run this, shift enter, and our tokenizer is defined.

291
00:27:05,960 --> 00:27:10,420
And this is where we run into our first task, which is to tokenize an email.

292
00:27:10,780 --> 00:27:18,920
So our task says that we want to print the full email, test email, and then we're going to print the results of the tokenized version of that email.

293
00:27:19,580 --> 00:27:21,140
So here I'm just going to do

294
00:27:27,880 --> 00:27:31,040
print tokenizer test email.

295
00:27:31,520 --> 00:27:33,860
And we can go ahead and run that, shift enter.

296
00:27:34,020 --> 00:27:39,660
We can see the original email that we have, East Asian Fonts and Lenny, thanks for your support.

297
00:27:39,740 --> 00:28:00,140
We can see, just like in our slides, we have East Asian Font Lenny, slightly changed, thank instead of thanks, support install, because we can have installing installed with an ed at the end, or I will install this software, which is just a standard install.

298
00:28:01,340 --> 00:28:03,920
We see unsubscribe has been changed.

299
00:28:04,020 --> 00:28:13,940
And two of the things that I want to point out is every single word in here is unique, except for the word unsubscribe, which shows up twice, and list.deviant.org, which shows up twice.

300
00:28:16,360 --> 00:28:18,440
So we're going to take a look at our training data.

301
00:28:18,440 --> 00:28:28,980
What this is going to do is identify the directories where our spam files are, our ham files are, and our test files are.

302
00:28:28,980 --> 00:28:40,380
We're going to get a count, because that's one of the five things that we needed to keep track of, was the count of, actually we want to keep track of the count of words, we don't care how many emails we have.

303
00:28:40,480 --> 00:28:48,900
So I'm going to go ahead and run this, and what we're going to do is put those emails into what is known as a corpus.

304
00:28:48,900 --> 00:28:52,180
So we're going to create two list arrays, so corpus and labels.

305
00:28:52,180 --> 00:29:01,620
Corpus is going to hold all of our emails for spam and ham, and then the labels is going to be each label for our data.

306
00:29:01,680 --> 00:29:10,780
So the first thing we're going to do is corpus equals a new list, labels equals a new list, and then we're going to do the same thing for spam and ham.

307
00:29:10,780 --> 00:29:20,380
We're going to load the email bodies from the ham directories and the spam directory into the corpus array, and then we're going to load the labels for each email into the labels array.

308
00:29:20,980 --> 00:29:31,400
So what that's going to look like is we're going to use a for loop here, so we can say for each in os.listdir, and then we have our data directory, which is defined above.

309
00:29:31,900 --> 00:29:36,080
We're going to look at our ham directory for this first round.

310
00:29:36,760 --> 00:29:44,920
With each open file, we're going to look at our data directory plus ham plus os.listdir.

311
00:29:44,940 --> 00:29:52,240
We're going to look at each , because each is the name of our file, and we're going to want to read those as the file descriptor f.

312
00:29:52,420 --> 00:29:59,320
So for our corpus, we can do an append and then f.read.

313
00:29:59,320 --> 00:30:12,380
So this will read each email into our corpus and then labels.append ham, because we know that all of the emails in the ham directory are part of the ham data set.

314
00:30:12,460 --> 00:30:17,780
And so we can easily do the same thing for spam which is we need to be careful on how we change this.

315
00:30:17,780 --> 00:30:24,140
So instead of slash ham, it's going to be slash spam, and then of course our label is also going to be spam.

316
00:30:26,060 --> 00:30:32,000
So I'm going to let this sit for a minute and let you guys go ahead and make sure that you copy this down correctly.

317
00:30:33,520 --> 00:30:36,920
This bit is pretty easy if you're familiar with Python.

318
00:30:36,920 --> 00:30:40,580
It might be a little confusing if you are not, but that's okay.

319
00:30:40,580 --> 00:30:47,990
If you ever get lost at any point, just remember that there are a completed solutions manual in the workbooks directory.

320
00:30:50,580 --> 00:30:54,820
So I'm going to go ahead and run this.

321
00:30:54,820 --> 00:30:56,930
So shift enter to run.

322
00:30:57,740 --> 00:31:00,980
And I spelled labels wrong.

323
00:31:02,680 --> 00:31:05,660
Labels, shift enter to run.

324
00:31:05,660 --> 00:31:07,810
Loading ham, loading spam, done.

325
00:31:08,940 --> 00:31:18,240
So what this is going to do is graphically load or graphically visualize how many ham emails we have versus how many spam emails we have.

326
00:31:18,240 --> 00:31:23,280
And this is something we want to generally do with our data just to see if there's any sort of imbalanced data sets.

327
00:31:23,380 --> 00:31:28,600
This one we have about 50% spam per ham.

328
00:31:29,280 --> 00:31:32,560
It's like 2 to 1 ratio of ham to spam.

329
00:31:32,560 --> 00:31:51,580
And so this I'm not going to consider an imbalanced data set for this example, but if we have something such as like credit card fraud where it happens less than a fraction of a percent of the time, that's something that we really want to dive into simply because if we try and use this exact example on credit card fraud,

330
00:31:51,580 --> 00:32:01,540
it's not going to work because what we will do is create a model that is 99% accurate because it just classifies all credit card transactions as legitimate.

331
00:32:01,820 --> 00:32:05,340
Now if we want to identify credit card fraud , that's not very helpful for us.

332
00:32:05,340 --> 00:32:11,060
So this just allows us to kind of explore the data and see how much of what we have.

333
00:32:11,060 --> 00:32:17,780
So in this case we have 2,359 ham emails and about 1,100 spam emails.

334
00:32:19,700 --> 00:32:23,020
So task 2A, we're going to view our training data.

335
00:32:23,020 --> 00:32:27,260
So this one's just to kind of see what our data looks like.

336
00:32:27,860 --> 00:32:30,880
We generally have an idea, but we just kind of want to play around with it.

337
00:32:30,880 --> 00:32:37,200
And this is another thing that you would want to do in a data discovery process is just kind of see, okay, what is it that we're actually working with?

338
00:32:37,600 --> 00:32:44,360
So in this case, I'm going to pick an email ID, just email ID 5.

339
00:32:44,360 --> 00:32:46,640
You can pick 5 or you can pick any other.

340
00:32:47,520 --> 00:32:52,560
And then I'm going to grab the email, which is part of the corpus array.

341
00:32:53,000 --> 00:32:55,920
Corpus and then our email ID.

342
00:32:56,880 --> 00:33:09,920
And then I'm going to print that email, and then I'm going to do the same thing and print the tokenized version using a tokenizer function of that email.

343
00:33:09,920 --> 00:33:11,700
So we can see how it's transformed.

344
00:33:14,180 --> 00:33:20,760
So just so I don't go too fast, I want to make sure that you're able to follow along and copy this down.

345
00:33:21,280 --> 00:33:22,900
It's fairly simple.

346
00:33:23,120 --> 00:33:26,040
So I will wait just a few seconds.

347
00:33:27,060 --> 00:33:29,460
How are we doing on questions?

348
00:33:31,180 --> 00:33:40,460
No one has been posting some helpful links, but hasn't had too many questions so far.

349
00:33:40,500 --> 00:33:42,500
So thank you very much.

350
00:33:42,500 --> 00:33:43,420
No winner.

351
00:33:44,540 --> 00:33:45,300
Okay.

352
00:33:45,300 --> 00:33:48,780
So when we run that, what we have is the email body.

353
00:33:48,780 --> 00:33:53,900
We see product review, vert tools, dev, 2.0 URL, and then this URL.

354
00:33:54,280 --> 00:34:03,920
We see product review, vert tool, dev, 2.0 URL as one word, HTTP, and then the full URL.

355
00:34:03,920 --> 00:34:09,620
So this is how we can identify and kind of track our data as it's being transformed over time.

356
00:34:11,980 --> 00:34:14,700
So task 3 is where we're going to start using our vectorizers.

357
00:34:14,700 --> 00:34:25,880
What we want to do is get a count of each unique word, and we want to get a count of each unique word per class so that we can use that in our classification.

358
00:34:25,980 --> 00:34:27,180
So there's two ways we can do that.

359
00:34:27,180 --> 00:34:36,920
We can use the straight count vectorizer, which will get the count of our words, and we can use a TF-IDF vectorizer, which will give us a weight of each count.

360
00:34:39,480 --> 00:34:42,980
So no winner says he's behind.

361
00:34:43,520 --> 00:34:46,760
He's catching up and he'll catch up later and follow along.

362
00:34:49,500 --> 00:34:54,620
I can try to figure out a system of helping the winner.

363
00:34:54,700 --> 00:35:00,680
I can send you a Zoom link or something and look over your shoulder metaphorically.

364
00:35:01,100 --> 00:35:04,980
Also , in the workbooks, you can just open up the...

365
00:35:04,980 --> 00:35:06,000
that's the wrong one...

366
00:35:06,000 --> 00:35:12,560
you can just open up the completed spam filter SK Learn, and it's completely filled out for you.

367
00:35:12,560 --> 00:35:16,960
So if you ever do fall behind and you would like to catch up, you can look at the answers there.

368
00:35:18,820 --> 00:35:23,000
There is a way to follow along and make sure your code works.

369
00:35:23,880 --> 00:35:25,140
Yeah, absolutely.

370
00:35:25,140 --> 00:35:26,780
And I like using the completed workbook.

371
00:35:26,780 --> 00:35:32,040
You can just run everything and see how things are supposed to be at each step of the process.

372
00:35:33,540 --> 00:35:35,540
So we're going to go ahead and train our vectorizers.

373
00:35:35,540 --> 00:35:37,620
And the instructions tell us exactly how to do this.

374
00:35:37,620 --> 00:35:39,520
So we're going to use the vectorizer.

375
00:35:39,520 --> 00:35:46,020
We're going to call it cvec for the count vectorizer, tvec for the tfidf vectorizer, and we're going to perform fit transform.

376
00:35:46,780 --> 00:35:50,840
Fit in scikit-learn world is code for train.

377
00:35:50,880 --> 00:35:55,160
So this is going to, quote unquote, train our vectorizers so they can get the count of each words.

378
00:35:55,160 --> 00:35:59,460
And then we're going to save the results as count x and tfidf x.

379
00:36:00,400 --> 00:36:06,160
So what that looks like is cvec equals count vectorizer.

380
00:36:06,160 --> 00:36:09,560
And then we want to use our tokenizer that we defined above.

381
00:36:09,560 --> 00:36:14,540
So the tokenizer that we're going to use is called tokenizer.

382
00:36:14,540 --> 00:36:16,740
I know, not confusing at all.

383
00:36:16,740 --> 00:36:23,140
But we're going to use our count x.

384
00:36:23,140 --> 00:36:25,020
And pay attention to your capitals.

385
00:36:25,040 --> 00:36:28,200
Scikit-learn is very picky and so is my workbook.

386
00:36:28,500 --> 00:36:33,260
We're going to use cvec.fit transform.

387
00:36:33,260 --> 00:36:39,400
And then we're going to train transform or count the words for our corpus.

388
00:36:40,160 --> 00:36:43,620
And then we're going to do the same thing with our tfidf vectorizer.

389
00:36:43,820 --> 00:36:48,100
tvec equals tfidf vectorizer.

390
00:36:48,100 --> 00:36:52,220
And we're going to use our tokenizer called tokenizer.

391
00:36:52,220 --> 00:37:01,080
And then our tfidf x equals tvec.fit transform.

392
00:37:01,080 --> 00:37:04,160
And it's also going to be on our corpus.

393
00:37:04,160 --> 00:37:09,100
So I'm going to let this sit for a minute and let you guys be able to catch up.

394
00:37:12,720 --> 00:37:13,820
And this takes a little bit.

395
00:37:13,820 --> 00:37:27,440
This takes about two minutes to run because what it's doing is it's reading every single email and it is counting the words in those emails in order to get the number of unique words.

396
00:37:30,060 --> 00:37:34,620
So I'm going to go ahead and run this because it's going to take a little bit.

397
00:37:35,020 --> 00:37:39,020
But in case you need, here is the code while this is operating.

398
00:37:39,020 --> 00:37:43,920
And so we can see that it is currently training the count vectorizer, kind of down here at the bottom.

399
00:37:44,380 --> 00:37:48,620
And then once the count vectorizer is complete, it will start training the tfidf vectorizer.

400
00:37:48,720 --> 00:37:51,740
And then it will let us know when that is complete.

401
00:38:02,120 --> 00:38:06,840
So while this is going, I'm going to go ahead and move on to the next code block.

402
00:38:06,840 --> 00:38:12,980
We can write the code while this is still training, but we won't be able to run it because it can run only one code block at a time.

403
00:38:13,500 --> 00:38:18,060
So in this case, we have task 3a, which is to count the test email tokens.

404
00:38:18,060 --> 00:38:22,420
So we're going to manually count out each of the tokens.

405
00:38:22,420 --> 00:38:33,040
So what we're going to do is say for i in and this is a bit of a trick so that we can keep the count as well as the words in order.

406
00:38:33,040 --> 00:38:35,620
So we're going to use a list of a dictionary

407
00:38:38,820 --> 00:38:44,880
and we're going to want to grab the tokenized version of our email.

408
00:38:44,880 --> 00:38:54,580
So let's call this tokenized email equals tokenizer test email.

409
00:38:54,720 --> 00:38:59,900
And then our from keys we're going to have tokenized email which we saw above is going to be a list .

410
00:38:59,900 --> 00:39:06,860
So we'll just call that tokenized email and then we're going to count the words .

411
00:39:06,860 --> 00:39:23,000
So print, do some Python format, And then we're going to look at our tokenized email.count of each word, and then the word itself.

412
00:39:23,440 --> 00:39:29,660
So we see our vectorizing is complete, so we are ready to run this.

413
00:39:29,660 --> 00:39:36,060
I'm going to let this sit for about 30 more seconds and let you guys catch up.

414
00:39:36,060 --> 00:39:36,960
And

415
00:39:43,710 --> 00:39:50,970
then, of course, after this workshop, I have a whole series of workbooks, as well as their solutions.

416
00:39:50,970 --> 00:40:00,050
So if you're interested in other ways to apply machine learning and security to different problems, and using different algorithms, don't forget to take a look at those.

417
00:40:02,630 --> 00:40:08,350
So I'm going to go ahead and run this, Shift-Enter, and we can see the counts.

418
00:40:08,350 --> 00:40:11,490
So we have East Asian font, Lenny, thanks, support.

419
00:40:11,830 --> 00:40:17,710
Each of these, we have our unique words, they only show up once in our email.

420
00:40:17,710 --> 00:40:23,830
The only two that are outliers is unsubscribe, or unsubscrib, after our stemmer.

421
00:40:23,830 --> 00:40:28,890
And this list.debian.org, they show up twice in our email.

422
00:40:28,890 --> 00:40:33,030
So this is if we manually counted these, but what if we used our vectorizers?

423
00:40:33,770 --> 00:40:47,430
So what we're going to do here in Task 3B is create new vectorizers, the count vectorizer and TF-IDF vectorizer, and then we're going to print out the counts that each of them represent.

424
00:40:47,670 --> 00:40:57,150
So in the first one here, I'll just call this example-cvec, and then we're going to define a new vectorizer.

425
00:41:02,220 --> 00:41:13,760
And I'll just call this example-x equals example-cvec.fit-transform.

426
00:41:14,340 --> 00:41:20,700
And then it expects a list, our email is a string, so we're going to wrap it in these square brackets.

427
00:41:21,000 --> 00:41:23,600
And we're just going to use our raw test email.

428
00:41:26,080 --> 00:41:30,960
And then we're going to want to print our example-x.

429
00:41:33,140 --> 00:41:38,180
And we can easily just copy and paste, do the same thing for our TF-IDF code.

430
00:41:38,180 --> 00:41:52,840
We'll just call this example-tvec, we'll use the TF-IDF vectorizer, tokenizer is tokenizer, we change this to tvec, and then we're already done with our example-x, so we don't need to change the name, we can just leave it as is.

431
00:41:53,980 --> 00:41:59,940
Example-tvec, fit-transform, and then square brackets, test-email, so that it's a list.

432
00:42:00,860 --> 00:42:01,760
Yes.

433
00:42:04,900 --> 00:42:07,800
Oh, it sounded like there was possibly a question.

434
00:42:13,190 --> 00:42:18,510
So Voto on the stream is asking, where are those workbooks again?

435
00:42:24,050 --> 00:42:28,070
Can one of the TAs post in the stream?

436
00:42:28,070 --> 00:42:29,750
There you go, perfect.

437
00:42:30,270 --> 00:42:32,170
Machine learning for a security analyst.

438
00:42:33,530 --> 00:42:37,830
Okay, so I'm going to go ahead and run this.

439
00:42:38,570 --> 00:42:41,030
Scroll up, let's take a look at the count vectorizer.

440
00:42:41,130 --> 00:42:44,890
So instead of seeing the words, we see these numbers, and that's exactly what we want.

441
00:42:44,890 --> 00:42:47,090
We've created what are called tokens.

442
00:42:47,330 --> 00:42:53,750
So the words themselves, the tokenizers will be able, or the vectorizers will be able to match them together.

443
00:42:55,090 --> 00:43:03,150
So all of them are one in their count, all of them are unique, except for token 16 and token 9.

444
00:43:03,150 --> 00:43:12,910
We don't know which one, either unsubscribe or list.wn.org, maps to 16 or 9, but what we do know is that both of them show up twice.

445
00:43:12,910 --> 00:43:21,990
So given that, list.wn.org is either 16 or 9, and then the other one belongs to unsubscribe, or the word unsubscribe.

446
00:43:21,990 --> 00:43:27,010
So this looks exactly the same as if we had done our raw count.

447
00:43:27,650 --> 00:43:32,470
If we look at our TF-IDF vectorizer, though, we see something a little differently.

448
00:43:32,470 --> 00:43:39,950
TF-IDF, what it does is use term frequency to give each term or token a weight.

449
00:43:40,030 --> 00:43:53,790
And so in this case, we see all of the ones that showed up once have a weight of 0.204, token 9 and token 16 have a weight of 0.408.

450
00:43:54,270 --> 00:44:00,930
So you don't really need to understand the underlying math, you just need to know that there are different ways to vectorize your data.

451
00:44:01,590 --> 00:44:08,130
So there's TF-IDF, count vectorizer, word2vec, and a couple others, bag of words.

452
00:44:08,870 --> 00:44:12,470
And so there's different ones that we can try and see which one works better for us.

453
00:44:14,170 --> 00:44:19,090
So let's hop down and start to load our testing data.

454
00:44:19,090 --> 00:44:30,450
So loading our testing data, what this is going to do is just show us the names of each of our files in our testing directory.

455
00:44:31,430 --> 00:44:36,650
And we have train email.txt, and then at the end we have the class that it belongs to.

456
00:44:36,650 --> 00:44:39,030
So we have ham, we have spam.

457
00:44:39,750 --> 00:44:45,310
So what we're going to want to do as part of loading our testing data is we're going to want to strip off these labels.

458
00:44:45,910 --> 00:44:55,010
So we're going to read the full email into our corpus, and we're going to strip off these labels, and those are going to be the new labels for our testing data.

459
00:44:56,410 --> 00:45:00,110
So that hops us into task 4 to load the testing data.

460
00:45:00,110 --> 00:45:06,810
Just like loading the training data, we're going to create two list arrays, our corpus, we're going to call this test corpus, and our test labels.

461
00:45:06,970 --> 00:45:14,310
And then we're going to load each email body, and then strip off the label and throw that into the test labels array.

462
00:45:16,450 --> 00:45:20,810
So what that's going to look like is...

463
00:45:25,550 --> 00:45:28,530
So we're going to look at... what was the name of that?

464
00:45:28,730 --> 00:45:30,770
Test corpus.

465
00:45:30,950 --> 00:45:37,270
And we're just going to set that as a blank list, and then test labels as a blank list.

466
00:45:37,990 --> 00:45:50,130
For loading each of the emails, we're going to do just like we did before, filename in ls.listdir, and then our data directory plus the test directory.

467
00:45:50,130 --> 00:46:08,230
So we're going to look at each filename, we're going to open each file, so data.dir plus test plus filename, and then we want to read those as the file descriptor F.

468
00:46:09,190 --> 00:46:19,830
So our test corpus, we're going to append F.read, which is going to read that email into our test corpus array, and then we want to grab the label.

469
00:46:19,830 --> 00:46:23,950
And in this case, I'm just going to use a simple regular expression.

470
00:46:25,390 --> 00:46:36,710
We're going to use txt, and then this is a backslash period, and filename 1.

471
00:46:36,710 --> 00:46:49,690
I'm going to scroll up real quick, because the reason why I want to do that is what this is saying is, where we see txt.period, we're going to split, so we have this full train txt.period, and then we have ham.

472
00:46:49,910 --> 00:47:02,610
And then the bracket 1, what that is going to do is say, this is bracket 0 once it's split, and then this is bracket 1, so we're just going to keep our labels, which are at the end of that split.

473
00:47:05,230 --> 00:47:17,350
And then with that label, or each label that we grab from the filename, we're going to append it to our labels directory, or our labels array.

474
00:47:19,210 --> 00:47:21,450
So this is what our code looks like.

475
00:47:21,450 --> 00:47:25,990
Double checking for errors, looks good to me.

476
00:47:26,390 --> 00:47:31,630
And I'm going to let this sit for a little bit, so you guys can go ahead and copy that down.

477
00:47:39,290 --> 00:47:41,170
How are we doing?

478
00:47:45,460 --> 00:47:50,780
No questions in Twitch stream, are there any questions in the Discord channel?

479
00:47:57,980 --> 00:48:01,420
Okay, I'm going to go ahead, shift, enter, and run.

480
00:48:01,420 --> 00:48:04,040
So this is going to quickly load the emails.

481
00:48:06,240 --> 00:48:09,400
And then let's run this next code block.

482
00:48:09,400 --> 00:48:13,060
So what this is going to do is allow us to visualize our data, just like we did before.

483
00:48:13,060 --> 00:48:29,160
So we see in our testing set, we have 590 ham emails, 276 spam emails, and the graph itself looks nearly identical to the one above, and that's because we did a random 80-20 split for our training and testing.

484
00:48:30,300 --> 00:48:33,020
And that was already done beforehand.

485
00:48:34,460 --> 00:48:51,780
So the next thing we want to do is use the vectorizers that we created, the cvec and the tvec vectorizer, and take a look at the test corpus, and get our counts of each word in the new emails.

486
00:48:52,560 --> 00:49:06,420
So let's go ahead, testCountX equals, and we're already going to use cvec, we're not going to use fitTransform, instead we're going to use transform, because we're not trying to train it, we're just trying to save the words themselves.

487
00:49:07,680 --> 00:49:20,040
We'll do testCorpus, and then testTFIDFX equals tvec.transform testCorpus.

488
00:49:20,280 --> 00:49:25,940
And that's all we need to do when we're using a piece of testing data.

489
00:49:27,700 --> 00:49:33,620
So I'm going to let that sit for a second, actually I'm going to run it because this is going to take a little bit as well.

490
00:49:36,620 --> 00:49:47,460
Slightly shorter than when we were training it, but it still needs to go through all the emails and apply the counts, and then the TFIDF, it needs to apply the weights.

491
00:49:47,460 --> 00:49:51,640
We already know what the counts are, we already know what the weights are, we just need to apply them to each token.

492
00:49:52,220 --> 00:49:53,840
So vectorizing is complete.

493
00:49:54,520 --> 00:49:59,220
And now we're finally on to testing and evaluating our model.

494
00:49:59,640 --> 00:50:05,460
So you remember how I said data collection and preprocessing is what's going to take the most amount of time?

495
00:50:05,460 --> 00:50:07,640
Well, that was about 80% of this workbook.

496
00:50:08,140 --> 00:50:11,340
If not, maybe a little more, maybe 85%.

497
00:50:12,320 --> 00:50:14,740
But yeah, data science in a nutshell.

498
00:50:15,680 --> 00:50:20,620
So here we have a very helpful report generator function.

499
00:50:20,920 --> 00:50:35,120
It's not important to know what's going on in this, it's just going to be a helper function, but we do need to know that it's going to take in our confusion matrix, our score, which is our accuracy, and the classification report, and then it's going to print out all the stats that we need.

500
00:50:36,520 --> 00:50:48,340
So that hops us into task 6A, which is going to be training and evaluating our first model, which is going to be multinomial Naive Bayesian using the TFIDF vectorizer.

501
00:50:49,980 --> 00:50:53,940
So there's a lot of steps to this, but we're going to walk through it step by step.

502
00:50:54,200 --> 00:51:08,000
First, what we're going to do is create a multinomial Naive Bayesian constructor, so that's going to look like mnb, tf, idf, and the names of these variables are very important because that's what's being used in the generator function.

503
00:51:08,980 --> 00:51:31,300
So I'm going to call this Numial Naive Bayesian, and then we're going to train the multinomial Naive Bayesian, as I already misspelled something, mnb, tf, idf, .fit, which is scikit-learn's code for train, and then we're going to use our TFIDF, our vectorized corpus,

504
00:51:31,300 --> 00:51:33,440
with the TFIDF array.

505
00:51:35,360 --> 00:51:48,740
So now we're going to get a couple stats, so we're going to get our score, mnb, tf, idf, equals mnb, tf, idf, .score.

506
00:51:48,740 --> 00:51:57,380
This is going to give us our accuracy, and we want to compare this against our test, tf, idf, x, and our test labels.

507
00:51:58,260 --> 00:52:14,760
And then we want to take a look at the raw predictions, so we're going to have our predictions, mnb, tf, idf, equals mnb, tf, idf, .predict, using the predict function, on our test, tf, idf, x.

508
00:52:15,540 --> 00:52:16,400
Cool.

509
00:52:16,400 --> 00:52:27,540
And with those results, what we are going to do is generate a classification, a confusion matrix, so we can see our true positives, false positives, true negatives, false negatives.

510
00:52:27,580 --> 00:52:50,800
So cmatrix, mnb, tf, idf, equals our confusion matrix, and then we're going to use our test labels and our predictions, mnb, tf, idf, which is our raw predictions.

511
00:52:50,800 --> 00:53:09,320
We're going to do the same thing for our classification report, so we'll see report, mnb, tf, idf, classification report, and in the same order, test labels and our predictions, mnb, tf, idf.

512
00:53:10,680 --> 00:53:17,520
So I'm going to let this sit for a second, give you a chance to copy all of that down.

513
00:53:19,580 --> 00:53:34,240
So what's really helpful here is seeing how at each of these tasks, 6a, 6b, 6c, and 6d, we're only going to need to change very few things.

514
00:53:34,240 --> 00:53:43,500
We're going to need to change, instead of tf, idf, we're going to change this to count to do our train and evaluate the multinomial Naive Bayesian with the count model.

515
00:53:43,720 --> 00:53:51,280
And then we're going to change this from the multinomial Naive Bayesian constructor to the logistic regression constructor.

516
00:53:51,400 --> 00:53:53,840
But other than that, the rest of our code is going to be identical.

517
00:53:53,840 --> 00:53:56,280
The rest of our preprocessing is going to be identical.

518
00:53:56,480 --> 00:54:05,620
And so this will allow us to train a total of four algorithms using two separate machine learning algorithms as well as two separate vectorizers.

519
00:54:06,340 --> 00:54:09,100
So this is going to be very helpful for us.

520
00:54:10,400 --> 00:54:14,040
So I'm going to go ahead and run this.

521
00:54:14,040 --> 00:54:16,260
So shift-enter to run.

522
00:54:16,260 --> 00:54:20,240
It looks like I misspelled something.

523
00:54:23,120 --> 00:54:24,560
Multinomial.

524
00:54:24,840 --> 00:54:26,660
There we go.

525
00:54:27,280 --> 00:54:29,440
Required argument y.

526
00:54:29,900 --> 00:54:31,400
That one's new.

527
00:54:32,980 --> 00:54:36,580
Give me one second.

528
00:54:36,900 --> 00:54:39,320
Oh, that is what I forgot.

529
00:54:39,360 --> 00:54:56,560
So we want to train our multinomial Naive Bayesian with our data as well as our labels so that it knows what belongs to spam, what belongs to ham, just kind of in our pen and paper example, what belonged to sports and what belonged to not sports.

530
00:54:57,020 --> 00:54:58,940
So now I can run that.

531
00:55:00,460 --> 00:55:04,360
And I see where we ran into that problem yesterday.

532
00:55:04,580 --> 00:55:05,260
Okay.

533
00:55:06,600 --> 00:55:08,740
I need to swap.

534
00:55:08,740 --> 00:55:11,800
This is the problem that I had with scikit-learn earlier.

535
00:55:13,140 --> 00:55:15,560
The test labels and the predictions.

536
00:55:15,560 --> 00:55:20,500
We start with the predictions and then we follow up with the test labels instead of the other way around.

537
00:55:21,240 --> 00:55:23,100
So I go ahead and run that.

538
00:55:23,800 --> 00:55:24,840
There we go.

539
00:55:24,840 --> 00:55:29,200
So we know that we had 590 ham emails.

540
00:55:29,200 --> 00:55:34,920
We know that we had, if we add these two numbers together, 276 spam emails.

541
00:55:34,920 --> 00:55:39,960
So we can see the actual number of ham emails, the actual number of spam emails in those columns.

542
00:55:40,800 --> 00:55:42,640
But let me scroll up real quick.

543
00:55:42,760 --> 00:55:45,660
So this is our classification report.

544
00:55:45,700 --> 00:55:51,500
Classification report we're not really going to use in this example, but it's good to know.

545
00:55:51,500 --> 00:55:56,800
There are other ways to measure how good a model is performing, beyond accuracy.

546
00:55:57,500 --> 00:56:03,780
Precision, recall, and F1 score are all used in their own ways.

547
00:56:03,820 --> 00:56:07,700
I'm not going to go into that, but it's definitely worth understanding.

548
00:56:08,040 --> 00:56:10,780
Instead, we're just going to look at the straight accuracy.

549
00:56:10,780 --> 00:56:19,860
For this multinomial with TF-IDF, we have a .875 or a 87.5% accuracy.

550
00:56:19,860 --> 00:56:28,920
If we look at our true positives and our true negatives, it correctly classified or predicted all ham emails as being part of the ham class.

551
00:56:28,960 --> 00:56:32,460
So these are all of our true positives for what is ham.

552
00:56:32,540 --> 00:56:40,280
And then our true negatives, what is spam, it only classified 168 out of the 276 that we have.

553
00:56:40,720 --> 00:56:46,660
The other half of the spam emails it also thought were ham or regular emails.

554
00:56:46,660 --> 00:56:54,920
So our predictor is letting some spam through, but the good news is it's not blocking any of our legitimate emails.

555
00:56:58,820 --> 00:57:02,660
Moving on to the next example, we're going to do the same thing.

556
00:57:02,660 --> 00:57:07,700
We're going to use multinomial naivete with count vectorizer.

557
00:57:07,700 --> 00:57:11,020
This is going to be the same, and then be count.

558
00:57:13,020 --> 00:57:27,020
I am actually a little lazy, so what I'm going to do is just copy this and I can show you all you need to do is just change a few things around and you have a working model.

559
00:57:27,020 --> 00:57:31,980
So instead of multinomial tfidf, I'm going to be using multinomial count.

560
00:57:31,980 --> 00:57:33,800
We're going to use the same constructor.

561
00:57:34,780 --> 00:57:41,580
So in this case it's called count, and instead of using the tfidfx, we're going to use the countx.

562
00:57:44,320 --> 00:57:56,680
score, count, mnb, count, and our test labels stay the same because we didn't vectorize those.

563
00:57:56,680 --> 00:57:58,920
There's no reason for us to change this.

564
00:58:00,120 --> 00:58:04,160
I'm just going to go ahead and ctrl-c and ctrl-v some stuff.

565
00:58:05,900 --> 00:58:11,100
So all I'm doing is just changing the word tfidf to the word count.

566
00:58:11,100 --> 00:58:14,300
Everything else is staying the same.

567
00:58:15,040 --> 00:58:22,840
We have our predictions, mnb, count, our test labels, that stays the same, we don't modify it.

568
00:58:22,900 --> 00:58:29,760
mnb, count, and then we change our predictions.

569
00:58:30,600 --> 00:58:33,680
So I'm going to go ahead and let that sit for a moment.

570
00:58:33,680 --> 00:58:42,700
I'll let you guys either manually type that in or do the copy-and-paste track that I did, which I highly recommend.

571
00:58:42,700 --> 00:58:44,980
It just saves quite a bit of typing.

572
00:58:46,280 --> 00:58:51,860
And then we'll type out the logistic regression in a moment because that's the next model that we'll be making.

573
00:58:54,780 --> 00:58:57,600
How are we doing on questions?

574
00:59:01,160 --> 00:59:03,520
I think we're all doing well.

575
00:59:04,300 --> 00:59:06,980
Everything's quiet on the Discord chat, too.

576
00:59:11,580 --> 00:59:12,380
Awesome.

577
00:59:12,380 --> 00:59:14,040
So I'm going to go ahead and run this.

578
00:59:14,040 --> 00:59:16,340
I'm going to do Shift-Enter.

579
00:59:17,640 --> 00:59:25,220
And if we look at our accuracy, our multinomial with the count vectorizer, we have a 94% accuracy.

580
00:59:25,380 --> 00:59:32,700
So quite a decent bump compared to our tfidf vectorizer using the same multinomial algorithm.

581
00:59:32,700 --> 00:59:36,760
What we see is about 26 of our emails...

582
00:59:36,760 --> 00:59:41,900
not about, it's exactly 26 of our emails in our test set got blocked.

583
00:59:41,900 --> 00:59:43,880
They were considered spam.

584
00:59:44,480 --> 00:59:46,940
These are the regular ham emails.

585
00:59:46,940 --> 00:59:53,900
But we have significantly fewer spam emails that are getting through our predictor.

586
00:59:53,900 --> 00:59:57,340
And so more of them are being classified as spam correctly.

587
00:59:57,340 --> 01:00:00,040
And so this is what we want to see is a dark blue over in this corner.

588
01:00:00,040 --> 01:00:09,080
And a darker blue or this lighter blue given the count over in the bottom right corner.

589
01:00:09,560 --> 01:00:15,840
So this kind of hops into a conversation about false positives and false negatives.

590
01:00:15,840 --> 01:00:19,000
Which would you like to prioritize?

591
01:00:19,980 --> 01:00:22,340
And so it really depends on the model.

592
01:00:22,340 --> 01:00:26,860
In security, one of the things that we're concerned about is impacting the users.

593
01:00:26,860 --> 01:00:38,660
So if you look at something like an antivirus, it looks for things that are probably talking out to the internet, trying to take over your webcam, uploading and downloading files.

594
01:00:38,720 --> 01:00:40,820
Well, what else does that?

595
01:00:41,400 --> 01:00:44,980
Chrome, Internet Explorer, Firefox.

596
01:00:45,760 --> 01:00:56,660
So when you're building an antivirus, it's better to have what they consider false negatives.

597
01:00:57,480 --> 01:01:03,180
Which is positive is it is malware and negative is it is not malware.

598
01:01:03,180 --> 01:01:04,200
It's benign.

599
01:01:04,220 --> 01:01:11,560
So it's better to allow some viruses to run instead of blocking legitimate applications.

600
01:01:11,560 --> 01:01:14,140
Because you don't want to impact the user experience.

601
01:01:14,140 --> 01:01:17,980
Because if you impact the user experience, it's just going to uninstall your antivirus software.

602
01:01:17,980 --> 01:01:23,320
And that leaves them unprotected and that's bad business for you when people are uninstalling your software.

603
01:01:24,960 --> 01:01:35,000
So even for us, we want to consider and weigh the factors of is it better for us to allow some spam to get through as long as we're not blocking any of the legitimate emails?

604
01:01:35,000 --> 01:01:42,480
Or do we want to make sure that absolutely no spam gets through and if we block legitimate emails, well that's just collateral damage.

605
01:01:42,560 --> 01:01:47,600
So these are considerations that you need to make when you're building models like these and as a data scientist.

606
01:01:49,180 --> 01:01:55,360
So for the multinomial naïve Bayesian with the count vectorizer, we have 94% accuracy.

607
01:01:55,740 --> 01:01:58,640
So let's try a different model.

608
01:01:58,640 --> 01:02:00,840
This one we're going to use the logistic regression.

609
01:02:01,120 --> 01:02:08,960
So if we look at the instructions, we're going to use the logistic regression constructor, but we're going to use this LBFGS solver.

610
01:02:08,960 --> 01:02:16,360
And this is one of the hyperparameters, or at least I would consider it a hyperparameter for logistic regression itself.

611
01:02:16,980 --> 01:02:22,020
There's a couple different solvers that you can use but in this case we're going to use the LBFGS.

612
01:02:22,360 --> 01:02:26,700
So what this is going to look like is LGS TF IDF equals

613
01:02:30,340 --> 01:02:53,640
logistic regression and then we're going to set our solver equals LBFGS and then of course, just like we did with the multinomial naïve Bayesian, we're going to fit our train, our classifier we're going to use the TF IDF X, which is our test data, and then we're going to use our training data,

614
01:02:53,640 --> 01:02:55,980
sorry, and then we're going to use our labels.

615
01:02:57,620 --> 01:03:05,260
Next we're going to hop in and do the score predictions classification matrix and, sorry, confusion matrix and classification reports.

616
01:03:05,260 --> 01:03:19,460
We're going to use score LGS TF IDF LGS TF IDF.score We're just going to look at our test TF IDF X and our test labels.

617
01:03:19,460 --> 01:03:44,360
We're going to have our predictions LGS TF IDF equals LGS TF IDF.predict That's going to give us our raw predictions and then I'm going to stop saying exactly what I'm typing but we're going to do our confusion matrix

618
01:03:59,660 --> 01:04:02,820
and our classification report

619
01:04:17,280 --> 01:04:18,540
Awesome.

620
01:04:18,540 --> 01:05:14,620
So I'm going to let this sit here for a little bit, let you guys kind of copy and follow along Hopefully we don't run into the same problems that we did last time Bender is fantastic I love it, but it will time out after about five minutes and once it times out you've got to kind of restart the whole thing or at least you've got to restart the notebook which is kind of annoying Another thing you will notice is the logistic regression will take a little longer to train and give us our predictions and it's just the way that logistic regression operates it does a little bit more mathematical or complicated mathematical formulas than the multinomial Naive Bayesian so it just takes a little bit longer to train but not too much longer,

621
01:05:14,620 --> 01:05:39,200
so that's good One of them is matrix multiplication which is a whole array of floats that you have to process versus Naive Bayes is literally counting and some logs when you're doing the math correctly which is very quick It's just counting Yeah, it's just counting and logs,

622
01:05:39,200 --> 01:06:15,100
the latest x86 stuff have built-in processes for doing integer logs and stuff like that because it's so common to use logs for numerical accuracy in machine learning You'll also notice the same thing with neural networks because neural networks do a lot of matrix multiplication which is why GPUs work really well for neural networks because you can parallelize matrix multiplications like that So I'm going to go ahead and run this Shift-Enter And it looks like it's running.

623
01:06:15,100 --> 01:06:17,220
Hopefully it doesn't take too long.

624
01:06:19,260 --> 01:06:23,920
Another comparison to make is a neural network versus a gradient-boosted tree.

625
01:06:23,920 --> 01:06:28,580
A gradient-boosted tree is all just if-then statements.

626
01:06:29,460 --> 01:06:34,000
Versus a neural network is a huge pain in the ass to multiply everything.

627
01:06:34,000 --> 01:06:43,060
So a GBT could be run in milliseconds versus a neural network taking seconds in 100% of your CPU.

628
01:06:45,000 --> 01:06:48,120
Actually, speaking of neural networks, I'm kind of curious about this.

629
01:06:48,120 --> 01:06:50,440
How well this would be received at DEF CON.

630
01:06:50,440 --> 01:06:51,940
Would anybody be interested?

631
01:06:51,940 --> 01:06:54,220
And please vote in the Discord channel.

632
01:06:54,240 --> 01:07:05,920
If you would be interested in learning how to build neural networks from scratch or how to build deep learning security tools I think both would be really interesting.

633
01:07:05,920 --> 01:07:09,120
Neural networks from scratch is actually a lot easier than you would imagine.

634
01:07:09,120 --> 01:07:11,480
You can do it in about 11 lines of Python code.

635
01:07:14,640 --> 01:07:33,520
Today we are able to simplify and easily explain the math whereas a lot of these algorithms were invented in the 50s where the people who would do this kind of work were called statisticians or people who would apply these in business scenarios were called business intelligence.

636
01:07:33,960 --> 01:07:39,560
But today with computer science and these abstraction libraries it's a lot easier to understand.

637
01:07:39,580 --> 01:07:47,200
So let me know in the Discord chat if there's any interest in either of those as potentially future workshops for future DEF CON events.

638
01:07:47,220 --> 01:07:49,280
Or even just AI Village events.

639
01:07:49,840 --> 01:07:53,020
So KDAMinMax has a question on Twitch chat.

640
01:07:53,020 --> 01:07:54,980
I'll just read it here.

641
01:07:54,980 --> 01:07:58,380
I have a possibly stupid question about input structures.

642
01:07:58,380 --> 01:08:01,860
If you're learning, there's no real stupid questions.

643
01:08:04,720 --> 01:08:08,860
An email predictor acts on strings or a set of strings.

644
01:08:08,860 --> 01:08:12,600
Are there any machine learning approaches that act reasonably well on trees?

645
01:08:12,640 --> 01:08:16,020
For example, abstract syntax tree representations of applications.

646
01:08:16,600 --> 01:08:28,340
For that, you might want to use a graph convolutional network or, honestly, you can interpret language as an abstract syntax tree.

647
01:08:28,340 --> 01:08:31,020
So you might want to reach in and use some of the...

648
01:08:31,020 --> 01:08:44,680
There's a huge backlog of literature that people largely ignore because of the success of deep learning of how to do language processing using abstract syntax trees.

649
01:08:44,900 --> 01:08:52,660
So you might want to reach into some old school machine learning from before 2010 when neural networks destroyed the world.

650
01:08:53,960 --> 01:08:56,760
I mean, built our jobs.

651
01:08:59,300 --> 01:09:00,500
That was a good answer.

652
01:09:00,500 --> 01:09:01,420
Thank you.

653
01:09:04,600 --> 01:09:09,660
So what we're looking at is logistic regression with TF-IDF vectorizer.

654
01:09:09,660 --> 01:09:11,540
We have a slightly better...

655
01:09:11,540 --> 01:09:16,020
It's a small bump, but it's slightly better than the multinomial with the count vectorizer.

656
01:09:16,020 --> 01:09:20,140
We have a 95.9% accuracy.

657
01:09:20,400 --> 01:09:27,280
We see that we have a little bit more than the 26 ham emails that were incorrectly classified as spam.

658
01:09:27,280 --> 01:09:31,900
We have significantly fewer of the spam emails being classified as ham.

659
01:09:32,100 --> 01:09:44,840
So this would be a perfect instance of moving towards having absolutely zero spam get through our filter at the risk of losing some of our regular emails.

660
01:09:46,020 --> 01:09:52,880
But what I want to show is how the accuracy improves just changing the algorithm itself.

661
01:09:52,880 --> 01:09:59,640
And then looking at multinomial Naive Bayesian compared to logistic regression both using TF-IDF.

662
01:09:59,640 --> 01:10:06,720
We had 87.5% for the multinomial and then we have 95.9% for the logistic regression.

663
01:10:06,720 --> 01:10:14,100
So far, logistic regression is kind of beating out multinomial in the TF-IDF and the count metrics.

664
01:10:14,100 --> 01:10:16,680
So we're going to do our last one which is going to be

665
01:10:19,980 --> 01:10:23,760
logistic regression with count vectorizer.

666
01:10:23,760 --> 01:10:28,860
Count vectorizer worked well for multinomial so let's see if it works well for logistic regression.

667
01:10:28,860 --> 01:10:30,960
So we're going to do LGS count.

668
01:10:36,550 --> 01:10:42,810
And I'm just going to type this out but you can use the same copy-paste trick that I used for the multinomial with the count vectorizer.

669
01:10:44,670 --> 01:10:52,130
I'm just doing this as a way to kind of slow myself down so that you all have an opportunity to copy the code.

670
01:10:52,350 --> 01:10:54,150
So LBFGS.

671
01:10:54,570 --> 01:10:58,190
We're going to fit our

672
01:11:02,540 --> 01:11:03,500
labels.

673
01:11:03,500 --> 01:11:08,940
And we got to do our scores, predictions, confusion matrix, and classification report.

674
01:11:18,730 --> 01:11:33,370
Yeah, one of the things that you kind of reminded me of, Sven, when you've been talking about the different trees and algorithms that have been overtaken by neural networks and deep neural networks is random forests.

675
01:11:33,370 --> 01:11:39,530
Random forests operate very well on a lot of different scenarios.

676
01:11:40,310 --> 01:11:46,810
But because random forests are a cluster of decision trees people go, oh, they're just if-then statements.

677
01:11:46,810 --> 01:11:47,370
This is dumb.

678
01:11:47,370 --> 01:11:48,870
This isn't machine learning.

679
01:11:49,670 --> 01:12:11,190
Yeah, but for us, we have a highly shared data where, for example, on the Ember malware dataset, the lightweight, incredibly cheap gradient decision tree built with LightGBM outperformed any neural network you can throw at it.

680
01:12:11,190 --> 01:12:21,870
So the LightGBM gets 99.8% accuracy or something like that, and the neural network gets 98.7% accuracy, which is significantly worse.

681
01:12:24,350 --> 01:12:43,930
For some tasks, a gradient-based decision tree or something like that is the correct thing mathematically because neural networks have biases because it's easier for them to learn some things than others and GBTs have biases where it's easier for them to learn some things than others.

682
01:12:43,930 --> 01:13:00,810
And one of the great secrets of machine learning is match up your problem with both the requirements, how powerful your predictor needs to be, and the inherent biases of those predictors.

683
01:13:02,130 --> 01:13:02,910
Yeah.

684
01:13:06,480 --> 01:13:08,120
Yeah, that's pretty interesting.

685
01:13:08,120 --> 01:13:09,300
I didn't know that.

686
01:13:10,700 --> 01:13:14,260
But that also reminds me of another thing with neural networks.

687
01:13:14,260 --> 01:13:16,720
Neural networks are really good at learning superstitions.

688
01:13:17,060 --> 01:13:26,700
I don't know if that's a machine learning as a whole thing or specifically in neural networks but I remember there was a dataset that they were using to identify skin cancer cells.

689
01:13:26,940 --> 01:13:34,640
They used a convolutional neural network and based off of that it was learning that if there was a... it was performing too well.

690
01:13:34,640 --> 01:13:47,640
It got nearly 100% and they kind of dug into what it was doing and it said, OK, well if I see a ruler in the picture which the scientists, how they were collecting the images, they had a ruler next to the skin cancer cells.

691
01:13:47,640 --> 01:13:51,080
If I see a ruler in the picture that's how I know it's cancer.

692
01:13:51,200 --> 01:13:53,480
Not the little blob next to it.

693
01:13:55,100 --> 01:14:03,140
So I don't know if that's a whole machine learning problem or specifically a neural net problem but it's just kind of an interesting anecdote.

694
01:14:04,060 --> 01:14:07,740
Be careful how you collect and preprocess and use your data.

695
01:14:09,060 --> 01:14:11,640
A friend of mine did his PhD.

696
01:14:11,640 --> 01:14:26,140
His PhD thesis was inducing the bias that's sort of like the inherent model bias that's in convolutional neural networks into other models and it slightly improved.

697
01:14:29,200 --> 01:14:31,620
By inducing a bias?

698
01:14:31,840 --> 01:14:58,500
Yeah, you can induce a pixel locality bias like a great green tree or a support vector machine and it improves that's one of the assumptions that people make about convolutional neural networks is it allows you to make small pixel local decisions and then those can get glued together in the right way.

699
01:15:00,160 --> 01:15:14,500
And so he went and built little modifications to a dozen other algorithms that have sort of fallen out of favor and all of them improved slightly on images when you induced a pixel local bias.

700
01:15:14,980 --> 01:15:17,160
Okay, interesting.

701
01:15:17,160 --> 01:15:24,900
Yeah , that's that's actually really interesting.

702
01:15:27,480 --> 01:15:33,840
Alright, so hopefully I gave you guys enough time to make sure that this got copied down.

703
01:15:33,840 --> 01:15:36,200
I'm going to go ahead and run it and shift enter.

704
01:15:36,360 --> 01:16:00,020
This is going to take a little bit because it's going through its matrix multiplications but in the meantime, we're about to hop into our final task and what we have is a real spam email I pulled this out of my inbox or my spam folder last May but we see your latest issue is available now if you don't want issue notifications,

705
01:16:00,020 --> 01:16:02,860
click here to unsubscribe Hi George.

706
01:16:02,900 --> 01:16:11,840
My name is not George but we're going to look at this email and we're going to try it against all four of our spam classifiers and see how well they do.

707
01:16:12,080 --> 01:16:15,560
But what's also good about this exercise is it doesn't really hold your hand.

708
01:16:15,560 --> 01:16:19,980
This is usually something that I give plenty of time in a classroom setting.

709
01:16:19,980 --> 01:16:23,220
This allows people to ask questions as I go through.

710
01:16:23,220 --> 01:16:32,180
But this last task is going to say OK, we have a new email we've never seen before run it against our classifiers that we trained.

711
01:16:33,180 --> 01:16:47,000
And so you have to go through the whole process of using the vectorizers to vectorize the tokens or the words in the email and then putting them through the predict function and then printing out the prediction that it gives.

712
01:16:47,960 --> 01:16:56,380
So before we hop into that, I really want this to finish I want it to just be on its own time but

713
01:16:59,600 --> 01:17:21,180
we're going to compare how well these models did once this last one is complete and we're going to see if or how much that comparison has an influence on our test email So we're just going to give this another Probably be another 30 seconds or so.

714
01:17:21,180 --> 01:17:22,860
So it's not too long.

715
01:17:22,860 --> 01:17:24,000
How's the Discord chat?

716
01:17:27,820 --> 01:17:29,460
Discord chat's doing well.

717
01:17:36,560 --> 01:17:40,940
I really appreciate how much discussion we have in this version of the workshop.

718
01:17:41,180 --> 01:17:45,000
Also, and this is something that I want to note for people who are seeing this for the first time.

719
01:17:45,000 --> 01:17:53,140
So the last time I did my workshop on Friday, unfortunately my webcam froze in a very unflattering position, so I just had my mouth gaping open.

720
01:17:54,540 --> 01:17:59,600
So for this one I just put a static image of myself today.

721
01:17:59,940 --> 01:18:04,940
That way you can still see my beautiful face, but you don't get a very unflattering image of it.

722
01:18:05,780 --> 01:18:08,320
Alright, so this one is complete.

723
01:18:08,320 --> 01:18:13,780
We are now at the best model that we have of the ones that we've trained.

724
01:18:13,780 --> 01:18:18,200
We have a 97.69% accuracy.

725
01:18:18,200 --> 01:18:21,100
We can round that up to a 97.7% accuracy.

726
01:18:21,100 --> 01:18:39,680
We see that only 8 out of our 582 ham emails were misclassified as spam, and then only 12 out of our 264 spam emails were misclassified as ham, making this the best model that we have out of the 4 that we've trained.

727
01:18:41,400 --> 01:18:43,980
So that's pretty solid.

728
01:18:44,960 --> 01:19:00,440
So moving forward, what I would do is take this as kind of the better of the models, and I would go ahead and deploy this in production, or add this model into an application that's designed to kind of wrap around the idea of spam and ham.

729
01:19:00,960 --> 01:19:25,540
So if you are simply an email provider, you can use this to filter email, or if you are a spam company, or not a spam company, but like a spam research company, you would use this to grab the spam messages and then collect that as data, and then you can use that to be training the next version of your spam filter.

730
01:19:27,680 --> 01:19:31,580
So let's take a look at our test spam email.

731
01:19:32,180 --> 01:19:42,360
We have our final task, which is to use the vectorizer and the models that we created to perform predictions on our test email and our test spam email.

732
01:19:42,360 --> 01:19:53,420
So to make things easier, what I'm going to do is use a variable, I'll call this working email, and this will allow us to switch pretty quickly.

733
01:19:54,180 --> 01:20:19,620
So I have test email, and then we're going to want to vectorize our email, so we're going to use our test email tfidf, yes, tvec.transform, and this is going to be, remember it needs to be in a list, so we've got to use these square brackets, our working email,

734
01:20:20,280 --> 01:20:27,660
and then we'll do the same thing with our test email count, tvec.transform,

735
01:20:30,760 --> 01:20:32,180
working email.

736
01:20:33,180 --> 01:20:36,520
Now we need to get the raw predictions for every single one of them.

737
01:20:36,520 --> 01:20:42,960
The raw predictions, we already have the variable names here, kind of at the bottom, so I'm just going to go ahead and write them out.

738
01:20:42,960 --> 01:21:06,800
So we have test email mnb with tfidf, which is going to be our mnbtfidf.predict, on our test email with the tfidf vectorized info, we're going to do the same thing for count, using our count.predict,

739
01:21:14,490 --> 01:21:23,850
with our count vectorized, test email lgs, tfidf.predict,

740
01:21:32,020 --> 01:21:38,480
tfidf.data, and our last one, test email lgs.count,

741
01:21:44,520 --> 01:21:45,560
count.

742
01:21:47,040 --> 01:21:52,800
And then, just for fun and games, we can print out our working email.

743
01:21:53,020 --> 01:21:55,240
So this is going to print our test email.

744
01:21:55,240 --> 01:21:59,940
Our test email was the one that we saw, EastAsianFonts and Lenny, thanks for your support.

745
01:22:02,500 --> 01:22:06,640
So you can go ahead and copy this down.

746
01:22:06,640 --> 01:22:11,780
I'm going to go ahead and run this, because we are running close to time.

747
01:22:11,780 --> 01:22:18,140
So this is going to be our test email, and ideally, this should say that it is a regular email, not spam.

748
01:22:18,140 --> 01:22:21,100
It's going to label all four of these as ham.

749
01:22:21,120 --> 01:22:22,620
So let's run it.

750
01:22:22,920 --> 01:22:29,540
Working is not defined, because I forgot to call this working email.

751
01:22:30,200 --> 01:22:31,560
So we're going to run it.

752
01:22:32,280 --> 01:22:33,720
Here's our email.

753
01:22:34,380 --> 01:22:39,500
And all four of our classifiers correctly classified this as ham.

754
01:22:40,360 --> 01:22:48,280
I'm going to scroll back up, and we saved our email as our test spam email.

755
01:22:48,280 --> 01:22:52,520
So I'm going to grab that, and I'll run it.

756
01:22:53,260 --> 01:22:58,200
Oh, I forgot to run the code block that actually defined our test spam email.

757
01:22:59,120 --> 01:23:00,960
And now we can run this.

758
01:23:01,560 --> 01:23:03,060
And we see something interesting here.

759
01:23:03,060 --> 01:23:14,000
Our worst performing model, our MNB TF-IDF, which was 87.5% accurate, misclassified this email as being legitimate, as a ham email.

760
01:23:14,240 --> 01:23:18,100
But the other three of ours classified this correctly as a spam email.

761
01:23:18,740 --> 01:23:20,700
So there's a couple things we can do with this.

762
01:23:20,700 --> 01:23:27,380
If we were so inclined, we can use what's called ensemble learning, which we use all four algorithms.

763
01:23:27,380 --> 01:23:34,820
We take the average of them, or in other words, they give a vote on what they think the class is.

764
01:23:34,820 --> 01:23:38,740
And then in this case, three out of the four say it's spam, so it would be spam.

765
01:23:38,740 --> 01:23:52,340
Or we can use this as an example and say, hey, the MNB TF-IDF, we probably shouldn't use that simply because it's misclassifying known spam emails at a high rate.

766
01:23:52,340 --> 01:24:03,760
87% means that we have about a 12.5% chance of misclassifying spam emails.

767
01:24:05,020 --> 01:24:08,960
So this is kind of pretty close to the conclusion of the workshop.

768
01:24:08,960 --> 01:24:17,520
This shows how you can take raw emails, raw data, transform it in a way that can be used in machine learning models.

769
01:24:17,720 --> 01:24:22,660
And then we build out our machine learning models, we evaluate them, and we can test them.

770
01:24:22,660 --> 01:24:26,220
And this last task shows you how we can properly deploy them.

771
01:24:26,220 --> 01:24:37,580
So we can build a piece of software or an application around it and say every time we get an email, just swap out this working email, and then you can leave everything else the same.

772
01:24:38,800 --> 01:24:41,900
Now, before I let you go, there's one more thing that I want to show you.

773
01:24:42,260 --> 01:24:44,440
I mentioned hyperparameter tuning.

774
01:24:44,440 --> 01:24:48,620
So let's take a look and see what hyperparameter tuning can do.

775
01:24:48,980 --> 01:24:55,620
So when we were talking about the multinomial Naive Bayesian, we saw that we had that smoothing function, the alpha, right?

776
01:24:56,000 --> 01:24:58,280
And it was normally 1.

777
01:24:58,580 --> 01:25:00,980
So we can add that in here.

778
01:25:00,980 --> 01:25:03,360
So we have alpha equals 1.

779
01:25:03,360 --> 01:25:07,900
We see that an 87.5% accuracy.

780
01:25:08,280 --> 01:25:09,720
We go ahead and run that.

781
01:25:09,720 --> 01:25:12,260
I'm just doing this to show you that there are no tricks up my sleeve.

782
01:25:12,260 --> 01:25:15,460
Alpha is by default 1 for this model.

783
01:25:16,240 --> 01:25:20,060
But what if we change that to maybe something a little smaller, right?

784
01:25:20,060 --> 01:25:21,480
Maybe a 0.1.

785
01:25:21,480 --> 01:25:24,200
Let's reduce the impact of that smoothing filter.

786
01:25:24,200 --> 01:25:28,200
So I'm going to go ahead and press Shift-Enter and run that.

787
01:25:28,860 --> 01:25:30,300
And we see something interesting.

788
01:25:30,300 --> 01:25:43,060
So this went from our worst model, 87.5% accuracy, to actually our second best model, 96.5% accuracy.

789
01:25:43,740 --> 01:25:50,720
And so it slightly misclassified some of the legitimate ham emails as spam.

790
01:25:50,720 --> 01:26:00,260
But it drastically reduced the, I think it was 108 spam emails being misclassified down to just 22.

791
01:26:01,740 --> 01:26:04,240
So this is the power of hyperparameter tuning.

792
01:26:04,240 --> 01:26:16,440
And if we go all the way back down to our final task, task 7, and we run this, where the multinomial manipulation with TF-IDF incorrectly classified that as ham, we go ahead and rerun it.

793
01:26:16,760 --> 01:26:20,500
Now it's correctly classifying it as spam because its accuracy is much better.

794
01:26:20,500 --> 01:26:23,840
And the only thing that we did to change it was that hyperparameter tuning.

795
01:26:26,920 --> 01:26:29,060
So that's all I have for you guys today.

796
01:26:29,060 --> 01:26:37,720
I have a couple good resources, but the biggest one that's going to be helpful is if you have any questions, feel free to reach out to me.

797
01:26:37,720 --> 01:26:39,540
I am gtklondike on Gmail.

798
01:26:39,540 --> 01:26:41,760
I'm also gtklondike on Twitter.

799
01:26:42,020 --> 01:26:47,540
I'm not as active on Twitter, but I will see your direct messages or any at mentions.

800
01:26:48,340 --> 01:26:57,000
And then the GitHub code for all of these workbooks, this and all the workbooks, are on github.com slash netsecexplained.

801
01:26:59,060 --> 01:27:01,540
So that is all I have for you guys.

802
01:27:01,540 --> 01:27:04,660
It looks like we just barely made it.

803
01:27:04,780 --> 01:27:05,740
1231.

804
01:27:06,960 --> 01:27:09,160
And I will let you go.

805
01:27:09,160 --> 01:27:16,140
I will stay in the chat channel for a little bit in case any of you would like to continue to discuss.

806
01:27:16,140 --> 01:27:20,100
And I'll also stay in the voice channel for a little bit for the same reason.

807
01:27:20,220 --> 01:27:22,480
So I will see you there.

808
01:27:22,720 --> 01:27:23,900
Have a good day.