WEBVTT 00:03.800 --> 00:06.200 Are we ready to go? 00:07.320 --> 00:08.580 We are. 00:08.700 --> 00:09.740 Perfect. 00:10.340 --> 00:14.280 Hello and welcome to Machine Learning for Security Analysts. 00:14.280 --> 00:19.200 This is our first workshop here at the AI village, DEF CON Safe Mode. 00:19.620 --> 00:24.520 So the goal here is to kind of take machine learning out of the buzzword and into the mainstream. 00:25.200 --> 00:36.620 So buzzwords like machine learning and AI have been in the industry for a couple of years now, but for most people, it's not really well understood what these actually mean. 00:36.900 --> 00:52.340 So my goal is to kind of introduce you to machine learning, and we're going to do this by kind of going over what machine learning is, how it fits in the broader concept of AI, and why it's important for security analysts and hackers to understand machine learning, 00:52.340 --> 00:56.000 as well as what the machine learning process looks like. 00:56.000 --> 01:03.380 And then we're going to take all of that and put that into building our first machine learning model, which will be something very familiar. 01:03.820 --> 01:06.020 So let's go ahead and get started. 01:07.600 --> 01:08.700 So who am I? 01:08.700 --> 01:09.520 My name is Gavin. 01:09.520 --> 01:11.940 I go by the pseudonym GT Klondike. 01:11.940 --> 01:15.220 I'm an independent security researcher and a security consultant. 01:15.220 --> 01:19.980 I have a passion for network security, both attack and defense. 01:20.080 --> 01:33.020 And so through that passion, I run a project called NetSec Explained, which is a blog and a YouTube channel where I use intermediate and advanced-level concepts and try to explain them in a very easy-to-understand way. 01:35.260 --> 01:43.100 So it's usually at this point where, in person, I would kind of ask the audience, when you hear the word machine learning, what comes to mind? 01:43.260 --> 01:58.580 Some of the common answers that I get are, oh, you need to have a PhD in multivariable calculus and linear algebra, or Skynet's going to take over the world, or pattern matching. 01:59.980 --> 02:08.600 So these are all pretty interesting answers, but what we're going to walk through in this workshop is kind of show you that not all of that is necessary. 02:08.600 --> 02:19.520 You don't need a PhD in multivariable calculus or linear algebra, especially with some of the abstraction libraries like Scikit-learn, Keras, TensorFlow, and things like that. 02:19.520 --> 02:24.880 And then as far as machines taking over the world, not quite. 02:27.140 --> 02:32.300 So machine learning and how it fits in the broader idea of artificial intelligence. 02:32.300 --> 02:41.360 So artificial intelligence is this big umbrella term that kind of is a catch-all for different types of ways that machines operate. 02:41.360 --> 02:48.680 So back in the past, when they first invented digital calculators, they thought that was machine intelligence or artificial intelligence. 02:48.680 --> 02:51.400 And then it came to video game AI. 02:52.420 --> 02:58.560 Today we also see things like rules-based artificial intelligence, or classic image recognition. 02:58.560 --> 03:03.320 So there's a bunch of different artificial intelligence technologies that aren't necessarily machine learning. 03:03.560 --> 03:17.060 Machine learning itself is more about applied statistics, and a merge between what used to be considered business intelligence, and statisticians would do this job, and computer science. 03:17.060 --> 03:25.880 With the improvement of computing resources such as memory and CPU, a lot more power can be put behind some of these algorithms. 03:26.640 --> 03:30.260 And then deep learning is specifically talking about deep neural networks. 03:30.260 --> 03:38.140 Neural networks are a type of machine learning algorithm, and then deep learning allows us to stack layers on top of these. 03:38.520 --> 03:45.900 So deep learning, usually these are referred to for convolutional neural networks, which are used for image recognition, primarily. 03:45.900 --> 03:51.940 And then recurrent neural networks, which are used for cyclical pattern recognition, such as heartbeat monitors. 03:54.060 --> 04:05.900 But if we were to find all of this, I would say machine learning is a set of statistical techniques that enables the process of information mining, pattern discovery, and drawing inferences from data. 04:05.900 --> 04:15.300 The idea is that machines learn from the data, instead of us having to program signatures, or heuristics, or invent new algorithms for different problems. 04:15.860 --> 04:29.400 For example, deep learning, we have convolutional neural networks where deciding between whether something is a bike or a bus is the exact same algorithm and exact same problem as deciding whether or not something is a tiger or a cheetah. 04:31.300 --> 04:37.920 So a couple examples, especially in the security space, we have the ability to classify domain generation algorithms. 04:37.920 --> 04:46.540 These are primarily used by botnets as a way for the bot herder to stay in contact with their networks. 04:47.360 --> 05:00.680 Web application firewalls, these we will usually see with being able to perform some anomaly detection and see, okay, this is what normal user traffic looks like compared to abnormal user traffic. 05:00.980 --> 05:11.460 So in this case, the machine learning algorithm identified a potential SQL injection in the parameter because it's very abnormal from what it's used to seeing. 05:12.400 --> 05:14.240 And then, of course, network anomaly detection. 05:14.240 --> 05:25.860 This is one of the big ones, especially in the security side, the defender side of things, where we can predict what we think the network traffic will look like over a period of time and then compare that to what the actual is. 05:25.860 --> 05:35.340 Once it's beyond a certain delta, then that's something that we should pass off to a human to kind of dig in a little deeper and kind of figure out what's going on. 05:36.920 --> 05:38.840 So why is this important? 05:39.100 --> 05:45.260 Well, today we have over a quarter of security products that are used for detection that use some form of machine learning. 05:45.400 --> 05:49.120 And we see machine learning being applied to more and more things every day. 05:49.120 --> 05:57.360 We see facial recognition, we see advanced spam filtering, data heuristics and persona collection. 05:58.100 --> 06:11.260 And so in order for security analysts to properly deploy and manage these machine learning products, they need to have a better understanding of how they operate so that they can ensure that they work efficiently. 06:15.060 --> 06:18.040 So the machine learning process, it follows seven steps. 06:18.240 --> 06:22.340 This is not a step one, step two, step three, but this is really a cycle. 06:22.460 --> 06:27.460 So you start at step one, you go all the way down to step seven, and then you hop back up to step one. 06:27.520 --> 06:30.360 So you start off with gathering the data. 06:30.360 --> 06:36.520 So first you want to identify if I'm going to build a network anomaly detection system, if I'm going to build some sort of hacking tool. 06:36.860 --> 06:40.800 I need to gather data that applies to that, and then I need to prepare the data. 06:40.800 --> 06:47.740 So I can't just take raw information, I need to look at things like metadata and pre-process that. 06:48.220 --> 06:51.220 This step is also known as pre-processing. 06:51.840 --> 06:56.080 We need to choose the model, and the model is going to depend... 06:56.080 --> 06:57.160 There are some heuristics. 06:57.160 --> 07:05.180 Some models work better for different problem sets, but generally we're going to try a couple different models and see what works best. 07:05.640 --> 07:16.580 We're going to train the model, we're going to evaluate the model, and then if we think that the model can perform a little bit better than it's actually performing right now, we can do what's called hyperparameter tuning. 07:16.680 --> 07:20.100 And so this lets us make little tweaks here and there. 07:20.360 --> 07:22.220 And then of course we deploy it. 07:22.220 --> 07:30.820 And as I said earlier, this is a bit of a cyclical process, because machine learning algorithms stop learning as soon as you stop training them. 07:30.820 --> 07:38.960 And so you want to continue to gather new data and train new models and build them into newer and better generations. 07:42.020 --> 07:48.620 So before we get started, what we're going to do is hop in and learn how to build a basic spam filter. 07:48.620 --> 07:54.420 Spam filters are fantastic first examples, especially for a lot of the newbies, because we already know how spam filters work. 07:54.620 --> 08:02.540 If it says something that looks spammy, like buy Viagra now, then that's going to be automatically thrown away into our spam filter. 08:02.720 --> 08:07.660 If we have something that seems legitimate, then that's going to be passed on to the user. 08:08.320 --> 08:13.740 So before we get into that, are there any questions on any of the stuff that I've gone over so far? 08:22.970 --> 08:23.850 Okay. 08:26.810 --> 08:28.030 One more thing. 08:28.190 --> 08:32.330 I totally forgot this, but this is a bit of a story that really... 08:32.330 --> 08:33.310 Yes. 08:36.290 --> 08:37.170 Okay. 08:37.250 --> 08:39.870 This is a story that really made things click for me. 08:40.490 --> 08:46.010 So a friend of mine, this was a few years ago, was approached by a non-profit organization. 08:46.010 --> 08:53.830 And what this non-profit does is they have cameras set up all over Africa and they take pictures to see if there's a cheetah. 08:54.170 --> 08:55.950 And they're tracking cheetah population. 08:55.990 --> 09:03.170 So they'll take pictures, it'll get sent off to a human analyst, and the human will look at it and say, okay, is there a cheetah in the picture or not? 09:03.810 --> 09:08.170 And so they approached my friend and they said, hey, is there a way that we can improve this process? 09:08.170 --> 09:09.690 What's something that we can do here? 09:09.690 --> 09:13.630 And so he said, okay, let me try a couple things. 09:13.910 --> 09:26.050 So what he did was use a very common image library and said, okay, let's train a machine learning algorithm to identify images, if there's a cheetah in the picture or not. 09:26.450 --> 09:35.550 If it reached above a 5% confidence, says, okay, I'm 5% confident or 6% confident that there's a cheetah in this image, it got sent over to an analyst. 09:35.810 --> 09:39.650 If it was less than 5% confident, it would just throw it away. 09:40.630 --> 09:45.210 Now, 5%, that doesn't really sound like a lot, and it's not. 09:46.350 --> 09:51.790 But the ROI of what this model was able to do is pretty astounding. 09:52.090 --> 09:58.270 So what normally would have taken them a year of labor to perform, they were able to do in one month. 09:58.450 --> 10:01.750 That is a 1,200% increase in productivity. 10:02.750 --> 10:04.190 So that's pretty powerful. 10:04.190 --> 10:05.970 And that's just a very simple model. 10:05.970 --> 10:14.630 So as we can develop more and more advanced things, these are ways that we can really improve our processes to move at machine speed and scale. 10:16.730 --> 10:17.670 GT? 10:17.670 --> 10:18.470 Yeah. 10:18.470 --> 10:29.250 We have a question, and it's, any links to information pertaining to creating your own models rather than training existing ones? 10:30.670 --> 10:34.590 Creating your own models rather than training existing ones? 10:34.890 --> 10:35.310 Yeah. 10:35.310 --> 10:38.550 I think that you're going to be showing them how to create their own model. 10:38.550 --> 10:40.370 We are going to create our own model. 10:41.610 --> 10:45.610 So there are existing models that you can use, what's called transfer learning. 10:45.610 --> 10:50.170 So you take a problem, for example, ImageNet, which is a challenge. 10:50.170 --> 10:59.630 It can recognize hundreds of different categories of images, and you can take that idea, train your own model so that it can recognize whether or not there's a cheetah in an image. 11:00.270 --> 11:03.030 But what we're going to do is actually build a model from scratch. 11:03.030 --> 11:10.630 There's no previously trained information, and I'm going to walk you through how to do that, and we're going to see a workbook in a little bit. 11:11.650 --> 11:12.330 Thank you. 11:12.330 --> 11:14.050 Hopefully that helps answer the question. 11:16.750 --> 11:17.670 Okay. 11:18.170 --> 11:26.790 So before we get started, machine learning, it uses statistics, and so what we need to do is grab information and turn them into numbers. 11:27.090 --> 11:38.410 But if we wanted to have the idea of a spam email versus a regular email or a ham email, we need to kind of understand how we're going to interpret words into numbers. 11:39.270 --> 11:46.690 So here's a couple sentences that refer to sports and not sports, or realistically, elections. 11:46.690 --> 11:51.690 So we see a great game, and that talks about sports. 11:51.950 --> 11:55.150 And then we see the election was over, and that talks about not sports. 11:55.790 --> 12:01.130 So the idea is how do we take a sentence that we haven't seen before, because that's what machine learning is supposed to do. 12:01.130 --> 12:06.690 We were trained on certain information, but we want to predict on information that we haven't seen before. 12:07.010 --> 12:13.230 How do we take a new sentence and then try and make a decision whether or not it's talking about sports or not sports? 12:14.950 --> 12:18.350 So to do that, we're going to use what's called Bayes' theorem. 12:18.390 --> 12:42.130 Bayes' theorem is a very common probability theorem that basically says, okay, so if we have a sentence that we already know, and given the probability that this sentence is talking about sports, versus here's a new sentence, what is the probability that it's talking about sports given that new sentence? 12:42.590 --> 12:47.990 So the probability of A given B equals probability of B given A, blah, blah, blah. 12:47.990 --> 12:54.250 We don't need to know too much about this, but it's important to get a good foundation of this understanding. 12:54.250 --> 13:00.230 So the idea is, what is the probability that the sentence, a very close game, is talking about sports? 13:01.050 --> 13:02.430 We don't know the answer to that. 13:02.430 --> 13:10.090 We've never come across the sentence of a very close game, but we would need to fill it into this algorithm here. 13:10.930 --> 13:19.050 So instead, what we can do is actually break up a very close game into their individual words, a, very, close, and game. 13:19.110 --> 13:26.510 And then we can turn that probability into, what's the probability that A is talking about sports? 13:26.510 --> 13:28.990 What's the probability that very is talking about sports? 13:28.990 --> 13:33.810 What's the probability of close talking about sports, and game talking about sports? 13:35.970 --> 13:39.030 So this winds up becoming a bit of a counting game. 13:39.230 --> 13:42.630 How many times do we see A, and it's talking about sports? 13:42.630 --> 13:43.550 Twice. 13:43.910 --> 13:52.790 And of course we need to do the same thing for not sports, because we need to find the probability of A happening, the probability of B happening, and we take whichever is higher. 13:52.910 --> 14:02.250 So the probability that A is talking about sports, we see it twice in two of our sentences, or twice across all of our sports sentences. 14:02.250 --> 14:03.470 Got to be precise. 14:03.770 --> 14:10.110 We don't see A in not sports at all, so that's going to be zero across all of our not sports sentences. 14:10.870 --> 14:15.150 Very shows up in sports once, does not show up in not sports. 14:15.150 --> 14:25.810 Close shows up in not sports once, does not show up in sports, and then game shows up in sports twice, and it doesn't show up in not sports. 14:27.110 --> 14:29.170 So we take those numbers and we plug them in. 14:29.170 --> 14:36.350 We saw for sports, A showed up twice, Very showed up once, Close didn't show up at all, and Game showed up twice. 14:36.450 --> 14:41.890 And then if we count the number of words that we have across all of the sports sentences, we have a total of 11. 14:41.890 --> 14:46.870 So it just becomes a simple 2 divided by 11, 1 divided by 11, and so on. 14:47.550 --> 14:50.370 But we can't use straight, naive Bayesian. 14:50.430 --> 14:53.530 And the reason why is because we have these zeros. 14:53.910 --> 15:01.350 Now, close is a word that we've never seen in sports, so we get zero, and that winds up being zero divided by 11, which is zero. 15:01.350 --> 15:12.950 Zero times everything is zero, and so unfortunately the probability of this sentence being about sports, if we just use straight Bayesian, is zero. 15:13.530 --> 15:15.550 So that doesn't really work for us. 15:16.590 --> 15:20.230 So instead, what we're going to use is called multinomial naive Bayesian. 15:20.230 --> 15:25.050 It's a very similar formula, but we have what's called a smoothing filter, and that's this alpha. 15:25.050 --> 15:27.070 We can set this alpha to whatever we want. 15:27.070 --> 15:28.650 Usually we just set it to 1. 15:29.830 --> 15:36.430 But we can modify that, and this will be a hyperparameter that we'll take a look at later in our workbook. 15:38.590 --> 15:42.250 So with the smoothing filter, we do the exact same word counting. 15:42.250 --> 15:43.750 We see A showed up twice in sports. 15:43.750 --> 15:45.070 Very showed up once in sports. 15:45.070 --> 15:46.210 Close, zero. 15:46.650 --> 15:48.610 Game showed up twice in sports. 15:48.650 --> 15:50.590 And then we do the same thing for not sports. 15:50.810 --> 15:53.530 But you see here where it says close for sports? 15:53.830 --> 15:58.710 That smoothing filter makes it so that that number never becomes zero. 15:58.810 --> 16:00.810 It's always greater than zero. 16:00.810 --> 16:03.090 So our probabilities will always fit. 16:03.530 --> 16:06.450 And then this is just simple addition and then division. 16:07.450 --> 16:18.810 And if we took out a calculator, did all our math, we can see that AVeryCloseGame has a probability of .0000461 that it's talking about sports. 16:18.810 --> 16:25.790 And then AVeryCloseGame has a probability of 0000143 that it's talking about not sports. 16:25.970 --> 16:35.970 So we take whichever is higher, in this case sports, and so our classifier has decided that AVeryCloseGame is talking about sports. 16:39.430 --> 16:41.670 So I know that was a lot of information. 16:41.930 --> 16:43.690 Let me know if there's any questions. 16:44.090 --> 16:51.130 I'm going to kind of hop forward a little bit, but TAs, feel free to chime in if there's any questions on this. 16:51.170 --> 16:56.110 I know this bit's a little confusing sometimes, but it's not something we need to harp on too much. 16:56.110 --> 16:58.930 It's just an underlying foundation that's good to know. 17:01.450 --> 17:19.870 So the five things that we need to keep track of in order to do our pen and pencil math that we did earlier, is we need to keep track of the total number of unique words, the total number of words in spam, the total number of words in ham, and these two were what we've put in the denominator. 17:20.450 --> 17:24.490 The count of each word in spam and the count of each word in ham. 17:24.490 --> 17:30.310 So this is where we had to count the word A, count the word very, count close, count game. 17:31.290 --> 17:34.090 And so these are the five things that we need to pay attention to. 17:35.290 --> 17:37.610 Are there any questions so far? 17:41.050 --> 17:43.730 It looks like we're clear on questions right now. 17:43.770 --> 17:44.650 Awesome. 17:45.770 --> 17:50.490 Okay, so how do we take this idea and expand it to a full email? 17:51.810 --> 17:54.210 Well, here's a spam email. 17:55.110 --> 17:56.910 Yeah, no, this is not a spam email. 17:56.910 --> 17:58.550 This is a regular email. 17:58.550 --> 18:02.310 And we're going to use this as a common example throughout the rest of our workbook. 18:02.650 --> 18:07.390 So in this example email, we have re, re, East Asian fonts in Lenny. 18:07.390 --> 18:08.410 Thanks for your support. 18:08.410 --> 18:10.350 Installing Unifonts did it well for me. 18:11.870 --> 18:24.210 Now, we could take this entire email and look at it, but there are some things that we want to probably remove, things that don't really make a lot of sense, especially in the English language. 18:24.350 --> 18:26.950 So punctuation, punctuation doesn't really make sense. 18:26.950 --> 18:29.610 It doesn't add any context to a sentence. 18:30.950 --> 18:39.870 Small words, which are called stop words, such as in, of, on, for, the, they also don't add any context. 18:39.870 --> 18:46.970 So we can just go ahead and remove that, and that'll reduce the amount of words that we're looking at in this email, because we want to look at keywords. 18:48.810 --> 18:54.990 Things like unsubscribe and unsubscribed, or thanks, or thank you. 18:54.990 --> 18:57.310 They mean the same thing. 18:57.390 --> 19:02.090 A really good example, especially in English, is congrats and congratulations. 19:02.470 --> 19:04.050 They mean the same thing. 19:04.050 --> 19:08.350 And if we want to narrow this down to keywords, we should do what's called stemming. 19:08.350 --> 19:23.290 And so what stemming does is it shortens the word down so that you have I thanked somebody today, as in past tense, thank-ed, thanks, thank with an S, and thank you, where it's just plain thank. 19:23.290 --> 19:25.710 We want to count those all as the same word. 19:27.630 --> 19:38.450 And then, of course, we want to set everything to lowercase, because capital words such as unsubscribe, all caps, is the exact same as unsubscribe, lowercase. 19:38.710 --> 19:41.370 So this is our preprocessing step. 19:41.370 --> 19:43.370 And what we did was we removed all the stop words. 19:43.370 --> 19:47.070 We removed re, in, for, your, did, it, for. 19:48.130 --> 19:50.350 And then we're going to stem the words. 19:50.350 --> 19:56.490 I didn't show the stemming in here, and we'll see in the workbook as we go along what the stemming actually looks like. 19:56.770 --> 20:05.270 But we can see that we kind of shrunk the email a little bit, and this is going to be a little easier for us to compare emails to each other. 20:06.430 --> 20:12.730 So I think this is the point where we can go ahead and open up our workbook. 20:13.430 --> 20:17.110 So I'm going to escape out of there, open this up. 20:17.410 --> 20:20.870 So for our workshops, we're going to do this for this workshop and for the other workshops. 20:20.870 --> 20:22.490 We're going to use a system called MyBender. 20:22.490 --> 20:24.310 MyBender is fantastic. 20:24.310 --> 20:27.870 It uses Jupyter notebooks in a Docker container. 20:27.870 --> 20:36.590 So I'm going to go ahead and copy this and send it off into Discord somehow. 20:37.810 --> 20:41.790 Actually, is there a way... 20:41.790 --> 20:44.870 I'm going to kind of hide my screen for a second. 20:45.730 --> 20:50.530 Yeah, just post it to me, and I'll put it in all the places for you. 20:51.950 --> 20:53.590 Yeah, absolutely. 20:55.690 --> 20:56.570 Okay. 20:56.830 --> 20:58.290 All right, perfect. 20:58.290 --> 20:59.410 Got it. 21:00.150 --> 21:01.450 Display capture. 21:01.490 --> 21:03.870 I'm still getting used to OBS and Twitch streaming. 21:03.870 --> 21:06.590 I haven't done this before, but I'm really excited. 21:07.990 --> 21:16.810 So MyBender, once you get the link, you can just grab that link, paste it in here, or you can click launch, and you can do this with any GitHub repository. 21:17.210 --> 21:20.850 So this is going to launch some Jupyter notebooks. 21:20.850 --> 21:25.150 It takes about a minute or two to spin up. 21:25.430 --> 21:34.030 So I'm going to let you guys get that link from the TA, from the workshops channel, and then I'll let this spin up for me. 21:34.030 --> 21:38.710 I'll wait a little bit longer so that it spins up for you, and then we can get moving. 21:40.130 --> 21:42.570 While this is happening, are there any questions? 21:42.570 --> 21:46.270 Feel free to just write them in the channel, and the TAs will relay them. 22:02.570 --> 22:08.810 And once yours spins up the way that I have this laid out, go into the workbooks directory. 22:09.070 --> 22:18.550 If you would like to follow along in the live coding session, we're going to use the workbook spam filter sklearn document. 22:18.550 --> 22:26.310 If you are not that familiar with Python, and you just want to follow along, you can go ahead and use the completed workbook. 22:27.870 --> 22:32.830 I would give it just another couple of minutes to make sure theirs are spun up. 22:32.830 --> 22:33.950 Yes, absolutely. 22:33.950 --> 22:34.690 Absolutely. 22:36.050 --> 22:39.170 So I just want to check something real quick on my other screen. 23:06.800 --> 23:11.480 And these workbooks outside of this workshop, you can play with them on your own. 23:11.680 --> 23:15.300 I have a couple others if you want to take a look. 23:16.160 --> 23:23.600 But yeah, for the purpose of this workshop, we're going to be using the spam filter sklearn, or the completed spam filter sklearn. 23:32.700 --> 23:33.620 Okay. 23:35.280 --> 23:38.120 So I'm going to go ahead and start. 23:39.020 --> 23:43.660 The first couple things that I'm going to go over are just next, next, next run. 23:43.880 --> 23:47.560 If you're still waiting for your instance to spin up, don't feel bad. 23:47.560 --> 23:49.400 You'll be able to catch up very quickly. 23:51.080 --> 23:54.600 So I'm going to go ahead and open up my spam filter sklearn. 23:54.600 --> 24:00.100 And in this one, we are going to use scikit-learn, which is a wonderful abstraction library. 24:00.240 --> 24:01.660 Great for beginners. 24:03.500 --> 24:07.740 So we're going to go ahead and install a couple tools. 24:07.900 --> 24:17.980 So for these blocks, in order to run them, you can click on them and press shift enter, or you can press run button up here. 24:17.980 --> 24:20.260 You can see that it's running all the Python code. 24:20.260 --> 24:30.140 In this one, I'm just installing the libraries and I am configuring the data directory where all of our spam and our ham emails will be located. 24:31.680 --> 24:36.900 So that's going to take a little bit while that is going. 24:37.820 --> 24:41.380 Just kind of a quick overview of this first block. 24:41.380 --> 24:42.700 We have our imports. 24:42.700 --> 24:44.580 So in this case, we're going to use matplotlib. 24:45.180 --> 24:48.820 Matplotlib is going to let us print some nice graphs. 24:48.820 --> 24:53.020 NumPy and Pandas is going to let us grab some of the information that we need. 24:53.380 --> 24:59.660 We're going to use a couple of the others as some support libraries. 24:59.660 --> 25:04.580 For example, OS, we're going to use that later to kind of grab all of the files. 25:05.420 --> 25:07.480 We want to use NLTK. 25:07.480 --> 25:10.580 NLTK stands for Natural Language Toolkit. 25:10.820 --> 25:16.960 And so this will allow us to identify our stop words and perform our stemming functions. 25:18.100 --> 25:21.820 From scikit-learn, we're going to use a train-test split. 25:21.820 --> 25:35.840 The way that you typically want to train a machine learning model is you have a bunch of clean labeled data and you're going to take probably about 80 or 70% of that and use it to train the model. 25:35.840 --> 25:37.700 But then you want to see how well the model is doing. 25:37.700 --> 25:43.060 So you're going to use the other 20 or 30% as your testing and you're going to use this as evaluation. 25:43.440 --> 25:45.960 It's kind of the same way we teach students in elementary school. 25:45.960 --> 25:50.840 Here's the bulk of the information and now we're going to test you to see how well you understand it. 25:52.620 --> 25:57.840 And then our TF-IDF and count vectorizers, we will get into that in a bit. 25:57.840 --> 26:03.880 These are going to be used for transforming our words into numbers that we can work with. 26:04.960 --> 26:15.980 And then the two models that we're actually going to build out are going to be the multinomial Naive Bayesian, which we went over in the slides, and then one that is completely unfamiliar, logistic regression. 26:16.020 --> 26:23.720 And this is to show how incredible machine learning can be where you can just swap out algorithms. 26:23.780 --> 26:33.180 So we can use SVM, random forest, neural networks, things of that nature without massively changing the way that our data is represented. 26:33.980 --> 26:49.880 And then to kind of represent how our models are doing, we're going to use Seaborn as a graphic representation, and we're going to use a confusion matrix and a classification report to identify exactly where our model is doing well, where it's not doing well, 26:49.880 --> 26:53.920 and how mild changes are going to affect it. 26:54.580 --> 26:57.540 So I'm going to go ahead, shift-enter, run that. 26:58.160 --> 27:02.040 And that's just going to do the import, and we'll see libraries imported. 27:02.440 --> 27:04.660 And then here's our test email from the slides. 27:04.660 --> 27:07.100 This is the East Asian fonts in Lenny. 27:07.100 --> 27:14.440 We're going to see every step of the way how this email is manipulated and transformed into workable data for our machine learning models. 27:14.660 --> 27:18.080 So I'm going to hit shift-enter and run that. 27:20.840 --> 27:22.840 TAs, are there any questions so far? 27:22.840 --> 27:25.740 Is everybody able to get their stuff set up? 27:26.660 --> 27:28.060 No questions so far. 27:28.060 --> 27:29.660 Looks like everything is going well. 27:29.660 --> 27:30.540 Perfect. 27:31.720 --> 27:39.260 So the tokenizer, this is the first step where we want to define a function that will pre-process our information. 27:39.740 --> 27:46.300 So, as we saw in the slides, we want to make everything lowercase, we want to remove the stop words, and we want to stem the words. 27:46.880 --> 27:59.280 So this function is already defined for you, but we can see that we grab all the punctuation, we grab all the stop words using nl2k, and then we define the porter stemmer, which is going to stem our words. 27:59.780 --> 28:19.700 And then we're going to create a list of tokens by taking all of our words, setting them to lowercase, and using the word tokenizer, which will just take a sentence, split it up by its individual words, we're going to strip out the punctuations, and then we are going to merge everything together. 28:20.480 --> 28:22.360 So we're going to go ahead and run that. 28:22.360 --> 28:28.360 It's going to define our function, and we're going to hop into our first task. 28:28.560 --> 28:30.180 So our first task is really easy. 28:30.180 --> 28:37.680 It's just print the full test email, and then we're going to print the tokenized version of the test email. 28:37.820 --> 28:49.880 So in Python we can just do print test email, and then we can do print tokenizer test email. 28:50.160 --> 28:53.040 And then shift-enter, going to go ahead and run that. 28:53.040 --> 28:56.220 So we see the original email, East Asian fonts in Leni. 28:56.380 --> 28:58.340 Then we can see the stop words have been removed. 28:58.340 --> 29:01.660 So we're looking at East Asian font Leni. 29:01.800 --> 29:04.800 Thanks has been turned into thank. 29:05.240 --> 29:06.780 Support install. 29:06.780 --> 29:09.980 Install is missing an L because we see installing. 29:09.980 --> 29:14.440 You have I installed something, or I am going to install something. 29:14.440 --> 29:18.460 And so we're going to stem that so that all of those words are counted the same. 29:20.060 --> 29:21.960 We see unsubscribe. 29:21.960 --> 29:24.540 Unsubscribe shows up twice. 29:24.540 --> 29:30.700 And then we see list.debian.org, which also shows up twice. 29:30.700 --> 29:34.960 So this is the way that our tokenizer modifies the email as we're going through. 29:36.420 --> 29:44.840 And again, just for those that are trying to catch up, all we did was print the test email and then print the tokenizer version of the test email. 29:46.040 --> 29:50.520 There's a question from Twitch chat. 29:55.220 --> 30:01.520 So why are we assuming the punctuation is not relevant? 30:01.800 --> 30:05.180 In our case, we're looking at spam emails. 30:05.180 --> 30:09.320 And so the punctuation, you'll look at commas, hyphens, question marks. 30:09.320 --> 30:15.660 And so in the context, if we're looking at keywords, the punctuation doesn't really matter. 30:15.660 --> 30:25.260 So if you see buy a Viagra now, period, it doesn't make any difference between buy Viagra now, exclamation point, or buy Viagra now with a question mark at the end. 30:25.780 --> 30:28.300 So in the context of the sentence, 30:31.940 --> 30:38.440 that one sentence would likely be classified as spam because it's talking about Viagra and it's very unlikely. 30:38.440 --> 30:41.480 That's a very common spam word we'll see in a lot of spam messages. 30:41.480 --> 30:47.480 So punctuation is just not relevant because it doesn't add any context to the message. 30:48.720 --> 30:50.560 And so that's something that we can remove. 30:50.820 --> 30:58.740 If you would like, you can follow through on this workbook and then try it without removing the punctuation and see if you notice any differences. 30:59.960 --> 31:04.460 But I don't think there would be any differences just because of that context. 31:04.640 --> 31:10.480 There are better things to weigh than periods, question marks, commas, things of that nature. 31:10.480 --> 31:21.080 And then as you see, emails or websites, such as lists.debian.org, they still have their punctuation because that's not the word lists, debian, and org. 31:21.080 --> 31:24.460 That is a URL, lists.debian.org. 31:24.460 --> 31:26.360 And so that's kind of the difference there. 31:26.360 --> 31:30.120 If we had hyphenated words, we would see them included in here as well. 31:33.660 --> 31:36.920 So we're going to go on and load the training data. 31:36.920 --> 31:42.820 This one is just going to give us the counts that we need. 31:42.820 --> 31:44.500 This is just done for you. 31:44.500 --> 31:54.940 Remember, one of those five things that we needed to keep track was all of the word counts in the ham emails, all the word counts in the spam emails. 31:55.200 --> 32:07.600 But what this is going to do is count the number of emails that we have in ham, count the number of emails that we have in spam, and then count the total number of emails that we have in our testing set. 32:07.980 --> 32:12.180 So for here, our testing and training split is already defined. 32:12.180 --> 32:16.660 It's 80-20, and so we'll see that in a moment. 32:17.440 --> 32:22.200 And we go ahead and hop into our second task, which is to load the training data. 32:22.600 --> 32:27.260 So task two, it asks us to create two lists, and we're going to have corpus. 32:28.040 --> 32:33.240 And it's important to get these words identical because of how we use them later on. 32:33.240 --> 32:35.880 So we're going to have a list called corpus. 32:35.880 --> 32:38.700 We're going to have a list called labels. 32:39.300 --> 32:49.680 And then the next bit is we want to load each of the email bodies from the data directory slash ham and data directory slash spam into the corpus array. 32:49.840 --> 33:02.280 So what we're going to do is for each in os.listdir, data.dir, this was defined above, slash ham. 33:03.860 --> 33:09.660 And then we're going to say with each open file, 33:15.290 --> 33:20.970 ham slash plus each, because that's what we defined in our for loop. 33:20.970 --> 33:36.850 We're going to read them as a file named f, corpus.append, f.read, and then labels.append the word ham. 33:37.450 --> 33:42.950 And the reason why is because this third task is load the labels for each email into the labels array. 33:42.950 --> 33:47.710 So in this first code block, we're going to read every single ham email in the ham directory. 33:47.710 --> 33:56.050 We're going to add the bodies to the corpus array, and then we're going to add the label ham into the labels array. 33:56.050 --> 34:00.790 Then we can go ahead and do the exact same thing for spam. 34:00.790 --> 34:09.210 We're just going to have to change each thing to spam, spam, and the labels to spam. 34:09.630 --> 34:12.390 And this will go through and read all of the emails. 34:12.490 --> 34:17.290 So I'm going to leave this up for a moment, let you guys copy that down. 34:28.440 --> 34:32.000 And then I'm going to press shift enter and run it. 34:32.000 --> 34:39.080 So this was really quick, just read all the ham emails, all the spam emails, and now it's loaded into our arrays. 34:39.820 --> 34:53.880 If we want to get an idea of what our class count looks like, we have a total of 2,359 ham emails, 1,100 spam emails, and then just as a graph we can see here. 34:54.340 --> 35:08.240 And I think it's important to graph these counts because sometimes you will have what are considered unbalanced data sets, and so you need to perform other changes here and there just to make sure that the data sets are balanced. 35:08.420 --> 35:15.820 Especially for things like credit card fraud, where that happens less than a percent of the time. 35:15.820 --> 35:25.800 You can build a model that's 99% accurate by just saying everything is a legitimate transaction, but unfortunately that doesn't really help when you're trying to stop credit card fraud. 35:26.080 --> 35:29.660 So when you have imbalanced data sets, there's additional things that you do. 35:29.660 --> 35:35.900 In this case, it looks a little unbalanced, but we'll be able to work with it no problems. 35:38.360 --> 35:41.120 So we are on task 2A. 35:41.460 --> 35:48.400 In this one we want to see the full text body of a random email in our corpus array, and then we want to see the tokenized result of that same email. 35:48.400 --> 35:56.380 So this is going to be very similar to what we did for task 1, which was printing out the test email, and then printing out the tokenized version of the test email. 35:56.380 --> 35:58.720 So we'll just go ahead and grab a random email. 35:58.720 --> 36:02.140 So email equals corpus. 36:03.240 --> 36:04.980 Actually, let me go ahead and do this. 36:04.980 --> 36:09.500 Let's have email ID equals just the number 5. 36:09.500 --> 36:10.280 Why not? 36:10.920 --> 36:15.060 And then we're going to grab email ID. 36:15.760 --> 36:17.820 So this is going to be our random email. 36:18.480 --> 36:20.340 We're going to... 36:20.340 --> 36:23.280 probably going to print a... 36:23.280 --> 36:24.780 No, I already have a divider in there. 36:24.780 --> 36:25.380 Cool. 36:25.380 --> 36:27.100 So let's just print that email. 36:28.180 --> 36:36.500 And then, same thing, we're going to do print tokenizer, and then that same email. 36:36.680 --> 36:41.360 So this line and this line is exactly the same that we did in task 1. 36:41.360 --> 36:44.260 This is the only thing that changes so that we grab a random email. 36:44.260 --> 36:49.000 And then you don't have to put 5, you can put any number, but I'm just picking 5. 36:49.280 --> 36:51.520 So go ahead, Shift-Enter. 36:52.440 --> 36:54.800 And this one's pretty nice. 36:54.800 --> 37:06.380 So we see the body of the email, product review, virttools, dev, URL, product review, virttool, dev, 2.0 URL. 37:06.600 --> 37:12.420 We see the full path for the URL here. 37:12.940 --> 37:15.600 Let's see if there's anything interesting. 37:15.780 --> 37:30.100 Maybe features, feature is shortened to feature, because you can say featured in a motion picture with an ed at the end, features as they have here, or we will feature with just an e at the end. 37:31.320 --> 37:41.440 So this is the way that our tokenizer operates, but we still need to turn these words into numbers, and this is where our vectorizers come in. 37:41.980 --> 37:44.440 So our vectorizers, we're going to use two kinds of vectorizer. 37:44.440 --> 37:47.600 We're going to use the count vectorizer, which just takes the straight word count. 37:47.600 --> 37:49.760 It's exactly what we did in the slides. 37:49.760 --> 37:56.460 Then we're going to use a TF-IDF vectorizer that applies a mathematical weight to each of the words. 37:56.680 --> 38:04.440 And they work slightly different, but let's go ahead and define them, and we'll show you what the difference looks like. 38:04.440 --> 38:08.140 So we're going to name this CVEC as our count vectorizer. 38:08.460 --> 38:12.580 Count vectorizer, and the capitalizations are important. 38:12.680 --> 38:16.500 Then we're going to use our tokenizer function above as our tokenizer function here. 38:16.500 --> 38:20.840 So we're going to have tokenizer equals tokenizer. 38:21.620 --> 38:22.660 Awesome. 38:23.180 --> 38:30.980 And then we want to vectorize our words, so this is going to be saved as countX. 38:30.980 --> 38:36.500 So we're going to have count underscore capital X equals... 38:37.020 --> 38:38.920 Oops, did not mean to open that. 38:42.860 --> 38:46.660 CVEC, our vectorizer, and then we're going to use fit transform. 38:47.900 --> 38:51.320 Fit is Scikit-learn's way of saying train. 38:51.360 --> 38:56.000 So in this instance, we're going to train our vectorizer. 38:56.320 --> 39:02.580 So fit transform, and then our purpose, which is just our list of words. 39:02.980 --> 39:06.640 And then we're going to do the same thing for our TF-IDF vectorizer. 39:06.640 --> 39:10.920 So we're going to take TVEC equals TF-IDF. 39:11.460 --> 39:14.420 Vectorizer and capitalization is important. 39:14.420 --> 39:17.720 Tokenizer equals tokenizer. 39:17.720 --> 39:31.280 And then our TF-IDF X capital X equals TVEC dot fit transform corpus. 39:32.180 --> 39:33.460 Cool. 39:33.460 --> 39:37.520 So I'm going to let that sit there for a second while you guys are typing in. 39:43.580 --> 39:46.000 And then I'm going to go ahead and run it. 39:46.600 --> 39:56.380 So this takes about a minute because what it's doing is it is counting all of the words across all of the emails, the ham emails and the spam emails. 39:56.660 --> 40:06.460 And so as it's counting each of these words, it's applying a word count for the count vectorizer, and then it applies a weight for the TF-IDF vectorizer. 40:06.460 --> 40:08.900 And the weight takes all the words into account. 40:08.900 --> 40:10.620 So it becomes a whole process. 40:10.620 --> 40:15.720 But this takes about 60 to 120 seconds. 40:15.960 --> 40:18.520 So we'll go ahead and let that run. 40:18.600 --> 40:27.320 While that's running, we're just going to get started on task 3A, which is just to print each token from the test email. 40:27.840 --> 40:29.460 Now this one can be a little tricky. 40:29.460 --> 40:34.700 So there's a bit of a trick to identify the unique words and the counts. 40:34.700 --> 40:39.800 So what we're going to do is create a list of dictionary keys. 40:39.800 --> 40:47.100 So what that's going to look like is for i in list dict from keys. 40:48.480 --> 40:55.440 Tokenized email was defined above, and what that is is the tokenized version of our test email. 40:55.820 --> 40:57.900 So we're going to go ahead and print. 40:58.480 --> 41:01.280 I'm just using the Python format. 41:04.220 --> 41:12.520 Tokenized email dot count of each word, and then the word in the tokenized email. 41:12.880 --> 41:16.220 So I'm going to verify vectorizing is complete. 41:17.360 --> 41:22.100 And I'm going to let you guys run your vectorizers, let that sit for a little bit. 41:22.100 --> 41:24.280 And then we're going to run this, because we need... 41:25.680 --> 41:28.720 Oh, I don't believe we do need the previous one to run this. 41:28.720 --> 41:38.200 So if your vectorizer is still going, your code here will not run right away, but it will run as soon as the vectorizer completes. 41:38.440 --> 41:40.540 So I'm going to go ahead and run mine. 41:40.560 --> 41:45.900 So shift-enter, and tokenizer email is not correct. 41:45.900 --> 41:50.400 I think it's tokenized email, that makes a little bit more sense. 41:50.580 --> 41:51.400 I'm going to run that. 41:51.440 --> 41:54.140 Tokenized email is not defined. 41:55.140 --> 41:56.660 Okay, give me a second. 41:56.660 --> 42:01.220 I'm going to scroll up to our tokenizer. 42:07.040 --> 42:09.680 I guess I don't have that defined. 42:12.820 --> 42:14.940 Where is that? 42:22.370 --> 42:24.530 Ah, I was supposed to save it up above. 42:24.530 --> 42:25.550 That's okay. 42:25.690 --> 42:28.590 So we'll go ahead and create it here. 42:29.750 --> 42:36.370 Tokenized email equals tokenizer test email. 42:36.370 --> 42:38.450 So we're going to go ahead and save that. 42:40.790 --> 42:43.910 And here we have the word counts of our test email. 42:43.910 --> 42:46.550 We see the familiar East Asian font, Lenny. 42:46.550 --> 42:53.110 We know unsubscribe showed up twice, and we know list.debian.org showed up twice. 42:53.110 --> 42:56.690 Everything else in here only showed up once. 42:56.690 --> 43:02.790 I know I went very quickly, but this was the line that I needed to add. 43:02.790 --> 43:06.130 Tokenized email equals tokenizer test email. 43:07.130 --> 43:15.110 And then we grab the list of the dictionary from that list of the tokenized email, and we print out the word counts. 43:18.390 --> 43:26.130 So going on to 3b, what we want to see is, okay, we know what the word count looks like if we just did a straight word count. 43:26.130 --> 43:34.130 How does the count vectorizer and the tf.idf vectorizer transform our test email as we're going through? 43:34.610 --> 43:41.590 So we're going to go ahead and use our example CVEC. 43:41.590 --> 43:48.110 And the reason why we're using a separate example version is because we want to redefine a count vectorizer. 43:50.330 --> 43:54.310 Don't forget our tokenizer equals tokenizer. 43:55.330 --> 44:08.290 And then we're going to have our exampleX using our exampleCVEC.fitTransform. 44:08.450 --> 44:11.510 And then it expects a list. 44:12.110 --> 44:20.330 Our test email is a simple string, so we're going to have to wrap it in these brackets so that it becomes a list. 44:20.330 --> 44:22.970 And then we're going to have our test email. 44:24.310 --> 44:25.410 Cool. 44:25.690 --> 44:31.560 And then we're going to go ahead and print our example email. 44:32.470 --> 44:33.510 Oops. 44:33.770 --> 44:35.370 ExampleX. 44:36.050 --> 44:40.170 And then here, we're going to do the same thing. 44:40.170 --> 44:47.270 We're going to have exampleTVEC equals tf.idf vectorizer. 44:47.270 --> 44:49.310 Capitals are important. 44:49.350 --> 44:51.430 Tokenizer equals tokenizer. 44:51.430 --> 44:57.710 And then our example, tf.idf. 44:57.710 --> 45:00.110 I'll just call it exampleX because it doesn't matter. 45:00.110 --> 45:03.790 Once we print out our original exampleX, we don't need it anymore. 45:07.830 --> 45:09.890 ExampleTVEC.fitTransform. 45:13.630 --> 45:17.890 And then the brackets, test email. 45:18.470 --> 45:22.750 And then we can print our exampleX. 45:23.670 --> 45:26.030 So I'm going to let that sit for a moment. 45:26.130 --> 45:28.230 Let you guys copy that down. 45:32.690 --> 45:35.210 I'm going to just look ahead real quick. 45:35.310 --> 45:36.990 We are doing great. 45:40.670 --> 45:41.370 Okay. 45:41.370 --> 45:43.310 So I'm going to go ahead and run this now. 45:43.310 --> 45:44.870 I pressed Shift-Enter. 45:45.910 --> 45:53.090 And if we look at our count vectorizer of test email, we see these different numbers. 45:53.090 --> 45:56.150 And what these numbers represent are tokens. 45:56.150 --> 45:59.390 So, for example, a very nice game. 45:59.390 --> 46:01.170 A would be token 1. 46:01.170 --> 46:02.750 Very would be token 2. 46:02.750 --> 46:04.130 Nice would be token 3. 46:04.130 --> 46:05.970 Game would be token 4. 46:06.530 --> 46:11.850 So in this case, Scikit-Learn actually knows the best way that it wants to tokenize things. 46:11.850 --> 46:17.810 So the letters, for example, 0, this does not necessarily represent East. 46:18.930 --> 46:23.310 And so, basically, Scikit-Learn knows how these tokens are used. 46:23.310 --> 46:24.950 We don't know how these tokens are used. 46:24.950 --> 46:32.490 But something that we can point out is we see token 16 shows up twice and token 9 shows up twice. 46:32.490 --> 46:41.850 So based on what we saw up above, we can guarantee that either token 16 or token 9 is unsubscribe. 46:41.850 --> 46:45.950 And then the other one is list.debian.org. 46:46.350 --> 46:47.510 And so these show up twice. 46:47.510 --> 46:50.730 So this is exactly the same as if we had done the word count ourselves. 46:51.390 --> 46:58.830 The TF-IDF vectorizer, however, uses a bit of a weight algorithm. 46:58.830 --> 47:03.270 And so everything that was set as 1 has this 0.204 weight. 47:03.450 --> 47:10.410 The 2, which is token 9 and token 16, have this 0.408 weight. 47:10.650 --> 47:14.810 And this doesn't necessarily happen in every instance. 47:14.810 --> 47:22.090 But in this instance, you can see the 0.408 shows up twice as much as the 0.204 exactly. 47:22.710 --> 47:28.030 And so this is the way that count vectorizer operates and TF-IDF vectorizer operates. 47:29.270 --> 47:33.810 The underlying, what happens underneath the hood, we don't really need to worry about. 47:33.810 --> 47:40.070 We just need to know how these work and the differences between the two. 47:42.950 --> 47:46.030 So moving on, we're going to load the testing data. 47:46.030 --> 47:47.390 This is already written for us. 47:47.390 --> 47:54.130 So we're just going to grab the first 10 files in the testing directory and print those out. 47:54.130 --> 47:57.930 So we see we just have all these test emails. 47:57.970 --> 48:02.870 And at the end of the file name, we have the label that it's supposed to belong to. 48:02.870 --> 48:06.110 So we have ham, ham, spam, spam, spam, ham. 48:08.830 --> 48:12.130 And so what we're going to do is read each of these emails. 48:12.130 --> 48:17.890 And then we're going to pull off the last little bit of the file name and use that as our label. 48:17.950 --> 48:22.950 And we want to have the testing separate from the training so that we can evaluate how well our model is doing. 48:23.450 --> 48:26.310 So we're going to load the testing data. 48:26.310 --> 48:30.630 And this one's a little complicated, so we have a couple tasks to run through. 48:30.630 --> 48:32.970 So we're going to create two list arrays, kind of like we did earlier. 48:32.970 --> 48:37.150 We're going to have test corpus, and we're going to set that as an empty list. 48:37.150 --> 48:42.610 And we're going to have test labels, which is also going to be an empty list. 48:43.390 --> 48:48.650 And then we are going to load all of the emails from the test directory. 48:48.650 --> 49:05.650 So we're going to say, for filename in os.listdir, our data directory, plus test, which is where it's located, with each file that is open. 49:05.650 --> 49:14.830 And we need to tell it which file test plus filename are. 49:15.470 --> 49:26.050 For read, as filenamed f, we are going to do test corpus.append, f.read. 49:26.690 --> 49:32.610 And then we are going to do something a little weird here, so bear with me. 49:32.610 --> 49:37.110 I'm just going to use re.split, just to split this up. 49:37.310 --> 49:39.690 I'm looking for txt. 49:41.750 --> 49:46.250 And then we're just going to grab the last bit in that split. 49:46.250 --> 49:51.990 So this is going to split the file trainemail.txt. 49:52.450 --> 49:57.210 So it's going to split it to this section, and then to this section. 49:58.050 --> 50:05.830 So this would be considered index 0, and this last little bit would be considered index 1, which is why we have this 1 here at the end. 50:07.570 --> 50:09.290 And then we're going to append the labels. 50:09.290 --> 50:21.290 So test.labels.appendLabel And so I'm going to give you guys a little bit to go ahead and get this all written up. 50:21.410 --> 50:27.090 Be careful, there is a backslash before this period, because this is a regular expression. 50:29.650 --> 50:31.590 And everything should be good. 50:32.390 --> 50:35.670 So I'm just going to let that sit for a little bit, let you guys do your thing. 50:35.670 --> 50:37.850 In the meantime, I'm just going to look ahead real quick. 50:39.950 --> 50:41.850 The rest of this gets so much easier. 50:42.030 --> 50:44.750 It's pretty crazy how easy it gets. 50:46.030 --> 50:48.890 In the meantime, TAs, are there any questions in the chat? 50:57.320 --> 50:59.620 We appear to be good so far. 51:00.060 --> 51:01.020 Awesome. 51:03.200 --> 51:06.260 So I'm going to go ahead and move on. 51:06.260 --> 51:07.720 I'm going to press Shift-Enter. 51:07.720 --> 51:10.440 It's going to load all of those emails. 51:10.440 --> 51:19.660 So this is just reading all of the test emails into our corpus array and grabbing those labels and putting them into the end of our test labels array. 51:20.160 --> 51:24.060 And then just like last time, what this is going to do is print out the number of test emails. 51:24.060 --> 51:27.780 We have 590 ham emails, 276 spam emails. 51:27.780 --> 51:31.200 And here's just a nice pretty graph representing what that looks like. 51:31.860 --> 51:37.900 And as I said earlier, this information or this data has already been split up in an 80-20 split. 51:37.900 --> 51:43.900 So 80% of it is being used for our training and then 20% of it is being used for our testing. 51:46.080 --> 51:51.460 And then this one, it might sound complicated, but this is actually really easy. 51:51.460 --> 51:55.760 We have already trained our CVEC and our TVEC. 51:55.760 --> 52:01.700 And now all we need to do is collect our testing values. 52:01.700 --> 52:06.660 We need to tokenize our testing values using these already trained vectorizers. 52:06.660 --> 52:08.800 So this is super easy. 52:10.800 --> 52:19.860 testCountX equals CVEC.transformTestCorpus. 52:20.140 --> 52:21.460 And that's it. 52:21.580 --> 52:23.060 That's all we've got to do for that. 52:23.480 --> 52:25.320 We can just copy and paste here. 52:25.580 --> 52:31.760 Say our testTFIDFX is our TVEC.transformTestCorpus. 52:32.560 --> 52:43.120 The reason this works is if you remember in the slides how we had the sentences that we already knew about, and then we had to perform a counting game for a very close game. 52:43.180 --> 52:50.820 We had to count how many times a showed up in the sentences that we were already aware of, how many times very showed up, how many times close showed up, how many times game showed up. 52:50.820 --> 52:54.260 So this transform allows us to do just that. 52:54.260 --> 53:02.040 So what this will do is grab each email and it will say, okay, here's the tokenized words of the email that I have. 53:02.040 --> 53:04.500 How many times does it show up in the count vectorizer? 53:04.500 --> 53:08.940 And then what is the weight that it shows up in the TFIDF vectorizer? 53:09.740 --> 53:12.640 So I'm going to go ahead, Shift-Enter, run that. 53:13.180 --> 53:15.260 And just like last time, this is going to take a little bit. 53:15.340 --> 53:18.900 It's counting all the words and associating them with the different tokens. 53:19.280 --> 53:21.980 This is going to be significantly faster than training it. 53:21.980 --> 53:23.680 See, it's almost done already. 53:25.020 --> 53:26.720 And so we're going to go ahead and move on. 53:28.580 --> 53:32.700 So this bit, we're going to get ready to test and evaluate our models. 53:32.700 --> 53:36.840 So this is just a helper function that's going to show us a bunch of really neat statistics. 53:37.100 --> 53:40.940 The details of this helper function you can look into on your own. 53:41.180 --> 53:50.400 But for now, what we need to use as an input is our confusion matrix, our score, and our classification report. 53:51.660 --> 53:54.600 So we're just going to go ahead, Shift-Enter, run that. 53:54.960 --> 53:57.780 Definition defined. 53:59.220 --> 54:09.300 And then here's where we get into tasks 6a, 6b, 6c, and 6d, which is going to be us training and evaluating our different models. 54:09.380 --> 54:22.040 So we're going to start with multinomial Naive Bayesian with a TF-IDF vectorizer, with a count vectorizer, and then we're going to do the same for logistic regression with TF-IDF and logistic regression with a count vectorizer. 54:24.020 --> 54:26.560 So we're going to start out just one thing at a time. 54:27.320 --> 54:30.420 We're going to define our constructor. 54:30.420 --> 54:36.480 So this is going to be MNB TF-IDF equals multinomial NB. 54:36.480 --> 54:38.660 Watch your capitalization. 54:39.580 --> 54:42.360 We're going to define that, and then we're going to train it. 54:43.740 --> 54:52.220 MNB TF-IDF dot fit, which is train in scikit-learn world. 54:52.220 --> 55:00.080 And we're going to use our TF-IDF X, which is our preprocessed information, and then we're going to compare it to our labels. 55:02.060 --> 55:03.860 So that's going to train our model. 55:03.860 --> 55:07.300 Now we're going to grab a couple really neat statistics. 55:07.300 --> 55:12.020 So the first one we're going to get is our score, and this wants us to call it score. 55:14.380 --> 55:18.880 Score MNB TF-IDF predictions using our predict function. 55:18.880 --> 55:27.440 We're going to develop a confusion matrix called C matrix, MNB TF-IDF, and then classification report, C report, MNB TF-IDF. 55:28.540 --> 55:30.280 So we're going to start with our score. 55:30.280 --> 55:37.380 Score MNB TF-IDF equals MNB TF-IDF dot score. 55:37.980 --> 55:43.280 And then we're going to use our test values, because we've trained it on our training data, and we want to test it with our testing. 55:43.280 --> 55:54.060 So we're going to do test TF-IDF X, because that's the way that our system was vectorizing these. 55:54.500 --> 55:58.840 So we don't want to use count vectorizer with a TF-IDF train, because that's not going to work. 55:59.140 --> 56:01.540 And then we're going to use test labels. 56:03.980 --> 56:05.200 That's going to give us our score. 56:05.200 --> 56:06.860 So we're going to get our predictions. 56:07.180 --> 56:14.760 Predictions MNB TF-IDF equals MNB TF-IDF dot predict. 56:14.900 --> 56:18.600 And this is going to give us our raw predictions. 56:21.360 --> 56:29.720 This is going to tell us what it thinks each email is, whereas our score is going to see how accurate it is. 56:29.720 --> 56:33.880 And then this is going to be C matrix. 56:34.880 --> 56:42.680 MNB TF-IDF equals confusion matrix. 56:42.980 --> 56:48.620 We're going to use our test labels and our predictions. 56:48.780 --> 56:51.280 MNB TF-IDF. 56:51.280 --> 57:01.300 So this is going to tell us what it thinks the value is, if it's a spam or ham email, and then what it actually is. 57:01.500 --> 57:05.740 So it's going to give us our true positives, our false positives, our true negatives, and our false negatives. 57:06.080 --> 57:07.700 And then the C report. 57:07.700 --> 57:18.080 We're not going to do too much with this, but what a classification report will do is show us some more metrics, such as the recall and the precision, which we don't need to worry about too much. 57:18.080 --> 57:20.340 We're just going to focus on the accuracy for now. 57:20.340 --> 57:28.740 But for data science, it's important to know what the recall and the precision are as we go through things. 57:30.340 --> 57:34.800 Report, test labels, and then we're going to use our predictions again. 57:36.180 --> 57:45.740 And this is just like our C matrix MNB TF-IDF, in the order that we put things. 57:45.740 --> 57:48.020 We have test labels, test labels, predictions, predictions. 57:48.820 --> 57:49.920 And that is it. 57:49.920 --> 57:52.240 And then we're going to use our helper function generate report. 57:53.560 --> 57:55.680 So I know that's a lot of typing. 57:55.720 --> 58:00.420 I'm going to go ahead and sit here as you guys kind of follow through. 58:00.420 --> 58:03.880 So we have our MNB TF-IDF constructor. 58:03.880 --> 58:07.120 We are going to train it using the fit function. 58:07.120 --> 58:13.980 We're going to train it on our testing data, which is stored as TF-IDFX, and our testing labels, which is just stored as labels. 58:13.980 --> 58:22.560 We are going to use score, MNB TF-IDF.score, to evaluate our accuracy. 58:22.880 --> 58:25.740 We're going to use our predictions to get the raw predictions. 58:25.740 --> 58:35.620 And then with the raw predictions, we are going to develop a confusion matrix and a classification report, which will tell us how well our model is doing visually. 58:38.920 --> 58:41.100 So I'm going to let that sit for one more minute. 58:41.100 --> 58:43.220 TA, are there any questions? 58:52.160 --> 58:56.320 No, I think... well, hang on, let me go back one here. 58:57.940 --> 58:59.060 Nope, still got it. 58:59.060 --> 59:00.300 So I think we're good. 59:00.340 --> 59:00.920 Awesome. 59:01.140 --> 59:01.680 Awesome. 59:01.680 --> 59:02.460 Fantastic. 59:03.500 --> 59:08.100 So I am going to go ahead and move on. 59:08.100 --> 59:09.360 I'm going to press Shift-Enter. 59:09.360 --> 59:10.400 It's going to run. 59:11.260 --> 59:16.800 And now we have our... up here at the top is our classification report. 59:17.280 --> 59:20.940 And so what this is going to do is show us our precision and our recall. 59:20.940 --> 59:23.460 We don't need to worry too much about this one. 59:23.620 --> 59:26.140 Instead, I want you to focus on the lower chart. 59:26.140 --> 59:28.680 So here we have an accuracy of 87.5%. 59:29.420 --> 59:35.840 We can see that all of the actual ham emails were predicted as ham. 59:35.840 --> 59:37.800 So these are our true positives. 59:38.520 --> 59:43.300 A little more than half of our spam emails were correctly predicted as spam. 59:43.300 --> 59:45.760 So these are our true negatives. 59:45.760 --> 59:51.020 And then, unfortunately, it misclassified... 59:51.760 --> 59:52.720 Oh, my mistake. 59:52.720 --> 59:53.760 I have that backwards. 59:53.760 --> 01:00:08.580 So it says 108 of our ham emails it thinks are spam emails, whereas all of the spam emails, so the actual spam emails, were correctly classified as spam. 01:00:08.580 --> 01:00:17.880 And then all of our ham emails, what it thought were ham emails, were ham. 01:00:20.760 --> 01:00:26.440 So we're going to go ahead and do the same thing for 6b, 6c, and 6d. 01:00:27.460 --> 01:00:39.540 One of the beautiful things, and I'm going to show you this just as a super quick and easy way of going about and doing things, is I'm just going to copy the code that I'd already written. 01:00:39.540 --> 01:00:51.740 This is why machine learning is awesome, and very different from programming normal algorithms, is all we need to do is just change these tf.ids to count, and then everything else is the same. 01:00:52.880 --> 01:01:02.140 So we use mnbcount to define our multinomial Naive Bayesian, as we have been asked of us here in 6b. 01:01:03.080 --> 01:01:07.560 mnbcount.fit, and we're going to use our countX because that's how it's been vectorized. 01:01:08.120 --> 01:01:13.980 We're going to change our score mnbcount using mnbcount.score. 01:01:13.980 --> 01:01:25.580 We're going to change our tf.idsx to testCountX, and then likewise kind of all throughout here. 01:01:29.310 --> 01:01:41.310 TestLabels is going to stay the same, predictions is going to change because it is no longer predictions.tf.ids, it is predictionsCount, and then we have our confusion matrix and our classification report. 01:01:41.670 --> 01:01:53.410 So just to go over it one more time, we define our multinomial Naive Bayesian constructor, we train it using the fit function and the count vectorizer, our testing data, our training data. 01:01:54.250 --> 01:02:10.350 We develop our score off of our testCount data using our testLabels, and then we have our predictions, we have our confusion matrix, and our classification report. 01:02:10.350 --> 01:02:16.910 So I'm going to let this sit for just a little bit as you guys go through and copy that down. 01:02:25.810 --> 01:02:32.370 What's really cool is in the GitHub repository, there are other workbooks. 01:02:32.370 --> 01:02:45.510 So for example, we use the exact same methodology to build a malicious URL predictor, so things that look like it's possibly part of a botnet. 01:02:47.510 --> 01:02:49.590 All of this code is exactly the same. 01:02:49.590 --> 01:02:54.990 Once we vectorize our information, we just copy and paste this code in, and it does the exact same thing. 01:02:55.250 --> 01:03:05.610 And so that's the benefit of machine learning, is it learns from the data itself instead of us having to manually program, okay, this solves problem A, but now we have to rewrite this to solve problem B. 01:03:05.610 --> 01:03:07.450 It just all does the same thing. 01:03:09.290 --> 01:03:13.390 So I am going to go ahead and move on. 01:03:13.390 --> 01:03:15.390 I'm going to press Shift-Enter. 01:03:16.710 --> 01:03:18.710 And we see that we got a little bit better, right? 01:03:18.710 --> 01:03:21.090 We are at a 94% accuracy. 01:03:21.630 --> 01:03:28.150 It thinks that some of our ham emails are still spam, but significantly less. 01:03:28.150 --> 01:03:37.750 We went from 108 down to 23, but now it thinks that some of our spam emails are actually ham. 01:03:37.810 --> 01:03:38.670 And so that's a problem. 01:03:38.670 --> 01:03:49.250 So we have a few more emails going through our filter than we would like, but we have significantly fewer of our ham emails from being blocked. 01:03:50.590 --> 01:03:54.110 GT, we've got a question from Twitch chat. 01:03:54.110 --> 01:03:54.650 Absolutely. 01:03:54.650 --> 01:03:56.670 So Chris... 01:03:56.670 --> 01:03:59.270 I'm not going to try and pronounce the name. 01:03:59.270 --> 01:04:08.610 Did you intentionally avoid having spam emails classified as ham when you created the model, or is it simply a try and error kind of thing when training the model? 01:04:09.550 --> 01:04:13.530 Intentionally avoid having spam emails classified as ham? 01:04:14.170 --> 01:04:16.430 Is that what they said? 01:04:16.430 --> 01:04:17.310 Yeah. 01:04:17.990 --> 01:04:22.050 Yeah, we wouldn't want spam emails to be classified as ham. 01:04:22.050 --> 01:04:30.990 We want to draw a line and separate so that you stop getting emails about Viagra and you start getting the regular emails from your school or your work. 01:04:31.130 --> 01:04:34.330 And so this is just kind of the way that spam filters operate. 01:04:34.870 --> 01:04:49.990 So we could build a model that would classify more spam emails as being part of ham or regular emails, or we can build a model that says, okay, we want absolutely zero spam emails. 01:04:49.990 --> 01:04:57.390 If we wind up losing regular emails and classifying those as spam as a mistake, we're okay with that. 01:04:57.770 --> 01:05:01.990 A really good example, actually I'm glad you brought this up, is antivirus. 01:05:02.170 --> 01:05:18.250 So the way that antivirus companies work is they can err on the side of, oh, I think this is a virus, therefore it must be a virus, or this could be benign, or I kind of think this is a virus, but instead of blocking it, I'm just going to let it go through. 01:05:18.870 --> 01:05:20.790 And so you think about the way that malware works. 01:05:23.110 --> 01:05:31.490 You have malware that will make calls out to the internet, it'll probably take over your webcam, it'll try and write files, it'll probably take over your microphone. 01:05:31.750 --> 01:05:33.150 What else does that? 01:05:33.470 --> 01:05:37.910 Chrome, IE, Firefox, browsers. 01:05:38.070 --> 01:05:55.870 And so the idea is instead of blocking what could be legitimate applications, and so that would really interrupt the user experience, they kind of err on the side of, we don't want to block the user experience because then you'll uninstall the antivirus. 01:05:55.990 --> 01:06:04.570 So they are more willing to let viruses go through than blocking legitimate applications. 01:06:05.450 --> 01:06:15.730 And so in this case, these are trade-offs that we have to make as data scientists of if we are okay with letting a few spam emails through versus blocking all of our regular emails. 01:06:16.790 --> 01:06:20.670 So he's specifically asking about task 6A. 01:06:20.670 --> 01:06:22.150 So if you scroll up a little bit. 01:06:22.150 --> 01:06:23.330 Task 6A? 01:06:23.710 --> 01:06:38.750 Yeah, I think the answer is a bit of that and a bit of the question of when you build these models, the training process literally punishes the model for getting it wrong every single time. 01:06:41.510 --> 01:06:43.610 And in this case... 01:06:45.730 --> 01:06:50.770 Yeah, so Chris, you're asking about a false negative. 01:06:54.010 --> 01:07:11.370 A false negative is when you've got something that's malicious, that's misclassified as benign, which is what the point of the model is, but what GT brought up is probably more important is when you have something that's benign that you classify as malicious, 01:07:11.370 --> 01:07:20.890 that is much more common because you get more benign things on users' devices, as GT brought up. 01:07:22.370 --> 01:07:25.450 So in this case, I don't think this was intentional. 01:07:26.930 --> 01:07:28.530 No, this wasn't intentional. 01:07:28.530 --> 01:07:31.090 This is just the way that the model learned. 01:07:32.270 --> 01:07:35.810 And then as we go through the other models, you'll see how these shift. 01:07:36.410 --> 01:07:40.130 Yeah, but this is honestly the way it looks like. 01:07:40.130 --> 01:07:47.470 You've got the ham stuff that are classified as spam is very low so that all the user experience isn't impacted that badly. 01:07:47.510 --> 01:07:55.790 And the spam stuff that's classified as ham is relatively high, but you've got a trade-off between the false positives and the false negatives. 01:07:55.810 --> 01:07:56.970 Yeah, exactly. 01:07:57.430 --> 01:08:01.010 And I hope that answered your question, Christopher. 01:08:01.470 --> 01:08:08.390 I think going through the other models will kind of help answer it a little bit more too. 01:08:10.830 --> 01:08:12.390 So accuracy, 87%. 01:08:12.950 --> 01:08:14.630 87.5%. 01:08:14.630 --> 01:08:16.390 94.3%. 01:08:17.870 --> 01:08:23.450 In this one, we're going to use a model that we haven't used before, logistic regression. 01:08:23.690 --> 01:08:25.970 And so with Scikit-learn, it's very easy. 01:08:25.970 --> 01:08:27.570 It's the exact same thing. 01:08:27.570 --> 01:08:34.990 The difference is we need to use a LBFGS solver. 01:08:35.650 --> 01:08:45.170 So what that's going to look like is LGSTFIDF equals logistic regression. 01:08:45.390 --> 01:08:50.030 Then we're going to set our solver LBFGS. 01:08:50.030 --> 01:08:54.110 And this is just a thing with logistic regression specifically. 01:08:54.430 --> 01:09:00.470 We saw in the slides that multinomial naivetation has a variable called alpha. 01:09:00.470 --> 01:09:01.650 We can tweak that. 01:09:01.650 --> 01:09:03.050 That's one of our hyperparameters. 01:09:03.050 --> 01:09:04.990 This is considered a hyperparameter. 01:09:04.990 --> 01:09:08.090 It's the way that logistic regression interprets the information. 01:09:08.750 --> 01:09:10.890 But for this case, we're just going to use LBFGS. 01:09:11.490 --> 01:09:12.630 We're going to do the same thing. 01:09:12.630 --> 01:09:17.770 We're going to fit LGSTFIDF dot fit 01:09:20.970 --> 01:09:24.110 TFIDFX and then it enables. 01:09:26.570 --> 01:09:28.310 Let me just hop down here. 01:09:28.510 --> 01:09:30.310 We're going to do score equals 01:09:34.090 --> 01:09:46.370 LG S TFIDF equals LGSTFIDF dot score testTFIDFX Pay attention to your capitals. 01:09:46.370 --> 01:09:48.870 Test labels. 01:09:48.870 --> 01:10:11.750 Then we're going to do predictions shuns LGSTFIDF equals LGSTFIDF dot predict and then our test data Typing TFIDF so much it's very confusing for my hands. 01:10:12.010 --> 01:10:12.710 T matrix 01:10:17.850 --> 01:10:48.070 we're going to use our confusion matrix test labels and predictions LGSTFIDF and then our C report LGSTFIDF equals location report test labels predictions TFIDF That should be good. 01:10:48.070 --> 01:10:51.010 Just checking for errors. 01:10:52.550 --> 01:10:54.710 That looks right. 01:10:56.370 --> 01:11:00.930 So, go ahead copy this down. 01:11:00.930 --> 01:11:07.630 I will sit and wait for a bit, but you'll notice a lot of similarities to the multinomial Naive Bayesian. 01:11:07.630 --> 01:11:21.350 In fact, we could have just copied and pasted and just changed it from LGS to MNB or from MNB to LGS and then change our constructor to be a logistic regression with the LBFGS solver. 01:11:24.030 --> 01:11:27.430 We have our logistic regression constructor. 01:11:27.430 --> 01:11:33.050 We train it using the fit function using our TFIDF vectorized data and our labels. 01:11:33.470 --> 01:11:40.670 We use our test data and our test labels to generate a score and to generate predictions. 01:11:41.150 --> 01:11:46.670 And then, off of those predictions, we generate a confusion matrix and a classification report. 01:11:48.070 --> 01:11:53.750 And then, of course, our nice little helper function at the bottom that's going to print us out some helpful stats. 01:11:54.270 --> 01:11:57.330 So , I'm just going to let this sit for a little bit longer. 01:11:58.930 --> 01:12:04.090 Kometh, did Chris ask any more questions, follow-up, or did we take care of them? 01:12:06.010 --> 01:12:11.150 A bunch of people piled on and I helped answer the question because of what he asked. 01:12:11.450 --> 01:12:17.650 Honestly, one of the most important things is the false positive versus false negative trade-off. 01:12:17.650 --> 01:12:18.170 Yeah. 01:12:20.710 --> 01:12:25.670 Yeah, a lot of people have strong opinions about that because it's a lot of... 01:12:25.670 --> 01:12:26.830 It's important. 01:12:27.110 --> 01:12:30.090 It's definitely important, especially in data science. 01:12:30.090 --> 01:12:31.670 This , I'm... 01:12:33.030 --> 01:12:34.970 That might have been a good talk. 01:12:35.350 --> 01:12:41.550 So, here's the interesting thing, is when I first started out, this I thought was amazing. 01:12:41.550 --> 01:12:47.570 Now, as I've learned more over time, I kind of feel like, oh, no, this is a little clunky and a little ham-fisted. 01:12:47.570 --> 01:12:50.770 I'm ignoring this, I'm ignoring this, I'm ignoring this, I'm ignoring this. 01:12:50.770 --> 01:12:54.470 But this is a great way for beginners to understand machine learning. 01:12:55.030 --> 01:13:10.070 And then from there, you can kind of hop into a little bit more of the data science considerations, like imbalanced data sets, true positives, false positives, and some of those trade-offs, as well as more hyperparameter tuning. 01:13:13.790 --> 01:13:21.970 If you want to look at a malware, you keep bringing it up, you can look at the Google Endgame Ember. 01:13:21.970 --> 01:13:25.470 I'll drop a link in the Twitch chat in a sec. 01:13:25.510 --> 01:13:45.010 But there's a malware model that you can look into, like the open-source code, and the final thing at the end of that is it sets the false-positive rate to 0.05% or 0.5% at the end, because that's what you should do in production, because customers care more about false positives, 01:13:45.010 --> 01:13:54.870 because false positives lead to alert fatigue, and alert fatigue leads to just ignoring every single alert, and then all the false negatives get through anyway. 01:13:54.870 --> 01:13:56.770 All the true negatives get through anyway. 01:13:57.910 --> 01:13:59.930 It is something... 01:14:00.750 --> 01:14:04.030 Yeah, and even that model is a little clunky and hampers. 01:14:05.090 --> 01:14:17.770 Well, and it kind of leads to with the alert fatigue, it kind of leads to the common thing that we hear, especially in the technical community from non-technical people, of why does my thing not work? 01:14:18.610 --> 01:14:19.410 Okay. 01:14:20.270 --> 01:14:21.990 So that kind of helps address it. 01:14:21.990 --> 01:14:24.410 But yeah, these are all data science considerations. 01:14:25.410 --> 01:14:27.150 So I'm going to go ahead and run this. 01:14:27.430 --> 01:14:28.650 Shift-Enter. 01:14:29.690 --> 01:14:40.150 And this is going to take a little bit longer than the multinomial Naive Bayesian, because logistic regression interprets the data a little differently, and so it needs to go through and kind of count things properly. 01:14:41.650 --> 01:14:44.810 This takes, I think, about 30 seconds? 01:14:46.310 --> 01:14:48.610 Yeah, it's not too bad. 01:14:48.610 --> 01:14:50.430 It's 30 to 60 seconds. 01:14:51.430 --> 01:14:53.210 So I'm just going to go ahead and run this. 01:14:55.210 --> 01:15:04.850 And then for the logistic regression count, we're going to do the same thing we did with the multinomial Naive Bayesian count and just copy and change words because I'm lazy. 01:15:09.390 --> 01:15:22.710 But yeah, I'm really excited to see that Twitch chat after I hop off to this workshop and see just how passionate people are about the false positive, true positive, and XYZ. 01:15:23.290 --> 01:15:25.750 Oh, we're going to erase it before you get off. 01:15:26.910 --> 01:15:28.250 Ruin my day. 01:15:33.270 --> 01:15:35.630 Yeah, it's taking a little bit longer than I'm used to. 01:15:37.310 --> 01:15:38.630 But that's okay. 01:15:38.630 --> 01:15:46.970 While this is going, I'm just going to go ahead and hop into the next bit and paste this. 01:15:46.970 --> 01:15:51.930 All we need to do is trade this TF-IDF with COUNT. 01:15:53.030 --> 01:15:54.750 So I'm going to go ahead and copy that. 01:15:54.750 --> 01:15:57.910 We're going to use the same constructor and the same solver. 01:15:57.930 --> 01:15:59.730 We're going to use COUNTEX. 01:15:59.730 --> 01:16:02.050 We're going to use COUNTFIT. 01:16:03.630 --> 01:16:07.850 COUNTCOUNT COUNTCOUNT COUNTCOUNT 01:16:12.880 --> 01:16:14.860 Test labels is the same. 01:16:18.410 --> 01:16:20.070 These are predictions. 01:16:21.010 --> 01:16:24.890 And just one quick look over. 01:16:27.010 --> 01:16:28.830 Yep, I have everything in. 01:16:29.430 --> 01:16:31.350 That's exactly how it's supposed to be. 01:16:31.910 --> 01:16:33.230 Scroll up. 01:16:33.870 --> 01:16:35.190 Oh, this is still going. 01:16:35.190 --> 01:16:35.830 Look at that. 01:16:38.990 --> 01:16:39.870 Weird. 01:16:41.250 --> 01:16:42.890 I'm going to let that sit for a bit then. 01:16:43.270 --> 01:16:45.090 So this one, I'm going to let you guys write down. 01:16:45.090 --> 01:16:47.590 This is task 6D. 01:16:48.310 --> 01:16:53.130 So go ahead and make sure that you have these written down while the other one is still going. 01:16:53.130 --> 01:16:59.970 I'm going to run this and then take a look at a logistic regression with the TF-IDF . 01:17:04.570 --> 01:17:06.790 We're actually doing really good on time. 01:17:06.790 --> 01:17:08.610 We have one more task after this. 01:17:08.610 --> 01:17:12.350 It is going to be the most challenging task. 01:17:12.630 --> 01:17:20.990 But it is going to take everything from A to Z and show us what we learned. 01:17:23.940 --> 01:17:27.140 So I just want to check one last time. 01:17:27.220 --> 01:17:28.940 And that is still going. 01:17:32.740 --> 01:17:35.880 So I'm going to go ahead and run this. 01:17:37.060 --> 01:17:40.880 This was 6D that I just clicked to run. 01:17:44.420 --> 01:17:46.840 It says the kernel is idle. 01:17:46.840 --> 01:17:48.020 That's not good. 01:17:52.780 --> 01:17:54.980 Let me see if I can pause this. 01:17:59.080 --> 01:18:00.720 Let me just interrupt it. 01:18:00.980 --> 01:18:01.920 Interrupt the kernel. 01:18:05.540 --> 01:18:07.780 So I'm going to run this because that took a second. 01:18:11.470 --> 01:18:13.330 Uh oh, I hope I didn't time out. 01:18:14.190 --> 01:18:15.650 That would be very bad. 01:18:30.950 --> 01:18:32.370 Let's try and run it again. 01:18:34.230 --> 01:18:35.690 Yeah, that's not good. 01:18:42.270 --> 01:18:44.150 Restart and clear output. 01:18:47.180 --> 01:18:49.620 So this is going to go all the way to the beginning. 01:18:51.140 --> 01:18:52.680 Let's see if I can get this to run. 01:18:52.680 --> 01:18:54.120 Okay, so that runs now. 01:18:55.180 --> 01:18:57.520 So I ran into a bit of a technical difficulty. 01:18:57.640 --> 01:19:04.860 Instead of going through and filling everything out, I'm just going to quickly grab the completed workbook. 01:19:06.260 --> 01:19:15.480 And I'm going to just go ahead, run all. 01:19:17.180 --> 01:19:21.120 This is going to run everything for me. 01:19:22.860 --> 01:19:34.980 So while I'm waiting for some of this stuff to run, let's see that packages are being downloaded, tokenization is defined, everything's good, everything's good, everything's good. 01:19:42.490 --> 01:19:44.010 Where is it stuck now? 01:19:44.010 --> 01:19:46.790 It is stuck... 01:19:50.540 --> 01:19:52.520 Looks like it's stuck reading the data. 01:19:53.720 --> 01:19:55.400 Training spam, training ham. 01:20:03.010 --> 01:20:05.270 That one's done, done. 01:20:08.370 --> 01:20:11.970 Yeah, when you see numbers here, that means that that was the order it was ran in. 01:20:11.970 --> 01:20:14.890 When you see a star, that means that it's still running. 01:20:15.170 --> 01:20:23.730 I'm not sure why the logistic regression was causing problems, but let's take a quick look at... 01:20:28.190 --> 01:20:29.430 Oh. 01:20:30.310 --> 01:20:32.450 Looks like there's some stuff missing. 01:20:37.160 --> 01:20:38.820 Well, that's unfortunate. 01:20:43.000 --> 01:20:47.180 So I'm just going to go ahead and create new code blocks. 01:20:47.700 --> 01:20:55.940 I'm just going to copy the code we had from task 6D and 6C. 01:20:57.080 --> 01:21:00.220 I apologize for scrolling so fast. 01:21:01.300 --> 01:21:07.040 So let's go ahead and grab all of that, paste it in, run it. 01:21:07.940 --> 01:21:10.460 Logistic regression is not defined. 01:21:12.000 --> 01:21:13.180 That's weird. 01:21:17.240 --> 01:21:18.860 Oh, it's not even in here. 01:21:18.860 --> 01:21:21.440 That is something I'm going to have to fix. 01:21:21.440 --> 01:21:23.400 I'm not sure... Oh. 01:21:24.640 --> 01:21:27.220 Because that's a whole different workbook. 01:21:27.220 --> 01:21:30.960 I need complete spam filter sklearn. 01:21:30.960 --> 01:21:32.640 That's the one that I need. 01:21:33.840 --> 01:21:35.560 My mistake. 01:21:36.100 --> 01:21:38.920 So kernel, restart, run all. 01:21:41.880 --> 01:21:44.800 So this is going to run all of them. 01:21:46.000 --> 01:21:49.100 Let's see where it gets stuck. 01:21:49.760 --> 01:21:51.480 Prints out the emails. 01:21:52.340 --> 01:21:54.160 It's training the vectorizers. 01:21:54.160 --> 01:21:56.780 I remember that took about a minute or so. 01:21:59.000 --> 01:22:12.560 But what we want to see after task 6D, we'll look at 6C and 6D, is at the end of this, what we're going to do is grab a spam email. 01:22:12.560 --> 01:22:17.680 This one is actually one that I grabbed directly from my spam box last year. 01:22:18.020 --> 01:22:24.340 And I like it because we can use our spam email predictor to identify whether or not it's spam. 01:22:24.340 --> 01:22:31.300 And then we can look and utilize any of the models to really identify that. 01:22:33.820 --> 01:22:40.980 So this was supposed to be a fill-in-the-blank kind of thing like we've been going through. 01:22:41.460 --> 01:22:56.740 But this last task, it just asks us use the vectorizers and the models created to perform predictions on the test email that we created up at the top and the test spam email which is the real world email that I pulled from a spam inbox. 01:22:57.620 --> 01:23:03.340 And so what we want to do is grab that as a working email. 01:23:03.340 --> 01:23:08.660 And this will let us change this to test email and test spam email. 01:23:10.200 --> 01:23:19.600 Remember, we want to use the tf.idf and the count vectorizers in order to use the working email. 01:23:19.600 --> 01:23:31.140 We need to use that to preprocess and because it's a string, we just wrap it in these square brackets in order to classify it as a list because the transform function expects a list. 01:23:31.800 --> 01:23:36.520 We have that as test email, tf.idf, and test email count. 01:23:37.940 --> 01:23:41.100 And then we use each of our models to form a prediction. 01:23:41.100 --> 01:23:50.860 So we have the multinomial Naive Bayesian with tf.idf, multinomial with count, logistic regression tf.idf, and logistic regression with count. 01:23:50.860 --> 01:23:53.000 They all use the same predict function. 01:23:53.000 --> 01:24:07.580 They're all using either the tf.idf vectorized version of the email or the count vectorized version of the email based on whether or not it's the tf.idf or the count version of the model. 01:24:08.820 --> 01:24:20.800 And then we make our predictions, we print the email itself, and then we can print what each model thinks it is. 01:24:22.660 --> 01:24:29.940 So if you're still on the fill in the blank bit, I encourage you to fill this out. 01:24:30.360 --> 01:24:42.940 If you are using the completed workbook, like I am, it's already filled out and we're just going to go ahead and let everything else run in the background until we get to this point. 01:24:45.840 --> 01:24:47.800 So, sorry about that. 01:24:49.360 --> 01:24:55.900 In the meantime, are there any questions, any thoughts, concerns, ideas? 01:24:55.900 --> 01:24:57.640 I'd love to hear from you. 01:25:15.770 --> 01:25:17.210 All right. 01:25:17.630 --> 01:25:26.510 Psychdude says, on the completed workbook 6a, show the top right and bottom left numbers swapped in 6a. 01:25:26.510 --> 01:25:27.790 Wait a second. 01:25:27.810 --> 01:25:33.530 But the one you were working on earlier had those flipped. 01:25:33.530 --> 01:25:40.430 The fill in the blank and the completed, were they different somehow? 01:25:40.430 --> 01:26:00.950 Yeah, so I'm thinking the reason why they're swapped is because there was a way that Scikit-learn documented how they did their confusion matrix versus how they actually did their confusion matrix, and it was swapped from what you would normally see. 01:26:01.370 --> 01:26:06.610 And so, following what you would normally see is what I did, and that's where the mistake came in. 01:26:06.610 --> 01:26:08.650 And I can actually show you that. 01:26:08.950 --> 01:26:11.410 I was hoping nobody noticed, but good eye. 01:26:12.930 --> 01:26:18.090 So if you look at 6, actually before we go there, let's take a look and see the count. 01:26:18.090 --> 01:26:25.090 So we have 590 ham emails in our test dataset, 276 spam emails in our test dataset. 01:26:25.290 --> 01:26:38.830 And so if we look at 6A, we see 590 ham emails, but as far as actual label, we should see in the column for spam a total of the 276. 01:26:38.830 --> 01:26:40.870 We don't see that. 01:26:40.950 --> 01:26:51.750 And the reason why is because this is supposed to be the actual, see 108 plus 168 is 276, and then this was supposed to be the predicted. 01:26:51.790 --> 01:27:00.750 The problem is the way that scikit-learn swapped them, and I guess I did not swap them back. 01:27:02.290 --> 01:27:13.630 To get the correct result, you just need to switch the test labels and the predictions MNB on both sides for the confusion matrix and the classification report. 01:27:14.630 --> 01:27:15.490 So, yeah. 01:27:15.490 --> 01:27:17.350 That's why they seem swapped. 01:27:17.350 --> 01:27:29.650 The one on the completed workbook is correct, because I believe the completed workbook shows in the actual label column the spam emails, and if you total the two numbers, you get 276. 01:27:29.670 --> 01:27:31.510 So, good eye on that. 01:27:34.980 --> 01:27:36.740 Go ahead and check this. 01:27:36.980 --> 01:27:37.760 Perfect. 01:27:41.220 --> 01:27:48.100 Before we ran into the problems, we're going to look at 6C and 6D with a logistic regression. 01:27:48.720 --> 01:27:58.440 Remember, we had 0.85 or 0.87 0.5% accuracy for the TF-IDF multinomial Naive Bayesian. 01:27:58.440 --> 01:28:02.580 For the count multinomial Naive Bayesian, we have 0.94. 01:28:03.480 --> 01:28:06.320 For the logistic regression, we get a little bit better. 01:28:06.320 --> 01:28:10.300 We're at 0.959 as our accuracy. 01:28:10.580 --> 01:28:19.260 We have fewer ham emails being classified as spam and fewer spam emails being classified as ham. 01:28:19.260 --> 01:28:21.040 We're getting better. 01:28:23.060 --> 01:28:29.320 With the task 6D, this is logistic regression with count vectorizer. 01:28:29.520 --> 01:28:34.040 This is going to be our best model at 0.976. 01:28:34.800 --> 01:28:42.420 Here we only have 8 of the ham emails being classified as spam and 12 of the spam emails being classified as ham. 01:28:42.420 --> 01:28:50.340 Of the four models that we trained, this is the better one in terms of accuracy. 01:28:51.320 --> 01:28:58.940 This is why we want to try different algorithms, different machine learning algorithms, as well as different ways that we preprocess or vectorize our data. 01:28:59.520 --> 01:29:05.620 That way we can create different models, see which ones perform the best, and then we can gravitate towards that one. 01:29:09.800 --> 01:29:15.300 As far as the test spam email, this is a brand new email. 01:29:15.400 --> 01:29:17.060 It comes in on our email server. 01:29:17.060 --> 01:29:20.380 We want to use our models to classify it. 01:29:22.480 --> 01:29:25.140 Let's go ahead and make a prediction. 01:29:26.140 --> 01:29:33.340 As I went over earlier, we're going to have our test spam email, which is called working email, so we can swap this out pretty seamlessly. 01:29:33.580 --> 01:29:39.200 We're going to use our vectorizers that we already defined to transform our working email. 01:29:39.500 --> 01:29:42.300 And then of course, being a string , we need to put it in a list. 01:29:42.300 --> 01:29:44.100 That's what the square brackets are for. 01:29:44.460 --> 01:29:51.260 Then we use all four of our models, the multinomial and the logistic regression with the tf and the count vectorizers. 01:29:51.260 --> 01:29:55.860 We use the predict function on the vectorized emails. 01:29:57.540 --> 01:30:00.940 And then we just go ahead and print out the working email itself. 01:30:01.120 --> 01:30:02.960 Your latest issue is available now. 01:30:02.960 --> 01:30:05.920 If you don't want issue notifications, click here to unsubscribe. 01:30:07.440 --> 01:30:13.800 And then we print out the values that are predicted from each of our models. 01:30:14.140 --> 01:30:16.480 If we look at it, we're going to see something interesting. 01:30:20.020 --> 01:30:27.560 Our best performing model, which is the logistic regression with the count vectorizer, correctly classified this as a spam email. 01:30:27.640 --> 01:30:38.660 But our worst performing model, even though it still had an 87.5% accuracy, our multinomial Naive Bayesian with tf.idf incorrectly classified this as a ham email. 01:30:39.620 --> 01:30:45.760 And so this is also why we want to try building different models and get things that predict much better. 01:30:51.020 --> 01:30:53.700 Let me know if there's any questions on that. 01:30:54.220 --> 01:30:55.920 We are short on time. 01:30:55.920 --> 01:30:59.840 We're actually running a little over, but before I let you go, there's one more thing I want to show you. 01:31:00.040 --> 01:31:03.420 And that is the hyperparameter tuning. 01:31:03.660 --> 01:31:13.940 So if we jump up to that worst performing model, that 0.85%, the multinomial with tf.idf, sorry, 0.875%. 01:31:15.160 --> 01:31:20.500 And we use that alpha that we saw in our formulas on the slides. 01:31:21.320 --> 01:31:23.040 Alpha is set to 1. 01:31:23.040 --> 01:31:24.980 We can go ahead and run that. 01:31:25.060 --> 01:31:27.560 And we see that we're at 0.875. 01:31:28.060 --> 01:31:31.740 But what if we shrink that alpha to 0.1? 01:31:31.740 --> 01:31:33.560 This is hyperparameter tuning. 01:31:33.560 --> 01:31:34.840 We can change it. 01:31:35.780 --> 01:31:42.320 Now our accuracy changed from one of the worst models that we have to one of the better models that we have. 01:31:42.640 --> 01:31:44.480 In fact, this is second place. 01:31:44.660 --> 01:31:48.060 So we have 0.965% accuracy. 01:31:49.480 --> 01:31:55.960 And so you can see how hyperparameter tuning can really help out in tweaking the way that our models operate. 01:32:03.130 --> 01:32:04.870 Before I let you go, 01:32:08.830 --> 01:32:12.550 here's some references for more information. 01:32:12.770 --> 01:32:34.210 And if you want the slides to this and more information from me and the GitHub project that this is all part of, check out github.com.netsec explain and you can shoot me an email with any questions, thoughts, comments at gtklondike at gmail.com So that's all I have for you guys. 01:32:34.210 --> 01:32:40.750 If there's any discussion, TAs, do we have a separate room for the workshops or are we just going to use this Discord? 01:32:41.570 --> 01:32:45.310 Nope, there is a separate room for the workshops here. 01:32:45.310 --> 01:32:49.010 Let me post that in Twitch. 01:32:51.430 --> 01:32:51.910 Um... 01:32:53.190 --> 01:32:54.350 There you go, folks. 01:32:54.350 --> 01:33:05.170 If you have any other questions or want to continue talking about this workshop, you can find us at that channel in the AI Village in the DC Discord. 01:33:06.070 --> 01:33:07.030 Perfect. 01:33:07.670 --> 01:33:10.370 And then I'm going to go ahead and end the stream. 01:33:10.370 --> 01:33:13.690 I'm going to hop into that channel and I will see you guys there. 01:33:15.110 --> 01:33:16.310 Thanks a lot, Gavin. 01:33:16.490 --> 01:33:17.370 Absolutely. 01:33:17.370 --> 01:33:18.230 Thank you.