start	end	text
3800	6200	Are we ready to go?
7320	8580	We are.
8700	9740	Perfect.
10340	14280	Hello and welcome to Machine Learning for Security Analysts.
14280	19200	This is our first workshop here at the AI village, DEF CON Safe Mode.
19620	24520	So the goal here is to kind of take machine learning out of the buzzword and into the mainstream.
25200	30560	So buzzwords like machine learning and AI have been in the industry for a couple of years now,
30560	36620	but for most people, it's not really well understood what these actually mean.
36900	39660	So my goal is to kind of introduce you to machine learning,
39660	43880	and we're going to do this by kind of going over what machine learning is,
43880	46860	how it fits in the broader concept of AI,
47080	52340	and why it's important for security analysts and hackers to understand machine learning,
52340	56000	as well as what the machine learning process looks like.
56000	61340	And then we're going to take all of that and put that into building our first machine learning model,
61340	63380	which will be something very familiar.
63820	66020	So let's go ahead and get started.
67600	71940	So who am I? My name is Gavin. I go by the pseudonym GT Klondike.
71940	75220	I'm an independent security researcher and a security consultant.
75220	79980	I have a passion for network security, both attack and defense.
80080	83880	And so through that passion, I run a project called NetSec Explained,
83880	89420	which is a blog and a YouTube channel where I use intermediate and advanced-level concepts
89420	93020	and try to explain them in a very easy-to-understand way.
95260	99600	So it's usually at this point where, in person, I would kind of ask the audience,
99600	103100	when you hear the word machine learning, what comes to mind?
103260	106660	Some of the common answers that I get are,
106660	112220	oh, you need to have a PhD in multivariable calculus and linear algebra,
112220	118580	or Skynet's going to take over the world, or pattern matching.
119980	123460	So these are all pretty interesting answers,
123460	126500	but what we're going to walk through in this workshop is kind of show you that
126500	128600	not all of that is necessary.
128600	132200	You don't need a PhD in multivariable calculus or linear algebra,
132200	139520	especially with some of the abstraction libraries like Scikit-learn, Keras, TensorFlow, and things like that.
139520	144880	And then as far as machines taking over the world, not quite.
147140	152300	So machine learning and how it fits in the broader idea of artificial intelligence.
152300	157200	So artificial intelligence is this big umbrella term that kind of is a catch-all
157200	161360	for different types of ways that machines operate.
161360	165120	So back in the past, when they first invented digital calculators,
165120	168680	they thought that was machine intelligence or artificial intelligence.
168680	171400	And then it came to video game AI.
172420	176480	Today we also see things like rules-based artificial intelligence,
176480	178560	or classic image recognition.
178560	181080	So there's a bunch of different artificial intelligence technologies
181080	183320	that aren't necessarily machine learning.
183560	187960	Machine learning itself is more about applied statistics,
187960	192140	and a merge between what used to be considered business intelligence,
192140	195120	and statisticians would do this job,
195120	197060	and computer science.
197060	202160	With the improvement of computing resources such as memory and CPU,
202380	205880	a lot more power can be put behind some of these algorithms.
206640	210260	And then deep learning is specifically talking about deep neural networks.
210260	213680	Neural networks are a type of machine learning algorithm,
213680	218140	and then deep learning allows us to stack layers on top of these.
218520	223020	So deep learning, usually these are referred to for convolutional neural networks,
223020	225900	which are used for image recognition, primarily.
225900	230240	And then recurrent neural networks, which are used for cyclical pattern recognition,
230240	231940	such as heartbeat monitors.
234060	236060	But if we were to find all of this,
236060	239800	I would say machine learning is a set of statistical techniques
239800	243200	that enables the process of information mining,
243200	245900	pattern discovery, and drawing inferences from data.
245900	248500	The idea is that machines learn from the data,
248500	252480	instead of us having to program signatures, or heuristics,
252480	255300	or invent new algorithms for different problems.
255860	258860	For example, deep learning, we have convolutional neural networks
258860	262140	where deciding between whether something is a bike or a bus
262140	264800	is the exact same algorithm and exact same problem
264800	269400	as deciding whether or not something is a tiger or a cheetah.
271300	274220	So a couple examples, especially in the security space,
274220	277920	we have the ability to classify domain generation algorithms.
277920	281440	These are primarily used by botnets as a way for the bot herder
281440	286540	to stay in contact with their networks.
287360	290940	Web application firewalls, these we will usually see
290940	295180	with being able to perform some anomaly detection
295180	298180	and see, okay, this is what normal user traffic looks like
298180	300680	compared to abnormal user traffic.
300980	303380	So in this case, the machine learning algorithm
303380	307840	identified a potential SQL injection in the parameter
307840	311460	because it's very abnormal from what it's used to seeing.
312400	314240	And then, of course, network anomaly detection.
314240	316640	This is one of the big ones, especially in the security side,
316640	319980	the defender side of things, where we can predict
319980	323500	what we think the network traffic will look like over a period of time
323500	325860	and then compare that to what the actual is.
325860	329500	Once it's beyond a certain delta, then that's something
329500	333160	that we should pass off to a human to kind of dig in a little deeper
333160	335340	and kind of figure out what's going on.
336920	338840	So why is this important?
339100	341940	Well, today we have over a quarter of security products
341940	345260	that are used for detection that use some form of machine learning.
345400	349120	And we see machine learning being applied to more and more things every day.
349120	353640	We see facial recognition, we see advanced spam filtering,
353640	357360	data heuristics and persona collection.
358100	362660	And so in order for security analysts to properly deploy
362660	365180	and manage these machine learning products,
365180	368400	they need to have a better understanding of how they operate
368400	371260	so that they can ensure that they work efficiently.
375060	378040	So the machine learning process, it follows seven steps.
378240	380820	This is not a step one, step two, step three,
380820	382340	but this is really a cycle.
382460	385480	So you start at step one, you go all the way down to step seven,
385480	387460	and then you hop back up to step one.
387520	390360	So you start off with gathering the data.
390360	392180	So first you want to identify if I'm going to build
392300	394080	a network anomaly detection system,
394080	396520	if I'm going to build some sort of hacking tool.
396860	399020	I need to gather data that applies to that,
399020	400800	and then I need to prepare the data.
400800	404200	So I can't just take raw information,
404200	407740	I need to look at things like metadata and pre-process that.
408220	411220	This step is also known as pre-processing.
411840	416080	We need to choose the model, and the model is going to depend...
416080	417160	There are some heuristics.
417160	420560	Some models work better for different problem sets,
420560	423120	but generally we're going to try a couple different models
423120	425180	and see what works best.
425640	429000	We're going to train the model, we're going to evaluate the model,
429000	432120	and then if we think that the model can perform a little bit better
432120	433940	than it's actually performing right now,
433940	436580	we can do what's called hyperparameter tuning.
436680	440100	And so this lets us make little tweaks here and there.
440360	442220	And then of course we deploy it.
442220	446380	And as I said earlier, this is a bit of a cyclical process,
446380	448680	because machine learning algorithms stop learning
448680	450820	as soon as you stop training them.
450820	453360	And so you want to continue to gather new data
453360	458960	and train new models and build them into newer and better generations.
462020	465800	So before we get started, what we're going to do is hop in
465800	468620	and learn how to build a basic spam filter.
468620	470960	Spam filters are fantastic first examples,
470960	472340	especially for a lot of the newbies,
472340	474420	because we already know how spam filters work.
474620	478760	If it says something that looks spammy, like buy Viagra now,
478760	482540	then that's going to be automatically thrown away into our spam filter.
482720	485720	If we have something that seems legitimate,
485720	487660	then that's going to be passed on to the user.
488320	491180	So before we get into that, are there any questions
491180	493740	on any of the stuff that I've gone over so far?
502970	503850	Okay.
506810	508030	One more thing.
508190	512330	I totally forgot this, but this is a bit of a story that really...
512330	513310	Yes.
516290	517170	Okay.
517250	519870	This is a story that really made things click for me.
520490	522930	So a friend of mine, this was a few years ago,
522930	526010	was approached by a non-profit organization.
526010	529490	And what this non-profit does is they have cameras set up all over Africa
529910	533830	and they take pictures to see if there's a cheetah.
534170	535950	And they're tracking cheetah population.
535990	539310	So they'll take pictures, it'll get sent off to a human analyst,
539310	540530	and the human will look at it and say,
540530	543170	okay, is there a cheetah in the picture or not?
543810	546270	And so they approached my friend and they said,
546270	548170	hey, is there a way that we can improve this process?
548170	549690	What's something that we can do here?
549690	553630	And so he said, okay, let me try a couple things.
553910	557990	So what he did was use a very common image library
557990	562070	and said, okay, let's train a machine learning algorithm
562070	566050	to identify images, if there's a cheetah in the picture or not.
566450	569010	If it reached above a 5% confidence,
569010	571830	says, okay, I'm 5% confident or 6% confident
571830	575550	that there's a cheetah in this image, it got sent over to an analyst.
575810	579650	If it was less than 5% confident, it would just throw it away.
580630	585210	Now, 5%, that doesn't really sound like a lot, and it's not.
586350	591790	But the ROI of what this model was able to do is pretty astounding.
592090	596130	So what normally would have taken them a year of labor to perform,
596130	598270	they were able to do in one month.
598450	601750	That is a 1,200% increase in productivity.
602750	604190	So that's pretty powerful.
604190	605970	And that's just a very simple model.
605970	609550	So as we can develop more and more advanced things,
609550	611810	these are ways that we can really improve our processes
611810	614630	to move at machine speed and scale.
616730	617670	GT?
617670	618470	Yeah.
618470	622010	We have a question, and it's,
622010	626130	any links to information pertaining to creating your own models
626130	629250	rather than training existing ones?
630670	634590	Creating your own models rather than training existing ones?
634890	635310	Yeah.
635310	638550	I think that you're going to be showing them how to create their own model.
638550	640370	We are going to create our own model.
641610	644010	So there are existing models that you can use,
644010	645610	what's called transfer learning.
645610	650170	So you take a problem, for example, ImageNet, which is a challenge.
650170	653190	It can recognize hundreds of different categories of images,
653190	655650	and you can take that idea, train your own model
655650	659630	so that it can recognize whether or not there's a cheetah in an image.
660270	663030	But what we're going to do is actually build a model from scratch.
663030	666870	There's no previously trained information,
666870	668710	and I'm going to walk you through how to do that,
668710	670630	and we're going to see a workbook in a little bit.
671650	672330	Thank you.
672330	674050	Hopefully that helps answer the question.
676750	677670	Okay.
678170	681910	So before we get started, machine learning, it uses statistics,
681910	686790	and so what we need to do is grab information and turn them into numbers.
687090	690350	But if we wanted to have the idea of a spam email
690350	693490	versus a regular email or a ham email,
694110	698410	we need to kind of understand how we're going to interpret words into numbers.
699270	704470	So here's a couple sentences that refer to sports and not sports,
704470	706690	or realistically, elections.
706690	711690	So we see a great game, and that talks about sports.
711950	715150	And then we see the election was over, and that talks about not sports.
715790	719470	So the idea is how do we take a sentence that we haven't seen before,
719470	721130	because that's what machine learning is supposed to do.
721130	723470	We were trained on certain information,
723470	726690	but we want to predict on information that we haven't seen before.
727010	730170	How do we take a new sentence and then try and make a decision
730170	733230	whether or not it's talking about sports or not sports?
734950	738350	So to do that, we're going to use what's called Bayes' theorem.
738390	743250	Bayes' theorem is a very common probability theorem that basically says,
743250	748230	okay, so if we have a sentence that we already know,
748230	754890	and given the probability that this sentence is talking about sports,
754890	758390	versus here's a new sentence,
758390	762130	what is the probability that it's talking about sports given that new sentence?
762590	767990	So the probability of A given B equals probability of B given A, blah, blah, blah.
767990	770650	We don't need to know too much about this,
770650	774250	but it's important to get a good foundation of this understanding.
774250	777070	So the idea is, what is the probability that the sentence,
777130	780230	a very close game, is talking about sports?
781050	782430	We don't know the answer to that.
782430	785850	We've never come across the sentence of a very close game,
785850	790090	but we would need to fill it into this algorithm here.
790930	794570	So instead, what we can do is actually break up a very close game
794570	799050	into their individual words, a, very, close, and game.
799110	801510	And then we can turn that probability into,
801510	806510	what's the probability that A is talking about sports?
806510	808990	What's the probability that very is talking about sports?
808990	811310	What's the probability of close talking about sports,
811310	813810	and game talking about sports?
815970	819030	So this winds up becoming a bit of a counting game.
819230	822630	How many times do we see A, and it's talking about sports?
822630	823550	Twice.
823910	826250	And of course we need to do the same thing for not sports,
826250	829130	because we need to find the probability of A happening,
829130	830970	the probability of B happening,
830970	832790	and we take whichever is higher.
832910	835310	So the probability that A is talking about sports,
835310	839050	we see it twice in two of our sentences,
839050	842250	or twice across all of our sports sentences.
842250	843470	Got to be precise.
843770	846870	We don't see A in not sports at all,
846870	850110	so that's going to be zero across all of our not sports sentences.
850870	855150	Very shows up in sports once, does not show up in not sports.
855150	859550	Close shows up in not sports once,
859550	861350	does not show up in sports,
861350	863830	and then game shows up in sports twice,
863830	865810	and it doesn't show up in not sports.
867110	869170	So we take those numbers and we plug them in.
869170	871670	We saw for sports, A showed up twice,
871670	872790	Very showed up once,
872790	874830	Close didn't show up at all,
874830	876350	and Game showed up twice.
876450	878650	And then if we count the number of words that we have
878650	880570	across all of the sports sentences,
880570	881890	we have a total of 11.
881890	884150	So it just becomes a simple 2 divided by 11,
884150	886870	1 divided by 11, and so on.
887550	890370	But we can't use straight, naive Bayesian.
890430	893530	And the reason why is because we have these zeros.
893910	897250	Now, close is a word that we've never seen in sports,
897250	899630	so we get zero, and that winds up being
899630	901350	zero divided by 11, which is zero.
901350	903310	Zero times everything is zero,
903310	905650	and so unfortunately the probability of
905650	908890	this sentence being about sports,
908890	911190	if we just use straight Bayesian,
911770	912950	is zero.
913530	915550	So that doesn't really work for us.
916590	918330	So instead, what we're going to use is called
918330	920230	multinomial naive Bayesian.
920230	922130	It's a very similar formula, but we have
922130	923710	what's called a smoothing filter,
923710	925050	and that's this alpha.
925050	927070	We can set this alpha to whatever we want.
927070	928650	Usually we just set it to 1.
929830	932070	But we can modify that, and this will be
933130	934830	a hyperparameter that we'll take a look at
934830	936430	later in our workbook.
938590	940550	So with the smoothing filter,
940550	942250	we do the exact same word counting.
942250	943750	We see A showed up twice in sports.
943750	945070	Very showed up once in sports.
945070	946210	Close, zero.
946650	948610	Game showed up twice in sports.
948650	950590	And then we do the same thing for not sports.
950810	953530	But you see here where it says close for sports?
953830	955590	That smoothing filter makes it so that
955590	958710	that number never becomes zero.
958810	960810	It's always greater than zero.
960810	963090	So our probabilities will always fit.
963530	965090	And then this is just simple addition
965090	966450	and then division.
967450	969490	And if we took out a calculator,
969490	971030	did all our math, we can see that
971650	976410	AVeryCloseGame has a probability of .0000461
976410	978810	that it's talking about sports.
978810	980770	And then AVeryCloseGame has a probability of
982530	985790	0000143 that it's talking about not sports.
985970	987250	So we take whichever is higher,
987250	989270	in this case sports, and so our classifier
990790	993510	has decided that AVeryCloseGame
993510	995970	is talking about sports.
999430	1001670	So I know that was a lot of information.
1001930	1003690	Let me know if there's any questions.
1004090	1007290	I'm going to kind of hop forward a little bit,
1007290	1009450	but TAs, feel free to chime in
1009450	1011130	if there's any questions on this.
1011170	1013990	I know this bit's a little confusing sometimes,
1013990	1016110	but it's not something we need to harp on too much.
1016110	1017750	It's just an underlying foundation
1017750	1018930	that's good to know.
1021450	1023430	So the five things that we need to keep track of
1023430	1026450	in order to do our pen and pencil math
1026450	1028330	that we did earlier, is we need to keep track
1028330	1030590	of the total number of unique words,
1030590	1032910	the total number of words in spam,
1032910	1035410	the total number of words in ham,
1035410	1039870	and these two were what we've put in the denominator.
1040450	1042530	The count of each word in spam
1042530	1044490	and the count of each word in ham.
1044490	1047230	So this is where we had to count the word A,
1047230	1050310	count the word very, count close, count game.
1051290	1052550	And so these are the five things
1052550	1054090	that we need to pay attention to.
1055290	1057610	Are there any questions so far?
1061050	1063730	It looks like we're clear on questions right now.
1063770	1064650	Awesome.
1065770	1067970	Okay, so how do we take this idea
1067970	1070490	and expand it to a full email?
1071810	1074210	Well, here's a spam email.
1075110	1076910	Yeah, no, this is not a spam email.
1076910	1078550	This is a regular email.
1078550	1080550	And we're going to use this as a common example
1080550	1082310	throughout the rest of our workbook.
1082650	1084330	So in this example email,
1084330	1087390	we have re, re, East Asian fonts in Lenny.
1087390	1088410	Thanks for your support.
1088410	1090350	Installing Unifonts did it well for me.
1091870	1094530	Now, we could take this entire email
1094530	1096530	and look at it,
1096530	1099030	but there are some things that we want to
1099650	1100790	probably remove,
1100790	1102430	things that don't really make a lot of sense,
1102430	1104210	especially in the English language.
1104350	1106950	So punctuation, punctuation doesn't really make sense.
1106950	1109610	It doesn't add any context to a sentence.
1110950	1114110	Small words, which are called stop words,
1114110	1117510	such as in, of, on, for, the,
1117510	1119870	they also don't add any context.
1119870	1121610	So we can just go ahead and remove that,
1121610	1123850	and that'll reduce the amount of words
1123850	1125330	that we're looking at in this email,
1125330	1126970	because we want to look at keywords.
1128810	1131990	Things like unsubscribe and unsubscribed,
1131990	1134990	or thanks, or thank you.
1134990	1137310	They mean the same thing.
1137390	1139590	A really good example, especially in English,
1139590	1142090	is congrats and congratulations.
1142470	1144050	They mean the same thing.
1144050	1146370	And if we want to narrow this down to keywords,
1146370	1148350	we should do what's called stemming.
1148350	1149530	And so what stemming does
1149530	1151350	is it shortens the word down
1151350	1154270	so that you have I thanked somebody today,
1154270	1156410	as in past tense, thank-ed,
1157490	1160830	thanks, thank with an S,
1160830	1163290	and thank you, where it's just plain thank.
1163290	1165710	We want to count those all as the same word.
1167630	1168450	And then, of course,
1168450	1170610	we want to set everything to lowercase,
1170610	1174510	because capital words such as unsubscribe,
1174510	1178450	all caps, is the exact same as unsubscribe, lowercase.
1178710	1181370	So this is our preprocessing step.
1181370	1183370	And what we did was we removed all the stop words.
1183370	1187070	We removed re, in, for, your, did, it, for.
1188130	1190350	And then we're going to stem the words.
1190350	1192570	I didn't show the stemming in here,
1192570	1194590	and we'll see in the workbook as we go along
1194590	1196490	what the stemming actually looks like.
1196770	1200130	But we can see that we kind of shrunk the email a little bit,
1200130	1201910	and this is going to be a little easier
1201910	1205270	for us to compare emails to each other.
1206430	1209850	So I think this is the point
1209850	1212730	where we can go ahead and open up our workbook.
1213430	1217110	So I'm going to escape out of there, open this up.
1217410	1218670	So for our workshops,
1218670	1219790	we're going to do this for this workshop
1219790	1220870	and for the other workshops.
1220870	1222490	We're going to use a system called MyBender.
1222490	1224310	MyBender is fantastic.
1224310	1227870	It uses Jupyter notebooks in a Docker container.
1227870	1231570	So I'm going to go ahead and copy this
1231570	1236590	and send it off into Discord somehow.
1237810	1241790	Actually, is there a way...
1241790	1244870	I'm going to kind of hide my screen for a second.
1245730	1247210	Yeah, just post it to me,
1247210	1250530	and I'll put it in all the places for you.
1251950	1253590	Yeah, absolutely.
1255690	1256570	Okay.
1256830	1258290	All right, perfect.
1258290	1259410	Got it.
1260150	1261450	Display capture.
1261490	1263870	I'm still getting used to OBS and Twitch streaming.
1263870	1266590	I haven't done this before, but I'm really excited.
1267990	1269610	So MyBender, once you get the link,
1269610	1272430	you can just grab that link, paste it in here,
1272430	1274270	or you can click launch,
1274270	1276810	and you can do this with any GitHub repository.
1277210	1280850	So this is going to launch some Jupyter notebooks.
1280850	1285150	It takes about a minute or two to spin up.
1285430	1288870	So I'm going to let you guys get that link from the TA,
1288870	1290450	from the workshops channel,
1290450	1294030	and then I'll let this spin up for me.
1294030	1296370	I'll wait a little bit longer so that it spins up for you,
1296370	1298710	and then we can get moving.
1300130	1302570	While this is happening, are there any questions?
1302570	1304310	Feel free to just write them in the channel,
1304310	1306270	and the TAs will relay them.
1322570	1325910	And once yours spins up the way that I have this laid out,
1325910	1328810	go into the workbooks directory.
1329070	1332930	If you would like to follow along in the live coding session,
1332930	1338550	we're going to use the workbook spam filter sklearn document.
1338550	1341630	If you are not that familiar with Python,
1341630	1343570	and you just want to follow along,
1343570	1346310	you can go ahead and use the completed workbook.
1347870	1352830	I would give it just another couple of minutes to make sure theirs are spun up.
1352830	1354690	Yes, absolutely. Absolutely.
1356050	1359170	So I just want to check something real quick on my other screen.
1386800	1389140	And these workbooks outside of this workshop,
1389140	1391480	you can play with them on your own.
1391680	1395300	I have a couple others if you want to take a look.
1396160	1398120	But yeah, for the purpose of this workshop,
1398120	1400660	we're going to be using the spam filter sklearn,
1400660	1403600	or the completed spam filter sklearn.
1412700	1413620	Okay.
1415280	1418120	So I'm going to go ahead and start.
1419020	1423660	The first couple things that I'm going to go over are just next, next, next run.
1423880	1427560	If you're still waiting for your instance to spin up, don't feel bad.
1427560	1429400	You'll be able to catch up very quickly.
1431080	1434600	So I'm going to go ahead and open up my spam filter sklearn.
1434600	1437500	And in this one, we are going to use scikit-learn,
1437500	1440100	which is a wonderful abstraction library.
1440240	1441660	Great for beginners.
1443500	1447740	So we're going to go ahead and install a couple tools.
1447900	1451580	So for these blocks, in order to run them,
1451580	1455300	you can click on them and press shift enter,
1455300	1457980	or you can press run button up here.
1457980	1460260	You can see that it's running all the Python code.
1460260	1462080	In this one, I'm just installing the libraries
1462080	1465140	and I am configuring the data directory
1465880	1470140	where all of our spam and our ham emails will be located.
1471680	1476900	So that's going to take a little bit while that is going.
1477820	1481380	Just kind of a quick overview of this first block.
1481380	1482700	We have our imports.
1482700	1484580	So in this case, we're going to use matplotlib.
1485180	1488820	Matplotlib is going to let us print some nice graphs.
1488820	1490360	NumPy and Pandas is going to let us
1490360	1493020	grab some of the information that we need.
1493380	1496280	We're going to use a couple of the others
1497520	1499660	as some support libraries.
1499660	1501560	For example, OS, we're going to use that later
1501560	1504580	to kind of grab all of the files.
1505420	1507480	We want to use NLTK.
1507480	1510580	NLTK stands for Natural Language Toolkit.
1510820	1514220	And so this will allow us to identify our stop words
1514220	1516960	and perform our stemming functions.
1518100	1521820	From scikit-learn, we're going to use a train-test split.
1521820	1525440	The way that you typically want to train a machine learning model
1525440	1528580	is you have a bunch of clean labeled data
1529340	1533280	and you're going to take probably about 80 or 70% of that
1533280	1535840	and use it to train the model.
1535840	1537700	But then you want to see how well the model is doing.
1537700	1541260	So you're going to use the other 20 or 30% as your testing
1541260	1543060	and you're going to use this as evaluation.
1543440	1545960	It's kind of the same way we teach students in elementary school.
1545960	1548520	Here's the bulk of the information and now we're going to test you
1548520	1550840	to see how well you understand it.
1552620	1555880	And then our TF-IDF and count vectorizers,
1555880	1557840	we will get into that in a bit.
1557840	1561220	These are going to be used for transforming our words
1561220	1563880	into numbers that we can work with.
1564960	1567280	And then the two models that we're actually going to build out
1567280	1570280	are going to be the multinomial Naive Bayesian,
1570280	1571640	which we went over in the slides,
1571640	1575980	and then one that is completely unfamiliar, logistic regression.
1576020	1581240	And this is to show how incredible machine learning can be
1581240	1583720	where you can just swap out algorithms.
1583780	1588680	So we can use SVM, random forest, neural networks,
1588680	1591460	things of that nature without massively changing the way
1591460	1593180	that our data is represented.
1593980	1598660	And then to kind of represent how our models are doing,
1598660	1601980	we're going to use Seaborn as a graphic representation,
1601980	1603160	and we're going to use a confusion matrix
1603590	1606670	and a classification report to identify exactly
1607200	1609880	where our model is doing well, where it's not doing well,
1609880	1613920	and how mild changes are going to affect it.
1614580	1617540	So I'm going to go ahead, shift-enter, run that.
1618160	1619940	And that's just going to do the import,
1619940	1622040	and we'll see libraries imported.
1622440	1624660	And then here's our test email from the slides.
1624660	1627100	This is the East Asian fonts in Lenny.
1627100	1630560	We're going to see every step of the way how this email is manipulated
1630560	1634440	and transformed into workable data for our machine learning models.
1634660	1638080	So I'm going to hit shift-enter and run that.
1640840	1642840	TAs, are there any questions so far?
1642840	1645740	Is everybody able to get their stuff set up?
1646660	1649660	No questions so far. Looks like everything is going well.
1649660	1650540	Perfect.
1651720	1655340	So the tokenizer, this is the first step where we want to define
1656000	1659260	a function that will pre-process our information.
1659740	1662920	So, as we saw in the slides, we want to make everything lowercase,
1662920	1666300	we want to remove the stop words, and we want to stem the words.
1666880	1669180	So this function is already defined for you,
1669180	1671660	but we can see that we grab all the punctuation,
1671660	1674220	we grab all the stop words using nl2k,
1674220	1679280	and then we define the porter stemmer, which is going to stem our words.
1679780	1682160	And then we're going to create a list of tokens
1682160	1686920	by taking all of our words, setting them to lowercase,
1686920	1690460	and using the word tokenizer,
1690460	1694340	which will just take a sentence, split it up by its individual words,
1694340	1696340	we're going to strip out the punctuations,
1696340	1699700	and then we are going to merge everything together.
1700480	1702360	So we're going to go ahead and run that.
1702360	1703920	It's going to define our function,
1703920	1708360	and we're going to hop into our first task.
1708560	1710180	So our first task is really easy.
1710180	1713800	It's just print the full test email,
1713800	1717680	and then we're going to print the tokenized version of the test email.
1717820	1723220	So in Python we can just do print test email,
1723220	1729880	and then we can do print tokenizer test email.
1730160	1733040	And then shift-enter, going to go ahead and run that.
1733040	1736220	So we see the original email, East Asian fonts in Leni.
1736380	1738340	Then we can see the stop words have been removed.
1738340	1741660	So we're looking at East Asian font Leni.
1741800	1744800	Thanks has been turned into thank.
1745240	1749980	Support install. Install is missing an L because we see installing.
1749980	1754440	You have I installed something, or I am going to install something.
1754440	1758460	And so we're going to stem that so that all of those words are counted the same.
1760060	1764540	We see unsubscribe. Unsubscribe shows up twice.
1764540	1770700	And then we see list.debian.org, which also shows up twice.
1770700	1774960	So this is the way that our tokenizer modifies the email as we're going through.
1776420	1779280	And again, just for those that are trying to catch up,
1779280	1784840	all we did was print the test email and then print the tokenizer version of the test email.
1786040	1790520	There's a question from Twitch chat.
1795220	1801520	So why are we assuming the punctuation is not relevant?
1801800	1805180	In our case, we're looking at spam emails.
1805180	1809320	And so the punctuation, you'll look at commas, hyphens, question marks.
1809320	1812840	And so in the context, if we're looking at keywords,
1813560	1815660	the punctuation doesn't really matter.
1815660	1818200	So if you see buy a Viagra now, period,
1818200	1821940	it doesn't make any difference between buy Viagra now, exclamation point,
1821940	1825260	or buy Viagra now with a question mark at the end.
1825780	1828300	So in the context of the sentence,
1831940	1834940	that one sentence would likely be classified as spam
1834940	1838440	because it's talking about Viagra and it's very unlikely.
1838440	1841480	That's a very common spam word we'll see in a lot of spam messages.
1841480	1844280	So punctuation is just not relevant
1844280	1847480	because it doesn't add any context to the message.
1848720	1850560	And so that's something that we can remove.
1850820	1854100	If you would like, you can follow through on this workbook
1854100	1856400	and then try it without removing the punctuation
1856400	1858740	and see if you notice any differences.
1859960	1862220	But I don't think there would be any differences
1862220	1864460	just because of that context.
1864640	1869320	There are better things to weigh than periods, question marks, commas,
1869320	1870480	things of that nature.
1870480	1874360	And then as you see, emails or websites,
1874360	1877220	such as lists.debian.org, they still have their punctuation
1877220	1881080	because that's not the word lists, debian, and org.
1881080	1884460	That is a URL, lists.debian.org.
1884460	1886360	And so that's kind of the difference there.
1886360	1890120	If we had hyphenated words, we would see them included in here as well.
1893660	1896920	So we're going to go on and load the training data.
1896920	1902820	This one is just going to give us the counts that we need.
1902820	1904500	This is just done for you.
1904500	1906860	Remember, one of those five things that we needed to keep track
1906860	1912440	was all of the word counts in the ham emails,
1912440	1914940	all the word counts in the spam emails.
1915200	1918120	But what this is going to do is count the number of emails
1918120	1921760	that we have in ham, count the number of emails that we have in spam,
1921760	1924120	and then count the total number of emails
1924120	1927600	that we have in our testing set.
1927980	1932180	So for here, our testing and training split is already defined.
1932180	1936660	It's 80-20, and so we'll see that in a moment.
1937440	1940020	And we go ahead and hop into our second task,
1940020	1942200	which is to load the training data.
1942600	1945900	So task two, it asks us to create two lists,
1945900	1947260	and we're going to have corpus.
1948040	1951160	And it's important to get these words identical
1951160	1953240	because of how we use them later on.
1953240	1955880	So we're going to have a list called corpus.
1955880	1958700	We're going to have a list called labels.
1959300	1962420	And then the next bit is we want to load each of the email bodies
1962420	1967180	from the data directory slash ham and data directory slash spam
1967180	1969680	into the corpus array.
1969840	1976320	So what we're going to do is for each in os.listdir,
1976320	1982280	data.dir, this was defined above, slash ham.
1983860	1989660	And then we're going to say with each open file,
1995290	1998900	ham slash plus each,
1999310	2000970	because that's what we defined in our for loop.
2000970	2006590	We're going to read them as a file named f,
2006590	2010550	corpus.append, f.read,
2010550	2016850	and then labels.append the word ham.
2017450	2019570	And the reason why is because this third task
2019570	2022950	is load the labels for each email into the labels array.
2022950	2024910	So in this first code block, we're going to read
2024910	2027710	every single ham email in the ham directory.
2027710	2030770	We're going to add the bodies to the corpus array,
2030770	2036050	and then we're going to add the label ham into the labels array.
2036050	2040790	Then we can go ahead and do the exact same thing for spam.
2040790	2045730	We're just going to have to change each thing to spam,
2045730	2049210	spam, and the labels to spam.
2049630	2052390	And this will go through and read all of the emails.
2052490	2055090	So I'm going to leave this up for a moment,
2055090	2057290	let you guys copy that down.
2068440	2072000	And then I'm going to press shift enter and run it.
2072000	2075340	So this was really quick, just read all the ham emails,
2075340	2079080	all the spam emails, and now it's loaded into our arrays.
2079820	2084300	If we want to get an idea of what our class count looks like,
2084300	2088860	we have a total of 2,359 ham emails,
2088860	2093880	1,100 spam emails, and then just as a graph we can see here.
2094340	2097320	And I think it's important to graph these counts
2097320	2099800	because sometimes you will have what are considered
2099800	2103160	unbalanced data sets, and so you need to
2103160	2105720	perform other changes here and there
2105720	2108240	just to make sure that the data sets are balanced.
2108420	2112400	Especially for things like credit card fraud,
2112400	2115820	where that happens less than a percent of the time.
2115820	2118280	You can build a model that's 99% accurate
2118280	2121080	by just saying everything is a legitimate transaction,
2121080	2123100	but unfortunately that doesn't really help
2123100	2125800	when you're trying to stop credit card fraud.
2126080	2127920	So when you have imbalanced data sets,
2127920	2129660	there's additional things that you do.
2129660	2131780	In this case, it looks a little unbalanced,
2131780	2135900	but we'll be able to work with it no problems.
2138360	2141120	So we are on task 2A.
2141460	2143700	In this one we want to see the full text body
2143700	2145620	of a random email in our corpus array,
2145620	2148400	and then we want to see the tokenized result of that same email.
2148400	2152300	So this is going to be very similar to what we did for task 1,
2152300	2153760	which was printing out the test email,
2153760	2156380	and then printing out the tokenized version of the test email.
2156380	2158720	So we'll just go ahead and grab a random email.
2158720	2162140	So email equals corpus.
2163240	2164980	Actually, let me go ahead and do this.
2164980	2169500	Let's have email ID equals just the number 5.
2169500	2170280	Why not?
2170920	2175060	And then we're going to grab email ID.
2175760	2177820	So this is going to be our random email.
2178480	2180340	We're going to...
2180340	2183280	probably going to print a...
2183280	2185380	No, I already have a divider in there. Cool.
2185380	2187100	So let's just print that email.
2188180	2193660	And then, same thing, we're going to do print tokenizer,
2194360	2196500	and then that same email.
2196680	2201360	So this line and this line is exactly the same that we did in task 1.
2201360	2204260	This is the only thing that changes so that we grab a random email.
2204260	2206980	And then you don't have to put 5, you can put any number,
2206980	2209000	but I'm just picking 5.
2209280	2211520	So go ahead, Shift-Enter.
2212440	2214800	And this one's pretty nice.
2214800	2221620	So we see the body of the email, product review, virttools, dev, URL,
2221620	2226380	product review, virttool, dev, 2.0 URL.
2226600	2232420	We see the full path for the URL here.
2232940	2235600	Let's see if there's anything interesting.
2235780	2239340	Maybe features, feature is shortened to feature,
2240000	2243780	because you can say featured in a motion picture with an ed at the end,
2243780	2250100	features as they have here, or we will feature with just an e at the end.
2251320	2254060	So this is the way that our tokenizer operates,
2254060	2257460	but we still need to turn these words into numbers,
2257460	2261440	and this is where our vectorizers come in.
2261980	2264440	So our vectorizers, we're going to use two kinds of vectorizer.
2264440	2267600	We're going to use the count vectorizer, which just takes the straight word count.
2267600	2269760	It's exactly what we did in the slides.
2269760	2276460	Then we're going to use a TF-IDF vectorizer that applies a mathematical weight to each of the words.
2276680	2280820	And they work slightly different, but let's go ahead and define them,
2280820	2284440	and we'll show you what the difference looks like.
2284440	2288140	So we're going to name this CVEC as our count vectorizer.
2288460	2292580	Count vectorizer, and the capitalizations are important.
2292680	2296500	Then we're going to use our tokenizer function above as our tokenizer function here.
2296500	2300840	So we're going to have tokenizer equals tokenizer.
2301620	2302660	Awesome.
2303180	2310980	And then we want to vectorize our words, so this is going to be saved as countX.
2310980	2316500	So we're going to have count underscore capital X equals...
2317020	2318920	Oops, did not mean to open that.
2322860	2326660	CVEC, our vectorizer, and then we're going to use fit transform.
2327900	2331320	Fit is Scikit-learn's way of saying train.
2331360	2336000	So in this instance, we're going to train our vectorizer.
2336320	2342580	So fit transform, and then our purpose, which is just our list of words.
2342980	2346640	And then we're going to do the same thing for our TF-IDF vectorizer.
2346640	2350920	So we're going to take TVEC equals TF-IDF.
2351460	2354420	Vectorizer and capitalization is important.
2354420	2357720	Tokenizer equals tokenizer.
2357720	2371280	And then our TF-IDF X capital X equals TVEC dot fit transform corpus.
2372180	2373460	Cool.
2373460	2377520	So I'm going to let that sit there for a second while you guys are typing in.
2383580	2386000	And then I'm going to go ahead and run it.
2386600	2396380	So this takes about a minute because what it's doing is it is counting all of the words across all of the emails, the ham emails and the spam emails.
2396660	2406460	And so as it's counting each of these words, it's applying a word count for the count vectorizer, and then it applies a weight for the TF-IDF vectorizer.
2406460	2408900	And the weight takes all the words into account.
2408900	2410620	So it becomes a whole process.
2410620	2415720	But this takes about 60 to 120 seconds.
2415960	2418520	So we'll go ahead and let that run.
2418600	2427320	While that's running, we're just going to get started on task 3A, which is just to print each token from the test email.
2427840	2429460	Now this one can be a little tricky.
2429460	2434700	So there's a bit of a trick to identify the unique words and the counts.
2434700	2439800	So what we're going to do is create a list of dictionary keys.
2439800	2447100	So what that's going to look like is for i in list dict from keys.
2448480	2455440	Tokenized email was defined above, and what that is is the tokenized version of our test email.
2455820	2457900	So we're going to go ahead and print.
2458480	2461280	I'm just using the Python format.
2464220	2472520	Tokenized email dot count of each word, and then the word in the tokenized email.
2472880	2476220	So I'm going to verify vectorizing is complete.
2477360	2482100	And I'm going to let you guys run your vectorizers, let that sit for a little bit.
2482100	2484280	And then we're going to run this, because we need...
2485680	2488720	Oh, I don't believe we do need the previous one to run this.
2488720	2494840	So if your vectorizer is still going, your code here will not run right away,
2494840	2498200	but it will run as soon as the vectorizer completes.
2498440	2500540	So I'm going to go ahead and run mine.
2500560	2505900	So shift-enter, and tokenizer email is not correct.
2505900	2510400	I think it's tokenized email, that makes a little bit more sense.
2510580	2514140	I'm going to run that. Tokenized email is not defined.
2515140	2521220	Okay, give me a second. I'm going to scroll up to our tokenizer.
2527040	2529680	I guess I don't have that defined.
2532820	2534940	Where is that?
2542370	2545550	Ah, I was supposed to save it up above. That's okay.
2545690	2548590	So we'll go ahead and create it here.
2549750	2556370	Tokenized email equals tokenizer test email.
2556370	2558450	So we're going to go ahead and save that.
2560790	2563910	And here we have the word counts of our test email.
2563910	2566550	We see the familiar East Asian font, Lenny.
2566550	2573110	We know unsubscribe showed up twice, and we know list.debian.org showed up twice.
2573110	2576690	Everything else in here only showed up once.
2576690	2582790	I know I went very quickly, but this was the line that I needed to add.
2582790	2586130	Tokenized email equals tokenizer test email.
2587130	2595110	And then we grab the list of the dictionary from that list of the tokenized email, and we print out the word counts.
2598390	2606130	So going on to 3b, what we want to see is, okay, we know what the word count looks like if we just did a straight word count.
2606130	2614130	How does the count vectorizer and the tf.idf vectorizer transform our test email as we're going through?
2614610	2621590	So we're going to go ahead and use our example CVEC.
2621590	2628110	And the reason why we're using a separate example version is because we want to redefine a count vectorizer.
2630330	2634310	Don't forget our tokenizer equals tokenizer.
2635330	2648290	And then we're going to have our exampleX using our exampleCVEC.fitTransform.
2648450	2651510	And then it expects a list.
2652110	2660330	Our test email is a simple string, so we're going to have to wrap it in these brackets so that it becomes a list.
2660330	2662970	And then we're going to have our test email.
2664310	2665410	Cool.
2665690	2671560	And then we're going to go ahead and print our example email.
2672470	2673510	Oops.
2673770	2675370	ExampleX.
2676050	2680170	And then here, we're going to do the same thing.
2680170	2687270	We're going to have exampleTVEC equals tf.idf vectorizer.
2687270	2689310	Capitals are important.
2689350	2691430	Tokenizer equals tokenizer.
2691430	2697710	And then our example, tf.idf.
2697710	2700110	I'll just call it exampleX because it doesn't matter.
2700110	2703790	Once we print out our original exampleX, we don't need it anymore.
2707830	2709890	ExampleTVEC.fitTransform.
2713630	2717890	And then the brackets, test email.
2718470	2722750	And then we can print our exampleX.
2723670	2726030	So I'm going to let that sit for a moment.
2726130	2728230	Let you guys copy that down.
2732690	2735210	I'm going to just look ahead real quick.
2735310	2736990	We are doing great.
2740670	2741370	Okay.
2741370	2743310	So I'm going to go ahead and run this now.
2743310	2744870	I pressed Shift-Enter.
2745910	2753090	And if we look at our count vectorizer of test email, we see these different numbers.
2753090	2756150	And what these numbers represent are tokens.
2756150	2759390	So, for example, a very nice game.
2759390	2761170	A would be token 1.
2761170	2762750	Very would be token 2.
2762750	2764130	Nice would be token 3.
2764130	2765970	Game would be token 4.
2766530	2771850	So in this case, Scikit-Learn actually knows the best way that it wants to tokenize things.
2771850	2777810	So the letters, for example, 0, this does not necessarily represent East.
2778930	2783310	And so, basically, Scikit-Learn knows how these tokens are used.
2783310	2784950	We don't know how these tokens are used.
2784950	2792490	But something that we can point out is we see token 16 shows up twice and token 9 shows up twice.
2792490	2801850	So based on what we saw up above, we can guarantee that either token 16 or token 9 is unsubscribe.
2801850	2805950	And then the other one is list.debian.org.
2806350	2807510	And so these show up twice.
2807510	2810730	So this is exactly the same as if we had done the word count ourselves.
2811390	2818830	The TF-IDF vectorizer, however, uses a bit of a weight algorithm.
2818830	2823270	And so everything that was set as 1 has this 0.204 weight.
2823450	2830410	The 2, which is token 9 and token 16, have this 0.408 weight.
2830650	2834810	And this doesn't necessarily happen in every instance.
2834810	2842090	But in this instance, you can see the 0.408 shows up twice as much as the 0.204 exactly.
2842710	2848030	And so this is the way that count vectorizer operates and TF-IDF vectorizer operates.
2849270	2853810	The underlying, what happens underneath the hood, we don't really need to worry about.
2853810	2860070	We just need to know how these work and the differences between the two.
2862950	2866030	So moving on, we're going to load the testing data.
2866030	2867390	This is already written for us.
2867390	2874130	So we're just going to grab the first 10 files in the testing directory and print those out.
2874130	2877930	So we see we just have all these test emails.
2877970	2882870	And at the end of the file name, we have the label that it's supposed to belong to.
2882870	2886110	So we have ham, ham, spam, spam, spam, ham.
2888830	2892130	And so what we're going to do is read each of these emails.
2892130	2897890	And then we're going to pull off the last little bit of the file name and use that as our label.
2897950	2902950	And we want to have the testing separate from the training so that we can evaluate how well our model is doing.
2903450	2906310	So we're going to load the testing data.
2906310	2910630	And this one's a little complicated, so we have a couple tasks to run through.
2910630	2912970	So we're going to create two list arrays, kind of like we did earlier.
2912970	2917150	We're going to have test corpus, and we're going to set that as an empty list.
2917150	2922610	And we're going to have test labels, which is also going to be an empty list.
2923390	2928650	And then we are going to load all of the emails from the test directory.
2928650	2945650	So we're going to say, for filename in os.listdir, our data directory, plus test, which is where it's located, with each file that is open.
2945650	2954830	And we need to tell it which file test plus filename are.
2955470	2966050	For read, as filenamed f, we are going to do test corpus.append, f.read.
2966690	2972610	And then we are going to do something a little weird here, so bear with me.
2972610	2977110	I'm just going to use re.split, just to split this up.
2977310	2979690	I'm looking for txt.
2981750	2986250	And then we're just going to grab the last bit in that split.
2986250	2991990	So this is going to split the file trainemail.txt.
2992450	2997210	So it's going to split it to this section, and then to this section.
2998050	3005830	So this would be considered index 0, and this last little bit would be considered index 1, which is why we have this 1 here at the end.
3007570	3009290	And then we're going to append the labels.
3009290	3014190	So test.labels.appendLabel
3016430	3021290	And so I'm going to give you guys a little bit to go ahead and get this all written up.
3021410	3027090	Be careful, there is a backslash before this period, because this is a regular expression.
3029650	3031590	And everything should be good.
3032390	3035670	So I'm just going to let that sit for a little bit, let you guys do your thing.
3035670	3037850	In the meantime, I'm just going to look ahead real quick.
3039950	3041850	The rest of this gets so much easier.
3042030	3044750	It's pretty crazy how easy it gets.
3046030	3048890	In the meantime, TAs, are there any questions in the chat?
3057320	3059620	We appear to be good so far.
3060060	3061020	Awesome.
3063200	3066260	So I'm going to go ahead and move on.
3066260	3067720	I'm going to press Shift-Enter.
3067720	3070440	It's going to load all of those emails.
3070440	3079660	So this is just reading all of the test emails into our corpus array and grabbing those labels and putting them into the end of our test labels array.
3080160	3084060	And then just like last time, what this is going to do is print out the number of test emails.
3084060	3087780	We have 590 ham emails, 276 spam emails.
3087780	3091200	And here's just a nice pretty graph representing what that looks like.
3091860	3097900	And as I said earlier, this information or this data has already been split up in an 80-20 split.
3097900	3103900	So 80% of it is being used for our training and then 20% of it is being used for our testing.
3106080	3111460	And then this one, it might sound complicated, but this is actually really easy.
3111460	3115760	We have already trained our CVEC and our TVEC.
3115760	3121700	And now all we need to do is collect our testing values.
3121700	3126660	We need to tokenize our testing values using these already trained vectorizers.
3126660	3128800	So this is super easy.
3130800	3139860	testCountX equals CVEC.transformTestCorpus.
3140140	3143060	And that's it. That's all we've got to do for that.
3143480	3145320	We can just copy and paste here.
3145580	3151760	Say our testTFIDFX is our TVEC.transformTestCorpus.
3152560	3158740	The reason this works is if you remember in the slides how we had the sentences that we already knew about,
3158740	3163120	and then we had to perform a counting game for a very close game.
3163180	3166680	We had to count how many times a showed up in the sentences that we were already aware of,
3166680	3170820	how many times very showed up, how many times close showed up, how many times game showed up.
3170820	3174260	So this transform allows us to do just that.
3174260	3177740	So what this will do is grab each email and it will say,
3177740	3182040	okay, here's the tokenized words of the email that I have.
3182040	3184500	How many times does it show up in the count vectorizer?
3184500	3188940	And then what is the weight that it shows up in the TFIDF vectorizer?
3189740	3192640	So I'm going to go ahead, Shift-Enter, run that.
3193180	3195260	And just like last time, this is going to take a little bit.
3195340	3198900	It's counting all the words and associating them with the different tokens.
3199280	3201980	This is going to be significantly faster than training it.
3201980	3203680	See, it's almost done already.
3205020	3206720	And so we're going to go ahead and move on.
3208580	3212700	So this bit, we're going to get ready to test and evaluate our models.
3212700	3216840	So this is just a helper function that's going to show us a bunch of really neat statistics.
3217100	3220940	The details of this helper function you can look into on your own.
3221180	3227000	But for now, what we need to use as an input is our confusion matrix,
3227000	3230400	our score, and our classification report.
3231660	3234600	So we're just going to go ahead, Shift-Enter, run that.
3234960	3237780	Definition defined.
3239220	3244740	And then here's where we get into tasks 6a, 6b, 6c, and 6d,
3244740	3249300	which is going to be us training and evaluating our different models.
3249380	3253380	So we're going to start with multinomial Naive Bayesian with a TF-IDF vectorizer,
3253380	3258140	with a count vectorizer, and then we're going to do the same for logistic regression with TF-IDF
3258140	3262040	and logistic regression with a count vectorizer.
3264020	3266560	So we're going to start out just one thing at a time.
3267320	3270420	We're going to define our constructor.
3270420	3276480	So this is going to be MNB TF-IDF equals multinomial NB.
3276480	3278660	Watch your capitalization.
3279580	3282360	We're going to define that, and then we're going to train it.
3283740	3292220	MNB TF-IDF dot fit, which is train in scikit-learn world.
3292220	3297140	And we're going to use our TF-IDF X, which is our preprocessed information,
3297140	3300080	and then we're going to compare it to our labels.
3302060	3303860	So that's going to train our model.
3303860	3307300	Now we're going to grab a couple really neat statistics.
3307300	3312020	So the first one we're going to get is our score, and this wants us to call it score.
3314380	3318880	Score MNB TF-IDF predictions using our predict function.
3318880	3323100	We're going to develop a confusion matrix called C matrix, MNB TF-IDF,
3323100	3327440	and then classification report, C report, MNB TF-IDF.
3328540	3330280	So we're going to start with our score.
3330280	3337380	Score MNB TF-IDF equals MNB TF-IDF dot score.
3337980	3341660	And then we're going to use our test values, because we've trained it on our training data,
3341660	3343280	and we want to test it with our testing.
3343280	3347420	So we're going to do test TF-IDF X,
3347420	3354060	because that's the way that our system was vectorizing these.
3354500	3357520	So we don't want to use count vectorizer with a TF-IDF train,
3357520	3358840	because that's not going to work.
3359140	3361540	And then we're going to use test labels.
3363980	3365200	That's going to give us our score.
3365200	3366860	So we're going to get our predictions.
3367180	3374760	Predictions MNB TF-IDF equals MNB TF-IDF dot predict.
3374900	3378600	And this is going to give us our raw predictions.
3381360	3386080	This is going to tell us what it thinks each email is,
3386080	3389720	whereas our score is going to see how accurate it is.
3389720	3393880	And then this is going to be C matrix.
3394880	3402680	MNB TF-IDF equals confusion matrix.
3402980	3408620	We're going to use our test labels and our predictions.
3408780	3411280	MNB TF-IDF.
3411280	3416800	So this is going to tell us what it thinks the value is,
3416800	3418400	if it's a spam or ham email,
3418400	3421300	and then what it actually is.
3421500	3423940	So it's going to give us our true positives, our false positives,
3423940	3425740	our true negatives, and our false negatives.
3426080	3427700	And then the C report.
3427700	3430040	We're not going to do too much with this,
3430040	3434160	but what a classification report will do is show us some more metrics,
3434160	3436620	such as the recall and the precision,
3436620	3438080	which we don't need to worry about too much.
3438080	3440340	We're just going to focus on the accuracy for now.
3440340	3443840	But for data science, it's important to know what the recall
3443840	3448740	and the precision are as we go through things.
3450340	3454800	Report, test labels, and then we're going to use our predictions again.
3456180	3463440	And this is just like our C matrix MNB TF-IDF,
3464100	3465740	in the order that we put things.
3465740	3468020	We have test labels, test labels, predictions, predictions.
3468820	3469920	And that is it.
3469920	3472240	And then we're going to use our helper function generate report.
3473560	3475680	So I know that's a lot of typing.
3475720	3480420	I'm going to go ahead and sit here as you guys kind of follow through.
3480420	3483880	So we have our MNB TF-IDF constructor.
3483880	3487120	We are going to train it using the fit function.
3487120	3489440	We're going to train it on our testing data,
3489440	3491140	which is stored as TF-IDFX,
3491140	3493980	and our testing labels, which is just stored as labels.
3493980	3499960	We are going to use score, MNB TF-IDF.score,
3499960	3502560	to evaluate our accuracy.
3502880	3505740	We're going to use our predictions to get the raw predictions.
3505740	3507260	And then with the raw predictions,
3507260	3512260	we are going to develop a confusion matrix and a classification report,
3512260	3515620	which will tell us how well our model is doing visually.
3518920	3521100	So I'm going to let that sit for one more minute.
3521100	3523220	TA, are there any questions?
3532160	3536320	No, I think... well, hang on, let me go back one here.
3537940	3540300	Nope, still got it. So I think we're good.
3540340	3542460	Awesome. Awesome. Fantastic.
3543500	3548100	So I am going to go ahead and move on.
3548100	3550400	I'm going to press Shift-Enter. It's going to run.
3551260	3556800	And now we have our... up here at the top is our classification report.
3557280	3559780	And so what this is going to do is show us our precision
3559780	3563460	and our recall. We don't need to worry too much about this one.
3563620	3566140	Instead, I want you to focus on the lower chart.
3566140	3568680	So here we have an accuracy of 87.5%.
3569420	3575840	We can see that all of the actual ham emails were predicted as ham.
3575840	3577800	So these are our true positives.
3578520	3583300	A little more than half of our spam emails were correctly predicted as spam.
3583300	3585760	So these are our true negatives.
3585760	3591020	And then, unfortunately, it misclassified...
3591760	3593760	Oh, my mistake. I have that backwards.
3593760	3600320	So it says 108 of our ham emails it thinks are spam emails,
3600320	3604860	whereas all of the spam emails, so the actual spam emails,
3604860	3608580	were correctly classified as spam.
3608580	3617880	And then all of our ham emails, what it thought were ham emails, were ham.
3620760	3626440	So we're going to go ahead and do the same thing for 6b, 6c, and 6d.
3627460	3630460	One of the beautiful things, and I'm going to show you this
3630460	3636800	just as a super quick and easy way of going about and doing things,
3636800	3639540	is I'm just going to copy the code that I'd already written.
3639540	3642360	This is why machine learning is awesome,
3642360	3645740	and very different from programming normal algorithms,
3645740	3649240	is all we need to do is just change these tf.ids to count,
3649240	3651740	and then everything else is the same.
3652880	3657820	So we use mnbcount to define our multinomial Naive Bayesian,
3657820	3662140	as we have been asked of us here in 6b.
3663080	3665320	mnbcount.fit, and we're going to use our countX
3665320	3667560	because that's how it's been vectorized.
3668120	3673980	We're going to change our score mnbcount using mnbcount.score.
3673980	3680740	We're going to change our tf.idsx to testCountX,
3681320	3685580	and then likewise kind of all throughout here.
3689310	3691310	TestLabels is going to stay the same,
3691310	3695010	predictions is going to change because it is no longer predictions.tf.ids,
3695010	3696450	it is predictionsCount,
3696810	3701310	and then we have our confusion matrix and our classification report.
3701670	3703150	So just to go over it one more time,
3703150	3706590	we define our multinomial Naive Bayesian constructor,
3706590	3710670	we train it using the fit function and the count vectorizer,
3710670	3713410	our testing data, our training data.
3714250	3722470	We develop our score off of our testCount data using our testLabels,
3722470	3725910	and then we have our predictions,
3725910	3728650	we have our confusion matrix,
3728650	3730350	and our classification report.
3730350	3732910	So I'm going to let this sit for just a little bit
3732910	3736910	as you guys go through and copy that down.
3745810	3750290	What's really cool is in the GitHub repository,
3750290	3752370	there are other workbooks.
3752370	3755630	So for example, we use the exact same methodology
3755630	3761010	to build a malicious URL predictor,
3761010	3765510	so things that look like it's possibly part of a botnet.
3767510	3769590	All of this code is exactly the same.
3769590	3771230	Once we vectorize our information,
3771230	3773230	we just copy and paste this code in,
3773230	3774990	and it does the exact same thing.
3775250	3777550	And so that's the benefit of machine learning,
3777550	3778890	is it learns from the data itself
3778890	3781110	instead of us having to manually program,
3781110	3782670	okay, this solves problem A,
3782670	3785610	but now we have to rewrite this to solve problem B.
3785610	3787450	It just all does the same thing.
3789290	3793390	So I am going to go ahead and move on.
3793390	3795390	I'm going to press Shift-Enter.
3796710	3798710	And we see that we got a little bit better, right?
3798710	3801090	We are at a 94% accuracy.
3801630	3807030	It thinks that some of our ham emails are still spam,
3807030	3808150	but significantly less.
3808150	3810250	We went from 108 down to 23,
3810250	3815610	but now it thinks that some of our spam emails
3815610	3817750	are actually ham.
3817810	3818670	And so that's a problem.
3818670	3821450	So we have a few more emails going through our filter
3821450	3822870	than we would like,
3822870	3827590	but we have significantly fewer of our ham emails
3827590	3829250	from being blocked.
3830590	3834110	GT, we've got a question from Twitch chat.
3834110	3834650	Absolutely.
3834650	3836670	So Chris...
3836670	3839270	I'm not going to try and pronounce the name.
3839270	3842970	Did you intentionally avoid having spam emails classified as ham
3842970	3844330	when you created the model,
3844330	3847130	or is it simply a try and error kind of thing
3847130	3848610	when training the model?
3849550	3853530	Intentionally avoid having spam emails classified as ham?
3854170	3856430	Is that what they said?
3856430	3857310	Yeah.
3857990	3862050	Yeah, we wouldn't want spam emails to be classified as ham.
3862050	3864390	We want to draw a line and separate
3864390	3866930	so that you stop getting emails about Viagra
3866930	3870990	and you start getting the regular emails from your school or your work.
3871130	3874330	And so this is just kind of the way that spam filters operate.
3874870	3879610	So we could build a model that would classify more spam emails
3879610	3883610	as being part of ham or regular emails,
3883610	3887610	or we can build a model that says,
3887610	3889990	okay, we want absolutely zero spam emails.
3889990	3893130	If we wind up losing regular emails
3893130	3895830	and classifying those as spam as a mistake,
3895830	3897390	we're okay with that.
3897770	3900630	A really good example, actually I'm glad you brought this up,
3900630	3901990	is antivirus.
3902170	3904150	So the way that antivirus companies work
3904150	3907190	is they can err on the side of,
3907670	3911030	oh, I think this is a virus, therefore it must be a virus,
3911030	3913350	or this could be benign,
3913350	3915750	or I kind of think this is a virus,
3915750	3918250	but instead of blocking it, I'm just going to let it go through.
3918870	3920790	And so you think about the way that malware works.
3923110	3926450	You have malware that will make calls out to the internet,
3926450	3928190	it'll probably take over your webcam,
3928190	3929650	it'll try and write files,
3929650	3931490	it'll probably take over your microphone.
3931750	3933150	What else does that?
3933470	3937910	Chrome, IE, Firefox, browsers.
3938070	3940350	And so the idea is instead of blocking
3940350	3945170	what could be legitimate applications,
3945170	3948050	and so that would really interrupt the user experience,
3948050	3950930	they kind of err on the side of,
3950930	3952830	we don't want to block the user experience
3952830	3955870	because then you'll uninstall the antivirus.
3955990	3961310	So they are more willing to let viruses go through
3961310	3964570	than blocking legitimate applications.
3965450	3966630	And so in this case,
3966630	3969230	these are trade-offs that we have to make as data scientists
3969230	3973350	of if we are okay with letting a few spam emails through
3973350	3975730	versus blocking all of our regular emails.
3976790	3980670	So he's specifically asking about task 6A.
3980670	3982150	So if you scroll up a little bit.
3982150	3983330	Task 6A?
3983710	3987790	Yeah, I think the answer is a bit of that
3987790	3993650	and a bit of the question of when you build these models,
3993650	3996170	the training process literally punishes the model
3996170	3998750	for getting it wrong every single time.
4001510	4003610	And in this case...
4005730	4010770	Yeah, so Chris, you're asking about a false negative.
4014010	4018930	A false negative is when you've got something that's malicious,
4018930	4021670	that's misclassified as benign,
4021670	4024070	which is what the point of the model is,
4024070	4028110	but what GT brought up is probably more important
4028110	4029530	is when you have something that's benign
4029530	4031370	that you classify as malicious,
4031370	4034050	that is much more common
4034050	4039270	because you get more benign things on users' devices,
4039270	4040890	as GT brought up.
4042370	4045450	So in this case, I don't think this was intentional.
4046930	4048530	No, this wasn't intentional.
4048530	4051090	This is just the way that the model learned.
4052270	4053810	And then as we go through the other models,
4053810	4055810	you'll see how these shift.
4056410	4060130	Yeah, but this is honestly the way it looks like.
4060130	4063110	You've got the ham stuff that are classified as spam
4063110	4065330	is very low so that all the user experience
4065790	4067470	isn't impacted that badly.
4067510	4069250	And the spam stuff that's classified as ham
4069250	4071290	is relatively high,
4071290	4073230	but you've got a trade-off
4073230	4075790	between the false positives and the false negatives.
4075810	4076970	Yeah, exactly.
4077430	4081010	And I hope that answered your question, Christopher.
4081470	4085870	I think going through the other models
4085870	4088390	will kind of help answer it a little bit more too.
4090830	4092390	So accuracy, 87%.
4092950	4094630	87.5%.
4094630	4096390	94.3%.
4097870	4099610	In this one, we're going to use
4100410	4102210	a model that we haven't used before,
4102210	4103450	logistic regression.
4103690	4105970	And so with Scikit-learn, it's very easy.
4105970	4107570	It's the exact same thing.
4107570	4109590	The difference is we need to
4112470	4114990	use a LBFGS solver.
4115650	4117770	So what that's going to look like is
4119590	4120270	LGSTFIDF
4120270	4120990	equals
4122070	4122750	logistic
4124210	4125170	regression.
4125390	4127390	Then we're going to set our solver
4129010	4130030	LBFGS.
4130030	4131710	And this is just a thing with
4131710	4134110	logistic regression specifically.
4134430	4136290	We saw in the slides
4136290	4138150	that multinomial naivetation
4138150	4140470	has a variable called alpha.
4140470	4141650	We can tweak that.
4141650	4143050	That's one of our hyperparameters.
4143050	4144990	This is considered a hyperparameter.
4144990	4146330	It's the way that logistic regression
4146770	4148090	interprets the information.
4148750	4150890	But for this case, we're just going to use LBFGS.
4151490	4152630	We're going to do the same thing.
4152630	4154990	We're going to fit LGSTFIDF
4156730	4157090	dot
4157090	4157770	fit
4160970	4161330	TFIDFX
4162390	4163010	and then
4163010	4164110	it enables.
4166570	4168310	Let me just hop down here.
4168510	4170310	We're going to do score equals
4174090	4174450	LG
4174450	4174930	S
4176450	4177850	TFIDF equals
4178450	4180350	LGSTFIDF dot
4180350	4181110	score
4182950	4184130	testTFIDFX
4184130	4186370	Pay attention to your capitals.
4186370	4188870	Test labels.
4188870	4189770	Then we're going to do
4189980	4190570	predictions
4193530	4194710	shuns
4194710	4196550	LGSTFIDF
4196550	4197530	equals
4198450	4199790	LGSTFIDF
4199940	4201570	dot predict
4202430	4203470	and then
4203470	4204830	our test data
4207070	4209090	Typing TFIDF so much
4209090	4211750	it's very confusing for my hands.
4212010	4212710	T matrix
4217850	4218970	we're going to use our
4218970	4219910	confusion matrix
4220930	4222530	test labels
4223050	4225030	and predictions
4226610	4227410	LGSTFIDF
4227410	4229030	and then our
4229030	4230030	C report
4231770	4232570	LGSTFIDF
4232570	4233270	equals
4234770	4235970	location
4236570	4237270	report
4238570	4240310	test labels
4240310	4241830	predictions
4242570	4243910	TFIDF
4246290	4248070	That should be good.
4248070	4251010	Just checking for errors.
4252550	4254710	That looks right.
4256370	4258710	So, go ahead
4258710	4260930	copy this down.
4260930	4262930	I will sit and
4262930	4264910	wait for a bit, but you'll notice
4265090	4266930	a lot of similarities to the multinomial
4266930	4268970	Naive Bayesian. In fact, we could have just copied and
4268970	4271030	pasted and just changed it from
4271030	4272590	LGS to
4272590	4274970	MNB or from MNB
4274970	4276710	to LGS and then
4276710	4278990	change our constructor to be a logistic regression
4278990	4281350	with the LBFGS solver.
4284030	4287430	We have our logistic regression constructor.
4287430	4289290	We train it using the fit function
4289290	4291490	using our TFIDF vectorized
4291490	4293050	data and our labels.
4293470	4295550	We use our test data
4295550	4296970	and our test labels
4296970	4299250	to generate a score and to
4299250	4300670	generate predictions.
4301150	4302990	And then, off of those predictions, we generate
4303510	4305510	a confusion matrix and a
4305510	4308470	classification report.
4308470	4309450	And then, of course, our nice
4309450	4311490	little helper function at the bottom that's going to print us out
4312390	4314510	some helpful stats.
4314510	4315610	So, I'm just going to let this
4315610	4317330	sit for a little bit longer.
4318930	4319450	Kometh,
4319450	4321130	did Chris ask any more
4321130	4323390	questions, follow-up, or did we take care
4323390	4324090	of them?
4326010	4327130	A bunch of people
4327130	4329250	piled on and I helped answer the question
4329250	4331150	because of what he asked.
4331450	4333510	Honestly, one of the most important things
4333510	4335370	is the false positive
4335370	4337650	versus false negative trade-off.
4337650	4338170	Yeah.
4340710	4341330	Yeah,
4341470	4343410	a lot of people have strong opinions about that
4343410	4345670	because it's a lot of...
4345670	4347710	It's important. It's definitely
4347710	4350090	important, especially in data science.
4350090	4351670	This, I'm...
4353030	4353690	That might
4353690	4354970	have been a good talk.
4355350	4357710	So, here's the interesting thing, is when I
4357710	4359670	first started out, this
4359670	4361550	I thought was amazing.
4361550	4363530	Now, as I've learned
4363530	4365690	more over time, I kind of feel like, oh, no, this
4365690	4367570	is a little clunky and a little ham-fisted.
4367570	4369670	I'm ignoring this, I'm ignoring this, I'm ignoring this,
4369670	4371850	I'm ignoring this. But this is a great
4371850	4373650	way for beginners to understand machine
4373650	4375690	learning. And then from there,
4375690	4376970	you can kind of hop into
4378210	4379630	a little bit more of the data
4379630	4381250	science considerations, like
4381250	4383130	imbalanced data sets, true
4383650	4385170	positives, false positives,
4385170	4387310	and some of those trade-offs,
4387310	4390070	as well as more hyperparameter tuning.
4393790	4395590	If you want to look at
4395890	4397910	a malware, you keep bringing it up,
4397910	4398810	you can look at the
4399970	4401970	Google Endgame Ember.
4401970	4403850	I'll drop a link in the
4403850	4406030	Twitch chat in a sec. But there's
4406150	4408010	a malware model that you can look
4408010	4409930	into, like the open-source code,
4409930	4411670	and the final thing at the end of that is
4411670	4413570	it sets the false-positive rate to
4413570	4414850	0.05%
4416310	4418310	or 0.5%
4418310	4419930	at the end, because that's
4419930	4421610	what you should do in production,
4421610	4423690	because customers care more about
4423690	4425730	false positives, because
4426590	4427890	false positives lead to
4427890	4429930	alert fatigue, and alert fatigue leads to
4429930	4431870	just ignoring every single
4431870	4433830	alert, and then all the false negatives get
4433830	4435930	through anyway. All the true negatives get through
4435930	4436770	anyway.
4437910	4439930	It is something...
4440750	4441910	Yeah, and even that
4441910	4444030	model is a little clunky and hampers.
4445090	4445750	Well, and
4446690	4448050	it kind of leads to
4448050	4449910	with the alert fatigue, it kind of leads
4449910	4451690	to the common thing that we
4452570	4453930	hear, especially in the technical
4453930	4455990	community from non-technical people,
4455990	4457770	of why does my thing not work?
4458610	4459410	Okay.
4460270	4461990	So that kind of helps address it.
4461990	4464410	But yeah, these are all data science considerations.
4465410	4465930	So I'm
4465930	4467150	going to go ahead and run this.
4467430	4468650	Shift-Enter.
4469690	4472070	And this is going to take a little bit longer than the multinomial
4472070	4473630	Naive Bayesian, because logistic
4473630	4475830	regression interprets
4475830	4477750	the data a little differently, and so it needs to
4477750	4480150	go through and kind of count things properly.
4481650	4482050	This
4482050	4483190	takes, I think, about
4483650	4484810	30 seconds?
4486310	4486830	Yeah, it's
4487670	4488670	not too bad. It's
4488670	4490430	30 to 60 seconds.
4491430	4493210	So I'm just going to go ahead and run this.
4495210	4496790	And then for the logistic
4496790	4498890	regression count, we're going to do the same thing we did
4498890	4500830	with the multinomial Naive Bayesian count
4500830	4502490	and just copy and change words
4503830	4504850	because I'm lazy.
4509390	4510870	But yeah, I'm really excited to see that
4510870	4512750	Twitch chat after I hop off to this
4512750	4514410	workshop and see just how
4516990	4518310	passionate people are
4518310	4520750	about the false positive,
4520750	4522710	true positive, and XYZ.
4523290	4524730	Oh, we're going to erase it before
4524730	4525750	you get off.
4526910	4528250	Ruin my day.
4533270	4535630	Yeah, it's taking a little bit longer than I'm used to.
4537310	4539590	But that's okay. While this is going,
4539590	4541190	I'm just going to go ahead
4541190	4542410	and hop into the next bit
4545310	4545830	and
4545830	4547890	paste this. All we need to do
4547890	4549190	is trade this
4549190	4551930	TF-IDF with COUNT.
4553030	4554110	So I'm going to go ahead
4554110	4555630	and copy that. We're going to use the same
4555630	4557910	constructor and the same solver.
4557930	4559730	We're going to use COUNTEX.
4559730	4562050	We're going to use COUNTFIT.
4563630	4564150	COUNTCOUNT
4566090	4566610	COUNTCOUNT
4567330	4567850	COUNTCOUNT
4572880	4574860	Test labels is the same.
4578410	4579050	These are
4579050	4580070	predictions.
4581010	4582430	And just one
4583190	4584890	quick look over.
4587010	4588830	Yep, I have everything in.
4589430	4591350	That's exactly how it's supposed to be.
4591910	4593230	Scroll up.
4593870	4595190	Oh, this is still going.
4595190	4595830	Look at that.
4598990	4599870	Weird.
4601250	4602890	I'm going to let that sit for a bit then.
4603270	4605090	So this one, I'm going to let you guys write down.
4605090	4607590	This is task 6D.
4608310	4609030	So go ahead
4609030	4611030	and make sure that you have these written down
4611030	4613130	while the other one is still going.
4613130	4615090	I'm going to run this and then take a look at
4617750	4618150	a
4618150	4619530	logistic regression with the
4619530	4625390	TF-IDF.
4625390	4626930	We're actually doing really good
4626930	4628770	on time. We have one more task
4628770	4630530	after this. It is
4630530	4633730	going to be the most challenging task.
4633730	4636090	But it is going to
4633990	4635990	take
4636090	4637750	everything from A to Z
4637750	4640170	and show us what we
4640170	4640990	learned.
4643940	4646080	So I just want to check one last
4646080	4647820	time. And that is
4647820	4648940	still going.
4652740	4654440	So I'm going to go ahead and run
4654440	4655880	this.
4657060	4658520	This was 6D that I
4658520	4660880	just clicked to run.
4664420	4666840	It says the kernel is idle.
4666840	4668020	That's not good.
4672780	4673540	Let me see if I can
4673540	4674980	pause this.
4679080	4680720	Let me just interrupt it.
4680980	4681920	Interrupt the kernel.
4685540	4686540	So I'm going to run this
4686540	4687780	because that took a second.
4691470	4692330	Uh oh, I hope I didn't
4692330	4693330	time out.
4694190	4695650	That would be very bad.
4710950	4712370	Let's try and run it again.
4714230	4714670	Yeah,
4714670	4715690	that's not good.
4722270	4724150	Restart and clear output.
4727180	4729620	So this is going to go all the way to the beginning.
4731140	4732680	Let's see if I can get this to run.
4732680	4734120	Okay, so that runs now.
4735180	4737520	So I ran into a bit of a technical difficulty.
4737640	4739100	Instead of going through
4739100	4740880	and filling
4740880	4742860	everything out, I'm just going to quickly
4742860	4744860	grab the completed workbook.
4746260	4749700	And I'm
4749700	4750760	going to
4752380	4753060	just
4753060	4755480	go ahead, run all.
4757180	4759080	This is going to run
4759080	4761120	everything for me.
4762860	4763540	So
4763540	4765380	while I'm waiting for some of
4765380	4767620	this stuff to run,
4767620	4769100	let's see that packages are
4769100	4772920	being downloaded, tokenization is defined,
4772920	4773580	everything's good,
4773580	4774980	everything's good, everything's good.
4782490	4784010	Where is it stuck now?
4784010	4786790	It is stuck...
4790540	4792520	Looks like it's stuck reading the data.
4793720	4795400	Training spam, training ham.
4803010	4805270	That one's done, done.
4808370	4809210	Yeah, when you see
4809210	4811250	numbers here, that means that that was the order
4811250	4813130	it was ran in. When you see a star,
4813130	4814890	that means that it's still running.
4815170	4817090	I'm not sure why the logistic regression
4817090	4819730	was causing problems,
4819730	4821790	but let's take
4821950	4823730	a quick look at...
4828190	4829430	Oh.
4830310	4832450	Looks like there's some stuff missing.
4837160	4838820	Well, that's unfortunate.
4843000	4844220	So I'm just going to go
4844220	4846120	ahead and create new code
4846120	4848220	blocks. I'm just going
4848220	4849820	to copy the code we had
4851820	4854200	from task 6D
4854200	4855940	and 6C.
4857080	4858760	I apologize for scrolling
4858760	4860220	so fast.
4861300	4862860	So let's go ahead
4862860	4864900	and grab all of that,
4864900	4867040	paste it in, run it.
4867940	4868900	Logistic regression
4868900	4870460	is not defined.
4872000	4873180	That's weird.
4877240	4878860	Oh, it's not even in here.
4878860	4881440	That is something I'm going to have to fix.
4881440	4883400	I'm not sure... Oh.
4884640	4885920	Because that's
4885920	4887220	a whole different workbook.
4887220	4889540	I need complete spam filter
4889540	4892640	sklearn. That's the one that I need.
4893840	4895560	My mistake.
4896100	4897800	So kernel, restart,
4897800	4898920	run all.
4901880	4903400	So this is going to run
4903400	4904800	all of them.
4906000	4907580	Let's see where it
4907580	4909100	gets stuck.
4909760	4911480	Prints out the emails.
4912340	4914160	It's training the vectorizers.
4914160	4916780	I remember that took about a minute or so.
4919000	4920140	But what we want
4920140	4920820	to see
4922660	4924780	after task 6D,
4924780	4926920	we'll look at 6C and 6D,
4926920	4928240	is at the end
4928240	4929880	of this, what we're going to do is
4929880	4932100	grab a spam
4932100	4934180	email. This one is actually one that I grabbed
4934180	4935720	directly from my spam box
4936380	4937680	last year.
4938020	4940060	And I like it because we can use
4940060	4941860	our spam email predictor
4941860	4943680	to identify whether or not
4943680	4945500	it's spam. And then we can look and
4945500	4946460	utilize
4947940	4949100	any of the models
4949100	4951300	to really identify that.
4953820	4956060	So this was supposed to be
4957400	4958320	a fill-in-the-blank
4958320	4959800	kind of thing like we've been
4959800	4960980	going through.
4961460	4963160	But this last task, it just asks us
4963160	4965460	use the vectorizers and the models created to perform
4965460	4967320	predictions on the test email
4967840	4969640	that we created up at the top
4969640	4971240	and the test spam email
4972400	4973640	which is the real
4973640	4975580	world email that I pulled from a spam
4975580	4976740	inbox.
4977620	4979600	And so what we want to do
4979600	4981640	is grab that as
4982060	4983340	a working email.
4983340	4985640	And this will let us change this to
4985640	4987660	test email and test spam
4987660	4988660	email.
4990200	4991640	Remember, we want to
4991640	4993600	use the tf.idf
4993600	4995460	and the count vectorizers
4995460	4997180	in order to use
4998020	4999600	the working email.
4999600	5001320	We need to use that to preprocess
5002100	5003520	and because it's
5003960	5005540	a string, we just wrap it in these
5005540	5007640	square brackets in order to classify
5007640	5009680	it as a list because the transform function
5009680	5011140	expects a list.
5011800	5013860	We have that as test email,
5013860	5015580	tf.idf, and test email
5015580	5016520	count.
5017940	5019620	And then we use each of our models
5019620	5021540	to form a prediction. So we have
5021540	5022500	the multinomial
5023160	5025640	Naive Bayesian with tf.idf, multinomial
5025640	5027480	with count, logistic regression
5027480	5029740	tf.idf, and logistic regression
5029740	5031720	with count. They all use
5031720	5033880	the same predict function. They're all using
5033880	5036500	either the tf.idf
5036500	5037640	vectorized version
5037640	5039260	of the email or the count
5039260	5041540	vectorized version of the email
5043140	5044060	based on
5044060	5045560	whether or not it's the tf.idf
5045560	5047580	or the count version of the model.
5048820	5049700	And then
5049700	5053200	we make our predictions,
5053200	5053760	we print
5053760	5055560	the email itself, and then we can
5055560	5057480	print what
5057480	5059560	each model thinks
5059560	5060800	it is.
5062660	5063640	So if you're
5063640	5065720	still on the fill in the blank bit,
5065720	5067880	I encourage you to
5067880	5069940	fill this out.
5070360	5071580	If you are using
5071580	5073660	the completed workbook, like
5073660	5075620	I am, it's already filled out
5075620	5077520	and we're just going to go ahead and
5077520	5079360	let everything else
5079360	5081580	run in the background until we get
5081580	5082940	to this point.
5085840	5087800	So, sorry about that.
5089360	5091520	In the meantime, are there any
5091520	5094060	questions, any thoughts,
5094060	5095900	concerns, ideas?
5095900	5097640	I'd love to hear from you.
5115770	5117210	All right.
5117630	5118350	Psychdude
5118350	5120190	says, on the completed workbook
5120190	5122230	6a, show the
5122230	5124350	top right and bottom left
5124350	5126510	numbers swapped in 6a.
5126510	5127790	Wait a second.
5127810	5130590	But the one
5130590	5132350	you were working on earlier
5132350	5133530	had those flipped.
5133530	5136370	The fill in the blank and
5136370	5138510	the completed, were
5138510	5140430	they different somehow?
5140430	5142370	Yeah, so I'm
5142370	5144330	thinking the reason why they're swapped
5144330	5145490	is because
5146510	5147230	there was
5149450	5150970	a way that
5150970	5152570	Scikit-learn
5152570	5154350	documented how they did
5154350	5156090	their confusion matrix versus how they
5156090	5157990	actually did their confusion matrix,
5157990	5160950	and it was swapped from what you would normally see.
5161370	5162230	And so, following
5162230	5164590	what you would normally see is what I did,
5164590	5166610	and that's where the mistake came in.
5166610	5168650	And I can actually show you that.
5168950	5170410	I was hoping nobody noticed, but
5170410	5171410	good eye.
5172930	5174210	So if you look at 6,
5174210	5176310	actually before we go there, let's take a look
5176310	5178090	and see the count.
5178090	5180190	So we have 590 ham emails in our
5180190	5181950	test dataset, 276
5181950	5183590	spam emails in our
5183590	5185790	test dataset. And so if we
5185790	5188310	look at 6A,
5188310	5189750	we see 590 ham
5189750	5191630	emails, but
5191630	5193770	as far as actual label, we
5193770	5195650	should see in the column for spam
5195970	5197330	a total of the
5197950	5199890	276. We don't see
5199890	5201650	that. And the reason why
5201650	5203670	is because this is supposed to be
5203670	5205350	the actual, see
5205350	5208790	108 plus 168 is 276,
5208790	5209670	and then
5209670	5211750	this was supposed to be the predicted.
5211790	5213030	The problem is
5213410	5214730	the way that scikit-learn
5215630	5217730	swapped them,
5217730	5219650	and I guess I did not swap
5219650	5220750	them back.
5222290	5223630	To get the
5223630	5225530	correct result, you just need to switch the
5225530	5227750	test labels and the predictions
5227750	5229930	MNB on both sides
5229930	5231770	for the confusion matrix
5231770	5233630	and the classification report.
5234630	5235490	So, yeah.
5235490	5237350	That's why they seem swapped.
5237350	5239210	The one on the completed workbook
5239210	5241230	is correct, because I believe
5241230	5242490	the completed workbook
5243590	5245090	shows in the actual label
5245090	5247210	column the spam emails, and if you total
5247210	5249650	the two numbers, you get 276.
5249670	5251510	So, good eye on that.
5254980	5256740	Go ahead and check this.
5256980	5257760	Perfect.
5261220	5263200	Before we ran into the problems,
5263200	5264140	we're going to look at
5264740	5267000	6C and 6D with a logistic
5267000	5268100	regression.
5268720	5271020	Remember, we had 0.85
5272000	5273320	or 0.87
5273320	5275260	0.5% accuracy
5275260	5276660	for the TF-IDF
5276660	5278440	multinomial Naive Bayesian.
5278440	5280900	For the count multinomial Naive Bayesian,
5280900	5282580	we have 0.94.
5283480	5285180	For the logistic regression,
5285180	5286320	we get a little bit better.
5286320	5288240	We're at 0.959
5288700	5290300	as our accuracy.
5290580	5292680	We have fewer
5292680	5294940	ham emails being classified as
5294940	5296560	spam and fewer
5296560	5299260	spam emails being classified as ham.
5299260	5301040	We're getting better.
5303060	5304340	With the
5304940	5306820	task 6D, this is logistic
5306820	5309320	regression with count vectorizer.
5309520	5310780	This is going to be our
5310780	5314040	best model at 0.976.
5314800	5317020	Here we only have 8 of the
5317020	5318740	ham emails being classified as spam
5318740	5320900	and 12 of the spam emails being
5320900	5322420	classified as ham.
5322420	5324320	Of the four models that we
5324320	5326560	trained, this
5326560	5327480	is the better one
5328640	5330340	in terms of accuracy.
5331320	5332780	This is why we want to try
5332780	5334560	different algorithms, different machine
5334560	5336400	learning algorithms, as well as different ways
5336400	5338940	that we preprocess or vectorize our data.
5339520	5340660	That way we can create
5340660	5342380	different models, see which ones perform
5342380	5344380	the best, and then we can gravitate
5344380	5345620	towards that one.
5349800	5350480	As far
5350480	5351020	as the
5351020	5353000	test spam email,
5353000	5355300	this is a brand new email.
5355400	5357060	It comes in on our email server.
5357060	5358800	We want to use our models to
5358800	5360380	classify it.
5362480	5365140	Let's go ahead and make a prediction.
5366140	5368380	As I went over earlier, we're going to have
5368380	5370540	our test spam email, which is called working email,
5370540	5372440	so we can swap this out pretty
5372440	5374440	seamlessly. We're going to use our
5374440	5376140	vectorizers that we already defined
5376140	5378460	to transform our working
5378460	5380740	email. And then of course, being a
5380740	5382420	string, we need to put it in a list.
5382420	5384740	That's what the square brackets are for.
5384740	5387380	Then we use all four of our models,
5387380	5389300	the multinomial and the logistic
5389300	5391260	regression with the tf and the count
5391260	5394900	vectorizers. We use the predict function
5394900	5398080	on the vectorized
5398080	5398280	emails.
5398280	5399260	And then we just go
5399260	5402120	ahead and print out the working email itself.
5402120	5402420	Your latest issue is available
5402420	5404540	now. If you don't want issue notifications,
5404540	5405920	click here to unsubscribe.
5407440	5408240	And then
5408240	5410580	we print out the values
5411180	5412160	that are predicted
5412160	5413800	from each of our models.
5414140	5416480	If we look at it, we're going to see something interesting.
5420020	5420980	Our best
5420980	5422400	performing model, which is the
5422400	5425080	logistic regression with the count vectorizer,
5425080	5426760	correctly classified this as a
5426760	5428900	spam email. But our worst
5428900	5430460	performing model, even though it still had an
5430460	5432480	87.5% accuracy,
5433040	5434300	our multinomial Naive Bayesian
5434300	5436540	with tf.idf incorrectly classified
5436540	5438660	this as a ham email.
5439620	5440440	And so this is also
5440440	5441380	why we want to
5442020	5444280	try building different models and get things that
5444280	5445760	predict much better.
5451020	5452440	Let me know if there's any
5452440	5453700	questions on that.
5454220	5456380	We are short on time. We're actually running
5456500	5458460	a little over, but before I let you go, there's one more
5458460	5460740	thing I want to show you. And that is
5460740	5462460	the hyperparameter
5462460	5464420	tuning. So if we
5464420	5466620	jump up to that worst performing model,
5466620	5468100	that 0.85%,
5468100	5470660	the multinomial with tf.idf,
5471580	5472180	sorry,
5472180	5473940	0.875%.
5475160	5476400	And we use that
5476400	5478600	alpha that we saw in our
5478600	5480500	formulas on the slides.
5481320	5483040	Alpha is set to 1.
5483040	5484980	We can go ahead and run that.
5485060	5485940	And we see that we're at
5485940	5487560	0.875.
5488060	5489960	But what if we shrink that alpha
5489960	5491740	to 0.1?
5491740	5494120	This is hyperparameter tuning. We can change
5494120	5494840	it.
5495780	5497920	Now our accuracy changed from
5497920	5499680	one of the worst models that we have
5499680	5502320	to one of the better models that we have.
5502640	5504480	In fact, this is second place.
5504660	5506900	So we have 0.965%
5506900	5508060	accuracy.
5509480	5510000	And so you can
5510000	5511860	see how hyperparameter tuning can
5511860	5514100	really help out in tweaking
5514100	5515960	the way that our models operate.
5523130	5524870	Before I let you go,
5528830	5529830	here's some references
5530290	5532550	for more information.
5532770	5534050	And if you want the slides to this
5534050	5536290	and more information from me
5536290	5538530	and the GitHub
5538530	5540450	project that this is all part of,
5540450	5542430	check out github.com.netsec
5542430	5544450	explain and you can shoot me an email with any
5544450	5546710	questions, thoughts, comments
5546710	5548370	at gtklondike
5548370	5549930	at gmail.com
5552310	5554210	So that's all I have for you guys.
5554210	5556310	If there's any discussion,
5556310	5558270	TAs, do we have a separate room for the workshops
5558270	5560370	or are we just going to use this
5560370	5560750	Discord?
5561570	5564310	Nope, there is a separate room for
5564310	5565590	the workshops here. Let me
5566230	5569010	post that in Twitch.
5571430	5571910	Um...
5573190	5575090	There you go, folks. If you have any
5575090	5577110	other questions or want to continue talking
5577110	5578990	about this workshop, you can find us
5578990	5581130	at that channel
5581130	5582990	in the AI Village
5582990	5585170	in the DC Discord.
5586070	5587030	Perfect.
5587670	5588890	And then I'm going to go ahead
5588890	5591010	and end the stream. I'm going to hop into
5591010	5593690	that channel and I will see you guys there.
5595110	5597370	Thanks a lot, Gavin. Absolutely.
5597370	5598230	Thank you.