start	end	text
0	5620	I just want to double check this.
6280	10840	Alright, cool. I have all my stuff muted and we are good to go.
10960	15020	Hello and welcome to Machine Learning for Security Analysts.
15020	18640	This is going to be our final workshop for the AI village.
19020	21760	This is going to be a beginner and interactive workshop
21760	25260	for all those who are interested in machine learning and security
25260	28700	and what this whole crazy AI thing is.
28700	31980	So really, the major motivation for this workshop
31980	35800	was that we're seeing a lot of buzz in the security community
35800	38360	around AI and machine learning.
38360	42640	But for many of us, it's not really clear what that is.
43460	46620	So... oh, sorry.
46620	49020	If you want to participate and ask questions,
49020	52100	head over to the DevSchool channel and join the
52100	55780	AIVillage-General-Voice, AIV-General-Voice.
55960	58680	And you'll be able to ask us questions.
59280	62800	If there's too many people joining and it gets unorganized,
62800	63900	we're going to switch to Zoom.
63900	68140	But it's sort of more open than it was before.
69000	71920	Sorry, Gavin. Just needed to say that.
72260	74500	No worries. Last line of this statement is
74500	76600	I wanted to take, for the security community,
76600	79400	the idea of machine learning and AI out of the buzzword
79400	80720	and into the mainstream.
82040	85520	So, my name is Gavin. I go by the pseudonym GT Klondike.
85520	88680	I'm an independent security researcher and a security consultant.
88700	92720	I'm very passionate about network attack and defense.
92720	95400	And through that passion, I run a project called NetSecExplain,
95400	97340	which is a blog and a YouTube channel,
97340	99900	where I explain intermediate and advanced level
99900	103180	computer networking concepts and security concepts
103550	105800	in an easy-to-understand way.
107420	109920	So, before I get started, usually in person,
109920	111360	this is where I ask people,
111360	113280	what are your thoughts when you hear the word
113280	115380	machine learning? Right?
115380	117680	And I've gotten a range of answers in the past
117680	121480	that range from, oh, Skynet's going to take over the world,
121480	124760	to you need a PhD in multivariable calculus
124760	126620	and linear algebra.
127700	131640	And while there is a mild amount of truth
131640	134120	to the math background that you should have,
134600	136520	neither of those are really true.
137920	141220	Instead, machine learning is really a form of pattern matching.
141220	142740	It's applied statistics.
142740	147380	And with the increase in power behind the computer memory
147680	148520	and behind the CPU,
148520	150200	we're able to use these statistics
150200	154380	in a much quicker, more processed way.
155980	157620	So, first I want to kind of cover
157620	159900	this idea of machine learning,
159900	161920	artificial intelligence, and deep learning.
161920	165120	In a lot of areas, it's being used interchangeably,
165120	166960	and that's not really correct.
167240	169440	First, what we have is this large umbrella
169440	171620	of artificial intelligence.
171620	173360	And artificial intelligence is
174220	178340	kind of a philosophy, in my opinion.
178340	180320	Artificial intelligence used to be,
180320	183400	when the first digital calculators came out,
183400	185540	that was considered artificial intelligence.
185540	187380	A computer can do math.
187400	189820	As things became more advanced,
189820	193040	different concepts were considered artificial intelligence.
193040	195000	There's video game artificial intelligence,
195000	196820	rule-based artificial intelligence,
196820	198760	classical image recognition.
200200	202020	But today we're talking about machine learning,
202020	204840	which is a subset of artificial intelligence.
204840	207480	And it's focused a lot more on
208020	209680	grabbing inferences from data
209680	212320	and making statistical patterns.
213680	215720	And then inside of machine learning,
215720	218020	we have a concept called deep learning.
218020	221520	Deep learning is specifically deep neural networks.
221960	224680	And so neural networks is a machine learning algorithm
225420	227180	that's pretty popular.
227180	228500	But deep learning allows us
228500	230200	to take that a little bit further.
230200	232020	And so usually we see this in the form
232020	233800	of convolutional neural networks,
233800	236680	which are primarily used for image recognition,
236680	238380	or recurrent neural networks,
238380	241400	which are used for cyclical pattern recognition.
241400	245880	So think of things such as a heartbeat monitor,
245880	247400	where there's a pattern involved
248180	250020	that you kind of need to match up.
250020	252440	And if there's any sort of heart irregularities,
252440	256320	the pattern will diverge from predicted patterns.
257960	262020	So if we wanted to formally define machine learning,
262020	264140	I would say that it's a set of statistical techniques
264140	266600	that enable the process of information mining,
266600	269480	pattern discovery, and drawing inferences from data.
269540	272580	So the idea is that the algorithms learn
272580	274820	from past data to predict future outcomes,
274820	277740	instead of creating a different algorithm
277740	280080	for a different scenario.
280400	284580	And we'll see that as we go through the workbooks,
284580	287140	how we're able to create multiple different types
287140	290420	of algorithms around the same set of data,
290420	291860	or how we can use the same algorithms
291860	293640	with different types of data.
294440	296120	So some key examples,
296120	297540	especially in the security sphere,
297540	299920	is domain generation algorithms.
299920	302060	Domain generation algorithms are heavily used
302060	304120	by botnets so that the bot herder
304120	307760	can stay in contact with the botnet itself,
307760	311000	while government agencies or ISPs
311000	312840	are taking down the domains
312840	314580	that they use to communicate.
314940	317680	We also see this with web application firewalls.
317680	319240	We can perform anomaly detection
319240	325100	to identify normal web traffic,
325100	327880	normal parameters that are sent by a web client,
327880	330260	versus potentially malicious parameters.
330260	331780	In this example here,
331780	335180	it's showing a possible SQL injection,
335180	336940	and it's calling that out.
337220	339320	And then, of course, network anomaly detection,
339320	341840	where what we can do is predict
343300	346880	how the network will operate over a period of time,
346880	350780	and then keep track of how it is actually operating.
350780	353360	And if it's beyond a certain delta or a difference,
353360	355240	then we can flag that and send it off
355240	356740	to a security analyst.
358040	360300	So why is this important for security analysts,
360300	362200	and also important for hackers?
362700	365460	Well, today, over a quarter of security products
365460	368640	for detection have some form of machine learning.
368800	370260	So from an analyst's perspective,
370260	371600	what you want to do is understand
372840	373220	how these machines work,
373220	375440	so that you can ensure that they operate efficiently
376580	378280	and are working effectively.
379100	381360	Machine learning isn't this magical black box
381360	383180	that just solves all of our problems.
383920	385480	And I want to share a story with you,
385480	388240	and this is what really made the idea of machine learning
388240	391020	and the value of machine learning really stick out to me.
391480	392520	So a number of years ago,
392620	395420	a friend of mine was approached by a non-profit.
395420	397200	And what this non-profit did was track
397200	399180	cheetah population out in Africa.
399180	401360	So they set up a bunch of different cameras,
402840	404960	and they would take a picture at different intervals,
404960	406620	and it would be sent off to a human analyst
406620	409900	to identify whether or not there's a cheetah in the picture.
410420	412020	Not a job I would want to do,
412020	414880	but it needs done, and it's really important.
416100	417220	So they approached my friend,
417220	418400	and they were saying, hey, is there a way
418400	420260	that we can increase our efficiency?
420260	422680	Something that we can use to kind of automate
422680	424780	some of the manual labor that we're doing.
425180	426840	And so what he did was spend about a week,
426840	429860	and he used a very simple machine learning model
431300	433140	to identify images.
433200	435840	And what he told this model was,
435840	437440	hey, if you see an image,
437440	439940	and you are at least 5% confident
439940	442080	that there's a cheetah in the picture,
442080	443740	send it off to an analyst.
443740	445680	If you're less than 5% confident,
445680	446920	just throw it away.
447640	449640	Now 5%, that doesn't really sound like a lot,
449640	451160	and to be honest, it's not.
451320	453740	But what this non-profit was able to do
453740	455860	with this simple model was
457240	459680	what would normally take them a year of labor
459860	461980	they were able to complete in one month.
462620	465500	That is a 1200% increase in productivity
465500	467720	by just implementing a little bit of machine learning
467720	470640	and that fuzzy logic into their process
470640	472320	to kind of help automate it.
472860	474160	So this is what I want to do
474160	476520	to kind of push for the security analysts
476520	477600	and for the hacker community
478100	479500	to get a little bit more involved
479500	480740	in AI and machine learning
480740	483940	so that we can see and identify ways
483940	485860	that we can improve our own processes,
486660	487880	especially ones that require
487960	489060	a little bit of fuzzy logic,
489060	490480	things that traditional signatures
490480	494160	or scripting aren't sufficient in.
496380	498600	So the 7-step machine learning process
498600	500500	generally looks like this.
500660	502720	And this isn't a list
502720	505560	so much as a cycle, right?
506560	508080	So first we're going to start
508080	508980	with gathering the data.
508980	512240	We want to identify what it is
512240	513840	we want our machine learning algorithm
513840	515660	to classify.
515960	517520	So first we're going to gather the data
517880	518720	and then we're going to prepare the data
518720	520660	in such a way that is easy
520660	523100	for machine learning models to understand.
523300	524640	And in this case, it's usually turning it
524640	525740	into some form of number
525740	527440	or some form of signal.
528780	530840	Most of our time is spent gathering the data
530840	531860	and preparing the data.
531860	534280	The rest of this tends to be a little bit quicker.
535160	536860	So first, based on the data that we have
536860	538840	and the problem set that were presented,
538840	540260	we choose a model.
540260	542240	So some models perform better than others
542240	543420	in certain tasks.
543420	544720	And sometimes you just got to kind of
547880	548780	figure out maybe neural nets isn't
548780	550120	the best model for you.
550120	552620	Maybe logistic regression or SVM
552620	555220	or random forest classifier is better for you.
555680	557140	But once we've identified the model
557140	559560	or models, then we go ahead and train it.
559560	561540	And so what we'll usually do is perform
561540	563600	what's called a training testing split.
563600	565760	And in this case, we have a series
565760	569320	or a lot of clean labeled data.
569480	570900	And we take that data,
570900	573440	we separate 80% of it and 20% of it.
573440	575880	80% of it is what we use to train the model.
575880	577000	And then the other 20%
577880	580000	is what is used to evaluate the model
580000	581940	to see how well the model is performing.
582840	585220	And so it's kind of like how we teach
585220	587160	students in elementary school, right?
587160	589500	We train them on the bulk of information
589500	591380	and then we test them to see how well
591380	593140	they're understanding the material.
594040	595880	And so some of our models
596680	597740	will perform very well.
597740	600640	Some of our models may not.
600640	602000	And so if we have a model
602000	603640	that performs fairly well,
603640	604800	but we think it could do better,
604800	606500	then we hop into what is called
607100	608260	hyperparameter tuning.
608500	609900	And we're going to see what a hyperparameter
609900	611600	looks like in a little bit.
611880	613600	But hyperparameter tuning,
613600	616180	think of it as little knobs and levers
616180	617540	and switches that we can use
617540	618760	to kind of tweak our model
618760	620160	and tune it in such a way
620160	624160	that it is better at a certain type of problem.
624980	626440	And then once we've completed all this,
626440	627680	we're ready to deploy.
628000	630040	Now the reason why this is cyclical, right?
630040	631300	We want to gather more data,
631300	632900	prepare the data,
634800	636000	and use the same model
636000	638500	that we're already using after deploy
638500	640040	is because machine learning
640580	642860	stops learning as soon as it's trained.
643300	644840	So to kind of give you an idea,
644840	646820	I know that certain spam filters
646820	648660	like Facebook's Facebook spam filter,
648660	650260	they retrain it and rebuild it
650260	651380	every one to three days
651380	653960	because that's how quickly spam
653960	656460	is being processed through their pipelines
656460	658260	and through their networks.
658320	659580	And so it's a way for them
659580	661420	to kind of stay on top of that.
664280	666040	So we're going to hop into
666900	668880	understanding how machine learning works,
668880	670240	especially in the security space.
670240	672240	And the easiest example, in my opinion,
672240	674040	is building a spam filter.
674600	675520	Now spam filters,
675520	677900	we already know how they operate conceptually.
678180	680660	So it's a lower barrier of entry.
680660	682520	We're going to see specifically
682520	684580	how the machine learning fits in.
684760	686120	But before I hop into that,
686120	687340	are there any questions
687340	689700	on how machine learning,
689700	691380	AI, and deep learning
691380	692300	kind of intermingle
692300	693600	and why it's important
693600	695400	to understand machine learning
695400	696580	from a security standpoint
696580	699220	or in the hacking community?
706490	708330	I see on the Twitch chat,
708330	709250	reinforcement learning,
709250	710670	reinforcement learning, self-learning,
710670	712290	contextual regret minimization,
712290	714110	what differences between them?
714410	717210	I'm not familiar with counterfactual,
717210	718850	not familiar with counterfactual
718850	720810	regret minimization,
720810	724910	but typically you'll see
726430	727970	three types of machine learning.
727970	729450	You'll see classification
729450	730910	or supervised learning,
730910	732490	which is what we're going to be covering.
732490	734350	You'll see unsupervised learning,
734350	737430	which is very good for data exploration.
738870	740170	And unsupervised learning
740170	741990	uses a whole different set of algorithms.
741990	743550	And then there's reinforcement learning,
743550	746870	where we embed in a model
746870	749170	different rewards and punishments
749170	752410	based on how it's performing.
752690	753770	And so these are all different ways
753770	754870	of training the model,
754870	757590	and then we can make decisions off of those.
759830	762490	Hey, Sven, are there any questions
762490	765070	in the Discord chat before I move on?
775840	777780	Okay, I'm going to assume not.
779480	782620	So before we look at a full spam filter,
782620	784740	we want to kind of look at a smaller example.
784880	786000	So in this case,
786000	787200	we have a series of sentences
787200	790020	talking about sports or not sports,
790020	791800	which are really elections.
792260	793980	So here we have a couple sentences.
793980	794920	A, great games.
795100	796760	Sorry. Yes.
798260	800420	Well, I think you got this one.
801340	803200	But there was a question from Norwin.
803200	804680	I fell asleep on the job.
806340	807660	No worries.
807780	809280	It has been a long weekend.
810260	811780	So what was the question?
814340	818680	Norwin kindly repeated Glucorio's question.
820200	822060	So you already answered it.
822060	823320	Okay, okay.
825000	826300	So kind of going back,
826300	827820	we have a great game,
827820	829260	which is talking about sports.
829260	830800	We have the election was over,
830800	833200	which is talking about not sports.
833240	835680	Very clean match, talking about sports.
835680	837140	Clean but forgettable game,
837140	838400	talking about sports.
838400	840580	So if we wanted to take this idea
841780	842300	and specify a sentence
842300	843640	that we've never seen before
843640	846500	about sports and not sports,
846500	848500	how would we do this?
848700	851640	We would want to look at keywords.
851900	853900	So in this case,
853900	856380	we can see that it's talking about games.
856380	858060	We know it's probably about sports.
858060	859260	If it's talking about elections,
859260	861720	it's probably not talking about sports.
862500	864960	And so what we want to really hone in on
864960	866640	are keywords.
866940	869040	But to put this into a machine learning model,
869040	871040	we need to find a way
871040	873500	to turn this into numbers.
874660	875720	So the way that we're going to do this
875720	877920	is Bayes' Theorem.
877920	879380	Bayes' Theorem is...
880030	881760	it may look complex at first,
881760	882860	but it's actually a really easy
882860	884800	to understand probability theorem.
885480	886080	And in this case,
886080	887940	we see the probability of A given B
887940	890260	equals probability of B given A
890260	891660	times the probability of A
891660	893540	divided by the probability of B.
893540	894220	So in this case,
894220	895580	we want to see what is the probability
896660	899120	that this sentence is talking about sports
899120	901020	given that the sentence is
901020	902800	A very close game.
903280	906220	So we don't know the probability of sports
906220	907640	given A very close game,
907640	909440	but we can easily plug that into this formula
909440	911780	and say, well, we know the probability of sports.
911780	913900	We saw three out of our five sentences
913900	915120	were talking about sports.
915120	916820	And we can figure out the probability
916820	919060	of A very close game given sports
919060	921500	and the probability of A very close game.
922200	923140	But how do we do that?
924220	925540	How do we get a sentence like this?
926000	926940	Well, with Bayes theorem,
926940	929240	we can actually, or with probability,
929240	931020	actually, we can split this up.
931020	932520	So instead of A very close game
932520	933620	as a whole sentence,
933620	935500	we can look at the probability of A
935500	936740	times the probability of very
936740	939160	times close times game.
939640	941540	And so what this turns into
941540	944000	in our A very close game given sports
945060	947220	is probability of A in sports,
947220	948820	probability of very in sports,
948820	949840	and so on.
949840	950920	And of course, we're going to want to do
954220	955540	is say, okay,
955540	956920	whichever is highest,
956920	958040	sports or not sports,
958040	959500	that's the one that we're going to classify
959500	962140	this sentence as being part of.
963380	964180	So at this point,
964180	966260	it becomes a simple counting game, right?
966260	968520	So how many times does the word A
968520	970160	show up in sports?
970400	972140	Well, we see that it shows up twice,
972140	973040	once in the first sentence
973040	974520	and once in the fourth sentence.
974520	975940	And then how many times does it show up
975940	977340	in not sports?
977540	979600	Once, right there in the fifth sentence.
984220	985360	So A does not show up in sports.
985360	986920	It only shows up in not sports.
986920	988040	And then we see the word game
988040	990040	show up twice in sports.
991740	993020	So we just take those numbers
993020	994620	and we plug them in and we say,
994620	996720	okay, A showed up in sports twice,
996720	997940	very showed up once,
997940	999740	close showed up zero times,
999740	1001180	and game showed up twice.
1001280	1003120	And if we counted the number of unique words,
1003120	1005080	we would have had 11 unique words
1005080	1006240	across all of our sentences
1006640	1008020	talking about sports.
1008880	1010240	But we run into a problem
1014620	1016280	and the problem is that
1016280	1018100	what we want our machine learning model to do
1018100	1020560	is identify and classify sentences
1020560	1022400	that it has never seen before.
1022960	1024540	Unfortunately, it's never seen close
1024540	1026060	and so that winds up being a zero,
1026060	1028380	zero times everything is zero.
1028380	1030260	And we see that that easily cascades down
1030260	1031380	and just ruins our model.
1031380	1033120	So it's because we've never seen the word close,
1033120	1034280	there's a zero probability
1034780	1037180	that this exists in our sentence.
1038080	1039320	So that's a problem,
1039760	1041480	but we can get around that.
1042700	1043740	There's another algorithm
1044220	1046060	called multinomial Naive Bayes.
1046180	1048000	And so this is just a small modification
1048000	1049700	and what it does is add
1049700	1051140	what is called a smoothing filter.
1051140	1053080	And we can see this alpha here.
1053080	1055740	It's added to the numerator and the denominator.
1055800	1057460	And usually this alpha is one,
1057460	1059420	but this is going to be our hyperparameter
1059420	1061600	for Naive Bayesian,
1061600	1062820	multinomial Naive Bayesian
1062820	1064540	that we can tweak, right?
1064540	1066360	We can set it to one, we can set it to two,
1066360	1067900	we can set it to 0.1.
1068040	1070840	And so this is a thing that we can tweak
1070840	1072280	to see if we can get our model
1074220	1074660	to work for us.
1075440	1076780	So we take the same idea.
1076780	1077680	We do the word counts.
1077680	1079700	We saw that a showed up in sports twice,
1079700	1081360	very showed up once, close to zero.
1081360	1083180	But now we have this plus one.
1083580	1086620	So the probability of close given sports
1086620	1088480	is never going to reach zero.
1089180	1090760	And then of course, as I mentioned earlier,
1090760	1093060	we need to do the same thing for not sports.
1093060	1094820	And the total number of unique words
1094820	1096280	in not sports is nine.
1096280	1098600	The total number of unique words in sports is 11.
1098600	1100060	And the total number of unique words
1100060	1103320	across both classes is 14.
1106080	1108080	So we plug in our answers
1108080	1109360	and we do our calculations
1109360	1113680	and we get 0.0000461 for sports
1113680	1117880	and 0.0000143 for non-sports.
1118200	1119440	Classifying the sentence,
1119560	1122800	a very close game being about sports.
1125480	1128320	So that was a lot that I just kind of laid on you.
1129080	1130400	If you have any questions,
1130400	1131660	type them in the chat.
1132020	1133200	While you're doing that,
1134220	1135580	I'm going to show you how we can
1136720	1138380	take our raw emails
1139100	1140940	and transform them into
1140940	1142480	these types of counts
1142480	1143440	and these types of numbers
1144120	1146500	in a very clean and effective way.
1146960	1149180	So the five things we need to keep track of
1149180	1151920	are the total number of unique words,
1151920	1154240	the total number of words in spam,
1154240	1156420	the total number of words in ham,
1156420	1158080	or regular emails,
1158080	1160420	the count of each word in spam,
1160420	1162700	and the count of each word in ham.
1165960	1168000	So let's take a look at one of our emails.
1168720	1169500	We see here,
1169500	1171620	East Asian fonts in Lenny.
1171620	1172640	Thanks for your support.
1172640	1174760	Installing Unifonts did it well for me.
1175340	1177140	Now we want to take a sentence like this
1177140	1179920	and find a way to classify it
1179920	1182240	as being spam or ham.
1182700	1183860	But before we do that,
1183860	1185840	we want to kind of point out a couple things.
1185840	1187500	So we see here kind of towards the bottom,
1187500	1189340	we have the word unsubscribe,
1189340	1190400	all capitalized,
1190400	1192700	and then unsubscribe lowercase.
1192900	1195460	What we want to do is look at keywords.
1195500	1197540	And so in context,
1197540	1201620	it doesn't matter if unsubscribe is capitalized
1201620	1202860	or the first letter is capitalized
1202860	1204180	or it's all lowercase.
1204280	1206300	It means the same thing.
1206320	1210900	And so to kind of reduce our problem set,
1210900	1213420	what we're going to do is set everything to lowercase.
1213900	1215140	The other thing we want to do
1215140	1217180	is remove what are called stop words.
1217180	1218700	Stop words are words like
1218700	1221700	the, in, of, on, or...
1221700	1224420	They don't add any context to the sentence.
1224780	1227100	And we're going to do the same thing for punctuation.
1227200	1228580	So we're going to remove those.
1228580	1230300	We're going to keep our keywords.
1230300	1231800	And then one more thing that we're going to do
1231800	1233300	is called stemming.
1233580	1236060	So if we have something like the word
1236820	1238700	congrats or congratulations,
1239540	1240540	in the English language,
1240540	1241620	they mean the same thing.
1241620	1244200	So if we're looking at keywords,
1244200	1245820	we want to treat them the same.
1245820	1247000	And so what stemming does
1247000	1249600	is shrink those words down to their base form
1249600	1250960	or the root form.
1250960	1252860	And then we can count them the same.
1252940	1256120	Another really good example is the word thanks,
1256120	1257240	thank you,
1257240	1259840	where it's a just singular thank.
1259840	1261760	And I thanked somebody today
1261760	1263380	with an ed at the end.
1264040	1265940	As a keyword, they all mean the same thing.
1265940	1268360	And so stemming will reduce them all down to thank.
1268540	1270340	And so instead of three separate words,
1270340	1271760	it's going to be counted as one word.
1271760	1273500	And so that's going to drastically reduce
1273500	1277200	our dimension space.
1278300	1280560	So after our preprocessing,
1280560	1282000	which is what this is known as,
1282000	1284520	we're going to see East Asian fonts in Lenny,
1284520	1286080	or East Asian fonts Lenny,
1286080	1289240	thanks support installing Unifonts,
1289240	1290080	well me.
1291400	1294820	And then what isn't shown here
1294820	1296060	is what the stemming looks like.
1296060	1297700	So as we go through the workbook,
1298340	1300040	we'll take this exact same email
1300040	1301880	and see how it's transformed
1301880	1305840	at every step of our preprocessing.
1306480	1309260	So are there any questions so far?
1315420	1316680	I don't think so.
1316680	1317840	There's none on Twitch chat,
1317840	1319240	and there's none in the Discord.
1319280	1320320	Perfect.
1322320	1323940	So what we're going to have
1323940	1325860	are a series of workbooks.
1325860	1327820	We're going to use MyBender.
1327820	1329740	MyBender is really awesome.
1330700	1333620	TAs, if you don't mind sharing this link.
1333880	1336040	What it will do is spin up a Docker container
1336800	1338860	with the workbook linked to my GitHub
1338860	1342420	so that you can kind of code with me
1342420	1343400	as we walk through
1343400	1346740	and build our own machine learning classifiers.
1347980	1349260	While this is starting up,
1349260	1350460	what we're going to be using is
1350460	1352520	instead of coding everything raw
1352520	1353920	and doing the raw numbers like we did
1353920	1356820	on our pen and paper,
1356820	1357920	we're going to be using
1357920	1359600	the abstraction library Scikit-learn.
1359600	1360860	Scikit-learn is very useful
1360860	1363520	and it allows us to treat things
1363520	1364680	kind of like Legos,
1364680	1366800	where, OK, maybe I don't want
1366800	1368140	multinomial Naive Bayesian,
1368140	1369300	maybe I want logistic regression
1369300	1370700	or random forest.
1370700	1372680	We can just swap that out.
1374780	1376380	So once it's loaded,
1376380	1378780	and I'm going to give you a moment
1378780	1380780	to get this loaded up.
1381380	1383120	If it takes a little bit longer
1383120	1385280	than it should,
1385280	1387100	then you'll easily be able
1387100	1387980	to catch up to me
1387980	1389600	and I'll show you how.
1390160	1391760	So we're going to click on workbooks
1393540	1395860	spam filter sklearn.
1395860	1397840	If you're not very familiar
1397840	1399940	with Python, you can still follow along.
1399940	1401520	We have a completed workbook,
1401520	1402660	which is the workbook complete
1402660	1404780	spam filter sklearn.
1405720	1406800	And these are Jupyter notebooks.
1406800	1408120	If you've never used Jupyter notebooks,
1408120	1409260	I think they're the best thing
1409260	1410480	since sliced bread,
1410480	1412380	especially when it comes to data science.
1412820	1415260	But it allows us to interact
1415260	1416200	with code blocks
1416200	1418540	and visualize things in line.
1418540	1419740	And I think they work fantastic
1423520	1424040	together.
1425080	1427800	So let's go ahead, scroll down.
1427800	1430560	We're going to run this first code block.
1431100	1432380	And so what this is going to do
1432380	1433880	is install a couple of libraries
1433880	1434840	that we need,
1434840	1437020	as well as set our data directory
1437020	1439320	to where the data directory is
1439320	1440960	in the GitHub project.
1440960	1443400	So to run these, you can click run
1443400	1444360	up at the top,
1444360	1446560	or you can press shift enter,
1446560	1447900	which is what I'm pressing now.
1447900	1450560	And then you can see that it's running.
1450560	1452160	A star in the upper corner
1452160	1453780	means that it's currently running.
1453780	1455320	And then a number in the upper corner
1455320	1457560	means that that was the order
1457560	1459840	that code block was ran in.
1462560	1465040	So while that is going,
1465040	1465820	I'm going to
1466780	1468220	let it go until it stops
1468220	1469880	scrolling my thing.
1470900	1472140	So we're just going to import
1472320	1473180	a couple libraries.
1473180	1474800	We're going to import numpy,
1474800	1476820	matplotlib for graphs,
1476820	1478280	regular expressions we'll be using
1480560	1481360	and NLTK.
1481360	1482540	NLTK stands for
1482540	1484380	Natural Language Toolkit.
1484380	1486400	This will allow us to identify
1486400	1489100	and collect a list of stop words
1489100	1491560	to remove those things.
1491560	1493260	And it will also allow us
1493260	1494900	to use a stemmer function
1494900	1497060	to stem the words
1497060	1499780	and reduce our problem space.
1500800	1501940	Train test split,
1501940	1504320	I don't believe we're actually using that in here.
1504320	1506320	The data directories I have
1506320	1507740	are already split into
1507740	1509260	80% training,
1509260	1511060	20% testing.
1511300	1513320	And then we're going to use our vectorizers.
1513320	1515460	And we'll explain what the vectorizers do,
1515460	1516940	but we're going to take a look at
1516940	1519060	both of the TF-IDF vectorizer
1519060	1520320	and the count vectorizer
1520320	1523160	because they operate
1523160	1524140	slightly different.
1524140	1526580	Count vectorizer just takes a straight word count
1526580	1529300	like we did with our pen and paper example.
1529300	1530760	And then TF-IDF actually takes
1530760	1532040	all of the words into account
1532040	1534680	and gives them a weight.
1535800	1536600	Scikit-learn,
1536600	1539480	we're going to use logistic regression
1539480	1541060	and multinomial naivetation.
1541060	1542460	And the reason why we're going to use both of these
1542460	1544040	is because we don't actually know
1544040	1546140	which one's going to perform best.
1546180	1547420	And so like good data scientists,
1547420	1549140	we're going to try a couple different models
1549140	1551160	and take the one that performs best
1551160	1552960	with the problem set that we have.
1553560	1555780	And then for a couple metric
1555780	1557140	and visualizations,
1557140	1558800	we're going to use the Scikit-learn
1558800	1561640	confusion matrix and classification report.
1563040	1564060	So I'm going to go ahead
1564060	1565200	and press Shift-Enter
1565200	1566240	and that's going to run.
1566240	1568360	You will see libraries imported
1568360	1569960	once that's complete.
1571520	1573300	And we can move on to the next thing.
1573620	1574700	So this is our test email.
1574700	1575840	This is the exact same email
1575840	1577760	that we saw in the slides.
1577980	1579440	So as I mentioned earlier,
1579440	1580500	we're going to use this email
1580500	1582880	to see how it's transformed
1582880	1585040	at every step of preprocessing.
1585040	1586540	So I'm going to go ahead and run that.
1587980	1590000	And then this is our tokenizer function.
1590000	1592260	Tokenizer functions will allow us
1592260	1593500	to pull out the keywords
1593500	1595620	in the way that we would like them
1595620	1597060	to be extracted.
1597180	1598760	So in this case, we're going to grab
1598760	1601080	all the punctuation, all the stop words,
1601080	1602480	and we're going to create a
1602480	1605020	stemmer constructor.
1605020	1607640	We're going to use these to
1607640	1609260	remove the stop words from the email.
1609260	1611280	We're going to use these to remove
1611280	1612380	punctuation from the email,
1612380	1614120	and then we're going to stem the words.
1616540	1616940	We're going to move all the words
1616940	1618280	to lowercase so that they are
1618280	1620100	treated exactly the same.
1620980	1622100	So we're just going to go ahead
1622100	1623900	and run this, shift enter,
1623900	1625400	and our tokenizer is defined.
1625960	1627180	And this is where we run into
1627180	1629100	our first task, which is to
1629100	1630420	tokenize an email.
1630780	1632260	So our task says that we want
1632260	1634420	to print the full email, test email,
1634420	1635240	and then we're going to print the
1635240	1637680	results of the tokenized version
1637680	1638920	of that email.
1639580	1641140	So here I'm just going to do
1647880	1651040	print tokenizer test email.
1651520	1652820	And we can go ahead and run that,
1652820	1655000	shift enter. We can see the
1655360	1656880	original email that we have,
1656880	1658160	East Asian Fonts and Lenny,
1658160	1659660	thanks for your support.
1659740	1661160	We can see, just like in our
1661160	1663360	slides, we have East Asian Font
1663360	1665820	Lenny, slightly changed,
1665820	1668400	thank instead of thanks,
1668400	1670500	support install, because we can
1670500	1673900	have installing installed with
1673900	1676320	an ed at the end, or I
1676320	1678220	will install this software,
1678220	1680140	which is just a standard install.
1681340	1683080	We see unsubscribe has been
1683080	1684560	changed. And two of the things
1684560	1685480	that I want to point out is
1685480	1686740	every single word in here is
1686740	1687860	unique, except for the word
1687860	1689320	unsubscribe, which shows up
1689320	1692060	twice, and list.deviant.org,
1692060	1693940	which shows up twice.
1696360	1697400	So we're going to take a look
1697400	1698680	at our training data. What this
1698680	1701500	is going to do is identify the
1702140	1704980	directories where our spam
1704980	1707100	files are, our ham files are,
1707100	1709160	and our test files are. We're
1709160	1710180	going to get a count, because
1710180	1711420	that's one of the five things
1711420	1712400	that we needed to keep track
1712400	1715500	of, was the count of, actually
1716800	1717800	we want to keep track of the
1717800	1718740	count of words, we don't care
1718740	1720700	how many emails we have. So
1720700	1721680	I'm going to go ahead and run
1721680	1725180	this, and what we're going to
1725180	1727720	do is put those emails into
1727720	1728900	what is known as a corpus.
1728900	1729640	So we're going to create two
1729640	1731660	list arrays, so corpus and
1731660	1732840	labels. Corpus is going to
1732840	1734820	hold all of our emails for
1734820	1738440	spam and ham, and then the
1738440	1739800	labels is going to be each
1739800	1741900	label for our data. So the
1741900	1742700	first thing we're going to do
1742700	1746020	is corpus equals a new list,
1746020	1748000	labels equals a new list,
1748720	1749500	and then we're going to do
1749500	1750560	the same thing for spam and
1750560	1751540	ham. We're going to load the
1751540	1753980	email bodies from the ham
1753980	1754940	directories and the spam
1754940	1756520	directory into the corpus
1756520	1757120	array, and then we're going
1757120	1758260	to load the labels for each
1758260	1760380	email into the labels array.
1760980	1761720	So what that's going to
1761720	1762300	look like is we're going to
1762300	1763280	use a for loop here, so
1763280	1765200	we can say for each in
1765200	1767780	os.listdir, and then we
1767780	1769340	have our data directory,
1769340	1771400	which is defined above.
1771900	1773260	We're going to look at our
1773820	1775260	ham directory for this
1775260	1777480	first round. With each
1777480	1778800	open file, we're going to
1778800	1781700	look at our data directory
1781700	1784920	plus ham plus os.listdir.
1784940	1785320	We're going to look at
1785320	1786860	each, because each is the
1786860	1788280	name of our file, and
1788280	1788840	we're going to want to
1788840	1790840	read those as the file
1790840	1792980	descriptor f. So for
1792980	1795800	our corpus, we can do
1795800	1799320	an append and then f.read.
1799320	1800380	So this will read each
1800380	1801500	email into our corpus
1801500	1803420	and then labels.append
1804080	1806560	ham, because we know
1806560	1808380	that all of the emails
1808380	1809440	in the ham directory are
1809440	1812380	part of the ham data set.
1812460	1813660	And so we can easily do
1813660	1814920	the same thing for spam
1814940	1815940	which is we need to be
1815940	1817300	careful on how we change
1817300	1818320	this. So instead of
1818320	1819280	slash ham, it's going to
1819280	1821440	be slash spam, and then
1821440	1822640	of course our label is
1822640	1824140	also going to be spam.
1826060	1827260	So I'm going to let this
1827260	1828100	sit for a minute and let
1828100	1829760	you guys go ahead and
1829760	1830500	make sure that you copy
1830500	1832000	this down correctly.
1833520	1835520	This bit is pretty easy
1835520	1836220	if you're familiar with
1836220	1837600	Python. It might be a
1837600	1838800	little confusing if you
1838800	1840780	are not, but that's okay.
1840780	1842140	If you ever get lost at
1842140	1843800	any point, just remember
1843800	1844700	that there are a
1844700	1846060	completed solutions manual
1846060	1847990	in the workbooks directory.
1850580	1852840	So I'm going to go ahead
1852840	1855340	and run this. So shift
1855340	1856930	enter to run.
1857740	1859740	And I spelled labels
1859740	1864160	wrong. Labels, shift
1864160	1865980	enter to run. Loading
1865980	1867810	ham, loading spam, done.
1868940	1870040	So what this is going to
1870040	1872600	do is graphically load
1872600	1874680	or graphically visualize
1874680	1876260	how many ham emails we
1876260	1877140	have versus how many
1877140	1878240	spam emails we have.
1878240	1878820	And this is something we
1878820	1879720	want to generally do with
1879720	1881000	our data just to see if
1881000	1881680	there's any sort of
1881680	1883280	imbalanced data sets.
1883380	1884440	This one we have about
1884440	1888600	50% spam per ham.
1889280	1890320	It's like 2 to 1
1890320	1892560	ratio of ham to spam.
1892560	1894580	And so this I'm not
1894580	1895740	going to consider an
1895740	1896880	imbalanced data set for
1896880	1898980	this example, but if we
1898980	1900000	have something such as
1900000	1901080	like credit card fraud
1901080	1901940	where it happens less
1901940	1903220	than a fraction of a
1903220	1904080	percent of the time,
1904080	1904800	that's something that we
1904800	1905900	really want to dive into
1905900	1907500	simply because if we try
1907500	1909440	and use this exact
1909440	1911240	example on credit card
1911240	1912060	fraud, it's not going to
1912060	1913040	work because what we
1913040	1914700	will do is create a
1914700	1916980	model that is 99%
1917540	1918980	accurate because it just
1918980	1920460	classifies all credit
1920460	1921540	card transactions as
1921820	1922400	legitimate. Now if we
1922400	1923480	want to identify credit
1923480	1924340	card fraud, that's not
1924340	1925700	very helpful for us.
1925700	1926920	So this just allows us
1926920	1928260	to kind of explore the
1928260	1930100	data and see how much
1930100	1931400	of what we have.
1931400	1932260	So in this case we have
1932260	1935000	2,359 ham emails and
1935000	1937780	about 1,100 spam emails.
1939700	1941560	So task 2A, we're going
1941560	1942500	to view our training
1942500	1944520	data. So this one's just
1944520	1945640	to kind of see what our
1945640	1948260	data looks like. We
1948260	1949380	generally have an idea,
1949380	1950140	but we just kind of want
1950140	1950880	to play around with it.
1950880	1951620	And this is another thing
1951620	1952440	that you would want to
1952440	1953800	do in a data discovery
1953800	1954980	process is just kind of
1954980	1955840	see, okay, what is it
1955840	1956300	that we're actually
1956300	1958340	working with? So in this
1958340	1961220	case, I'm going to pick
1961220	1963200	an email ID, just email
1963200	1965180	ID 5. You can pick 5 or
1965180	1966640	you can pick any other.
1967520	1969000	And then I'm going to
1969000	1970560	grab the email, which
1970560	1971580	is part of the corpus
1971580	1974160	array. Corpus and then
1974160	1977340	our email ID. And then
1977340	1979740	I'm going to print that
1979740	1981560	email, and then I'm
1981560	1982400	going to do the same
1982400	1984520	thing and print the
1985460	1987300	tokenized version using
1987460	1989100	a tokenizer function of
1989100	1990460	that email. So we can
1990460	1991700	see how it's transformed.
1994180	1995840	So just so I don't go
1995840	1997240	too fast, I want to make
1997240	1998200	sure that you're able to
1998200	1999660	follow along and copy
1999660	2001940	this down. It's fairly
2001940	2004320	simple. So I will wait
2004320	2007620	just a few seconds. How
2007620	2008420	are we doing on
2008420	2009460	questions?
2011180	2012520	No one has been
2012520	2014160	posting some helpful
2014160	2018640	links, but hasn't had
2018640	2019640	too many questions so
2019640	2022140	far. So thank you very
2022140	2023420	much. No winner.
2024540	2026260	Okay. So when we run
2026260	2027760	that, what we have is
2027760	2029100	the email body. We see
2029100	2030280	product review, vert
2030280	2032320	tools, dev, 2.0 URL, and
2032320	2035020	then this URL. We see
2035020	2036560	product review, vert
2036560	2039280	tool, dev, 2.0 URL as
2039280	2042060	one word, HTTP, and then
2042060	2044600	the full URL. So this
2044600	2045460	is how we can identify
2045460	2047680	and kind of track our
2047680	2048520	data as it's being
2048520	2049620	transformed over time.
2051980	2053200	So task 3 is where
2053200	2053620	we're going to start
2053620	2054700	using our vectorizers.
2054700	2055940	What we want to do is
2055940	2057560	get a count of each
2057560	2058720	unique word, and we want
2058720	2060620	to get a count of each
2061460	2063540	unique word per class so
2063540	2064760	that we can use that in
2064760	2066140	our classification. So
2066140	2066920	there's two ways we can
2066920	2067760	do that. We can use the
2067760	2068940	straight count vectorizer,
2068940	2069900	which will get the count
2069900	2072560	of our words, and we can
2072560	2074020	use a TF-IDF vectorizer,
2074020	2075400	which will give us a
2075400	2076920	weight of each count.
2079480	2081070	So no winner says he's
2081820	2084320	behind. He's catching up
2084320	2085300	and he'll catch up
2085300	2086760	later and follow along.
2089500	2091440	I can try to figure out
2091760	2093800	a system of helping the
2093800	2095460	winner. I can send you a
2095460	2097500	Zoom link or something
2097500	2099140	and look over your
2099140	2101680	shoulder metaphorically.
2101680	2103060	Also, in the workbooks,
2103060	2104980	you can just open up
2104980	2106000	the... that's the wrong
2106000	2106660	one... you can just open
2106660	2108600	up the completed spam
2108600	2110640	filter SK Learn, and
2110640	2111680	it's completely filled out
2111680	2113080	for you. So if you ever
2113080	2113940	do fall behind and you
2113940	2114740	would like to catch up,
2114740	2115760	you can look at the
2115760	2116960	answers there.
2118820	2120400	There is a way to
2120400	2121840	follow along and make
2121840	2123000	sure your code works.
2123880	2125400	Yeah, absolutely. And I
2125400	2126220	like using the completed
2126220	2127140	workbook. You can just
2127140	2128260	run everything and see
2128260	2129320	how things are supposed
2129320	2131200	to be at each step of
2131200	2134140	the process. So we're
2134140	2134600	going to go ahead and
2134600	2135540	train our vectorizers.
2135540	2136380	And the instructions tell
2136380	2137240	us exactly how to do
2137240	2138340	this. So we're going to
2138340	2139520	use the vectorizer.
2139520	2140060	We're going to call it
2140060	2141040	cvec for the count
2141040	2142680	vectorizer, tvec for the
2142680	2144040	tfidf vectorizer, and
2144040	2144640	we're going to perform
2144640	2147440	fit transform. Fit in
2147440	2148980	scikit-learn world is
2149420	2151200	code for train. So this
2151200	2151820	is going to, quote
2151820	2152800	unquote, train our
2152800	2153900	vectorizers so they can
2153900	2154740	get the count of each
2154740	2155460	words. And then we're
2155460	2155880	going to save the
2155880	2157840	results as count x and
2158260	2160940	tfidf x. So what that
2160940	2162540	looks like is cvec
2163740	2164540	equals count
2165260	2166380	vectorizer. And then we
2166380	2167400	want to use our
2167400	2168300	tokenizer that we
2168300	2170580	defined above. So the
2170580	2171860	tokenizer that we're
2171860	2172920	going to use is called
2173720	2175100	tokenizer. I know, not
2175100	2176960	confusing at all. But
2178360	2179440	we're going to use our
2181920	2183340	count x. And pay
2183340	2183880	attention to your
2183880	2185380	capitals. Scikit-learn
2185380	2187060	is very picky and so is
2187060	2188880	my workbook. We're going
2188880	2190710	to use cvec.fit
2192380	2193660	transform. And then
2193660	2194520	we're going to train
2194520	2196340	transform or count the
2196340	2199400	words for our corpus.
2200160	2200600	And then we're going
2200600	2201280	to do the same thing
2201280	2202220	with our tfidf
2202220	2204540	vectorizer. tvec
2204540	2206600	equals tfidf
2206600	2208360	vectorizer. And we're
2208360	2209180	going to use our
2209180	2210720	tokenizer called
2210720	2212820	tokenizer. And then
2212820	2214680	our tfidf x
2214680	2219220	equals tvec.fit
2219220	2222020	transform. And it's
2222020	2222860	also going to be on
2222860	2224500	our corpus. So I'm
2224500	2225160	going to let this sit
2225160	2227240	for a minute and let
2227240	2228120	you guys be able to
2228120	2233060	catch up. And this
2233060	2233820	takes a little bit.
2233820	2235800	This takes about two
2235800	2237840	minutes to run because
2237840	2238620	what it's doing is it's
2238620	2239640	reading every single
2239640	2241000	email and it is
2241000	2242260	counting the words in
2242260	2245100	those emails in order
2245100	2246060	to get the number of
2246060	2250760	unique words. So I'm
2250760	2251680	going to go ahead and
2251680	2253320	run this because it's
2253320	2254100	going to take a little
2254100	2255800	bit. But in case you
2255800	2257240	need, here is the
2257240	2258440	code while this is
2258440	2259240	operating. And so we
2259240	2260560	can see that it is
2260560	2261560	currently training the
2261560	2262640	count vectorizer, kind
2262640	2263320	of down here at the
2263320	2265000	bottom. And then once
2265000	2265780	the count vectorizer is
2265780	2266540	complete, it will start
2266540	2267600	training the tfidf
2267600	2269280	vectorizer. And then it
2269280	2270420	will let us know when
2270420	2282480	that is complete. So
2282480	2283440	while this is going,
2283440	2284220	I'm going to go ahead
2284220	2285860	and move on to the
2285860	2286980	next code block. We
2286980	2287800	can write the code
2287800	2288500	while this is still
2288500	2289400	training, but we won't
2289400	2290100	be able to run it
2290100	2290860	because it can run
2290860	2292080	only one code block at
2292180	2293800	a time. So in this
2293800	2294720	case, we have task
2294720	2296240	3a, which is to count
2296240	2298060	the test email tokens.
2298060	2298900	So we're going to
2298900	2300120	manually count out
2300880	2302420	each of the tokens.
2302420	2303280	So what we're going to
2303280	2307480	do is say for i in
2307480	2308800	and this is a bit of
2308960	2309820	a trick so that we can
2309820	2311800	keep the count as well
2311800	2313040	as the words in order.
2313040	2313560	So we're going to use
2313740	2315620	a list of a dictionary
2318820	2320420	and we're going to
2320420	2323240	want to grab the
2323240	2324260	tokenized version of
2324260	2325080	our email. So let's
2325080	2328720	call this tokenized
2329360	2331280	email equals tokenizer
2332860	2335500	test email. And then
2335500	2336620	our from keys we're
2336620	2337300	going to have tokenized
2337300	2338380	email which we saw
2338380	2339660	above is going to be a
2339660	2340420	list. So we'll just
2340420	2343260	call that tokenized
2343260	2345040	email and then we're
2345040	2346580	going to count the
2346580	2349620	words. So print, do
2349620	2350400	some Python format,
2350400	2363000	And then we're going to look at our tokenized email.count of each word, and then the word itself.
2363440	2369660	So we see our vectorizing is complete, so we are ready to run this.
2369660	2376060	I'm going to let this sit for about 30 more seconds and let you guys catch up.
2376060	2390970	And then, of course, after this workshop, I have a whole series of workbooks, as well as their solutions.
2390970	2395830	So if you're interested in other ways to apply machine learning and security to different problems,
2395830	2400050	and using different algorithms, don't forget to take a look at those.
2402630	2408350	So I'm going to go ahead and run this, Shift-Enter, and we can see the counts.
2408350	2411490	So we have East Asian font, Lenny, thanks, support.
2411830	2417710	Each of these, we have our unique words, they only show up once in our email.
2417710	2423830	The only two that are outliers is unsubscribe, or unsubscrib, after our stemmer.
2423830	2428890	And this list.debian.org, they show up twice in our email.
2428890	2433030	So this is if we manually counted these, but what if we used our vectorizers?
2433770	2438470	So what we're going to do here in Task 3B is create new vectorizers,
2438470	2440770	the count vectorizer and TF-IDF vectorizer,
2440770	2447430	and then we're going to print out the counts that each of them represent.
2447670	2451950	So in the first one here, I'll just call this example-cvec,
2451950	2457150	and then we're going to define a new vectorizer.
2462220	2473760	And I'll just call this example-x equals example-cvec.fit-transform.
2474340	2478100	And then it expects a list, our email is a string,
2478100	2480700	so we're going to wrap it in these square brackets.
2481000	2483600	And we're just going to use our raw test email.
2486080	2490960	And then we're going to want to print our example-x.
2493140	2498180	And we can easily just copy and paste, do the same thing for our TF-IDF code.
2498180	2503700	We'll just call this example-tvec, we'll use the TF-IDF vectorizer,
2503700	2508120	tokenizer is tokenizer, we change this to tvec,
2508120	2510160	and then we're already done with our example-x,
2510160	2512840	so we don't need to change the name, we can just leave it as is.
2513980	2518440	Example-tvec, fit-transform, and then square brackets, test-email,
2518440	2519940	so that it's a list.
2520860	2521760	Yes.
2524900	2527800	Oh, it sounded like there was possibly a question.
2533190	2538510	So Voto on the stream is asking, where are those workbooks again?
2544050	2548070	Can one of the TAs post in the stream?
2548070	2549750	There you go, perfect.
2550270	2552170	Machine learning for a security analyst.
2553530	2557830	Okay, so I'm going to go ahead and run this.
2558570	2561030	Scroll up, let's take a look at the count vectorizer.
2561130	2563690	So instead of seeing the words, we see these numbers,
2563690	2564890	and that's exactly what we want.
2564890	2567090	We've created what are called tokens.
2567330	2570970	So the words themselves, the tokenizers will be able,
2570970	2573750	or the vectorizers will be able to match them together.
2575090	2579870	So all of them are one in their count, all of them are unique,
2579870	2583150	except for token 16 and token 9.
2583150	2587810	We don't know which one, either unsubscribe or list.wn.org,
2587810	2592910	maps to 16 or 9, but what we do know is that both of them show up twice.
2592910	2597410	So given that, list.wn.org is either 16 or 9,
2597410	2601990	and then the other one belongs to unsubscribe, or the word unsubscribe.
2601990	2607010	So this looks exactly the same as if we had done our raw count.
2607650	2610810	If we look at our TF-IDF vectorizer, though,
2610810	2612470	we see something a little differently.
2612470	2616030	TF-IDF, what it does is use term frequency
2616030	2619950	to give each term or token a weight.
2620030	2624310	And so in this case, we see all of the ones that showed up once
2624310	2629950	have a weight of 0.204, token 9 and token 16
2630450	2633790	have a weight of 0.408.
2634270	2636690	So you don't really need to understand the underlying math,
2636690	2640930	you just need to know that there are different ways to vectorize your data.
2641590	2645550	So there's TF-IDF, count vectorizer, word2vec,
2645550	2648130	and a couple others, bag of words.
2648870	2652470	And so there's different ones that we can try and see which one works better for us.
2654170	2659090	So let's hop down and start to load our testing data.
2659090	2662390	So loading our testing data, what this is going to do is
2662990	2670450	just show us the names of each of our files in our testing directory.
2671430	2674250	And we have train email.txt,
2674250	2676650	and then at the end we have the class that it belongs to.
2676650	2679030	So we have ham, we have spam.
2679750	2682770	So what we're going to want to do as part of loading our testing data
2682770	2685310	is we're going to want to strip off these labels.
2685910	2688530	So we're going to read the full email into our corpus,
2688530	2689850	and we're going to strip off these labels,
2689850	2695010	and those are going to be the new labels for our testing data.
2696410	2700110	So that hops us into task 4 to load the testing data.
2700110	2703530	Just like loading the training data, we're going to create two list arrays,
2703530	2706810	our corpus, we're going to call this test corpus, and our test labels.
2706970	2709790	And then we're going to load each email body,
2709790	2714310	and then strip off the label and throw that into the test labels array.
2716450	2720810	So what that's going to look like is...
2725550	2728530	So we're going to look at... what was the name of that?
2728730	2730770	Test corpus.
2730950	2732930	And we're just going to set that as a blank list,
2732930	2737270	and then test labels as a blank list.
2737990	2741010	For loading each of the emails, we're going to do just like we did before,
2741010	2744510	filename in ls.listdir,
2744510	2750130	and then our data directory plus the test directory.
2750130	2751430	So we're going to look at each filename,
2751430	2756010	we're going to open each file,
2756010	2763110	so data.dir plus test plus filename,
2763110	2768230	and then we want to read those as the file descriptor F.
2769190	2774090	So our test corpus, we're going to append F.read,
2774090	2777290	which is going to read that email into our test corpus array,
2777290	2779830	and then we want to grab the label.
2779830	2783950	And in this case, I'm just going to use a simple regular expression.
2785390	2790670	We're going to use txt, and then this is a backslash period,
2793330	2796710	and filename 1.
2796710	2800050	I'm going to scroll up real quick, because the reason why I want to do that is
2800050	2805350	what this is saying is, where we see txt.period, we're going to split,
2805350	2809690	so we have this full train txt.period, and then we have ham.
2809910	2814650	And then the bracket 1, what that is going to do is say,
2814650	2817890	this is bracket 0 once it's split, and then this is bracket 1,
2817890	2822610	so we're just going to keep our labels, which are at the end of that split.
2825230	2829990	And then with that label, or each label that we grab from the filename,
2829990	2837350	we're going to append it to our labels directory, or our labels array.
2839210	2841450	So this is what our code looks like.
2841450	2845990	Double checking for errors, looks good to me.
2846390	2848710	And I'm going to let this sit for a little bit,
2848710	2851630	so you guys can go ahead and copy that down.
2859290	2861170	How are we doing?
2865460	2870780	No questions in Twitch stream, are there any questions in the Discord channel?
2877980	2881420	Okay, I'm going to go ahead, shift, enter, and run.
2881420	2884040	So this is going to quickly load the emails.
2886240	2889400	And then let's run this next code block.
2889400	2893060	So what this is going to do is allow us to visualize our data, just like we did before.
2893060	2899200	So we see in our testing set, we have 590 ham emails, 276 spam emails,
2899200	2903040	and the graph itself looks nearly identical to the one above,
2903040	2909160	and that's because we did a random 80-20 split for our training and testing.
2910300	2913020	And that was already done beforehand.
2914460	2918280	So the next thing we want to do is use the vectorizers that we created,
2918280	2922640	the cvec and the tvec vectorizer,
2922640	2926280	and take a look at the test corpus,
2926280	2931780	and get our counts of each word in the new emails.
2932560	2938560	So let's go ahead, testCountX equals, and we're already going to use cvec,
2938560	2941900	we're not going to use fitTransform, instead we're going to use transform,
2941900	2946420	because we're not trying to train it, we're just trying to save the words themselves.
2947680	2960040	We'll do testCorpus, and then testTFIDFX equals tvec.transform testCorpus.
2960280	2965940	And that's all we need to do when we're using a piece of testing data.
2967700	2970320	So I'm going to let that sit for a second,
2970320	2973620	actually I'm going to run it because this is going to take a little bit as well.
2976620	2978900	Slightly shorter than when we were training it,
2978900	2984940	but it still needs to go through all the emails and apply the counts,
2984940	2987460	and then the TFIDF, it needs to apply the weights.
2987460	2989640	We already know what the counts are, we already know what the weights are,
2989640	2991640	we just need to apply them to each token.
2992220	2993840	So vectorizing is complete.
2994520	2999220	And now we're finally on to testing and evaluating our model.
2999640	3002280	So you remember how I said data collection and preprocessing
3002280	3005460	is what's going to take the most amount of time?
3005460	3007640	Well, that was about 80% of this workbook.
3008140	3011340	If not, maybe a little more, maybe 85%.
3012320	3014740	But yeah, data science in a nutshell.
3015680	3020620	So here we have a very helpful report generator function.
3020920	3024900	It's not important to know what's going on in this,
3024900	3026280	it's just going to be a helper function,
3026280	3029280	but we do need to know that it's going to take in our confusion matrix,
3029280	3032560	our score, which is our accuracy, and the classification report,
3032560	3035120	and then it's going to print out all the stats that we need.
3036520	3039860	So that hops us into task 6A,
3039860	3043400	which is going to be training and evaluating our first model,
3043400	3048340	which is going to be multinomial Naive Bayesian using the TFIDF vectorizer.
3049980	3053940	So there's a lot of steps to this, but we're going to walk through it step by step.
3054200	3058160	First, what we're going to do is create a multinomial Naive Bayesian constructor,
3058160	3061420	so that's going to look like mnb, tf, idf,
3061420	3064160	and the names of these variables are very important
3064160	3068000	because that's what's being used in the generator function.
3068980	3074320	So I'm going to call this Numial Naive Bayesian,
3074840	3078660	and then we're going to train the multinomial Naive Bayesian,
3078660	3084200	as I already misspelled something, mnb, tf, idf, .fit,
3084200	3087040	which is scikit-learn's code for train,
3087040	3091300	and then we're going to use our TFIDF, our vectorized corpus,
3091300	3093440	with the TFIDF array.
3095360	3098060	So now we're going to get a couple stats,
3098060	3100060	so we're going to get our score,
3100060	3108740	mnb, tf, idf, equals mnb, tf, idf, .score.
3108740	3110400	This is going to give us our accuracy,
3110400	3114500	and we want to compare this against our test, tf, idf, x,
3114500	3117380	and our test labels.
3118260	3121140	And then we want to take a look at the raw predictions,
3121140	3124680	so we're going to have our predictions, mnb, tf, idf,
3125400	3131600	equals mnb, tf, idf, .predict, using the predict function,
3131600	3134760	on our test, tf, idf, x.
3135540	3136400	Cool.
3136400	3139760	And with those results, what we are going to do
3139760	3143700	is generate a classification, a confusion matrix,
3143700	3145900	so we can see our true positives, false positives,
3145900	3147540	true negatives, false negatives.
3147580	3159780	So cmatrix, mnb, tf, idf, equals our confusion matrix,
3159780	3163720	and then we're going to use our test labels
3163720	3170800	and our predictions, mnb, tf, idf, which is our raw predictions.
3170800	3173220	We're going to do the same thing for our classification report,
3173220	3178680	so we'll see report, mnb, tf, idf,
3179940	3182300	classification report, and in the same order,
3182300	3189320	test labels and our predictions, mnb, tf, idf.
3190680	3193900	So I'm going to let this sit for a second,
3194580	3197520	give you a chance to copy all of that down.
3199580	3204600	So what's really helpful here is seeing how
3207200	3210720	at each of these tasks, 6a, 6b, 6c, and 6d,
3211320	3214240	we're only going to need to change very few things.
3214240	3217500	We're going to need to change, instead of tf, idf,
3217500	3220600	we're going to change this to count to do our train and evaluate
3220600	3223500	the multinomial Naive Bayesian with the count model.
3223720	3226620	And then we're going to change this from
3226620	3228520	the multinomial Naive Bayesian constructor
3228520	3231280	to the logistic regression constructor.
3231400	3233840	But other than that, the rest of our code is going to be identical.
3233840	3236280	The rest of our preprocessing is going to be identical.
3236480	3239240	And so this will allow us to train a total of four algorithms
3239960	3243860	using two separate machine learning algorithms
3243860	3245620	as well as two separate vectorizers.
3246340	3249100	So this is going to be very helpful for us.
3250400	3254040	So I'm going to go ahead and run this.
3254040	3256260	So shift-enter to run.
3256260	3260240	It looks like I misspelled something.
3263120	3264560	Multinomial.
3264840	3266660	There we go.
3267280	3269440	Required argument y.
3269900	3271400	That one's new.
3272980	3276580	Give me one second.
3276900	3279320	Oh, that is what I forgot.
3279360	3282500	So we want to train our multinomial Naive Bayesian
3282500	3286340	with our data as well as our labels
3286340	3290120	so that it knows what belongs to spam,
3290120	3293980	what belongs to ham, just kind of in our pen and paper example,
3293980	3296560	what belonged to sports and what belonged to not sports.
3297020	3298940	So now I can run that.
3300460	3304360	And I see where we ran into that problem yesterday.
3304580	3305260	Okay.
3306600	3308740	I need to swap.
3308740	3311800	This is the problem that I had with scikit-learn earlier.
3313140	3315560	The test labels and the predictions.
3315560	3318720	We start with the predictions and then we follow up with the test labels
3318720	3320500	instead of the other way around.
3321240	3323100	So I go ahead and run that.
3323800	3324840	There we go.
3324840	3329200	So we know that we had 590 ham emails.
3329200	3332660	We know that we had, if we add these two numbers together,
3333440	3334920	276 spam emails.
3334920	3338200	So we can see the actual number of ham emails, the actual number of spam emails
3338200	3339960	in those columns.
3340800	3342640	But let me scroll up real quick.
3342760	3345660	So this is our classification report.
3345700	3349240	Classification report we're not really going to use in this example,
3349240	3351500	but it's good to know.
3351500	3354980	There are other ways to measure how good a model is performing,
3354980	3356800	beyond accuracy.
3357500	3360400	Precision, recall, and F1 score
3360400	3363780	are all used in their own ways.
3363820	3367700	I'm not going to go into that, but it's definitely worth understanding.
3368040	3370780	Instead, we're just going to look at the straight accuracy.
3370780	3373400	For this multinomial with TF-IDF,
3373400	3375940	we have a .875
3375940	3379860	or a 87.5% accuracy.
3379860	3383000	If we look at our true positives and our true negatives,
3383000	3386500	it correctly classified or predicted all ham emails
3386500	3388920	as being part of the ham class.
3388960	3392460	So these are all of our true positives for what is ham.
3392540	3394720	And then our true negatives, what is spam,
3394720	3400280	it only classified 168 out of the 276 that we have.
3400720	3403080	The other half of the spam emails
3403080	3406660	it also thought were ham or regular emails.
3406660	3409720	So our predictor is letting some spam through,
3409720	3412880	but the good news is it's not blocking any of our
3412880	3414920	legitimate emails.
3418820	3422660	Moving on to the next example, we're going to do the same thing.
3422660	3423980	We're going to use multinomial naivete
3424540	3427700	with count vectorizer.
3427700	3431020	This is going to be the same, and then be count.
3433020	3435660	I am actually a little lazy,
3435660	3437640	so what I'm going to do
3437640	3440640	is just copy this
3440640	3442460	and I can show you all you need to do
3442460	3447020	is just change a few things around and you have a working model.
3447020	3449060	So instead of multinomial tfidf,
3449060	3451980	I'm going to be using multinomial count.
3451980	3453800	We're going to use the same constructor.
3454780	3456940	So in this case it's called count,
3456940	3461580	and instead of using the tfidfx, we're going to use the countx.
3464320	3465280	score,
3465280	3466760	count,
3469240	3470140	mnb,
3470140	3471260	count,
3471260	3476680	and our test labels stay the same because we didn't vectorize those.
3476680	3478920	There's no reason for us to change this.
3480120	3484160	I'm just going to go ahead and ctrl-c and ctrl-v some stuff.
3485900	3489360	So all I'm doing is just changing the word tfidf
3489360	3491100	to the word count.
3491100	3494300	Everything else is staying the same.
3495040	3497040	We have our predictions,
3497040	3498300	mnb, count,
3498300	3502840	our test labels, that stays the same, we don't modify it.
3502900	3505140	mnb, count,
3505140	3509760	and then we change our predictions.
3510600	3511540	So I'm going to go ahead
3511540	3513680	and let that sit for a moment.
3513680	3516460	I'll let you guys either manually type that in
3516460	3520940	or do the copy-and-paste track that I did,
3520940	3522700	which I highly recommend.
3522700	3524980	It just saves quite a bit of typing.
3526280	3528620	And then we'll type out the logistic regression
3528620	3531860	in a moment because that's the next model that we'll be making.
3534780	3537600	How are we doing on questions?
3541160	3543520	I think we're all doing well.
3544300	3546980	Everything's quiet on the Discord chat, too.
3551580	3552380	Awesome.
3552380	3554040	So I'm going to go ahead and run this.
3554040	3556340	I'm going to do Shift-Enter.
3557640	3560280	And if we look at our accuracy, our multinomial
3560280	3565220	with the count vectorizer, we have a 94% accuracy.
3565380	3567660	So quite a decent bump
3567660	3569920	compared to our tfidf vectorizer
3569920	3572700	using the same multinomial algorithm.
3572700	3576760	What we see is about 26 of our emails...
3576760	3579620	not about, it's exactly 26 of our emails
3579620	3581900	in our test set got blocked.
3581900	3583880	They were considered spam.
3584480	3586940	These are the regular ham emails.
3586940	3589480	But we have significantly fewer spam emails
3589480	3593900	that are getting through our predictor.
3593900	3597340	And so more of them are being classified as spam correctly.
3597340	3600040	And so this is what we want to see is a dark blue over in this corner.
3600040	3603720	And a darker blue
3603720	3607080	or this lighter blue given the count
3607080	3609080	over in the bottom right corner.
3609560	3612200	So this kind of hops into a conversation
3612200	3615840	about false positives and false negatives.
3615840	3619000	Which would you like to prioritize?
3619980	3622340	And so it really depends on the model.
3622340	3625220	In security, one of the things that we're concerned about
3625220	3626860	is impacting the users.
3626860	3630980	So if you look at something like an antivirus,
3630980	3633100	it looks for things that are probably
3633100	3636280	talking out to the internet, trying to take over your webcam,
3636280	3638660	uploading and downloading files.
3638720	3640820	Well, what else does that?
3641400	3644980	Chrome, Internet Explorer, Firefox.
3645760	3648100	So when you're building
3649260	3650720	an antivirus,
3650720	3653940	it's better to have
3653940	3656660	what they consider false negatives.
3657480	3660540	Which is positive is
3660540	3663180	it is malware and negative is it is not malware.
3663180	3665920	It's benign. So it's better to allow
3665920	3669000	some viruses to run instead of
3669820	3671560	blocking legitimate applications.
3671560	3674140	Because you don't want to impact the user experience.
3674140	3677980	Because if you impact the user experience, it's just going to uninstall your antivirus software.
3677980	3681120	And that leaves them unprotected and that's bad business for you
3681120	3683320	when people are uninstalling your software.
3684960	3687300	So even for us, we want to consider
3687300	3689460	and weigh the factors of
3689460	3692140	is it better for us to allow some spam to get through
3692140	3695000	as long as we're not blocking any of the legitimate emails?
3695000	3698180	Or do we want to make sure that absolutely no spam gets through
3698180	3702480	and if we block legitimate emails, well that's just collateral damage.
3702560	3704520	So these are considerations that you need to make
3704520	3707600	when you're building models like these and as a data scientist.
3709180	3712180	So for the multinomial naïve Bayesian
3712180	3715360	with the count vectorizer, we have 94% accuracy.
3715740	3718640	So let's try a different model.
3718640	3720840	This one we're going to use the logistic regression.
3721120	3723600	So if we look at the instructions, we're going to use the
3723600	3726780	logistic regression constructor, but we're going to use this
3727580	3729580	LBFGS solver. And this is one of the
3729580	3732900	hyperparameters, or at least I would consider it a hyperparameter
3733660	3736360	for logistic regression itself.
3736980	3739420	There's a couple different solvers that you can use
3739420	3742020	but in this case we're going to use the LBFGS.
3742360	3743540	So what this is going to look like is
3743540	3746700	LGS TF IDF equals
3750340	3751140	logistic regression
3751720	3753800	and then we're going to set our solver
3754340	3756140	equals LBFGS
3757940	3760200	and then of course, just like we did with the multinomial
3760200	3762680	naïve Bayesian, we're going to fit
3762680	3766000	our train, our classifier
3766000	3768600	we're going to use the TF IDF
3769260	3771860	X, which is our test data, and then we're going to
3771860	3774580	use our training data, sorry, and then we're going to use
3774580	3775980	our labels.
3777620	3780080	Next we're going to hop in and do the score predictions
3780530	3783780	classification matrix and, sorry, confusion matrix
3783780	3785640	and classification reports. We're going to use
3786360	3788680	score LGS TF IDF
3788680	3793000	LGS TF IDF.score
3793000	3795000	We're just going to look at our
3795000	3797800	test TF IDF X and our
3797800	3800920	test labels. We're going to have our
3800920	3802060	predictions
3803270	3805940	LGS TF IDF equals
3806680	3810120	LGS TF IDF.predict
3812920	3814520	That's going to give us our
3814520	3815620	raw predictions
3815620	3818740	and then I'm going to
3818740	3820540	stop saying exactly what I'm typing
3822180	3824360	but we're going to do our confusion matrix
3839660	3840860	and our
3841700	3842820	classification report
3857280	3859380	Awesome. So I'm going to let this sit
3859380	3862240	here for a little bit, let you guys
3862240	3864560	kind of copy and follow along
3867120	3868380	Hopefully we don't run into
3868380	3871060	the same problems that we did last time
3873740	3875140	Bender is fantastic
3875140	3877260	I love it, but it will time out after about
3877260	3880440	five minutes and once it
3880440	3882400	times out you've got to kind of restart the whole thing
3883440	3885300	or at least you've got to restart the notebook
3885300	3887160	which is kind of annoying
3888400	3891220	Another thing you will notice is the logistic regression
3892020	3894500	will take a little longer
3896060	3899280	to train and give us our predictions
3899280	3901760	and it's just the way that logistic regression operates
3901760	3903140	it does a little bit more mathematical
3904240	3906220	or complicated mathematical formulas
3907000	3908920	than the multinomial Naive Bayesian
3910080	3912220	so it just takes a little bit longer to train
3912220	3915140	but not too much longer, so that's good
3917360	3919180	One of them is matrix multiplication
3919180	3921320	which is a whole array of floats
3921320	3923500	that you have to process versus
3923500	3927080	Naive Bayes is literally counting and some logs
3928600	3930340	when you're doing the math correctly
3930340	3931640	which is very quick
3931640	3934540	It's just counting
3934540	3936960	Yeah, it's just counting
3936960	3942820	and logs, the latest x86 stuff
3942820	3945060	have built-in processes
3945060	3947100	for doing integer logs and stuff like that
3947100	3948400	because it's so common
3948400	3951820	to use logs for numerical accuracy
3951820	3953340	in machine learning
3954720	3957380	You'll also notice the same thing with neural networks
3957380	3959640	because neural networks do a lot of matrix multiplication
3960840	3963940	which is why GPUs work really well for neural networks
3963940	3968100	because you can parallelize matrix multiplications like that
3968780	3970220	So I'm going to go ahead and run this
3970220	3971120	Shift-Enter
3973320	3977220	And it looks like it's running. Hopefully it doesn't take too long.
3979260	3983920	Another comparison to make is a neural network versus a gradient-boosted tree.
3983920	3988580	A gradient-boosted tree is all just if-then statements.
3989460	3994000	Versus a neural network is a huge pain in the ass to multiply everything.
3994000	3999220	So a GBT could be run in milliseconds
3999220	4003060	versus a neural network taking seconds in 100% of your CPU.
4005000	4008120	Actually, speaking of neural networks, I'm kind of curious about this.
4008120	4010440	How well this would be received at DEF CON.
4010440	4014220	Would anybody be interested? And please vote in the Discord channel.
4014240	4018400	If you would be interested in learning how to build neural networks from scratch
4018400	4023180	or how to build deep learning security tools
4023180	4025920	I think both would be really interesting.
4025920	4029120	Neural networks from scratch is actually a lot easier than you would imagine.
4029120	4031480	You can do it in about 11 lines of Python code.
4034640	4038360	Today we are able to simplify and easily explain the math
4038360	4042200	whereas a lot of these algorithms were invented in the 50s
4043020	4047460	where the people who would do this kind of work were called statisticians
4047460	4053520	or people who would apply these in business scenarios were called business intelligence.
4053960	4057340	But today with computer science and these abstraction libraries
4057340	4059560	it's a lot easier to understand.
4059580	4063760	So let me know in the Discord chat if there's any interest in either of those
4063760	4067200	as potentially future workshops for future DEF CON events.
4067220	4069280	Or even just AI Village events.
4069840	4073020	So KDAMinMax has a question on Twitch chat.
4073020	4074980	I'll just read it here.
4074980	4078380	I have a possibly stupid question about input structures.
4078380	4081860	If you're learning, there's no real stupid questions.
4084720	4088860	An email predictor acts on strings or a set of strings.
4088860	4092600	Are there any machine learning approaches that act reasonably well on trees?
4092640	4096020	For example, abstract syntax tree representations of applications.
4096600	4100740	For that, you might want to use a graph convolutional network
4100740	4108340	or, honestly, you can interpret language as an abstract syntax tree.
4108340	4111020	So you might want to reach in and use some of the...
4111020	4116160	There's a huge backlog of literature that people largely ignore
4116160	4119020	because of the success of deep learning
4119020	4124680	of how to do language processing using abstract syntax trees.
4124900	4128640	So you might want to reach into some old school machine learning
4128640	4132660	from before 2010 when neural networks destroyed the world.
4133960	4136760	I mean, built our jobs.
4139300	4141420	That was a good answer. Thank you.
4144600	4149660	So what we're looking at is logistic regression with TF-IDF vectorizer.
4149660	4151540	We have a slightly better...
4151540	4153640	It's a small bump, but it's slightly better
4153640	4156020	than the multinomial with the count vectorizer.
4156020	4160140	We have a 95.9% accuracy.
4160400	4164880	We see that we have a little bit more than the 26 ham emails
4164880	4167280	that were incorrectly classified as spam.
4167280	4171900	We have significantly fewer of the spam emails being classified as ham.
4172100	4176260	So this would be a perfect instance of moving towards
4176600	4179900	having absolutely zero spam get through our filter
4179900	4184840	at the risk of losing some of our regular emails.
4186020	4189380	But what I want to show is how the accuracy improves
4189380	4192880	just changing the algorithm itself.
4192880	4195700	And then looking at multinomial Naive Bayesian
4195700	4199640	compared to logistic regression both using TF-IDF.
4199640	4202320	We had 87.5% for the multinomial
4202600	4206720	and then we have 95.9% for the logistic regression.
4206720	4209720	So far, logistic regression is kind of beating out multinomial
4209720	4214100	in the TF-IDF and the count metrics.
4214100	4216680	So we're going to do our last one which is going to be
4219980	4223760	logistic regression with count vectorizer.
4223760	4225840	Count vectorizer worked well for multinomial
4225840	4228860	so let's see if it works well for logistic regression.
4228860	4230960	So we're going to do LGS count.
4236550	4238070	And I'm just going to type this out
4238070	4241310	but you can use the same copy-paste trick that I used for the multinomial
4241310	4242810	with the count vectorizer.
4244670	4248030	I'm just doing this as a way to kind of slow myself down
4248030	4252130	so that you all have an opportunity to copy the code.
4252350	4254150	So LBFGS.
4254570	4263500	We're going to fit our labels.
4263500	4266500	And we got to do our scores, predictions,
4266500	4268940	confusion matrix, and classification report.
4278730	4279990	Yeah, one of the things
4281090	4283070	that you kind of reminded me of, Sven,
4283070	4285150	when you've been talking about the different trees and algorithms
4285150	4288230	that have been overtaken by
4290490	4292290	neural networks and deep neural networks
4292290	4295090	is random forests. Random forests operate very well
4295090	4299530	on a lot of different scenarios.
4300310	4301350	But because
4301350	4304710	random forests are a cluster of decision trees
4304710	4306810	people go, oh, they're just if-then statements.
4306810	4308870	This is dumb. This isn't machine learning.
4309670	4312970	Yeah, but for us, we have
4313050	4315810	a highly shared data where,
4315810	4318870	for example, on the Ember malware dataset,
4319470	4322330	the lightweight,
4322330	4323250	incredibly cheap
4325090	4327970	gradient decision tree built with LightGBM
4327970	4331190	outperformed any neural network you can throw at it.
4331190	4333910	So the LightGBM gets 99.8%
4333910	4336970	accuracy or something like that, and the neural network gets
4336970	4339910	98.7% accuracy, which is
4339910	4341870	significantly worse.
4344350	4346230	For some tasks,
4346410	4349650	a gradient-based decision tree or something like that
4349650	4352170	is the correct thing mathematically
4352590	4354290	because neural networks have biases
4354970	4358070	because it's easier for them to learn some things than others
4358070	4361050	and GBTs have biases where
4361050	4363930	it's easier for them to learn some things than others.
4363930	4366550	And one of the great secrets
4366550	4368510	of machine learning is
4369190	4372290	match up your problem with both
4372640	4375850	the requirements, how powerful your predictor
4375850	4378010	needs to be, and the inherent biases
4378010	4380810	of those predictors.
4382130	4382910	Yeah.
4386480	4389300	Yeah, that's pretty interesting. I didn't know that.
4390700	4392780	But that also reminds me of another thing
4392780	4396720	with neural networks. Neural networks are really good at learning superstitions.
4397060	4399520	I don't know if that's a machine learning as a whole thing
4399520	4401780	or specifically in neural networks
4401780	4404960	but I remember there was a dataset that they were using to identify
4404960	4408040	skin cancer cells. They used a convolutional
4408040	4411180	neural network and based off of that it was learning
4411180	4414640	that if there was a... it was performing too well.
4414640	4416960	It got nearly 100% and they
4417540	4420140	kind of dug into what it was doing
4420140	4422800	and it said, OK, well if I see a ruler in the picture
4422800	4425780	which the scientists, how they were collecting the images, they had a ruler
4425780	4428860	next to the skin cancer cells. If I see a ruler in the picture
4428860	4432380	that's how I know it's cancer. Not the little blob
4432380	4433480	next to it.
4435100	4437640	So I don't know if that's a whole machine learning problem
4437640	4439140	or specifically a neural net problem
4439140	4443140	but it's just kind of an interesting anecdote.
4444060	4447740	Be careful how you collect and preprocess and use your data.
4449060	4451640	A friend of mine did his PhD.
4451640	4454340	His PhD thesis was inducing the
4455220	4457560	bias that's sort of like the inherent
4458360	4461140	model bias that's in convolutional
4461140	4463980	neural networks into other models
4463980	4466140	and it slightly improved.
4469200	4471620	By inducing a bias?
4471840	4475340	Yeah, you can induce a pixel
4475340	4478100	locality bias like a great green
4478100	4482140	tree or a support vector machine
4483260	4484780	and it improves
4485700	4487740	that's one of the assumptions that people make
4487740	4490920	about convolutional neural networks is it allows you to make
4490920	4493900	small pixel local decisions
4494880	4496880	and then those can get glued together
4496880	4498500	in the right way.
4500160	4502900	And so he went and built little modifications
4502900	4505580	to a dozen other algorithms
4505580	4507260	that have sort of fallen out of favor
4507260	4511400	and all of them improved slightly on images
4511400	4514500	when you induced a pixel local bias.
4514980	4517160	Okay, interesting.
4517160	4520200	Yeah, that's
4523140	4523920	that's actually
4523920	4524900	really interesting.
4527480	4528920	Alright, so
4529520	4532620	hopefully I gave you guys enough time to make sure that this got
4532620	4534980	copied down. I'm going to go ahead and run it
4534980	4538220	and shift enter. This is going to take a little bit because it's going through
4538220	4540320	its matrix multiplications
4541200	4544100	but in the meantime, we're about to hop into
4544620	4546780	our final task
4546780	4549960	and what we have is a real spam email
4549960	4552580	I pulled this out of my inbox or my spam folder
4554220	4555100	last May
4555860	4558160	but we see your latest issue is available now
4558160	4560940	if you don't want issue notifications, click here to unsubscribe
4561660	4563840	Hi George. My name is not George
4565180	4567400	but we're going to look at this email
4567400	4570260	and we're going to try it against all four of our spam classifiers
4570260	4573360	and see how well they do. But what's also good about this exercise
4573360	4576400	is it doesn't really hold your hand. This is usually something
4576400	4579500	that I give plenty of time in a classroom
4579500	4581480	setting. This allows people to ask questions
4582100	4584860	as I go through. But this last task is going to say
4584860	4587360	OK, we have a new email we've never seen before
4589540	4591160	run it against our classifiers
4591160	4593960	that we trained. And so you have to go through
4593960	4597080	the whole process of using the
4597080	4600000	vectorizers to vectorize the tokens
4600000	4602940	or the words in the email and then putting them through the predict
4602940	4605880	function and then printing out the prediction
4605880	4608960	that it gives. So before
4608960	4611320	we hop into that, I really want this to finish
4611320	4614620	I want it to just be on its own time
4615780	4616380	but
4619600	4621240	we're going to
4621240	4623860	compare how well these models did
4625040	4626760	once this last one is complete
4626760	4629520	and we're going to see if or how much
4629520	4632660	that comparison has an influence on our test email
4635180	4637460	So we're just going to give this another
4638580	4641180	Probably be another 30 seconds or so.
4641180	4642860	So it's not too long.
4642860	4644000	How's the Discord chat?
4647820	4649460	Discord chat's doing well.
4656560	4660940	I really appreciate how much discussion we have in this version of the workshop.
4661180	4665000	Also, and this is something that I want to note for people who are seeing this for the first time.
4665000	4667820	So the last time I did my workshop on Friday,
4667820	4671280	unfortunately my webcam froze in a very unflattering position,
4671280	4673140	so I just had my mouth gaping open.
4674540	4679600	So for this one I just put a static image of myself today.
4679940	4681740	That way you can still see my beautiful face,
4681740	4684940	but you don't get a very unflattering image of it.
4685780	4688320	Alright, so this one is complete.
4688320	4693780	We are now at the best model that we have of the ones that we've trained.
4693780	4698200	We have a 97.69% accuracy.
4698200	4701100	We can round that up to a 97.7% accuracy.
4701100	4708180	We see that only 8 out of our 582 ham emails were misclassified as spam,
4708180	4714720	and then only 12 out of our 264 spam emails were misclassified as ham,
4714720	4719680	making this the best model that we have out of the 4 that we've trained.
4721400	4723980	So that's pretty solid.
4724960	4729860	So moving forward, what I would do is take this as kind of the better of the models,
4729860	4733140	and I would go ahead and deploy this in production,
4733140	4740440	or add this model into an application that's designed to kind of wrap around the idea of spam and ham.
4740960	4745520	So if you are simply an email provider, you can use this to filter email,
4745520	4753300	or if you are a spam company, or not a spam company, but like a spam research company,
4753300	4760920	you would use this to grab the spam messages and then collect that as data,
4760920	4765540	and then you can use that to be training the next version of your spam filter.
4767680	4771580	So let's take a look at our test spam email.
4772180	4776900	We have our final task, which is to use the vectorizer and the models that we created
4776900	4782360	to perform predictions on our test email and our test spam email.
4782360	4789500	So to make things easier, what I'm going to do is use a variable, I'll call this working email,
4790340	4793420	and this will allow us to switch pretty quickly.
4794180	4800620	So I have test email, and then we're going to want to vectorize our email,
4800620	4804960	so we're going to use our test email tfidf,
4807640	4811620	yes, tvec.transform,
4812520	4815940	and this is going to be, remember it needs to be in a list,
4815940	4819620	so we've got to use these square brackets, our working email,
4820280	4827660	and then we'll do the same thing with our test email count, tvec.transform,
4830760	4832180	working email.
4833180	4836520	Now we need to get the raw predictions for every single one of them.
4836520	4840880	The raw predictions, we already have the variable names here, kind of at the bottom,
4840880	4842960	so I'm just going to go ahead and write them out.
4842960	4846660	So we have test email mnb with tfidf,
4847840	4852600	which is going to be our mnbtfidf.predict,
4852600	4859840	on our test email with the tfidf vectorized info,
4860620	4864520	we're going to do the same thing for count,
4864520	4866800	using our count.predict,
4874490	4877830	with our count vectorized,
4878390	4883850	test email lgs, tfidf.predict,
4892020	4895220	tfidf.data, and our last one,
4895220	4898480	test email lgs.count,
4904520	4905560	count.
4907040	4909560	And then, just for fun and games,
4909560	4912800	we can print out our working email.
4913020	4915240	So this is going to print our test email.
4915240	4917400	Our test email was the one that we saw,
4917400	4919940	EastAsianFonts and Lenny, thanks for your support.
4922500	4926640	So you can go ahead and copy this down.
4926640	4929220	I'm going to go ahead and run this,
4929220	4931780	because we are running close to time.
4931780	4933820	So this is going to be our test email,
4933820	4935840	and ideally, this should say that
4935840	4938140	it is a regular email, not spam.
4938140	4941100	It's going to label all four of these as ham.
4941120	4942620	So let's run it.
4942920	4944780	Working is not defined,
4946540	4949540	because I forgot to call this working email.
4950200	4951560	So we're going to run it.
4952280	4953720	Here's our email.
4954380	4956140	And all four of our classifiers
4956140	4959500	correctly classified this as ham.
4960360	4962220	I'm going to scroll back up,
4962220	4968280	and we saved our email as our test spam email.
4968280	4970780	So I'm going to grab that,
4970780	4972520	and I'll run it.
4973260	4975520	Oh, I forgot to run the code block
4975520	4978200	that actually defined our test spam email.
4979120	4980960	And now we can run this.
4981560	4983060	And we see something interesting here.
4983060	4984760	Our worst performing model,
4984760	4986720	our MNB TF-IDF,
4986720	4989480	which was 87.5% accurate,
4990520	4992900	misclassified this email as being legitimate,
4992900	4994000	as a ham email.
4994240	4995600	But the other three of ours
4995600	4998100	classified this correctly as a spam email.
4998740	5000700	So there's a couple things we can do with this.
5000700	5002380	If we were so inclined,
5002380	5004540	we can use what's called ensemble learning,
5004540	5007380	which we use all four algorithms.
5007380	5009120	We take the average of them,
5009120	5011240	or in other words,
5011240	5014820	they give a vote on what they think the class is.
5014820	5016420	And then in this case,
5016420	5017820	three out of the four say it's spam,
5017820	5018740	so it would be spam.
5018740	5022240	Or we can use this as an example and say,
5022240	5024420	hey, the MNB TF-IDF,
5024420	5025680	we probably shouldn't use that
5025680	5027780	simply because it's misclassifying
5027780	5029680	known spam emails
5030420	5032340	at a high rate.
5032340	5035180	87% means that we have
5035180	5038840	about a 12.5%
5040640	5043760	chance of misclassifying spam emails.
5045020	5046740	So this is kind of
5046740	5048960	pretty close to the conclusion of the workshop.
5048960	5050380	This shows how you can take
5050380	5052540	raw emails, raw data,
5052540	5054100	transform it in a way
5054100	5055300	that can be used in
5055780	5057520	machine learning models.
5057720	5059800	And then we build out our machine learning models,
5059800	5060880	we evaluate them,
5060880	5062660	and we can test them.
5062660	5064020	And this last task shows you
5064020	5066220	how we can properly deploy them.
5066220	5068080	So we can build a piece of software
5068080	5069240	or an application around it
5069240	5070820	and say every time we get an email,
5072180	5073500	just swap out
5073500	5074960	this working email,
5074960	5077580	and then you can leave everything else the same.
5078800	5079960	Now, before I let you go,
5079960	5081900	there's one more thing that I want to show you.
5082260	5084440	I mentioned hyperparameter tuning.
5084440	5086000	So let's take a look and see
5086000	5088620	what hyperparameter tuning can do.
5088980	5090060	So when we were talking about
5090060	5091720	the multinomial Naive Bayesian,
5091720	5093560	we saw that we had that smoothing function,
5093560	5095620	the alpha, right?
5096000	5098280	And it was normally 1.
5098580	5100980	So we can add that in here.
5100980	5103360	So we have alpha equals 1.
5103360	5107900	We see that an 87.5% accuracy.
5108280	5109720	We go ahead and run that.
5109720	5110800	I'm just doing this to show you
5110800	5112260	that there are no tricks up my sleeve.
5112260	5115460	Alpha is by default 1 for this model.
5116240	5117740	But what if we change that
5117740	5120060	to maybe something a little smaller, right?
5120060	5121480	Maybe a 0.1.
5121480	5124200	Let's reduce the impact of that smoothing filter.
5124200	5125280	So I'm going to go ahead
5125280	5128200	and press Shift-Enter and run that.
5128860	5130300	And we see something interesting.
5130300	5134160	So this went from our worst model,
5134160	5136340	87.5% accuracy,
5136340	5140140	to actually our second best model,
5140140	5143060	96.5% accuracy.
5143740	5147300	And so it slightly misclassified
5148380	5150720	some of the legitimate ham emails as spam.
5150720	5152780	But it drastically reduced the,
5152780	5156300	I think it was 108 spam emails
5157160	5160260	being misclassified down to just 22.
5161740	5164240	So this is the power of hyperparameter tuning.
5164240	5165400	And if we go all the way back down
5165400	5167720	to our final task, task 7,
5167720	5169640	and we run this,
5169640	5172640	where the multinomial manipulation with TF-IDF
5173240	5174840	incorrectly classified that as ham,
5174840	5176440	we go ahead and rerun it.
5176760	5178860	Now it's correctly classifying it as spam
5178860	5180500	because its accuracy is much better.
5180500	5182040	And the only thing that we did to change it
5182040	5183840	was that hyperparameter tuning.
5186920	5189060	So that's all I have for you guys today.
5189060	5192440	I have a couple good resources,
5192440	5194320	but the biggest one that's going to be helpful
5195160	5196760	is if you have any questions,
5196760	5197720	feel free to reach out to me.
5197720	5199540	I am gtklondike on Gmail.
5199540	5201760	I'm also gtklondike on Twitter.
5202020	5203400	I'm not as active on Twitter,
5203400	5204780	but I will see your direct messages
5204780	5207540	or any at mentions.
5208340	5210120	And then the GitHub code
5210920	5212360	for all of these workbooks,
5212360	5214060	this and all the workbooks,
5214060	5217000	are on github.com slash netsecexplained.
5219060	5221540	So that is all I have for you guys.
5221540	5224660	It looks like we just barely made it.
5224780	5225740	1231.
5226960	5229160	And I will let you go.
5229160	5232420	I will stay in the chat channel
5232420	5233240	for a little bit
5233240	5236140	in case any of you would like to continue to discuss.
5236140	5237480	And I'll also stay in the voice channel
5237480	5240100	for a little bit for the same reason.
5240220	5242480	So I will see you there.
5242720	5243900	Have a good day.