start	end	text
5790	11750	Hi, and today with Young-Hoo Lee, I'll be presenting Detecting Handcrafted Social
11750	15510	Engineering Emails with a Bleeding Edge Neural Language Model.
19940	22700	A bit about us before getting into the talk.
22700	28680	Sophos is a security products and services company. We have a whole range of security
28680	35020	products ranging from firewalls, to mobile security, to email security, to endpoint security.
35020	43040	And the team that I manage, Sophos AI, is the AI shop inside of Sophos. We research and develop
43040	47000	the company's machine learning technology. And then we're also responsible...
47000	50340	Okay, I'm gonna mute myself. I'm gonna like definitely mute myself in here so I can listen
50340	56180	to the talk. I'm happy if you need something. Young-Hoo Lee is one of the star researchers
56180	63460	on my team. He's almost completely solely responsible for the machine learning subsystem
63460	72180	that protects our Android customers, at least responsible for the R&D part of that work.
72180	76000	He's also the principal author of the work presented today. I'm presenting in a supporting
76000	80180	role here. I'll mostly just be setting the stage for the original work that Young-Hoo is gonna
80180	90040	describe. So, the problem we're solving. Most generally, we're working on the problem of
90040	96520	detecting phishing emails. More specifically, we're working on detecting business email
96520	102380	compromise phishing emails and targeted phishing emails. So, to understand what this is, it's
102380	109100	useful to say what it's not. We're not focused on detecting mass campaigns where detection
109100	114340	reduces to a near-duplicate detection problem because attackers are just sending out millions
114340	120020	of copies of basically the same phishing email. Detecting those types of emails turns out to be
120200	127380	a side effect of our focus. But our real focus is on detecting new, custom-authored, bespoke
127380	133200	phishing emails that are based on research on a target. And this diagram does a good job of
133200	138840	getting across the kind of workflow we're looking at stopping. In this workflow, cyber criminals and
138840	147080	attackers identify targets through open-source research, usually on the web. Then they establish
147080	154980	contact with those targets. In step two, we call this grooming. In this case, they put out a lure,
154980	161280	an initial email, usually or sometimes an initial text message, and then build trust and authenticity
161280	169500	around some identity that they're impersonating with the mark. In step three, they cash out the
169500	174860	trust that they've built with their targets and make an ask of those targets. Oftentimes, that
174860	179500	will be around wiring money or sending credentials. And then in step four, they actually
181820	184050	receive money or receive the credentials.
187020	191540	So within targeted phishing, business email compromise, which is focused on
191540	197160	stealing money from businesses and other organizations, has been a growing trend.
197160	203220	So as you can see here, in July 2016, a few billion dollars were stolen, according to the FBI,
203220	207720	through business email compromise attacks. These are targeted phishing attacks that extort money
207720	215020	from organizations. In May 2017, that number had grown to almost six billion. And in 2018,
215020	219140	that number had grown to more than 12 billion. I don't have data for the last two years, but
219140	224940	I see no reason to believe that this trend has attenuated. I think it's likely it's continued
224940	230460	on a similar trajectory. This is a big problem. We see this in Sophos's customer base,
230460	235520	which is which is a pretty large sample. We also hear about it from other folks in the
235520	239940	cybersecurity space. And so we're very focused on it because it's affecting people. And it's
239940	244220	not just affecting large organizations. You can see on this on the axis on the right,
244220	252580	that something like 80,000 organizations had been hit by July 2018 by these attacks.
253900	257860	So there are lots of small and mid-sized organizations getting hit. And oftentimes,
257860	263100	financial damage can be in the hundreds of thousands of dollars and really impact people's
263100	269240	lives when these attacks happened. So again, just to reiterate, we're focused on primarily
269240	273220	these business email compromise use cases, but also more generally targeted phishing in which
274600	278920	a lot of manual labor goes into the phishing process on the criminal actors.
281520	285560	And I think it almost goes without saying, but we're focused on step two and three
285560	291300	of these criminal actors workflow, the steps that are mediated over email. So we see lots of
291300	298840	malicious emails exchanged in the grooming step. And then we also see, obviously, a malicious email
298840	302440	transmitted in the exchange of information step where the attacker makes the ask of the
303040	310920	target, usually employee. And this is just because we're focused on email as our signal.
311860	314500	And that's the scope of the work that I'll be talking about today.
316180	321760	Just to flesh this out a little bit, here's an example, phishing email sent in the later
324060	330900	epics of the grooming stage of an attacker's workflow. Here, the attacker has established
330900	338040	themselves as an impersonated chancellor of UC Berkeley. They are emailing an employee at UC
338040	342640	Berkeley, asking if they're available, looking to exchange messages with them, probably about to
342640	351510	make an ask around a money transfer. Now, I think it's important to highlight why phishing
351510	359570	detection is hard. And why in particular, detecting new previously unseen phishing emails written as
359570	367590	part of a manual phishing campaign, like what I've just described, is hard. And what this boils down
367590	373870	to, I think, is that classical natural language processing problems are hard. It's hard to
373870	378990	get computers to understand language in any meaningful way and reason about language in any
378990	386530	meaningful way. And detecting phishing emails really boils down to building models that have
386530	394670	some level of understanding of language. So to understand why algorithmically it's hard to make
394670	398910	sense of language, let's look at a few classical natural language processing problems. So one of
398910	406470	those is coreference resolution. To get a sense of what this problem means, consider the following
406470	411510	sentence. I went to the store for some milk, and based on the price, decided to buy it.
412470	417950	Now, from a grammatical perspective, it could refer to the store or to the milk, right? It's
417950	421470	possible that I went to the store for some milk, and based on the price of the store, I decided to
421470	429030	buy it. But it's more likely that I bought the milk. As humans, in solving this coreference
429030	437430	resolution problem, so resolving what it refers to, we have to plumb the depths of a number of
437430	443950	complex mental models, right? We need to deploy our syntactical model of the English language,
443950	447850	our semantic model of the English language, and our model of the world in which a person is more likely
447850	452930	to have bought milk at the store than have bought the store itself. And that's how we solve this
452930	459530	problem. Hopefully, it's clear that it's hard to get algorithms to do that. But to detect phishing
459530	464890	emails, we really need to understand language. And this is a problem that's constituent of the
464890	469030	problem of understanding language. Word polysemy is also a classic problem
469030	473850	in algorithmic understanding of language. So a sentence like, he drank a lot and was quite the
473850	480830	rake. Grammatically, it's valid to interpret this sentence as meaning that he drank a lot
480830	488590	and was quite the garden tool used to rake leaves off your lawn. That's clearly not the right,
488590	493070	that's not the sense in which the word rake is being used. The word rake is being used in the
493070	500970	sense of a drunk, semi-criminal, sort of dissolute individual here. But it takes a pretty deep
500970	507870	exercise of a human being's mental models to arrive at the sense in which this word was used.
507870	513150	And not easy to reproduce in the form of an automated agent, either machine learning or
513150	518430	based on regexes and rules. Sentiment detection is another hard problem in natural language
518430	523450	processing. So consider the sentence, I'm not angry at all. No, of course. Why would I be angry
523450	529130	that you spent our life savings on your mistress? So clearly, the speaker is angry here. And they're
529130	534630	being sarcastic. Detecting that they're angry and sarcastic is not trivial and requires that
534630	540010	we understand not only the syntactic structure of the sentence, but also the semantics of the
540010	545470	sentence. And requires that we have the reflex that this person is probably angry if their
545470	555050	interlocutor spent their life savings on their mistress. Okay, so to solve phishing means that
555050	559330	we need algorithms that can make sense of language. Making sense of language is hard, as these three
559330	568610	problems demonstrate. So a good solution to the phishing problem would model as intermediate
568610	574150	steps to detecting that an email is a phishing email. At some level, be able to solve these
574150	577930	problems and sit somewhere in the depths of this intermediate representation of the language that
577930	582410	it's looking at. The other challenge we have, obviously, in any cybersecurity problem, at least
582410	586350	any detection context in cybersecurity, is that we have adversaries who'd like to bypass our
586350	591490	detection. And that's also worth considering. So these are all reasons why the problem that we're
591490	596190	presenting here is a hard one, deserves... we haven't solved it completely, deserves attention
596190	605110	from our community. Now, the approach that we're using to attack the phishing problem
605610	611490	is based in neural networks and deep learning, and a specific advance that happened in the last
611490	619710	few years known as transformers. So transformers, or more specifically, transformer blocks, are a
619710	626950	new kind of construct in neural networks, much like convolutions were new, I think, as of the
626950	631730	90s or late 80s, and backpropagation was a new idea in neural networks, I think,
631730	636430	starting in the 80s. Transformers are a new idea that's come out in recent years,
637290	644370	and they help model language with a depth and fidelity that seems to be genuinely new and
644370	649090	represent a step function in our ability to model language. So they're very exciting. The big idea
649090	652670	behind the work that we're presenting today is that we're taking transformers and applying them
652670	657010	to a cybersecurity problem, which we haven't seen much of before. So I'm going to talk a bit about
657010	662270	what transformers are. A detailed discussion of how they work is beyond the scope of this talk,
662270	669130	but I'm going to give some intuition. And then I'll pass the mic metaphorically over to
669130	679540	Young, who will present on how we're using transformers. Okay, so here's an example,
679540	686940	which I think helps to illustrate some ways in which transformers represent a real breakthrough
686940	694580	in machine comprehension of language. So in this example, and I expect some of you have seen this
694580	702720	because this went viral last year, a researcher wrote a prompt. This prompt is given at the top
702720	710360	here in a shocking finding, that paragraph. And then a transformer model just took it from there
710360	718740	and wrote a story based on that prompt. And I think when you see the story that the machine wrote,
718740	726680	you'll see that it betrays a understanding of the syntax and semantics of language
727760	732560	that's quite striking. So I think it's worth reading this out loud. The human-written prompt
732560	737860	reads, in a shocking finding, scientists discovered a herd of unicorns living in a remote,
737860	742360	previously unexplored valley in the Andes Mountains. Even more surprising to the researchers
742360	747840	was the fact that the unicorn spoke perfect English. And now again, the challenge to the
747840	753440	left off and write something coherent. And if it can, that reflects something around its ability
753440	760740	to solve problems like the fundamental problems I described earlier. So the neural network
760740	765980	continues. The scientists named the population after their distinctive horn, Ovid's unicorn.
765980	770180	These four-horned, silver-white unicorns were previously unknown to science.
770180	775380	Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved.
775380	780440	Dr. Jorge Perez, an evolutionary biologist from the University of La Paz and several companions,
780440	785460	were exploring the Andes Mountains when they found a small valley with no other animals or humans.
785860	789860	Perez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks
789860	795360	of rock and silver snow. Perez and the others then ventured further into the valley. By the time we
795360	799700	reached the top of one peak, the water looked blue with some crystals on top, said Perez.
799700	807280	And the essay continues. But what interested Younghoo and I in seeing examples of what
807280	812720	transformers can do like this is that it seems that the transformers have, examples like these
812720	817680	suggest that transformers have understood the syntax and semantics of the English language
817680	825220	to some degree, and which begs the question, could the representations that the transformers
825220	832140	learn in their parameter structure be useful in detecting phishing emails, since these
832140	839080	transformer models seem to have distilled knowledge of how language works and what it means
840600	846220	within their parameter settings. Okay, so this is one example of, sort of impressive example,
846220	849960	of what transformers can do, that beg the question of whether or not they could be
849960	854580	applied usefully to phishing. And we'll show results from our experiments later in Younghoo's
854580	863840	section. Here's another example of just how sort of marvelous transformer models are with respect to
866380	872360	how much more sophisticated they seem to be. So here a researcher typed into this box
873100	876660	a description of some HTML that he would like the transformer models to generate.
876660	882200	He writes, a button with the color of Donald Trump's hair. And the model actually writes,
882820	889040	as a text completion, some HTML and CSS. It seems to understand that Donald Trump has yellow hair,
889040	897000	and it makes a button, and it writes valid HTML and CSS. Which is, I think, just
898720	902820	impressive in the absolute sense, but also for folks who've been in the NLP space for a while,
902820	907620	it's impressive relatively. Five, ten years ago, if you would have shown somebody that we could do
907620	913820	this in 2020, I think they would have been a bit incredulous. This is a big step for you.
913960	918280	Applications like these represent a big step forward in natural language processing. And
918280	924640	they're due, again, to this new idea called a transformer block. That's the key building
924640	929120	block out of which models like this model, which is called GPT-3. It's from OpenAI,
929700	939340	which is an AI research lab, are based. Okay. So, moving on.
939900	947180	So, I want to go back a little bit in the history of NLP as a way of describing how
947180	954040	transformers are new and different. So, I assume a substantial chunk of our audience today
954640	962720	has studied basic machine learning on text. And typically, in like a machine learning 101
963380	970200	course, you learn about the bag of words model and about discrete states, Markov models of language.
970860	975600	And I want to talk about these representations as a way of talking about how limited those
975600	979580	representations were, and then I'll talk about how transformers break us free of some of those
979580	985960	limitations. So, a bag of words model of a document is a way of representing a document
985960	990280	numerically for the purposes of machine learning, in which we just count up how many times each
990280	997600	word in that document appeared. And then we create a matrix out of all the documents we have,
997600	1003000	and all the words in all those documents. And the entries in those matrix are just word counts.
1003000	1010320	So, in the case on the left over here, the column vectors in our matrix are documents,
1010320	1016160	and they get one dimension per word in our vocabulary, and they get counts of how many
1016160	1021560	times words appeared in that vocabulary in the entries of that particular vector.
1022840	1027120	And hopefully it's intuitive that, you know, once you've represented your documents in a
1027120	1031120	vocabulary space in this way, you can compare documents by taking some distance measure
1031120	1036400	between pairs of documents. You can also train machine learning models on your document corpus
1036400	1044520	to say classified news articles is about sports or politics. But what you've done in the first
1044520	1052140	step of these models is drop out sequence information. So, you've forgotten about which
1052140	1058760	words come in which order in the document, and you've just represented your document as a bag
1058760	1063820	of words. Which is a useful simplifying assumption, and it's one that we still use
1063820	1068720	today in some of the modeling we do in my research group at Sophos. But it throws out
1068880	1072820	a ton of information that transformers and more modern models don't throw out.
1073740	1078460	In the model on the right, we have a discrete state Markov model of language.
1078740	1085160	Here each word is a state, and the concept of language given by the model
1086440	1092560	is kind of a cheesier adventure story in which the next words in an utterance depends only on
1092560	1097620	the current words, and you're just sort of drawing from a probability distribution and moving through
1097620	1102580	this graph to generate language. You could never have generated anything close to that unicorn
1102580	1109120	story using a Markov model, and yet as recently as the last five, ten years, there's lots of
1109120	1114840	papers coming out around like using hidden Markov models to parse sentences and that kind of thing.
1115160	1119060	These are still useful models, but we've gone far beyond the simplifying assumptions
1119520	1127540	in these original sort of simplified models of the world in NLP from the past few decades.
1129500	1134820	So let's contrast transformers now, and I'll get into more details about how transformers
1134820	1139400	work in a second, but let's contrast transformers with these earlier natural
1139400	1146320	language processing approaches. So pre-transformer, most machine learning
1146320	1154620	approaches didn't consider words in context. Many approaches made the simplifying bag of
1154620	1162980	words assumptions as kind of a first step in the modeling process, and then ran term vectors
1162980	1169980	through models like topic models, or logistic regression, or support vector machines.
1170620	1179240	Most models didn't model kind of like co-reference relationships between words,
1179240	1184520	or sort of which words pertain to which other words in a sentence, and I'll talk about what
1184520	1191140	that means later, but transformers do sort of solve that problem. Most approaches didn't
1193320	1200340	operate on either words or characters. Typically the way people use transformers,
1200340	1206600	we use well-chosen chunks of words, which allows us to model misspelling and this kind of thing,
1206600	1210960	so there's been an improvement there in the current generation of natural language processing models.
1212060	1222200	And older approaches tended not to use neural network technology. That's changed a lot in the
1222200	1229400	best ideas that have come out of the neural network revolution that's been ongoing since
1229400	1238980	around 2012. So transformers kind of kick apart a number of log jams in natural language modeling.
1238980	1244320	They give words contextual representations, they model attention, like the relationship between
1244320	1250260	words in a document, they use these smart partial word representations that allow for
1250260	1254340	misspellings and just the tower of babble of vernaculars that appear
1255800	1260500	like under the banner of, say, the English language on the internet, and they take advantage
1260500	1267160	of ideas like residual connections and modern optimizers and many of the really good ideas
1267160	1271980	that have come out of the neural network revolution. So these are all reasons why
1271980	1281540	we wanted to test their applicability to phishing detection. Okay, so if you want to get into detail
1281540	1287780	about how transformers work, I'd recommend this blog post by Jay Alomar. It's where I cribbed this
1288340	1294360	screenshot here. I'm not going to get into the details of the series of matrix multiplications
1294360	1301140	that comprise summations and various linear algebra and operations that comprise the
1301140	1305640	transformer block. I just want to give a little bit of intuition here before passing the mic over
1305640	1312300	to Younghoo. The basic idea behind a transformer block, which is kind of a Lego block,
1312860	1319540	out of which you build a transformer-based neural network, is we're taking in a sequence of words.
1320140	1324060	This diagram shows a very simple example where we're just taking a sequence of two words. Typically
1324060	1331400	we've taken a larger window, like 512 words. We pass them into the block, and the way they
1331400	1340760	get passed in is not just as entries in a term vector matrix, but actually as vectors themselves.
1340760	1347920	The words get a vector representation. These are known as embeddings. We pass them both into the
1347920	1355440	transformer network. The first thing that the transformer network does to our input word
1355440	1365520	sequence is model the attention relationships between the words. In a two-word case, it's a
1365520	1371180	little bit harder to describe here, but basically if you had a sequence of 15 words,
1371180	1377840	for every word, the attention mechanism would compute how much
1377840	1384240	that word pertains to the other 14 words in the sentence. You'll see how that has a
1384240	1388600	relationship with the coreference resolution problem I talked about earlier. But it's just
1388600	1394560	intuitive that there's a graph of word relationships in a sentence, and self-attention
1394560	1401820	kind of models that in terms of which subjects pertain to which objects,
1401820	1407760	which pronouns pertain to which people, etc. There's an addition and normalization step
1407760	1413680	that happens when we've run this self-attention process a number of times
1414920	1419460	and typically we don't just do self-attention once. We have a number of what are called heads and
1419460	1424520	we run attention a number of times. We combine all that together, we do some non-linear transformation
1424520	1429800	on it, and then we wind up with a new embedding of your original sequence of the same dimensionality
1429800	1436000	as this original embedding, except that now thinking is encoded in this new representation
1436000	1440680	in the context in which it appears, in the context of machines. Machines is encoded in the context of
1440680	1444400	thinking. Then typically we stack these transformer blocks that we actually do.
1444400	1447840	We have another transformer block that then sort of refines the representation and we keep
1447840	1452500	going. Young will show that we use a number of transformer blocks in our phishing detection
1452500	1459240	work in a few minutes. Here's some intuition as to what comes out of the attention mechanism
1460220	1465980	in a typical transformer. So here's what a transformer block has decided it
1467720	1471440	sort of relates to in an input sentence. So here we have the input sentence,
1471440	1474880	the animal didn't cross the street because it was too tired.
1476440	1483700	And the strongest attentional relationship here goes to the animal, which is interesting
1483700	1489140	because one could ask whether or not it refers to the street or the animal here.
1489780	1494200	One can interpret the weight of the connection to the animal, meaning that the transformer block
1494960	1500760	has decided, in their quotes, that it pertains to the animal, which is really interesting.
1501660	1505160	So hopefully you get some intuition as to how powerful this attentional representation is and
1505160	1511160	how important it is in machine learning models, getting some what we might call understanding
1511160	1518410	over the language that they're analyzing. Okay, so I want to put the intuition together
1518410	1523410	around how transformers pertain to the work that YoungHu and I are presenting today.
1524230	1531350	So basically, what we're going to do in our phishing model is embed an email as a sequence
1531350	1539370	of embedded character sequence vectors. And then we're going to run that through a series of
1539370	1546350	transformer blocks, like what I just showed, that are going to create a very refined attentional
1546350	1554790	representation of the word sequences, and then produce these contextual embeddings that get at
1554790	1560490	the meaning of the words in the context in which they appear. And then finally, our network is
1560490	1565790	going to solve a classification task, say whether or not the email is a phishing email or not.
1566310	1570190	How all that magic works, how the network gets trained, which there's some tricks there that
1570190	1576170	are really cool. I'll leave to YoungHu, and hope a lot of this makes sense, and happy to
1576170	1580550	take questions about my piece of this presentation later at the end of the talk.
1581970	1590850	Thank you, Joshi. Let me continue the second part of our talk. The second part will include
1591570	1599250	our design decisions for CatBot and performance result.
1599830	1608130	CatBot is the name of our email model, context-aware tiny bot. The model size is tiny,
1608130	1621580	but it is mighty bot. Modern NLP models all have a nice and friendly name. For example,
1622150	1631120	Elmer was introduced in 2018. The model used bidirectional LSTMs to
1631650	1636440	generate contextualized word embeddings.
1637650	1647980	And then, same year later, Google researchers introduced bot and achieved the state-of-art
1647980	1657120	performance in many English-understanding problems. Next year, 2019, Baidu researchers
1657120	1666660	introduced the Orni, and the model achieved another state-of-art performance in many
1666660	1676220	Chinese-language-understanding problems. They are all popular characters from Sesame Street.
1677180	1686880	This year, 2020, we introduced CatBot to tackle email security problems.
1692010	1700270	Transformer-based NLP models are powerful, but they are complex and heavy.
1700790	1709050	And it is challenging to deploy heavy models for real-time applications.
1709670	1718770	So our first design goal is to convert the heavy model into a lightweight model,
1718770	1725810	so we can reduce the number of parameters and then we can improve inference speed.
1726450	1733890	We downsized a baseline model called DistribBot, which has six transformer blocks.
1734590	1740650	We take half of transformer blocks from pre-trained model and then replace missing
1740650	1750110	transformers with simple adapters. For example here, we take transformer 1, 3, 5,
1750110	1758510	and then we added two adapters. Also, we can take other number of transformer blocks.
1759320	1764840	This approach allows simply significantly reduced number of parameters.
1769940	1779020	The second goal is to improve the model performance by combining additional input.
1780420	1790160	Standard NLP models only accept text data as input. However, we can extract additional
1790750	1798420	features from email headers and we can use the additional input to our model.
1798900	1807660	So the text input will be the input to the embedding and then we can add additional input
1807660	1813100	to the classification header and we added additional dense layers in the classification
1813100	1822160	to combine the input from transformer block and another header-related input.
1822680	1828880	With the additional input, we improved our model's performance further.
1834010	1838750	Let me talk about the details of our trans adapters.
1840350	1848310	We inserted two adapters here and each adapter has quite a simple architecture.
1849050	1857490	Each adapter will have two dense layers and there is one non-linear activation unit in between
1857490	1861710	and we have a skip connection.
1862930	1871390	The dimensionality of a dense unit is the same as the output of transformer blocks.
1871690	1880450	And the two dense layers are initialized with near-zero values. So as initially, the adapters
1880450	1892450	will act as identity blocks, but however, they will gradually change the data from the lower
1892450	1901130	transformer block to upper transformer block and to minimize classification loss.
1905580	1913380	We also modified the standard fine-tuning method by using a partial fine-tuning method.
1914800	1922740	Standard fine-tuning involves updating all parameters jointly. However, for partial
1922740	1931700	fine-tuning, we only update upper blocks, but we fix low blocks. For example, here,
1931700	1941100	the low blocks embedding the transformer 1 and 3 are fixed, but we update adapter 1 and 2
1941100	1951240	and transformer 5 and classification header. This approach was to minimize forgetting
1951240	1959400	problems of learned presentations from low transformer blocks.
1962230	1970430	As mentioned earlier, we have two set of features. One from text data and another one from
1971290	1980410	email headers. We can use multiple email header builders, for example, from two cc-reply
1981370	1991670	builders to extract additional context information. And we consider the subject and the text as
1992170	2005030	context input. The first set of features are from email text content features.
2005270	2015050	We extract text data from subject and plain text body. If only html content is available,
2015050	2023650	then we can also extract plain text from html data using a html parser. For example,
2023650	2037130	this one is a simple html hyperlink, but we only extract visible text visit bank site as output.
2037910	2044150	And then from the extracted text, we remove less informative characters.
2044870	2050770	We remove digit and punctuation characters. The second example, there are many
2050770	2061030	hyphens, but the hyphens will be removed in this step. Otherwise, each hyphen will be
2061810	2073230	individual token. Finally, we select 120 tokens as input to the transformer blocks.
2077290	2084870	We use a sub-word tokenized called WordPiece. The WordPiece tokenized can overcome
2084870	2093950	some of the limitations from character-level or word-level tokens. Character-level tokens are
2093950	2100190	too fine-grained, so it is hard to recognize word boundaries and meaning of words.
2100190	2106370	And word-level tokens often have auto-vocabulary problems.
2107320	2116090	The sub-word token tokenized can split complex or uncommon words into sub-tokens.
2116470	2127090	For example, CAPBOT can be divided into simple CAP and BOT, and double HES is an indicator
2127090	2139330	for sub-words. Similarly, SOPHOS can be divided into three tokens, SO, PH, and OS.
2140630	2147470	The sub-word tokenized reduce number of unknown tokens in our email data.
2151830	2159150	With the selected tokens, the tokens will be the input to the embedding layer,
2159150	2170450	and then we have three transformer blocks. And each transformer block has 12 multi-head
2170450	2179770	attention layers. And each individual attention layer runs contextual relationship between tokens.
2180890	2184270	And the lighter diagram shows attention rate between
2184830	2194230	tokens. The transfer token has multiple attention rate for other tokens.
2197240	2207480	In our email data, we have many non-English emails, and also non-English emails can include
2207480	2216340	English words. Also, English emails can include non-English words.
2217360	2226040	The non-English emails account for 25% of our total P9 and malicious emails.
2226040	2238240	So, we needed to support a multilingual model which can recognize different languages.
2241170	2249030	How we can support multilingual emails? The solution is BOT comes with two versions,
2249030	2256490	English and multilingual BOT. The English version was pre-trained with the English text
2256490	2264350	datasets, including Wikipedia, and has 30,000 English tokens.
2266210	2274370	The multilingual version was pre-trained with large text datasets from more than 100 languages.
2274970	2280670	And this version has four times large vocabulary size, which is 120 tokens,
2280670	2285790	which will cover many Unicode characters and Unicode words.
2286490	2294890	So, we fine-tuned a multilingual BOT for our multilingual emails.
2298270	2306190	The second set of results from email headers. We can extract multiple indicators from
2307170	2314970	email header builders. For example, the first one, we check whether the emails are from
2314970	2325090	internal or external. We can compare the domain of recipient and sender. For example, here,
2325580	2332750	the domain name is a similar looking one, but actually there are extra ads.
2332750	2342330	So, this one we consider as external email. And then, we can also use external reply
2342990	2352230	by comparing the domain of from and reply to. Often, targeted phishing attackers use
2353050	2360090	a different domain for reply to. And also, we collect the size of the recipients and the size
2360090	2367710	of the carbon copy recipients as additional indicators. It is obvious that targeted
2367710	2375210	phishing attacks will have only single recipient. However, many user
2377110	2385090	phishing emails will have multiple recipients or carbon copy recipients.
2390280	2395100	Next, we will have a look at the performance of our CatBot.
2400640	2404660	Let's have a look at the performance of a CatBot.
2406900	2412500	We used a data set of 10 million benign samples.
2412720	2422180	And the data set also includes 350 phishing emails and 1000 BC emails.
2423740	2433220	We used time split to allocate 70% of samples as training and remaining 30% for test samples.
2434010	2444140	We included two baseline models to compare CatBot. The first one is DistilBot, which has
2444140	2453280	six transformer blocks. And another one is LSTM Long Short Memory. The model is a recurrent neural
2453280	2465140	network architecture, which also uses BERT's same embedding layer. We trained three models on a
2465140	2472920	GPU instance from AWS. And we assigned high sample weight for the
2472920	2479380	PEC samples and oversample minor class malicious samples to allocate balanced
2480120	2489240	samples in each mini-matching. To compare performance, we use R-Curves and L-Re-Under-Curve.
2489240	2496240	Also, we compare inference speed and the model size as key performance metrics.
2500280	2511780	These R-Curves compare our CatBot model with two baseline models. The top blue one is CatBot,
2511780	2518140	and the second one is DistilBot, and the bottom green one is LSTM.
2518380	2523140	Our model outperformed the two baseline models.
2524340	2534600	And our model achieved 0.82% positive rate at 0.1% false positive rate.
2537410	2547610	Next, we compare the performance when we removed adapters and context input rated layers.
2548110	2557710	The top one is CatBot, and the second orange one is when we removed adapters. And the bottom one
2558060	2564550	is when context input was removed from the CatBot.
2565010	2573530	We can see significant performance drop when we remove either adapter or context input,
2573530	2583990	which demonstrates we can improve performance by using additional adapters and context rated layers.
2587760	2598010	Next, we compare the performance of three models with targeted BC samples.
2598010	2605450	We assigned high sample rate for BC samples, and we achieved high performance for detecting
2605450	2613050	those BC samples and the CatBot above the two baseline models.
2615460	2624420	Next, we compare performance for phishing emails. We divide the phishing emails into two groups,
2624420	2634760	English and non-English emails. Our CatBot outperformed for English and non-English emails.
2636000	2643720	And our model was based on the multilingual bot, so we can see significant performance
2643720	2654970	when we use the model for detecting non-English emails, when it compares with simple LST model.
2656570	2664330	Next, we compare the inference speed. This bot has six transformer blocks, and the CatBot has
2664330	2673210	three transformer blocks, so we achieve two times speed up in inference time when we measure the
2673210	2686090	performance on a CPU machine. As the number of blocks decrease, the inference time can be reduced.
2689560	2691900	Next, model size.
2693820	2697000	For comparison, we divide the model size into two
2698400	2703180	parts, embedding and transformer blocks.
2706190	2716990	The DistilBot has six transformers and has 92 million parameters for embedding and 42 million
2716990	2725010	parameters for transformer blocks. DistilBot and the CatBot have six transformer blocks.
2725010	2731550	We reuse the same embedding parameters, but we reduce the number of parameters for
2731550	2740070	transformer blocks by 50%. In total, our model size is 85% of baseline model.
2740650	2744390	When we apply the same mechanism for English version,
2745070	2753770	the English CatBot will have 71% of parameters from baseline model.
2758490	2765490	Next, we will inspect how CatBot generates outputs.
2768240	2779280	We use a LIME method to interpret our predictions. LIME is a
2779280	2786860	local interpretable model ergodastic explanation method which can be applied to any black box
2786860	2795140	models. We can understand a model by perturbing the input and understanding how the predictions
2795140	2810250	change. The first LIME example is for a benign email. The prediction score for this email
2811010	2817530	is close to zero. And we highlight legitimate tokens with
2819370	2826190	blue color and malicious ones as orange one. And this one, we don't have any high-rated
2826190	2837500	tokens for malicious ones. Next, we have a BC sample. The model prediction score for
2837500	2848860	maliciousness is close to one. And the model recognizes transfer and urgent payment as
2848860	2862170	high-rated tokens. Next, we have another BC sample, which is related with gift card.
2862170	2868970	And the model prediction score is close to one and card and urgently.
2870110	2883710	The tokens are high-rated for this email. Next, we have two handcrafted social engineering emails.
2883710	2892570	They look quite different, but if you read the text carefully, they are asking the same wire transfer.
2893450	2902850	And the model predictions are close to one for both emails and the highlight tokens are payment
2904050	2916470	and swift or as soon as possible. These examples demonstrated our model's ability to
2917190	2925390	understand complex texts and conceptually similar emails can be identified.
2928830	2937150	In conclusion, our CatBot is a carefully re-architectural transform-based model.
2937150	2943950	With this architecture, we achieve both high-speed and high-accuracy in detecting
2945010	2951290	handicrafted social engineering email attacks. In the future, we want to apply the same
2951290	2961850	design decisions to new GPT-3 model. Thank you. Do you have any questions?
2965180	2969600	If you do have any questions, please head over to the Discord channel.
2970640	2974980	They are currently there answering questions in the DEF CON Discord channel
2974980	2980640	under aiv-talks-text.