WEBVTT 00:05.790 --> 00:15.510 Hi, and today with Young-Hoo Lee, I'll be presenting Detecting Handcrafted Social Engineering Emails with a Bleeding Edge Neural Language Model. 00:19.940 --> 00:22.700 A bit about us before getting into the talk. 00:22.700 --> 00:27.360 Sophos is a security products and services company. 00:27.360 --> 00:35.020 We have a whole range of security products ranging from firewalls, to mobile security, to email security, to endpoint security. 00:35.020 --> 00:41.360 And the team that I manage, Sophos AI, is the AI shop inside of Sophos. 00:41.360 --> 00:45.720 We research and develop the company's machine learning technology. 00:45.760 --> 00:47.000 And then we're also responsible... 00:47.000 --> 00:48.040 Okay, I'm gonna mute myself. 00:48.040 --> 00:51.300 I'm gonna like definitely mute myself in here so I can listen to the talk. 00:52.000 --> 00:54.000 I'm happy if you need something. 00:54.000 --> 00:57.400 Young-Hoo Lee is one of the star researchers on my team. 00:57.400 --> 01:12.180 He's almost completely solely responsible for the machine learning subsystem that protects our Android customers, at least responsible for the R&D part of that work. 01:12.180 --> 01:14.780 He's also the principal author of the work presented today. 01:14.780 --> 01:16.780 I'm presenting in a supporting role here. 01:16.780 --> 01:21.320 I'll mostly just be setting the stage for the original work that Young-Hoo is gonna describe. 01:24.380 --> 01:26.540 So, the problem we're solving. 01:26.660 --> 01:31.940 Most generally, we're working on the problem of detecting phishing emails. 01:31.940 --> 01:40.340 More specifically, we're working on detecting business email compromise phishing emails and targeted phishing emails. 01:40.600 --> 01:43.620 So, to understand what this is, it's useful to say what it's not. 01:43.620 --> 01:57.280 We're not focused on detecting mass campaigns where detection reduces to a near-duplicate detection problem because attackers are just sending out millions of copies of basically the same phishing email. 01:57.480 --> 02:01.940 Detecting those types of emails turns out to be a side effect of our focus. 02:01.940 --> 02:11.100 But our real focus is on detecting new, custom-authored, bespoke phishing emails that are based on research on a target. 02:11.280 --> 02:16.220 And this diagram does a good job of getting across the kind of workflow we're looking at stopping. 02:16.380 --> 02:25.400 In this workflow, cyber criminals and attackers identify targets through open-source research, usually on the web. 02:25.560 --> 02:29.620 Then they establish contact with those targets. 02:29.980 --> 02:32.500 In step two, we call this grooming. 02:32.940 --> 02:45.620 In this case, they put out a lure, an initial email, usually or sometimes an initial text message, and then build trust and authenticity around some identity that they're impersonating with the mark. 02:45.800 --> 02:54.100 In step three, they cash out the trust that they've built with their targets and make an ask of those targets. 02:54.100 --> 02:58.360 Oftentimes, that will be around wiring money or sending credentials. 02:58.360 --> 03:04.050 And then in step four, they actually receive money or receive the credentials. 03:07.020 --> 03:17.160 So within targeted phishing, business email compromise, which is focused on stealing money from businesses and other organizations, has been a growing trend. 03:17.160 --> 03:25.520 So as you can see here, in July 2016, a few billion dollars were stolen, according to the FBI, through business email compromise attacks. 03:25.520 --> 03:28.880 These are targeted phishing attacks that extort money from organizations. 03:29.300 --> 03:33.120 In May 2017, that number had grown to almost six billion. 03:33.260 --> 03:37.180 And in 2018, that number had grown to more than 12 billion. 03:37.300 --> 03:43.880 I don't have data for the last two years, but I see no reason to believe that this trend has attenuated. 03:43.880 --> 03:46.400 I think it's likely it's continued on a similar trajectory. 03:46.700 --> 03:48.120 This is a big problem. 03:48.240 --> 03:52.780 We see this in Sophos's customer base, which is which is a pretty large sample. 03:52.960 --> 03:56.920 We also hear about it from other folks in the cybersecurity space. 03:57.020 --> 03:59.700 And so we're very focused on it because it's affecting people. 03:59.740 --> 04:01.820 And it's not just affecting large organizations. 04:01.820 --> 04:12.580 You can see on this on the axis on the right, that something like 80,000 organizations had been hit by July 2018 by these attacks. 04:13.900 --> 04:16.980 So there are lots of small and mid-sized organizations getting hit. 04:16.980 --> 04:25.060 And oftentimes, financial damage can be in the hundreds of thousands of dollars and really impact people's lives when these attacks happened. 04:25.060 --> 04:38.920 So again, just to reiterate, we're focused on primarily these business email compromise use cases, but also more generally targeted phishing in which a lot of manual labor goes into the phishing process on the criminal actors. 04:41.520 --> 04:50.160 And I think it almost goes without saying, but we're focused on step two and three of these criminal actors workflow, the steps that are mediated over email. 04:50.160 --> 04:55.720 So we see lots of malicious emails exchanged in the grooming step. 04:55.720 --> 05:04.820 And then we also see, obviously, a malicious email transmitted in the exchange of information step where the attacker makes the ask of the target, usually employee. 05:07.520 --> 05:10.920 And this is just because we're focused on email as our signal. 05:11.860 --> 05:14.500 And that's the scope of the work that I'll be talking about today. 05:16.180 --> 05:27.840 Just to flesh this out a little bit, here's an example, phishing email sent in the later epics of the grooming stage of an attacker's workflow. 05:27.840 --> 05:35.880 Here, the attacker has established themselves as an impersonated chancellor of UC Berkeley. 05:36.000 --> 05:45.720 They are emailing an employee at UC Berkeley, asking if they're available, looking to exchange messages with them, probably about to make an ask around a money transfer. 05:48.810 --> 05:53.510 Now, I think it's important to highlight why phishing detection is hard. 05:53.570 --> 06:05.730 And why in particular, detecting new previously unseen phishing emails written as part of a manual phishing campaign, like what I've just described, is hard. 06:06.050 --> 06:13.090 And what this boils down to, I think, is that classical natural language processing problems are hard. 06:13.090 --> 06:20.590 It's hard to get computers to understand language in any meaningful way and reason about language in any meaningful way. 06:20.590 --> 06:29.970 And detecting phishing emails really boils down to building models that have some level of understanding of language. 06:30.610 --> 06:38.110 So to understand why algorithmically it's hard to make sense of language, let's look at a few classical natural language processing problems. 06:38.110 --> 06:40.750 So one of those is coreference resolution. 06:41.210 --> 06:47.370 To get a sense of what this problem means, consider the following sentence. 06:47.410 --> 06:51.510 I went to the store for some milk, and based on the price, decided to buy it. 06:52.470 --> 06:57.710 Now, from a grammatical perspective, it could refer to the store or to the milk, right? 06:57.710 --> 07:02.290 It's possible that I went to the store for some milk, and based on the price of the store, I decided to buy it. 07:02.410 --> 07:05.290 But it's more likely that I bought the milk. 07:05.490 --> 07:19.190 As humans, in solving this coreference resolution problem, so resolving what it refers to, we have to plumb the depths of a number of complex mental models, right? 07:19.190 --> 07:32.090 We need to deploy our syntactical model of the English language, our semantic model of the English language, and our model of the world in which a person is more likely to have bought milk at the store than have bought the store itself. 07:32.090 --> 07:33.630 And that's how we solve this problem. 07:34.350 --> 07:38.370 Hopefully, it's clear that it's hard to get algorithms to do that. 07:38.510 --> 07:41.870 But to detect phishing emails, we really need to understand language. 07:41.870 --> 07:46.670 And this is a problem that's constituent of the problem of understanding language. 07:47.230 --> 07:51.530 Word polysemy is also a classic problem in algorithmic understanding of language. 07:51.610 --> 07:54.550 So a sentence like, he drank a lot and was quite the rake. 07:57.270 --> 08:06.390 Grammatically, it's valid to interpret this sentence as meaning that he drank a lot and was quite the garden tool used to rake leaves off your lawn. 08:07.310 --> 08:10.930 That's clearly not the right, that's not the sense in which the word rake is being used. 08:10.930 --> 08:18.350 The word rake is being used in the sense of a drunk, semi-criminal, sort of dissolute individual here. 08:18.570 --> 08:27.870 But it takes a pretty deep exercise of a human being's mental models to arrive at the sense in which this word was used. 08:27.870 --> 08:35.330 And not easy to reproduce in the form of an automated agent, either machine learning or based on regexes and rules. 08:35.990 --> 08:39.290 Sentiment detection is another hard problem in natural language processing. 08:39.950 --> 08:42.010 So consider the sentence, I'm not angry at all. 08:42.010 --> 08:42.770 No, of course. 08:42.770 --> 08:45.590 Why would I be angry that you spent our life savings on your mistress? 08:46.590 --> 08:48.810 So clearly, the speaker is angry here. 08:48.810 --> 08:50.370 And they're being sarcastic. 08:50.370 --> 09:00.930 Detecting that they're angry and sarcastic is not trivial and requires that we understand not only the syntactic structure of the sentence, but also the semantics of the sentence. 09:00.930 --> 09:10.410 And requires that we have the reflex that this person is probably angry if their interlocutor spent their life savings on their mistress. 09:11.290 --> 09:17.550 Okay, so to solve phishing means that we need algorithms that can make sense of language. 09:17.550 --> 09:20.670 Making sense of language is hard, as these three problems demonstrate. 09:22.190 --> 09:32.570 So a good solution to the phishing problem would model as intermediate steps to detecting that an email is a phishing email. 09:32.570 --> 09:38.950 At some level, be able to solve these problems and sit somewhere in the depths of this intermediate representation of the language that it's looking at. 09:39.170 --> 09:47.090 The other challenge we have, obviously, in any cybersecurity problem, at least any detection context in cybersecurity, is that we have adversaries who'd like to bypass our detection. 09:47.090 --> 09:48.430 And that's also worth considering. 09:48.430 --> 09:57.750 So these are all reasons why the problem that we're presenting here is a hard one, deserves... we haven't solved it completely, deserves attention from our community. 10:00.450 --> 10:15.070 Now, the approach that we're using to attack the phishing problem is based in neural networks and deep learning, and a specific advance that happened in the last few years known as transformers. 10:15.510 --> 10:32.870 So transformers, or more specifically, transformer blocks, are a new kind of construct in neural networks, much like convolutions were new, I think, as of the 90s or late 80s, and backpropagation was a new idea in neural networks, I think, starting in the 80s. 10:32.870 --> 10:47.230 Transformers are a new idea that's come out in recent years, and they help model language with a depth and fidelity that seems to be genuinely new and represent a step function in our ability to model language. 10:47.230 --> 10:48.390 So they're very exciting. 10:48.390 --> 10:55.750 The big idea behind the work that we're presenting today is that we're taking transformers and applying them to a cybersecurity problem, which we haven't seen much of before. 10:55.810 --> 10:58.410 So I'm going to talk a bit about what transformers are. 10:59.070 --> 11:03.730 A detailed discussion of how they work is beyond the scope of this talk, but I'm going to give some intuition. 11:04.150 --> 11:13.550 And then I'll pass the mic metaphorically over to Young, who will present on how we're using transformers. 11:16.900 --> 11:31.480 Okay, so here's an example, which I think helps to illustrate some ways in which transformers represent a real breakthrough in machine comprehension of language. 11:31.780 --> 11:40.540 So in this example, and I expect some of you have seen this because this went viral last year, a researcher wrote a prompt. 11:40.540 --> 11:46.180 This prompt is given at the top here in a shocking finding, that paragraph. 11:46.720 --> 11:55.000 And then a transformer model just took it from there and wrote a story based on that prompt. 11:55.780 --> 12:09.280 And I think when you see the story that the machine wrote, you'll see that it betrays a understanding of the syntax and semantics of language that's quite striking. 12:09.580 --> 12:11.260 So I think it's worth reading this out loud. 12:11.360 --> 12:20.460 The human-written prompt reads, in a shocking finding, scientists discovered a herd of unicorns living in a remote, previously unexplored valley in the Andes Mountains. 12:20.780 --> 12:25.340 Even more surprising to the researchers was the fact that the unicorn spoke perfect English. 12:25.340 --> 12:30.160 And now again, the challenge to the left off and write something coherent. 12:30.160 --> 12:37.540 And if it can, that reflects something around its ability to solve problems like the fundamental problems I described earlier. 12:37.540 --> 12:41.400 So the neural network continues. 12:41.400 --> 12:45.980 The scientists named the population after their distinctive horn, Ovid's unicorn. 12:45.980 --> 12:50.180 These four-horned, silver-white unicorns were previously unknown to science. 12:50.180 --> 12:55.380 Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved. 12:55.380 --> 12:55.740 Dr. 12:55.740 --> 13:05.460 Jorge Perez, an evolutionary biologist from the University of La Paz and several companions, were exploring the Andes Mountains when they found a small valley with no other animals or humans. 13:05.860 --> 13:11.820 Perez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow. 13:12.300 --> 13:14.940 Perez and the others then ventured further into the valley. 13:14.940 --> 13:19.700 By the time we reached the top of one peak, the water looked blue with some crystals on top, said Perez. 13:19.700 --> 13:21.920 And the essay continues. 13:22.820 --> 13:38.960 But what interested Younghoo and I in seeing examples of what transformers can do like this is that it seems that the transformers have, examples like these suggest that transformers have understood the syntax and semantics of the English language to some degree, 13:38.960 --> 14:03.100 and which begs the question, could the representations that the transformers learn in their parameter structure be useful in detecting phishing emails, since these transformer models seem to have distilled knowledge of how language works and what it means within their parameter settings. 14:03.720 --> 14:11.580 Okay, so this is one example of, sort of impressive example, of what transformers can do, that beg the question of whether or not they could be applied usefully to phishing. 14:11.580 --> 14:15.380 And we'll show results from our experiments later in Younghoo's section. 14:15.380 --> 14:28.460 Here's another example of just how sort of marvelous transformer models are with respect to how much more sophisticated they seem to be. 14:28.460 --> 14:36.660 So here a researcher typed into this box a description of some HTML that he would like the transformer models to generate. 14:36.660 --> 14:40.140 He writes, a button with the color of Donald Trump's hair. 14:40.640 --> 14:46.120 And the model actually writes, as a text completion, some HTML and CSS. 14:46.220 --> 14:53.980 It seems to understand that Donald Trump has yellow hair, and it makes a button, and it writes valid HTML and CSS. 14:54.780 --> 15:04.260 Which is, I think, just impressive in the absolute sense, but also for folks who've been in the NLP space for a while, it's impressive relatively. 15:04.260 --> 15:12.620 Five, ten years ago, if you would have shown somebody that we could do this in 2020, I think they would have been a bit incredulous. 15:12.620 --> 15:13.820 This is a big step for you. 15:13.960 --> 15:18.220 Applications like these represent a big step forward in natural language processing. 15:18.220 --> 15:22.980 And they're due, again, to this new idea called a transformer block. 15:23.200 --> 15:27.880 That's the key building block out of which models like this model, which is called GPT-3. 15:27.880 --> 15:34.360 It's from OpenAI, which is an AI research lab, are based. 15:35.460 --> 15:36.360 Okay. 15:36.840 --> 15:39.340 So, moving on. 15:39.900 --> 15:49.460 So, I want to go back a little bit in the history of NLP as a way of describing how transformers are new and different. 15:49.920 --> 15:59.760 So, I assume a substantial chunk of our audience today has studied basic machine learning on text. 16:00.580 --> 16:10.200 And typically, in like a machine learning 101 course, you learn about the bag of words model and about discrete states, Markov models of language. 16:10.860 --> 16:20.500 And I want to talk about these representations as a way of talking about how limited those representations were, and then I'll talk about how transformers break us free of some of those limitations. 16:20.780 --> 16:32.660 So, a bag of words model of a document is a way of representing a document numerically for the purposes of machine learning, in which we just count up how many times each word in that document appeared. 16:33.040 --> 16:39.820 And then we create a matrix out of all the documents we have, and all the words in all those documents. 16:39.920 --> 16:43.000 And the entries in those matrix are just word counts. 16:43.000 --> 17:01.560 So, in the case on the left over here, the column vectors in our matrix are documents, and they get one dimension per word in our vocabulary, and they get counts of how many times words appeared in that vocabulary in the entries of that particular vector. 17:02.840 --> 17:13.200 And hopefully it's intuitive that, you know, once you've represented your documents in a vocabulary space in this way, you can compare documents by taking some distance measure between pairs of documents. 17:13.200 --> 17:20.960 You can also train machine learning models on your document corpus to say classified news articles is about sports or politics. 17:21.800 --> 17:30.440 But what you've done in the first step of these models is drop out sequence information. 17:30.440 --> 17:39.600 So, you've forgotten about which words come in which order in the document, and you've just represented your document as a bag of words. 17:39.640 --> 17:47.420 Which is a useful simplifying assumption, and it's one that we still use today in some of the modeling we do in my research group at Sophos. 17:47.720 --> 17:52.820 But it throws out a ton of information that transformers and more modern models don't throw out. 17:53.740 --> 17:58.460 In the model on the right, we have a discrete state Markov model of language. 17:58.740 --> 18:19.840 Here each word is a state, and the concept of language given by the model is kind of a cheesier adventure story in which the next words in an utterance depends only on the current words, and you're just sort of drawing from a probability distribution and moving through this graph to generate language. 18:19.920 --> 18:34.840 You could never have generated anything close to that unicorn story using a Markov model, and yet as recently as the last five, ten years, there's lots of papers coming out around like using hidden Markov models to parse sentences and that kind of thing. 18:35.160 --> 18:47.540 These are still useful models, but we've gone far beyond the simplifying assumptions in these original sort of simplified models of the world in NLP from the past few decades. 18:49.500 --> 19:01.060 So let's contrast transformers now, and I'll get into more details about how transformers work in a second, but let's contrast transformers with these earlier natural language processing approaches. 19:01.100 --> 19:09.040 So pre-transformer, most machine learning approaches didn't consider words in context. 19:09.260 --> 19:29.980 Many approaches made the simplifying bag of words assumptions as kind of a first step in the modeling process, and then ran term vectors through models like topic models, or logistic regression, or support vector machines. 19:30.620 --> 19:47.880 Most models didn't model kind of like co-reference relationships between words, or sort of which words pertain to which other words in a sentence, and I'll talk about what that means later, but transformers do sort of solve that problem. 19:49.060 --> 19:56.420 Most approaches didn't operate on either words or characters. 19:57.120 --> 20:10.960 Typically the way people use transformers, we use well-chosen chunks of words, which allows us to model misspelling and this kind of thing, so there's been an improvement there in the current generation of natural language processing models. 20:12.060 --> 20:15.580 And older approaches tended not to use neural network technology. 20:15.580 --> 20:31.280 That's changed a lot in the best ideas that have come out of the neural network revolution that's been ongoing since around 2012. 20:31.280 --> 20:38.980 So transformers kind of kick apart a number of log jams in natural language modeling. 20:38.980 --> 20:57.380 They give words contextual representations, they model attention, like the relationship between words in a document, they use these smart partial word representations that allow for misspellings and just the tower of babble of vernaculars that appear like under the banner of, 20:57.380 --> 21:10.140 say, the English language on the internet, and they take advantage of ideas like residual connections and modern optimizers and many of the really good ideas that have come out of the neural network revolution. 21:10.600 --> 21:15.640 So these are all reasons why we wanted to test their applicability to phishing detection. 21:18.440 --> 21:26.380 Okay, so if you want to get into detail about how transformers work, I'd recommend this blog post by Jay Alomar. 21:26.380 --> 21:29.480 It's where I cribbed this screenshot here. 21:29.480 --> 21:41.660 I'm not going to get into the details of the series of matrix multiplications that comprise summations and various linear algebra and operations that comprise the transformer block. 21:41.660 --> 21:46.980 I just want to give a little bit of intuition here before passing the mic over to Younghoo. 21:47.240 --> 21:59.540 The basic idea behind a transformer block, which is kind of a Lego block, out of which you build a transformer-based neural network, is we're taking in a sequence of words. 22:00.140 --> 22:03.840 This diagram shows a very simple example where we're just taking a sequence of two words. 22:03.840 --> 22:07.440 Typically we've taken a larger window, like 512 words. 22:07.580 --> 22:20.760 We pass them into the block, and the way they get passed in is not just as entries in a term vector matrix, but actually as vectors themselves. 22:20.760 --> 22:22.140 The words get a vector representation. 22:22.140 --> 22:23.660 These are known as embeddings. 22:25.920 --> 22:29.100 We pass them both into the transformer network. 22:30.000 --> 22:42.920 The first thing that the transformer network does to our input word sequence is model the attention relationships between the words. 22:43.320 --> 23:02.500 In a two-word case, it's a little bit harder to describe here, but basically if you had a sequence of 15 words, for every word, the attention mechanism would compute how much that word pertains to the other 14 words in the sentence. 23:02.960 --> 23:08.280 You'll see how that has a relationship with the coreference resolution problem I talked about earlier. 23:08.280 --> 23:24.060 But it's just intuitive that there's a graph of word relationships in a sentence, and self-attention kind of models that in terms of which subjects pertain to which objects, which pronouns pertain to which people, etc. 23:25.700 --> 23:37.580 There's an addition and normalization step that happens when we've run this self-attention process a number of times and typically we don't just do self-attention once. 23:37.580 --> 23:41.660 We have a number of what are called heads and we run attention a number of times. 23:41.700 --> 23:57.780 We combine all that together, we do some non-linear transformation on it, and then we wind up with a new embedding of your original sequence of the same dimensionality as this original embedding, except that now thinking is encoded in this new representation in the context in which it appears, 23:57.780 --> 23:59.040 in the context of machines. 23:59.040 --> 24:01.420 Machines is encoded in the context of thinking. 24:01.620 --> 24:04.400 Then typically we stack these transformer blocks that we actually do. 24:04.400 --> 24:08.920 We have another transformer block that then sort of refines the representation and we keep going. 24:09.400 --> 24:14.200 Young will show that we use a number of transformer blocks in our phishing detection work in a few minutes. 24:15.720 --> 24:22.580 Here's some intuition as to what comes out of the attention mechanism in a typical transformer. 24:22.580 --> 24:30.020 So here's what a transformer block has decided it sort of relates to in an input sentence. 24:30.020 --> 24:34.880 So here we have the input sentence, the animal didn't cross the street because it was too tired. 24:36.440 --> 24:49.140 And the strongest attentional relationship here goes to the animal, which is interesting because one could ask whether or not it refers to the street or the animal here. 24:49.780 --> 25:00.760 One can interpret the weight of the connection to the animal, meaning that the transformer block has decided, in their quotes, that it pertains to the animal, which is really interesting. 25:01.660 --> 25:13.520 So hopefully you get some intuition as to how powerful this attentional representation is and how important it is in machine learning models, getting some what we might call understanding over the language that they're analyzing. 25:16.310 --> 25:23.410 Okay, so I want to put the intuition together around how transformers pertain to the work that YoungHu and I are presenting today. 25:24.230 --> 25:36.650 So basically, what we're going to do in our phishing model is embed an email as a sequence of embedded character sequence vectors. 25:36.650 --> 25:57.650 And then we're going to run that through a series of transformer blocks, like what I just showed, that are going to create a very refined attentional representation of the word sequences, and then produce these contextual embeddings that get at the meaning of the words in the context in which they appear. 25:57.870 --> 26:05.790 And then finally, our network is going to solve a classification task, say whether or not the email is a phishing email or not. 26:06.310 --> 26:11.310 How all that magic works, how the network gets trained, which there's some tricks there that are really cool. 26:11.770 --> 26:20.550 I'll leave to YoungHu, and hope a lot of this makes sense, and happy to take questions about my piece of this presentation later at the end of the talk. 26:21.970 --> 26:24.150 Thank you, Joshi. 26:24.150 --> 26:28.290 Let me continue the second part of our talk. 26:28.650 --> 26:39.250 The second part will include our design decisions for CatBot and performance result. 26:39.830 --> 26:46.310 CatBot is the name of our email model, context-aware tiny bot. 26:46.310 --> 26:50.950 The model size is tiny, but it is mighty bot. 26:54.040 --> 27:00.280 Modern NLP models all have a nice and friendly name. 27:00.280 --> 27:06.360 For example, Elmer was introduced in 2018. 27:06.860 --> 27:16.440 The model used bidirectional LSTMs to generate contextualized word embeddings. 27:17.650 --> 27:32.800 And then, same year later, Google researchers introduced bot and achieved the state-of-art performance in many English-understanding problems. 27:32.800 --> 27:50.340 Next year, 2019, Baidu researchers introduced the Orni, and the model achieved another state-of-art performance in many Chinese-language-understanding problems. 27:51.140 --> 27:56.220 They are all popular characters from Sesame Street. 27:57.180 --> 28:06.880 This year, 2020, we introduced CatBot to tackle email security problems. 28:12.010 --> 28:20.270 Transformer-based NLP models are powerful, but they are complex and heavy. 28:20.790 --> 28:29.050 And it is challenging to deploy heavy models for real-time applications. 28:29.670 --> 28:45.810 So our first design goal is to convert the heavy model into a lightweight model, so we can reduce the number of parameters and then we can improve inference speed. 28:46.450 --> 28:53.890 We downsized a baseline model called DistribBot, which has six transformer blocks. 28:54.590 --> 29:04.290 We take half of transformer blocks from pre-trained model and then replace missing transformers with simple adapters. 29:04.610 --> 29:14.050 For example here, we take transformer 1, 3, 5, and then we added two adapters. 29:14.510 --> 29:18.510 Also, we can take other number of transformer blocks. 29:19.320 --> 29:24.840 This approach allows simply significantly reduced number of parameters. 29:29.940 --> 29:39.020 The second goal is to improve the model performance by combining additional input. 29:40.420 --> 29:46.660 Standard NLP models only accept text data as input. 29:46.660 --> 29:58.420 However, we can extract additional features from email headers and we can use the additional input to our model. 29:58.900 --> 30:22.160 So the text input will be the input to the embedding and then we can add additional input to the classification header and we added additional dense layers in the classification to combine the input from transformer block and another header-related input. 30:22.680 --> 30:28.880 With the additional input, we improved our model's performance further. 30:34.010 --> 30:38.750 Let me talk about the details of our trans adapters. 30:40.350 --> 30:48.310 We inserted two adapters here and each adapter has quite a simple architecture. 30:49.050 --> 31:01.710 Each adapter will have two dense layers and there is one non-linear activation unit in between and we have a skip connection. 31:02.930 --> 31:11.390 The dimensionality of a dense unit is the same as the output of transformer blocks. 31:11.690 --> 31:17.410 And the two dense layers are initialized with near-zero values. 31:17.410 --> 31:41.130 So as initially, the adapters will act as identity blocks, but however, they will gradually change the data from the lower transformer block to upper transformer block and to minimize classification loss. 31:45.580 --> 31:53.380 We also modified the standard fine-tuning method by using a partial fine-tuning method. 31:54.800 --> 32:00.940 Standard fine-tuning involves updating all parameters jointly. 32:01.140 --> 32:09.700 However, for partial fine-tuning, we only update upper blocks, but we fix low blocks. 32:09.700 --> 32:25.480 For example, here, the low blocks embedding the transformer 1 and 3 are fixed, but we update adapter 1 and 2 and transformer 5 and classification header. 32:28.240 --> 32:39.400 This approach was to minimize forgetting problems of learned presentations from low transformer blocks. 32:42.230 --> 32:46.720 As mentioned earlier, we have two set of features. 32:46.720 --> 32:52.930 One from text data and another one from email headers. 32:52.930 --> 33:07.270 We can use multiple email header builders, for example, from two cc-reply builders to extract additional context information. 33:07.270 --> 33:14.170 And we consider the subject and the text as context input. 33:17.450 --> 33:25.030 The first set of features are from email text content features. 33:25.270 --> 33:31.150 We extract text data from subject and plain text body. 33:31.750 --> 33:42.310 If only html content is available, then we can also extract plain text from html data using a html parser. 33:42.310 --> 33:57.130 For example, this one is a simple html hyperlink, but we only extract visible text visit bank site as output. 33:57.910 --> 34:04.150 And then from the extracted text, we remove less informative characters. 34:04.870 --> 34:08.890 We remove digit and punctuation characters. 34:08.890 --> 34:17.230 The second example, there are many hyphens, but the hyphens will be removed in this step. 34:17.230 --> 34:24.410 Otherwise, each hyphen will be individual token. 34:24.850 --> 34:33.230 Finally, we select 120 tokens as input to the transformer blocks. 34:37.290 --> 34:42.390 We use a sub-word tokenized called WordPiece. 34:42.390 --> 34:51.650 The WordPiece tokenized can overcome some of the limitations from character-level or word-level tokens. 34:51.910 --> 35:00.190 Character-level tokens are too fine-grained, so it is hard to recognize word boundaries and meaning of words. 35:00.190 --> 35:06.370 And word-level tokens often have auto-vocabulary problems. 35:07.320 --> 35:16.090 The sub-word token tokenized can split complex or uncommon words into sub-tokens. 35:16.470 --> 35:29.770 For example, CAPBOT can be divided into simple CAP and BOT, and double HES is an indicator for sub-words. 35:30.650 --> 35:39.330 Similarly, SOPHOS can be divided into three tokens, SO, PH, and OS. 35:40.630 --> 35:47.470 The sub-word tokenized reduce number of unknown tokens in our email data. 35:51.830 --> 36:05.270 With the selected tokens, the tokens will be the input to the embedding layer, and then we have three transformer blocks. 36:05.270 --> 36:12.050 And each transformer block has 12 multi-head attention layers. 36:12.050 --> 36:19.770 And each individual attention layer runs contextual relationship between tokens. 36:20.890 --> 36:25.850 And the lighter diagram shows attention rate between tokens. 36:25.850 --> 36:34.230 The transfer token has multiple attention rate for other tokens. 36:37.240 --> 36:52.060 In our email data, we have many non-English emails, and also non-English emails can include English words. 36:52.060 --> 36:56.340 Also, English emails can include non-English words. 36:57.360 --> 37:06.040 The non-English emails account for 25% of our total P9 and malicious emails. 37:06.040 --> 37:18.240 So, we needed to support a multilingual model which can recognize different languages. 37:21.170 --> 37:24.830 How we can support multilingual emails? 37:24.830 --> 37:31.910 The solution is BOT comes with two versions, English and multilingual BOT. 37:31.910 --> 37:44.350 The English version was pre-trained with the English text datasets, including Wikipedia, and has 30,000 English tokens. 37:46.210 --> 37:54.370 The multilingual version was pre-trained with large text datasets from more than 100 languages. 37:54.970 --> 38:05.790 And this version has four times large vocabulary size, which is 120 tokens, which will cover many Unicode characters and Unicode words. 38:06.490 --> 38:14.890 So, we fine-tuned a multilingual BOT for our multilingual emails. 38:18.270 --> 38:21.850 The second set of results from email headers. 38:21.890 --> 38:29.410 We can extract multiple indicators from email header builders. 38:29.410 --> 38:36.870 For example, the first one, we check whether the emails are from internal or external. 38:36.870 --> 38:43.070 We can compare the domain of recipient and sender. 38:43.070 --> 38:52.750 For example, here, the domain name is a similar looking one, but actually there are extra ads. 38:52.750 --> 38:57.090 So, this one we consider as external email. 38:57.650 --> 39:07.330 And then, we can also use external reply by comparing the domain of from and reply to. 39:07.570 --> 39:16.090 Often, targeted phishing attackers use a different domain for reply to. 39:16.110 --> 39:24.650 And also, we collect the size of the recipients and the size of the carbon copy recipients as additional indicators. 39:25.110 --> 39:32.710 It is obvious that targeted phishing attacks will have only single recipient. 39:32.710 --> 39:45.090 However, many user phishing emails will have multiple recipients or carbon copy recipients. 39:50.280 --> 39:55.100 Next, we will have a look at the performance of our CatBot. 40:00.640 --> 40:04.660 Let's have a look at the performance of a CatBot. 40:06.900 --> 40:12.500 We used a data set of 10 million benign samples. 40:12.720 --> 40:22.180 And the data set also includes 350 phishing emails and 1000 BC emails. 40:23.740 --> 40:33.220 We used time split to allocate 70% of samples as training and remaining 30% for test samples. 40:34.010 --> 40:40.780 We included two baseline models to compare CatBot. 40:41.020 --> 40:46.420 The first one is DistilBot, which has six transformer blocks. 40:46.760 --> 40:50.660 And another one is LSTM Long Short Memory. 40:50.660 --> 41:00.760 The model is a recurrent neural network architecture, which also uses BERT's same embedding layer. 41:00.760 --> 41:07.900 We trained three models on a GPU instance from AWS. 41:08.040 --> 41:23.160 And we assigned high sample weight for the PEC samples and oversample minor class malicious samples to allocate balanced samples in each mini-matching. 41:23.680 --> 41:29.240 To compare performance, we use R-Curves and L-Re-Under-Curve. 41:29.240 --> 41:36.240 Also, we compare inference speed and the model size as key performance metrics. 41:40.280 --> 41:48.940 These R-Curves compare our CatBot model with two baseline models. 41:48.940 --> 41:58.140 The top blue one is CatBot, and the second one is DistilBot, and the bottom green one is LSTM. 41:58.380 --> 42:03.140 Our model outperformed the two baseline models. 42:04.340 --> 42:14.600 And our model achieved 0.82% positive rate at 0.1% false positive rate. 42:17.410 --> 42:27.610 Next, we compare the performance when we removed adapters and context input rated layers. 42:28.110 --> 42:36.030 The top one is CatBot, and the second orange one is when we removed adapters. 42:36.030 --> 42:44.550 And the bottom one is when context input was removed from the CatBot. 42:45.010 --> 43:03.990 We can see significant performance drop when we remove either adapter or context input, which demonstrates we can improve performance by using additional adapters and context rated layers. 43:07.760 --> 43:18.010 Next, we compare the performance of three models with targeted BC samples. 43:18.010 --> 43:33.050 We assigned high sample rate for BC samples, and we achieved high performance for detecting those BC samples and the CatBot above the two baseline models. 43:35.460 --> 43:39.780 Next, we compare performance for phishing emails. 43:40.120 --> 43:47.320 We divide the phishing emails into two groups, English and non-English emails. 43:48.140 --> 43:54.760 Our CatBot outperformed for English and non-English emails. 43:56.000 --> 44:14.970 And our model was based on the multilingual bot, so we can see significant performance when we use the model for detecting non-English emails, when it compares with simple LST model. 44:16.570 --> 44:20.330 Next, we compare the inference speed. 44:20.550 --> 44:36.270 This bot has six transformer blocks, and the CatBot has three transformer blocks, so we achieve two times speed up in inference time when we measure the performance on a CPU machine. 44:36.730 --> 44:46.090 As the number of blocks decrease, the inference time can be reduced. 44:49.560 --> 44:51.900 Next, model size. 44:53.820 --> 45:03.180 For comparison, we divide the model size into two parts, embedding and transformer blocks. 45:06.190 --> 45:20.090 The DistilBot has six transformers and has 92 million parameters for embedding and 42 million parameters for transformer blocks. 45:20.730 --> 45:25.010 DistilBot and the CatBot have six transformer blocks. 45:25.010 --> 45:34.770 We reuse the same embedding parameters, but we reduce the number of parameters for transformer blocks by 50%. 45:34.770 --> 45:40.070 In total, our model size is 85% of baseline model. 45:40.650 --> 45:53.770 When we apply the same mechanism for English version, the English CatBot will have 71% of parameters from baseline model. 45:58.490 --> 46:05.490 Next, we will inspect how CatBot generates outputs. 46:08.240 --> 46:15.420 We use a LIME method to interpret our predictions. 46:15.420 --> 46:28.400 LIME is a local interpretable model ergodastic explanation method which can be applied to any black box models. 46:28.400 --> 46:37.100 We can understand a model by perturbing the input and understanding how the predictions change. 46:39.430 --> 46:46.210 The first LIME example is for a benign email. 46:46.890 --> 46:53.750 The prediction score for this email is close to zero. 46:53.790 --> 47:03.250 And we highlight legitimate tokens with blue color and malicious ones as orange one. 47:03.250 --> 47:09.630 And this one, we don't have any high-rated tokens for malicious ones. 47:11.820 --> 47:15.460 Next, we have a BC sample. 47:15.620 --> 47:20.420 The model prediction score for maliciousness is close to one. 47:20.420 --> 47:31.440 And the model recognizes transfer and urgent payment as high-rated tokens. 47:34.730 --> 47:42.170 Next, we have another BC sample, which is related with gift card. 47:42.170 --> 47:48.970 And the model prediction score is close to one and card and urgently. 47:50.110 --> 47:54.590 The tokens are high-rated for this email. 47:58.070 --> 48:03.710 Next, we have two handcrafted social engineering emails. 48:03.710 --> 48:12.570 They look quite different, but if you read the text carefully, they are asking the same wire transfer. 48:13.450 --> 48:30.730 And the model predictions are close to one for both emails and the highlight tokens are payment and swift or as soon as possible. 48:31.030 --> 48:45.390 These examples demonstrated our model's ability to understand complex texts and conceptually similar emails can be identified. 48:48.830 --> 48:57.150 In conclusion, our CatBot is a carefully re-architectural transform-based model. 48:57.150 --> 49:08.550 With this architecture, we achieve both high-speed and high-accuracy in detecting handicrafted social engineering email attacks. 49:08.550 --> 49:16.330 In the future, we want to apply the same design decisions to new GPT-3 model. 49:18.770 --> 49:20.190 Thank you. 49:20.190 --> 49:21.850 Do you have any questions? 49:25.180 --> 49:29.600 If you do have any questions, please head over to the Discord channel. 49:30.640 --> 49:40.640 They are currently there answering questions in the DEF CON Discord channel under aiv-talks-text.