1
00:00:05,790 --> 00:00:15,510
Hi, and today with Young-Hoo Lee, I'll be presenting Detecting Handcrafted Social Engineering Emails with a Bleeding Edge Neural Language Model.

2
00:00:19,940 --> 00:00:22,700
A bit about us before getting into the talk.

3
00:00:22,700 --> 00:00:27,360
Sophos is a security products and services company.

4
00:00:27,360 --> 00:00:35,020
We have a whole range of security products ranging from firewalls, to mobile security, to email security, to endpoint security.

5
00:00:35,020 --> 00:00:41,360
And the team that I manage, Sophos AI, is the AI shop inside of Sophos.

6
00:00:41,360 --> 00:00:45,720
We research and develop the company's machine learning technology.

7
00:00:45,760 --> 00:00:47,000
And then we're also responsible...

8
00:00:47,000 --> 00:00:48,040
Okay, I'm gonna mute myself.

9
00:00:48,040 --> 00:00:51,300
I'm gonna like definitely mute myself in here so I can listen to the talk.

10
00:00:52,000 --> 00:00:54,000
I'm happy if you need something.

11
00:00:54,000 --> 00:00:57,400
Young-Hoo Lee is one of the star researchers on my team.

12
00:00:57,400 --> 00:01:12,180
He's almost completely solely responsible for the machine learning subsystem that protects our Android customers, at least responsible for the R&D part of that work.

13
00:01:12,180 --> 00:01:14,780
He's also the principal author of the work presented today.

14
00:01:14,780 --> 00:01:16,780
I'm presenting in a supporting role here.

15
00:01:16,780 --> 00:01:21,320
I'll mostly just be setting the stage for the original work that Young-Hoo is gonna describe.

16
00:01:24,380 --> 00:01:26,540
So, the problem we're solving.

17
00:01:26,660 --> 00:01:31,940
Most generally, we're working on the problem of detecting phishing emails.

18
00:01:31,940 --> 00:01:40,340
More specifically, we're working on detecting business email compromise phishing emails and targeted phishing emails.

19
00:01:40,600 --> 00:01:43,620
So, to understand what this is, it's useful to say what it's not.

20
00:01:43,620 --> 00:01:57,280
We're not focused on detecting mass campaigns where detection reduces to a near-duplicate detection problem because attackers are just sending out millions of copies of basically the same phishing email.

21
00:01:57,480 --> 00:02:01,940
Detecting those types of emails turns out to be a side effect of our focus.

22
00:02:01,940 --> 00:02:11,100
But our real focus is on detecting new, custom-authored, bespoke phishing emails that are based on research on a target.

23
00:02:11,280 --> 00:02:16,220
And this diagram does a good job of getting across the kind of workflow we're looking at stopping.

24
00:02:16,380 --> 00:02:25,400
In this workflow, cyber criminals and attackers identify targets through open-source research, usually on the web.

25
00:02:25,560 --> 00:02:29,620
Then they establish contact with those targets.

26
00:02:29,980 --> 00:02:32,500
In step two, we call this grooming.

27
00:02:32,940 --> 00:02:45,620
In this case, they put out a lure, an initial email, usually or sometimes an initial text message, and then build trust and authenticity around some identity that they're impersonating with the mark.

28
00:02:45,800 --> 00:02:54,100
In step three, they cash out the trust that they've built with their targets and make an ask of those targets.

29
00:02:54,100 --> 00:02:58,360
Oftentimes, that will be around wiring money or sending credentials.

30
00:02:58,360 --> 00:03:04,050
And then in step four, they actually receive money or receive the credentials.

31
00:03:07,020 --> 00:03:17,160
So within targeted phishing, business email compromise, which is focused on stealing money from businesses and other organizations, has been a growing trend.

32
00:03:17,160 --> 00:03:25,520
So as you can see here, in July 2016, a few billion dollars were stolen, according to the FBI, through business email compromise attacks.

33
00:03:25,520 --> 00:03:28,880
These are targeted phishing attacks that extort money from organizations.

34
00:03:29,300 --> 00:03:33,120
In May 2017, that number had grown to almost six billion.

35
00:03:33,260 --> 00:03:37,180
And in 2018, that number had grown to more than 12 billion.

36
00:03:37,300 --> 00:03:43,880
I don't have data for the last two years, but I see no reason to believe that this trend has attenuated.

37
00:03:43,880 --> 00:03:46,400
I think it's likely it's continued on a similar trajectory.

38
00:03:46,700 --> 00:03:48,120
This is a big problem.

39
00:03:48,240 --> 00:03:52,780
We see this in Sophos's customer base, which is which is a pretty large sample.

40
00:03:52,960 --> 00:03:56,920
We also hear about it from other folks in the cybersecurity space.

41
00:03:57,020 --> 00:03:59,700
And so we're very focused on it because it's affecting people.

42
00:03:59,740 --> 00:04:01,820
And it's not just affecting large organizations.

43
00:04:01,820 --> 00:04:12,580
You can see on this on the axis on the right, that something like 80,000 organizations had been hit by July 2018 by these attacks.

44
00:04:13,900 --> 00:04:16,980
So there are lots of small and mid-sized organizations getting hit.

45
00:04:16,980 --> 00:04:25,060
And oftentimes, financial damage can be in the hundreds of thousands of dollars and really impact people's lives when these attacks happened.

46
00:04:25,060 --> 00:04:38,920
So again, just to reiterate, we're focused on primarily these business email compromise use cases, but also more generally targeted phishing in which a lot of manual labor goes into the phishing process on the criminal actors.

47
00:04:41,520 --> 00:04:50,160
And I think it almost goes without saying, but we're focused on step two and three of these criminal actors workflow, the steps that are mediated over email.

48
00:04:50,160 --> 00:04:55,720
So we see lots of malicious emails exchanged in the grooming step.

49
00:04:55,720 --> 00:05:04,820
And then we also see, obviously, a malicious email transmitted in the exchange of information step where the attacker makes the ask of the target, usually employee.

50
00:05:07,520 --> 00:05:10,920
And this is just because we're focused on email as our signal.

51
00:05:11,860 --> 00:05:14,500
And that's the scope of the work that I'll be talking about today.

52
00:05:16,180 --> 00:05:27,840
Just to flesh this out a little bit, here's an example, phishing email sent in the later epics of the grooming stage of an attacker's workflow.

53
00:05:27,840 --> 00:05:35,880
Here, the attacker has established themselves as an impersonated chancellor of UC Berkeley.

54
00:05:36,000 --> 00:05:45,720
They are emailing an employee at UC Berkeley, asking if they're available, looking to exchange messages with them, probably about to make an ask around a money transfer.

55
00:05:48,810 --> 00:05:53,510
Now, I think it's important to highlight why phishing detection is hard.

56
00:05:53,570 --> 00:06:05,730
And why in particular, detecting new previously unseen phishing emails written as part of a manual phishing campaign, like what I've just described, is hard.

57
00:06:06,050 --> 00:06:13,090
And what this boils down to, I think, is that classical natural language processing problems are hard.

58
00:06:13,090 --> 00:06:20,590
It's hard to get computers to understand language in any meaningful way and reason about language in any meaningful way.

59
00:06:20,590 --> 00:06:29,970
And detecting phishing emails really boils down to building models that have some level of understanding of language.

60
00:06:30,610 --> 00:06:38,110
So to understand why algorithmically it's hard to make sense of language, let's look at a few classical natural language processing problems.

61
00:06:38,110 --> 00:06:40,750
So one of those is coreference resolution.

62
00:06:41,210 --> 00:06:47,370
To get a sense of what this problem means, consider the following sentence.

63
00:06:47,410 --> 00:06:51,510
I went to the store for some milk, and based on the price, decided to buy it.

64
00:06:52,470 --> 00:06:57,710
Now, from a grammatical perspective, it could refer to the store or to the milk, right?

65
00:06:57,710 --> 00:07:02,290
It's possible that I went to the store for some milk, and based on the price of the store, I decided to buy it.

66
00:07:02,410 --> 00:07:05,290
But it's more likely that I bought the milk.

67
00:07:05,490 --> 00:07:19,190
As humans, in solving this coreference resolution problem, so resolving what it refers to, we have to plumb the depths of a number of complex mental models, right?

68
00:07:19,190 --> 00:07:32,090
We need to deploy our syntactical model of the English language, our semantic model of the English language, and our model of the world in which a person is more likely to have bought milk at the store than have bought the store itself.

69
00:07:32,090 --> 00:07:33,630
And that's how we solve this problem.

70
00:07:34,350 --> 00:07:38,370
Hopefully, it's clear that it's hard to get algorithms to do that.

71
00:07:38,510 --> 00:07:41,870
But to detect phishing emails, we really need to understand language.

72
00:07:41,870 --> 00:07:46,670
And this is a problem that's constituent of the problem of understanding language.

73
00:07:47,230 --> 00:07:51,530
Word polysemy is also a classic problem in algorithmic understanding of language.

74
00:07:51,610 --> 00:07:54,550
So a sentence like, he drank a lot and was quite the rake.

75
00:07:57,270 --> 00:08:06,390
Grammatically, it's valid to interpret this sentence as meaning that he drank a lot and was quite the garden tool used to rake leaves off your lawn.

76
00:08:07,310 --> 00:08:10,930
That's clearly not the right, that's not the sense in which the word rake is being used.

77
00:08:10,930 --> 00:08:18,350
The word rake is being used in the sense of a drunk, semi-criminal, sort of dissolute individual here.

78
00:08:18,570 --> 00:08:27,870
But it takes a pretty deep exercise of a human being's mental models to arrive at the sense in which this word was used.

79
00:08:27,870 --> 00:08:35,330
And not easy to reproduce in the form of an automated agent, either machine learning or based on regexes and rules.

80
00:08:35,990 --> 00:08:39,290
Sentiment detection is another hard problem in natural language processing.

81
00:08:39,950 --> 00:08:42,010
So consider the sentence, I'm not angry at all.

82
00:08:42,010 --> 00:08:42,770
No, of course.

83
00:08:42,770 --> 00:08:45,590
Why would I be angry that you spent our life savings on your mistress?

84
00:08:46,590 --> 00:08:48,810
So clearly, the speaker is angry here.

85
00:08:48,810 --> 00:08:50,370
And they're being sarcastic.

86
00:08:50,370 --> 00:09:00,930
Detecting that they're angry and sarcastic is not trivial and requires that we understand not only the syntactic structure of the sentence, but also the semantics of the sentence.

87
00:09:00,930 --> 00:09:10,410
And requires that we have the reflex that this person is probably angry if their interlocutor spent their life savings on their mistress.

88
00:09:11,290 --> 00:09:17,550
Okay, so to solve phishing means that we need algorithms that can make sense of language.

89
00:09:17,550 --> 00:09:20,670
Making sense of language is hard, as these three problems demonstrate.

90
00:09:22,190 --> 00:09:32,570
So a good solution to the phishing problem would model as intermediate steps to detecting that an email is a phishing email.

91
00:09:32,570 --> 00:09:38,950
At some level, be able to solve these problems and sit somewhere in the depths of this intermediate representation of the language that it's looking at.

92
00:09:39,170 --> 00:09:47,090
The other challenge we have, obviously, in any cybersecurity problem, at least any detection context in cybersecurity, is that we have adversaries who'd like to bypass our detection.

93
00:09:47,090 --> 00:09:48,430
And that's also worth considering.

94
00:09:48,430 --> 00:09:57,750
So these are all reasons why the problem that we're presenting here is a hard one, deserves... we haven't solved it completely, deserves attention from our community.

95
00:10:00,450 --> 00:10:15,070
Now, the approach that we're using to attack the phishing problem is based in neural networks and deep learning, and a specific advance that happened in the last few years known as transformers.

96
00:10:15,510 --> 00:10:32,870
So transformers, or more specifically, transformer blocks, are a new kind of construct in neural networks, much like convolutions were new, I think, as of the 90s or late 80s, and backpropagation was a new idea in neural networks, I think, starting in the 80s.

97
00:10:32,870 --> 00:10:47,230
Transformers are a new idea that's come out in recent years, and they help model language with a depth and fidelity that seems to be genuinely new and represent a step function in our ability to model language.

98
00:10:47,230 --> 00:10:48,390
So they're very exciting.

99
00:10:48,390 --> 00:10:55,750
The big idea behind the work that we're presenting today is that we're taking transformers and applying them to a cybersecurity problem, which we haven't seen much of before.

100
00:10:55,810 --> 00:10:58,410
So I'm going to talk a bit about what transformers are.

101
00:10:59,070 --> 00:11:03,730
A detailed discussion of how they work is beyond the scope of this talk, but I'm going to give some intuition.

102
00:11:04,150 --> 00:11:13,550
And then I'll pass the mic metaphorically over to Young, who will present on how we're using transformers.

103
00:11:16,900 --> 00:11:31,480
Okay, so here's an example, which I think helps to illustrate some ways in which transformers represent a real breakthrough in machine comprehension of language.

104
00:11:31,780 --> 00:11:40,540
So in this example, and I expect some of you have seen this because this went viral last year, a researcher wrote a prompt.

105
00:11:40,540 --> 00:11:46,180
This prompt is given at the top here in a shocking finding, that paragraph.

106
00:11:46,720 --> 00:11:55,000
And then a transformer model just took it from there and wrote a story based on that prompt.

107
00:11:55,780 --> 00:12:09,280
And I think when you see the story that the machine wrote, you'll see that it betrays a understanding of the syntax and semantics of language that's quite striking.

108
00:12:09,580 --> 00:12:11,260
So I think it's worth reading this out loud.

109
00:12:11,360 --> 00:12:20,460
The human-written prompt reads, in a shocking finding, scientists discovered a herd of unicorns living in a remote, previously unexplored valley in the Andes Mountains.

110
00:12:20,780 --> 00:12:25,340
Even more surprising to the researchers was the fact that the unicorn spoke perfect English.

111
00:12:25,340 --> 00:12:30,160
And now again, the challenge to the left off and write something coherent.

112
00:12:30,160 --> 00:12:37,540
And if it can, that reflects something around its ability to solve problems like the fundamental problems I described earlier.

113
00:12:37,540 --> 00:12:41,400
So the neural network continues.

114
00:12:41,400 --> 00:12:45,980
The scientists named the population after their distinctive horn, Ovid's unicorn.

115
00:12:45,980 --> 00:12:50,180
These four-horned, silver-white unicorns were previously unknown to science.

116
00:12:50,180 --> 00:12:55,380
Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved.

117
00:12:55,380 --> 00:12:55,740
Dr.

118
00:12:55,740 --> 00:13:05,460
Jorge Perez, an evolutionary biologist from the University of La Paz and several companions, were exploring the Andes Mountains when they found a small valley with no other animals or humans.

119
00:13:05,860 --> 00:13:11,820
Perez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow.

120
00:13:12,300 --> 00:13:14,940
Perez and the others then ventured further into the valley.

121
00:13:14,940 --> 00:13:19,700
By the time we reached the top of one peak, the water looked blue with some crystals on top, said Perez.

122
00:13:19,700 --> 00:13:21,920
And the essay continues.

123
00:13:22,820 --> 00:13:38,960
But what interested Younghoo and I in seeing examples of what transformers can do like this is that it seems that the transformers have, examples like these suggest that transformers have understood the syntax and semantics of the English language to some degree,

124
00:13:38,960 --> 00:14:03,100
and which begs the question, could the representations that the transformers learn in their parameter structure be useful in detecting phishing emails, since these transformer models seem to have distilled knowledge of how language works and what it means within their parameter settings.

125
00:14:03,720 --> 00:14:11,580
Okay, so this is one example of, sort of impressive example, of what transformers can do, that beg the question of whether or not they could be applied usefully to phishing.

126
00:14:11,580 --> 00:14:15,380
And we'll show results from our experiments later in Younghoo's section.

127
00:14:15,380 --> 00:14:28,460
Here's another example of just how sort of marvelous transformer models are with respect to how much more sophisticated they seem to be.

128
00:14:28,460 --> 00:14:36,660
So here a researcher typed into this box a description of some HTML that he would like the transformer models to generate.

129
00:14:36,660 --> 00:14:40,140
He writes, a button with the color of Donald Trump's hair.

130
00:14:40,640 --> 00:14:46,120
And the model actually writes, as a text completion, some HTML and CSS.

131
00:14:46,220 --> 00:14:53,980
It seems to understand that Donald Trump has yellow hair, and it makes a button, and it writes valid HTML and CSS.

132
00:14:54,780 --> 00:15:04,260
Which is, I think, just impressive in the absolute sense, but also for folks who've been in the NLP space for a while, it's impressive relatively.

133
00:15:04,260 --> 00:15:12,620
Five, ten years ago, if you would have shown somebody that we could do this in 2020, I think they would have been a bit incredulous.

134
00:15:12,620 --> 00:15:13,820
This is a big step for you.

135
00:15:13,960 --> 00:15:18,220
Applications like these represent a big step forward in natural language processing.

136
00:15:18,220 --> 00:15:22,980
And they're due, again, to this new idea called a transformer block.

137
00:15:23,200 --> 00:15:27,880
That's the key building block out of which models like this model, which is called GPT-3.

138
00:15:27,880 --> 00:15:34,360
It's from OpenAI, which is an AI research lab, are based.

139
00:15:35,460 --> 00:15:36,360
Okay.

140
00:15:36,840 --> 00:15:39,340
So, moving on.

141
00:15:39,900 --> 00:15:49,460
So, I want to go back a little bit in the history of NLP as a way of describing how transformers are new and different.

142
00:15:49,920 --> 00:15:59,760
So, I assume a substantial chunk of our audience today has studied basic machine learning on text.

143
00:16:00,580 --> 00:16:10,200
And typically, in like a machine learning 101 course, you learn about the bag of words model and about discrete states, Markov models of language.

144
00:16:10,860 --> 00:16:20,500
And I want to talk about these representations as a way of talking about how limited those representations were, and then I'll talk about how transformers break us free of some of those limitations.

145
00:16:20,780 --> 00:16:32,660
So, a bag of words model of a document is a way of representing a document numerically for the purposes of machine learning, in which we just count up how many times each word in that document appeared.

146
00:16:33,040 --> 00:16:39,820
And then we create a matrix out of all the documents we have, and all the words in all those documents.

147
00:16:39,920 --> 00:16:43,000
And the entries in those matrix are just word counts.

148
00:16:43,000 --> 00:17:01,560
So, in the case on the left over here, the column vectors in our matrix are documents, and they get one dimension per word in our vocabulary, and they get counts of how many times words appeared in that vocabulary in the entries of that particular vector.

149
00:17:02,840 --> 00:17:13,200
And hopefully it's intuitive that, you know, once you've represented your documents in a vocabulary space in this way, you can compare documents by taking some distance measure between pairs of documents.

150
00:17:13,200 --> 00:17:20,960
You can also train machine learning models on your document corpus to say classified news articles is about sports or politics.

151
00:17:21,800 --> 00:17:30,440
But what you've done in the first step of these models is drop out sequence information.

152
00:17:30,440 --> 00:17:39,600
So, you've forgotten about which words come in which order in the document, and you've just represented your document as a bag of words.

153
00:17:39,640 --> 00:17:47,420
Which is a useful simplifying assumption, and it's one that we still use today in some of the modeling we do in my research group at Sophos.

154
00:17:47,720 --> 00:17:52,820
But it throws out a ton of information that transformers and more modern models don't throw out.

155
00:17:53,740 --> 00:17:58,460
In the model on the right, we have a discrete state Markov model of language.

156
00:17:58,740 --> 00:18:19,840
Here each word is a state, and the concept of language given by the model is kind of a cheesier adventure story in which the next words in an utterance depends only on the current words, and you're just sort of drawing from a probability distribution and moving through this graph to generate language.

157
00:18:19,920 --> 00:18:34,840
You could never have generated anything close to that unicorn story using a Markov model, and yet as recently as the last five, ten years, there's lots of papers coming out around like using hidden Markov models to parse sentences and that kind of thing.

158
00:18:35,160 --> 00:18:47,540
These are still useful models, but we've gone far beyond the simplifying assumptions in these original sort of simplified models of the world in NLP from the past few decades.

159
00:18:49,500 --> 00:19:01,060
So let's contrast transformers now, and I'll get into more details about how transformers work in a second, but let's contrast transformers with these earlier natural language processing approaches.

160
00:19:01,100 --> 00:19:09,040
So pre-transformer, most machine learning approaches didn't consider words in context.

161
00:19:09,260 --> 00:19:29,980
Many approaches made the simplifying bag of words assumptions as kind of a first step in the modeling process, and then ran term vectors through models like topic models, or logistic regression, or support vector machines.

162
00:19:30,620 --> 00:19:47,880
Most models didn't model kind of like co-reference relationships between words, or sort of which words pertain to which other words in a sentence, and I'll talk about what that means later, but transformers do sort of solve that problem.

163
00:19:49,060 --> 00:19:56,420
Most approaches didn't operate on either words or characters.

164
00:19:57,120 --> 00:20:10,960
Typically the way people use transformers, we use well-chosen chunks of words, which allows us to model misspelling and this kind of thing, so there's been an improvement there in the current generation of natural language processing models.

165
00:20:12,060 --> 00:20:15,580
And older approaches tended not to use neural network technology.

166
00:20:15,580 --> 00:20:31,280
That's changed a lot in the best ideas that have come out of the neural network revolution that's been ongoing since around 2012.

167
00:20:31,280 --> 00:20:38,980
So transformers kind of kick apart a number of log jams in natural language modeling.

168
00:20:38,980 --> 00:20:57,380
They give words contextual representations, they model attention, like the relationship between words in a document, they use these smart partial word representations that allow for misspellings and just the tower of babble of vernaculars that appear like under the banner of,

169
00:20:57,380 --> 00:21:10,140
say, the English language on the internet, and they take advantage of ideas like residual connections and modern optimizers and many of the really good ideas that have come out of the neural network revolution.

170
00:21:10,600 --> 00:21:15,640
So these are all reasons why we wanted to test their applicability to phishing detection.

171
00:21:18,440 --> 00:21:26,380
Okay, so if you want to get into detail about how transformers work, I'd recommend this blog post by Jay Alomar.

172
00:21:26,380 --> 00:21:29,480
It's where I cribbed this screenshot here.

173
00:21:29,480 --> 00:21:41,660
I'm not going to get into the details of the series of matrix multiplications that comprise summations and various linear algebra and operations that comprise the transformer block.

174
00:21:41,660 --> 00:21:46,980
I just want to give a little bit of intuition here before passing the mic over to Younghoo.

175
00:21:47,240 --> 00:21:59,540
The basic idea behind a transformer block, which is kind of a Lego block, out of which you build a transformer-based neural network, is we're taking in a sequence of words.

176
00:22:00,140 --> 00:22:03,840
This diagram shows a very simple example where we're just taking a sequence of two words.

177
00:22:03,840 --> 00:22:07,440
Typically we've taken a larger window, like 512 words.

178
00:22:07,580 --> 00:22:20,760
We pass them into the block, and the way they get passed in is not just as entries in a term vector matrix, but actually as vectors themselves.

179
00:22:20,760 --> 00:22:22,140
The words get a vector representation.

180
00:22:22,140 --> 00:22:23,660
These are known as embeddings.

181
00:22:25,920 --> 00:22:29,100
We pass them both into the transformer network.

182
00:22:30,000 --> 00:22:42,920
The first thing that the transformer network does to our input word sequence is model the attention relationships between the words.

183
00:22:43,320 --> 00:23:02,500
In a two-word case, it's a little bit harder to describe here, but basically if you had a sequence of 15 words, for every word, the attention mechanism would compute how much that word pertains to the other 14 words in the sentence.

184
00:23:02,960 --> 00:23:08,280
You'll see how that has a relationship with the coreference resolution problem I talked about earlier.

185
00:23:08,280 --> 00:23:24,060
But it's just intuitive that there's a graph of word relationships in a sentence, and self-attention kind of models that in terms of which subjects pertain to which objects, which pronouns pertain to which people, etc.

186
00:23:25,700 --> 00:23:37,580
There's an addition and normalization step that happens when we've run this self-attention process a number of times and typically we don't just do self-attention once.

187
00:23:37,580 --> 00:23:41,660
We have a number of what are called heads and we run attention a number of times.

188
00:23:41,700 --> 00:23:57,780
We combine all that together, we do some non-linear transformation on it, and then we wind up with a new embedding of your original sequence of the same dimensionality as this original embedding, except that now thinking is encoded in this new representation in the context in which it appears,

189
00:23:57,780 --> 00:23:59,040
in the context of machines.

190
00:23:59,040 --> 00:24:01,420
Machines is encoded in the context of thinking.

191
00:24:01,620 --> 00:24:04,400
Then typically we stack these transformer blocks that we actually do.

192
00:24:04,400 --> 00:24:08,920
We have another transformer block that then sort of refines the representation and we keep going.

193
00:24:09,400 --> 00:24:14,200
Young will show that we use a number of transformer blocks in our phishing detection work in a few minutes.

194
00:24:15,720 --> 00:24:22,580
Here's some intuition as to what comes out of the attention mechanism in a typical transformer.

195
00:24:22,580 --> 00:24:30,020
So here's what a transformer block has decided it sort of relates to in an input sentence.

196
00:24:30,020 --> 00:24:34,880
So here we have the input sentence, the animal didn't cross the street because it was too tired.

197
00:24:36,440 --> 00:24:49,140
And the strongest attentional relationship here goes to the animal, which is interesting because one could ask whether or not it refers to the street or the animal here.

198
00:24:49,780 --> 00:25:00,760
One can interpret the weight of the connection to the animal, meaning that the transformer block has decided, in their quotes, that it pertains to the animal, which is really interesting.

199
00:25:01,660 --> 00:25:13,520
So hopefully you get some intuition as to how powerful this attentional representation is and how important it is in machine learning models, getting some what we might call understanding over the language that they're analyzing.

200
00:25:16,310 --> 00:25:23,410
Okay, so I want to put the intuition together around how transformers pertain to the work that YoungHu and I are presenting today.

201
00:25:24,230 --> 00:25:36,650
So basically, what we're going to do in our phishing model is embed an email as a sequence of embedded character sequence vectors.

202
00:25:36,650 --> 00:25:57,650
And then we're going to run that through a series of transformer blocks, like what I just showed, that are going to create a very refined attentional representation of the word sequences, and then produce these contextual embeddings that get at the meaning of the words in the context in which they appear.

203
00:25:57,870 --> 00:26:05,790
And then finally, our network is going to solve a classification task, say whether or not the email is a phishing email or not.

204
00:26:06,310 --> 00:26:11,310
How all that magic works, how the network gets trained, which there's some tricks there that are really cool.

205
00:26:11,770 --> 00:26:20,550
I'll leave to YoungHu, and hope a lot of this makes sense, and happy to take questions about my piece of this presentation later at the end of the talk.

206
00:26:21,970 --> 00:26:24,150
Thank you, Joshi.

207
00:26:24,150 --> 00:26:28,290
Let me continue the second part of our talk.

208
00:26:28,650 --> 00:26:39,250
The second part will include our design decisions for CatBot and performance result.

209
00:26:39,830 --> 00:26:46,310
CatBot is the name of our email model, context-aware tiny bot.

210
00:26:46,310 --> 00:26:50,950
The model size is tiny, but it is mighty bot.

211
00:26:54,040 --> 00:27:00,280
Modern NLP models all have a nice and friendly name.

212
00:27:00,280 --> 00:27:06,360
For example, Elmer was introduced in 2018.

213
00:27:06,860 --> 00:27:16,440
The model used bidirectional LSTMs to generate contextualized word embeddings.

214
00:27:17,650 --> 00:27:32,800
And then, same year later, Google researchers introduced bot and achieved the state-of-art performance in many English-understanding problems.

215
00:27:32,800 --> 00:27:50,340
Next year, 2019, Baidu researchers introduced the Orni, and the model achieved another state-of-art performance in many Chinese-language-understanding problems.

216
00:27:51,140 --> 00:27:56,220
They are all popular characters from Sesame Street.

217
00:27:57,180 --> 00:28:06,880
This year, 2020, we introduced CatBot to tackle email security problems.

218
00:28:12,010 --> 00:28:20,270
Transformer-based NLP models are powerful, but they are complex and heavy.

219
00:28:20,790 --> 00:28:29,050
And it is challenging to deploy heavy models for real-time applications.

220
00:28:29,670 --> 00:28:45,810
So our first design goal is to convert the heavy model into a lightweight model, so we can reduce the number of parameters and then we can improve inference speed.

221
00:28:46,450 --> 00:28:53,890
We downsized a baseline model called DistribBot, which has six transformer blocks.

222
00:28:54,590 --> 00:29:04,290
We take half of transformer blocks from pre-trained model and then replace missing transformers with simple adapters.

223
00:29:04,610 --> 00:29:14,050
For example here, we take transformer 1, 3, 5, and then we added two adapters.

224
00:29:14,510 --> 00:29:18,510
Also, we can take other number of transformer blocks.

225
00:29:19,320 --> 00:29:24,840
This approach allows simply significantly reduced number of parameters.

226
00:29:29,940 --> 00:29:39,020
The second goal is to improve the model performance by combining additional input.

227
00:29:40,420 --> 00:29:46,660
Standard NLP models only accept text data as input.

228
00:29:46,660 --> 00:29:58,420
However, we can extract additional features from email headers and we can use the additional input to our model.

229
00:29:58,900 --> 00:30:22,160
So the text input will be the input to the embedding and then we can add additional input to the classification header and we added additional dense layers in the classification to combine the input from transformer block and another header-related input.

230
00:30:22,680 --> 00:30:28,880
With the additional input, we improved our model's performance further.

231
00:30:34,010 --> 00:30:38,750
Let me talk about the details of our trans adapters.

232
00:30:40,350 --> 00:30:48,310
We inserted two adapters here and each adapter has quite a simple architecture.

233
00:30:49,050 --> 00:31:01,710
Each adapter will have two dense layers and there is one non-linear activation unit in between and we have a skip connection.

234
00:31:02,930 --> 00:31:11,390
The dimensionality of a dense unit is the same as the output of transformer blocks.

235
00:31:11,690 --> 00:31:17,410
And the two dense layers are initialized with near-zero values.

236
00:31:17,410 --> 00:31:41,130
So as initially, the adapters will act as identity blocks, but however, they will gradually change the data from the lower transformer block to upper transformer block and to minimize classification loss.

237
00:31:45,580 --> 00:31:53,380
We also modified the standard fine-tuning method by using a partial fine-tuning method.

238
00:31:54,800 --> 00:32:00,940
Standard fine-tuning involves updating all parameters jointly.

239
00:32:01,140 --> 00:32:09,700
However, for partial fine-tuning, we only update upper blocks, but we fix low blocks.

240
00:32:09,700 --> 00:32:25,480
For example, here, the low blocks embedding the transformer 1 and 3 are fixed, but we update adapter 1 and 2 and transformer 5 and classification header.

241
00:32:28,240 --> 00:32:39,400
This approach was to minimize forgetting problems of learned presentations from low transformer blocks.

242
00:32:42,230 --> 00:32:46,720
As mentioned earlier, we have two set of features.

243
00:32:46,720 --> 00:32:52,930
One from text data and another one from email headers.

244
00:32:52,930 --> 00:33:07,270
We can use multiple email header builders, for example, from two cc-reply builders to extract additional context information.

245
00:33:07,270 --> 00:33:14,170
And we consider the subject and the text as context input.

246
00:33:17,450 --> 00:33:25,030
The first set of features are from email text content features.

247
00:33:25,270 --> 00:33:31,150
We extract text data from subject and plain text body.

248
00:33:31,750 --> 00:33:42,310
If only html content is available, then we can also extract plain text from html data using a html parser.

249
00:33:42,310 --> 00:33:57,130
For example, this one is a simple html hyperlink, but we only extract visible text visit bank site as output.

250
00:33:57,910 --> 00:34:04,150
And then from the extracted text, we remove less informative characters.

251
00:34:04,870 --> 00:34:08,890
We remove digit and punctuation characters.

252
00:34:08,890 --> 00:34:17,230
The second example, there are many hyphens, but the hyphens will be removed in this step.

253
00:34:17,230 --> 00:34:24,410
Otherwise, each hyphen will be individual token.

254
00:34:24,850 --> 00:34:33,230
Finally, we select 120 tokens as input to the transformer blocks.

255
00:34:37,290 --> 00:34:42,390
We use a sub-word tokenized called WordPiece.

256
00:34:42,390 --> 00:34:51,650
The WordPiece tokenized can overcome some of the limitations from character-level or word-level tokens.

257
00:34:51,910 --> 00:35:00,190
Character-level tokens are too fine-grained, so it is hard to recognize word boundaries and meaning of words.

258
00:35:00,190 --> 00:35:06,370
And word-level tokens often have auto-vocabulary problems.

259
00:35:07,320 --> 00:35:16,090
The sub-word token tokenized can split complex or uncommon words into sub-tokens.

260
00:35:16,470 --> 00:35:29,770
For example, CAPBOT can be divided into simple CAP and BOT, and double HES is an indicator for sub-words.

261
00:35:30,650 --> 00:35:39,330
Similarly, SOPHOS can be divided into three tokens, SO, PH, and OS.

262
00:35:40,630 --> 00:35:47,470
The sub-word tokenized reduce number of unknown tokens in our email data.

263
00:35:51,830 --> 00:36:05,270
With the selected tokens, the tokens will be the input to the embedding layer, and then we have three transformer blocks.

264
00:36:05,270 --> 00:36:12,050
And each transformer block has 12 multi-head attention layers.

265
00:36:12,050 --> 00:36:19,770
And each individual attention layer runs contextual relationship between tokens.

266
00:36:20,890 --> 00:36:25,850
And the lighter diagram shows attention rate between tokens.

267
00:36:25,850 --> 00:36:34,230
The transfer token has multiple attention rate for other tokens.

268
00:36:37,240 --> 00:36:52,060
In our email data, we have many non-English emails, and also non-English emails can include English words.

269
00:36:52,060 --> 00:36:56,340
Also, English emails can include non-English words.

270
00:36:57,360 --> 00:37:06,040
The non-English emails account for 25% of our total P9 and malicious emails.

271
00:37:06,040 --> 00:37:18,240
So, we needed to support a multilingual model which can recognize different languages.

272
00:37:21,170 --> 00:37:24,830
How we can support multilingual emails?

273
00:37:24,830 --> 00:37:31,910
The solution is BOT comes with two versions, English and multilingual BOT.

274
00:37:31,910 --> 00:37:44,350
The English version was pre-trained with the English text datasets, including Wikipedia, and has 30,000 English tokens.

275
00:37:46,210 --> 00:37:54,370
The multilingual version was pre-trained with large text datasets from more than 100 languages.

276
00:37:54,970 --> 00:38:05,790
And this version has four times large vocabulary size, which is 120 tokens, which will cover many Unicode characters and Unicode words.

277
00:38:06,490 --> 00:38:14,890
So, we fine-tuned a multilingual BOT for our multilingual emails.

278
00:38:18,270 --> 00:38:21,850
The second set of results from email headers.

279
00:38:21,890 --> 00:38:29,410
We can extract multiple indicators from email header builders.

280
00:38:29,410 --> 00:38:36,870
For example, the first one, we check whether the emails are from internal or external.

281
00:38:36,870 --> 00:38:43,070
We can compare the domain of recipient and sender.

282
00:38:43,070 --> 00:38:52,750
For example, here, the domain name is a similar looking one, but actually there are extra ads.

283
00:38:52,750 --> 00:38:57,090
So, this one we consider as external email.

284
00:38:57,650 --> 00:39:07,330
And then, we can also use external reply by comparing the domain of from and reply to.

285
00:39:07,570 --> 00:39:16,090
Often, targeted phishing attackers use a different domain for reply to.

286
00:39:16,110 --> 00:39:24,650
And also, we collect the size of the recipients and the size of the carbon copy recipients as additional indicators.

287
00:39:25,110 --> 00:39:32,710
It is obvious that targeted phishing attacks will have only single recipient.

288
00:39:32,710 --> 00:39:45,090
However, many user phishing emails will have multiple recipients or carbon copy recipients.

289
00:39:50,280 --> 00:39:55,100
Next, we will have a look at the performance of our CatBot.

290
00:40:00,640 --> 00:40:04,660
Let's have a look at the performance of a CatBot.

291
00:40:06,900 --> 00:40:12,500
We used a data set of 10 million benign samples.

292
00:40:12,720 --> 00:40:22,180
And the data set also includes 350 phishing emails and 1000 BC emails.

293
00:40:23,740 --> 00:40:33,220
We used time split to allocate 70% of samples as training and remaining 30% for test samples.

294
00:40:34,010 --> 00:40:40,780
We included two baseline models to compare CatBot.

295
00:40:41,020 --> 00:40:46,420
The first one is DistilBot, which has six transformer blocks.

296
00:40:46,760 --> 00:40:50,660
And another one is LSTM Long Short Memory.

297
00:40:50,660 --> 00:41:00,760
The model is a recurrent neural network architecture, which also uses BERT's same embedding layer.

298
00:41:00,760 --> 00:41:07,900
We trained three models on a GPU instance from AWS.

299
00:41:08,040 --> 00:41:23,160
And we assigned high sample weight for the PEC samples and oversample minor class malicious samples to allocate balanced samples in each mini-matching.

300
00:41:23,680 --> 00:41:29,240
To compare performance, we use R-Curves and L-Re-Under-Curve.

301
00:41:29,240 --> 00:41:36,240
Also, we compare inference speed and the model size as key performance metrics.

302
00:41:40,280 --> 00:41:48,940
These R-Curves compare our CatBot model with two baseline models.

303
00:41:48,940 --> 00:41:58,140
The top blue one is CatBot, and the second one is DistilBot, and the bottom green one is LSTM.

304
00:41:58,380 --> 00:42:03,140
Our model outperformed the two baseline models.

305
00:42:04,340 --> 00:42:14,600
And our model achieved 0.82% positive rate at 0.1% false positive rate.

306
00:42:17,410 --> 00:42:27,610
Next, we compare the performance when we removed adapters and context input rated layers.

307
00:42:28,110 --> 00:42:36,030
The top one is CatBot, and the second orange one is when we removed adapters.

308
00:42:36,030 --> 00:42:44,550
And the bottom one is when context input was removed from the CatBot.

309
00:42:45,010 --> 00:43:03,990
We can see significant performance drop when we remove either adapter or context input, which demonstrates we can improve performance by using additional adapters and context rated layers.

310
00:43:07,760 --> 00:43:18,010
Next, we compare the performance of three models with targeted BC samples.

311
00:43:18,010 --> 00:43:33,050
We assigned high sample rate for BC samples, and we achieved high performance for detecting those BC samples and the CatBot above the two baseline models.

312
00:43:35,460 --> 00:43:39,780
Next, we compare performance for phishing emails.

313
00:43:40,120 --> 00:43:47,320
We divide the phishing emails into two groups, English and non-English emails.

314
00:43:48,140 --> 00:43:54,760
Our CatBot outperformed for English and non-English emails.

315
00:43:56,000 --> 00:44:14,970
And our model was based on the multilingual bot, so we can see significant performance when we use the model for detecting non-English emails, when it compares with simple LST model.

316
00:44:16,570 --> 00:44:20,330
Next, we compare the inference speed.

317
00:44:20,550 --> 00:44:36,270
This bot has six transformer blocks, and the CatBot has three transformer blocks, so we achieve two times speed up in inference time when we measure the performance on a CPU machine.

318
00:44:36,730 --> 00:44:46,090
As the number of blocks decrease, the inference time can be reduced.

319
00:44:49,560 --> 00:44:51,900
Next, model size.

320
00:44:53,820 --> 00:45:03,180
For comparison, we divide the model size into two parts, embedding and transformer blocks.

321
00:45:06,190 --> 00:45:20,090
The DistilBot has six transformers and has 92 million parameters for embedding and 42 million parameters for transformer blocks.

322
00:45:20,730 --> 00:45:25,010
DistilBot and the CatBot have six transformer blocks.

323
00:45:25,010 --> 00:45:34,770
We reuse the same embedding parameters, but we reduce the number of parameters for transformer blocks by 50%.

324
00:45:34,770 --> 00:45:40,070
In total, our model size is 85% of baseline model.

325
00:45:40,650 --> 00:45:53,770
When we apply the same mechanism for English version, the English CatBot will have 71% of parameters from baseline model.

326
00:45:58,490 --> 00:46:05,490
Next, we will inspect how CatBot generates outputs.

327
00:46:08,240 --> 00:46:15,420
We use a LIME method to interpret our predictions.

328
00:46:15,420 --> 00:46:28,400
LIME is a local interpretable model ergodastic explanation method which can be applied to any black box models.

329
00:46:28,400 --> 00:46:37,100
We can understand a model by perturbing the input and understanding how the predictions change.

330
00:46:39,430 --> 00:46:46,210
The first LIME example is for a benign email.

331
00:46:46,890 --> 00:46:53,750
The prediction score for this email is close to zero.

332
00:46:53,790 --> 00:47:03,250
And we highlight legitimate tokens with blue color and malicious ones as orange one.

333
00:47:03,250 --> 00:47:09,630
And this one, we don't have any high-rated tokens for malicious ones.

334
00:47:11,820 --> 00:47:15,460
Next, we have a BC sample.

335
00:47:15,620 --> 00:47:20,420
The model prediction score for maliciousness is close to one.

336
00:47:20,420 --> 00:47:31,440
And the model recognizes transfer and urgent payment as high-rated tokens.

337
00:47:34,730 --> 00:47:42,170
Next, we have another BC sample, which is related with gift card.

338
00:47:42,170 --> 00:47:48,970
And the model prediction score is close to one and card and urgently.

339
00:47:50,110 --> 00:47:54,590
The tokens are high-rated for this email.

340
00:47:58,070 --> 00:48:03,710
Next, we have two handcrafted social engineering emails.

341
00:48:03,710 --> 00:48:12,570
They look quite different, but if you read the text carefully, they are asking the same wire transfer.

342
00:48:13,450 --> 00:48:30,730
And the model predictions are close to one for both emails and the highlight tokens are payment and swift or as soon as possible.

343
00:48:31,030 --> 00:48:45,390
These examples demonstrated our model's ability to understand complex texts and conceptually similar emails can be identified.

344
00:48:48,830 --> 00:48:57,150
In conclusion, our CatBot is a carefully re-architectural transform-based model.

345
00:48:57,150 --> 00:49:08,550
With this architecture, we achieve both high-speed and high-accuracy in detecting handicrafted social engineering email attacks.

346
00:49:08,550 --> 00:49:16,330
In the future, we want to apply the same design decisions to new GPT-3 model.

347
00:49:18,770 --> 00:49:20,190
Thank you.

348
00:49:20,190 --> 00:49:21,850
Do you have any questions?

349
00:49:25,180 --> 00:49:29,600
If you do have any questions, please head over to the Discord channel.

350
00:49:30,640 --> 00:49:40,640
They are currently there answering questions in the DEF CON Discord channel under aiv-talks-text.