Hi, and today with Young-Hoo Lee, I'll be presenting Detecting Handcrafted Social
Engineering Emails with a Bleeding Edge Neural Language Model.
A bit about us before getting into the talk.
Sophos is a security products and services company. We have a whole range of security
products ranging from firewalls, to mobile security, to email security, to endpoint security.
And the team that I manage, Sophos AI, is the AI shop inside of Sophos. We research and develop
the company's machine learning technology. And then we're also responsible...
Okay, I'm gonna mute myself. I'm gonna like definitely mute myself in here so I can listen
to the talk. I'm happy if you need something. Young-Hoo Lee is one of the star researchers
on my team. He's almost completely solely responsible for the machine learning subsystem
that protects our Android customers, at least responsible for the R&D part of that work.
He's also the principal author of the work presented today. I'm presenting in a supporting
role here. I'll mostly just be setting the stage for the original work that Young-Hoo is gonna
describe. So, the problem we're solving. Most generally, we're working on the problem of
detecting phishing emails. More specifically, we're working on detecting business email
compromise phishing emails and targeted phishing emails. So, to understand what this is, it's
useful to say what it's not. We're not focused on detecting mass campaigns where detection
reduces to a near-duplicate detection problem because attackers are just sending out millions
of copies of basically the same phishing email. Detecting those types of emails turns out to be
a side effect of our focus. But our real focus is on detecting new, custom-authored, bespoke
phishing emails that are based on research on a target. And this diagram does a good job of
getting across the kind of workflow we're looking at stopping. In this workflow, cyber criminals and
attackers identify targets through open-source research, usually on the web. Then they establish
contact with those targets. In step two, we call this grooming. In this case, they put out a lure,
an initial email, usually or sometimes an initial text message, and then build trust and authenticity
around some identity that they're impersonating with the mark. In step three, they cash out the
trust that they've built with their targets and make an ask of those targets. Oftentimes, that
will be around wiring money or sending credentials. And then in step four, they actually
receive money or receive the credentials.
So within targeted phishing, business email compromise, which is focused on
stealing money from businesses and other organizations, has been a growing trend.
So as you can see here, in July 2016, a few billion dollars were stolen, according to the FBI,
through business email compromise attacks. These are targeted phishing attacks that extort money
from organizations. In May 2017, that number had grown to almost six billion. And in 2018,
that number had grown to more than 12 billion. I don't have data for the last two years, but
I see no reason to believe that this trend has attenuated. I think it's likely it's continued
on a similar trajectory. This is a big problem. We see this in Sophos's customer base,
which is which is a pretty large sample. We also hear about it from other folks in the
cybersecurity space. And so we're very focused on it because it's affecting people. And it's
not just affecting large organizations. You can see on this on the axis on the right,
that something like 80,000 organizations had been hit by July 2018 by these attacks.
So there are lots of small and mid-sized organizations getting hit. And oftentimes,
financial damage can be in the hundreds of thousands of dollars and really impact people's
lives when these attacks happened. So again, just to reiterate, we're focused on primarily
these business email compromise use cases, but also more generally targeted phishing in which
a lot of manual labor goes into the phishing process on the criminal actors.
And I think it almost goes without saying, but we're focused on step two and three
of these criminal actors workflow, the steps that are mediated over email. So we see lots of
malicious emails exchanged in the grooming step. And then we also see, obviously, a malicious email
transmitted in the exchange of information step where the attacker makes the ask of the
target, usually employee. And this is just because we're focused on email as our signal.
And that's the scope of the work that I'll be talking about today.
Just to flesh this out a little bit, here's an example, phishing email sent in the later
epics of the grooming stage of an attacker's workflow. Here, the attacker has established
themselves as an impersonated chancellor of UC Berkeley. They are emailing an employee at UC
Berkeley, asking if they're available, looking to exchange messages with them, probably about to
make an ask around a money transfer. Now, I think it's important to highlight why phishing
detection is hard. And why in particular, detecting new previously unseen phishing emails written as
part of a manual phishing campaign, like what I've just described, is hard. And what this boils down
to, I think, is that classical natural language processing problems are hard. It's hard to
get computers to understand language in any meaningful way and reason about language in any
meaningful way. And detecting phishing emails really boils down to building models that have
some level of understanding of language. So to understand why algorithmically it's hard to make
sense of language, let's look at a few classical natural language processing problems. So one of
those is coreference resolution. To get a sense of what this problem means, consider the following
sentence. I went to the store for some milk, and based on the price, decided to buy it.
Now, from a grammatical perspective, it could refer to the store or to the milk, right? It's
possible that I went to the store for some milk, and based on the price of the store, I decided to
buy it. But it's more likely that I bought the milk. As humans, in solving this coreference
resolution problem, so resolving what it refers to, we have to plumb the depths of a number of
complex mental models, right? We need to deploy our syntactical model of the English language,
our semantic model of the English language, and our model of the world in which a person is more likely
to have bought milk at the store than have bought the store itself. And that's how we solve this
problem. Hopefully, it's clear that it's hard to get algorithms to do that. But to detect phishing
emails, we really need to understand language. And this is a problem that's constituent of the
problem of understanding language. Word polysemy is also a classic problem
in algorithmic understanding of language. So a sentence like, he drank a lot and was quite the
rake. Grammatically, it's valid to interpret this sentence as meaning that he drank a lot
and was quite the garden tool used to rake leaves off your lawn. That's clearly not the right,
that's not the sense in which the word rake is being used. The word rake is being used in the
sense of a drunk, semi-criminal, sort of dissolute individual here. But it takes a pretty deep
exercise of a human being's mental models to arrive at the sense in which this word was used.
And not easy to reproduce in the form of an automated agent, either machine learning or
based on regexes and rules. Sentiment detection is another hard problem in natural language
processing. So consider the sentence, I'm not angry at all. No, of course. Why would I be angry
that you spent our life savings on your mistress? So clearly, the speaker is angry here. And they're
being sarcastic. Detecting that they're angry and sarcastic is not trivial and requires that
we understand not only the syntactic structure of the sentence, but also the semantics of the
sentence. And requires that we have the reflex that this person is probably angry if their
interlocutor spent their life savings on their mistress. Okay, so to solve phishing means that
we need algorithms that can make sense of language. Making sense of language is hard, as these three
problems demonstrate. So a good solution to the phishing problem would model as intermediate
steps to detecting that an email is a phishing email. At some level, be able to solve these
problems and sit somewhere in the depths of this intermediate representation of the language that
it's looking at. The other challenge we have, obviously, in any cybersecurity problem, at least
any detection context in cybersecurity, is that we have adversaries who'd like to bypass our
detection. And that's also worth considering. So these are all reasons why the problem that we're
presenting here is a hard one, deserves... we haven't solved it completely, deserves attention
from our community. Now, the approach that we're using to attack the phishing problem
is based in neural networks and deep learning, and a specific advance that happened in the last
few years known as transformers. So transformers, or more specifically, transformer blocks, are a
new kind of construct in neural networks, much like convolutions were new, I think, as of the
90s or late 80s, and backpropagation was a new idea in neural networks, I think,
starting in the 80s. Transformers are a new idea that's come out in recent years,
and they help model language with a depth and fidelity that seems to be genuinely new and
represent a step function in our ability to model language. So they're very exciting. The big idea
behind the work that we're presenting today is that we're taking transformers and applying them
to a cybersecurity problem, which we haven't seen much of before. So I'm going to talk a bit about
what transformers are. A detailed discussion of how they work is beyond the scope of this talk,
but I'm going to give some intuition. And then I'll pass the mic metaphorically over to
Young, who will present on how we're using transformers. Okay, so here's an example,
which I think helps to illustrate some ways in which transformers represent a real breakthrough
in machine comprehension of language. So in this example, and I expect some of you have seen this
because this went viral last year, a researcher wrote a prompt. This prompt is given at the top
here in a shocking finding, that paragraph. And then a transformer model just took it from there
and wrote a story based on that prompt. And I think when you see the story that the machine wrote,
you'll see that it betrays a understanding of the syntax and semantics of language
that's quite striking. So I think it's worth reading this out loud. The human-written prompt
reads, in a shocking finding, scientists discovered a herd of unicorns living in a remote,
previously unexplored valley in the Andes Mountains. Even more surprising to the researchers
was the fact that the unicorn spoke perfect English. And now again, the challenge to the
left off and write something coherent. And if it can, that reflects something around its ability
to solve problems like the fundamental problems I described earlier. So the neural network
continues. The scientists named the population after their distinctive horn, Ovid's unicorn.
These four-horned, silver-white unicorns were previously unknown to science.
Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved.
Dr. Jorge Perez, an evolutionary biologist from the University of La Paz and several companions,
were exploring the Andes Mountains when they found a small valley with no other animals or humans.
Perez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks
of rock and silver snow. Perez and the others then ventured further into the valley. By the time we
reached the top of one peak, the water looked blue with some crystals on top, said Perez.
And the essay continues. But what interested Younghoo and I in seeing examples of what
transformers can do like this is that it seems that the transformers have, examples like these
suggest that transformers have understood the syntax and semantics of the English language
to some degree, and which begs the question, could the representations that the transformers
learn in their parameter structure be useful in detecting phishing emails, since these
transformer models seem to have distilled knowledge of how language works and what it means
within their parameter settings. Okay, so this is one example of, sort of impressive example,
of what transformers can do, that beg the question of whether or not they could be
applied usefully to phishing. And we'll show results from our experiments later in Younghoo's
section. Here's another example of just how sort of marvelous transformer models are with respect to
how much more sophisticated they seem to be. So here a researcher typed into this box
a description of some HTML that he would like the transformer models to generate.
He writes, a button with the color of Donald Trump's hair. And the model actually writes,
as a text completion, some HTML and CSS. It seems to understand that Donald Trump has yellow hair,
and it makes a button, and it writes valid HTML and CSS. Which is, I think, just
impressive in the absolute sense, but also for folks who've been in the NLP space for a while,
it's impressive relatively. Five, ten years ago, if you would have shown somebody that we could do
this in 2020, I think they would have been a bit incredulous. This is a big step for you.
Applications like these represent a big step forward in natural language processing. And
they're due, again, to this new idea called a transformer block. That's the key building
block out of which models like this model, which is called GPT-3. It's from OpenAI,
which is an AI research lab, are based. Okay. So, moving on.
So, I want to go back a little bit in the history of NLP as a way of describing how
transformers are new and different. So, I assume a substantial chunk of our audience today
has studied basic machine learning on text. And typically, in like a machine learning 101
course, you learn about the bag of words model and about discrete states, Markov models of language.
And I want to talk about these representations as a way of talking about how limited those
representations were, and then I'll talk about how transformers break us free of some of those
limitations. So, a bag of words model of a document is a way of representing a document
numerically for the purposes of machine learning, in which we just count up how many times each
word in that document appeared. And then we create a matrix out of all the documents we have,
and all the words in all those documents. And the entries in those matrix are just word counts.
So, in the case on the left over here, the column vectors in our matrix are documents,
and they get one dimension per word in our vocabulary, and they get counts of how many
times words appeared in that vocabulary in the entries of that particular vector.
And hopefully it's intuitive that, you know, once you've represented your documents in a
vocabulary space in this way, you can compare documents by taking some distance measure
between pairs of documents. You can also train machine learning models on your document corpus
to say classified news articles is about sports or politics. But what you've done in the first
step of these models is drop out sequence information. So, you've forgotten about which
words come in which order in the document, and you've just represented your document as a bag
of words. Which is a useful simplifying assumption, and it's one that we still use
today in some of the modeling we do in my research group at Sophos. But it throws out
a ton of information that transformers and more modern models don't throw out.
In the model on the right, we have a discrete state Markov model of language.
Here each word is a state, and the concept of language given by the model
is kind of a cheesier adventure story in which the next words in an utterance depends only on
the current words, and you're just sort of drawing from a probability distribution and moving through
this graph to generate language. You could never have generated anything close to that unicorn
story using a Markov model, and yet as recently as the last five, ten years, there's lots of
papers coming out around like using hidden Markov models to parse sentences and that kind of thing.
These are still useful models, but we've gone far beyond the simplifying assumptions
in these original sort of simplified models of the world in NLP from the past few decades.
So let's contrast transformers now, and I'll get into more details about how transformers
work in a second, but let's contrast transformers with these earlier natural
language processing approaches. So pre-transformer, most machine learning
approaches didn't consider words in context. Many approaches made the simplifying bag of
words assumptions as kind of a first step in the modeling process, and then ran term vectors
through models like topic models, or logistic regression, or support vector machines.
Most models didn't model kind of like co-reference relationships between words,
or sort of which words pertain to which other words in a sentence, and I'll talk about what
that means later, but transformers do sort of solve that problem. Most approaches didn't
operate on either words or characters. Typically the way people use transformers,
we use well-chosen chunks of words, which allows us to model misspelling and this kind of thing,
so there's been an improvement there in the current generation of natural language processing models.
And older approaches tended not to use neural network technology. That's changed a lot in the
best ideas that have come out of the neural network revolution that's been ongoing since
around 2012. So transformers kind of kick apart a number of log jams in natural language modeling.
They give words contextual representations, they model attention, like the relationship between
words in a document, they use these smart partial word representations that allow for
misspellings and just the tower of babble of vernaculars that appear
like under the banner of, say, the English language on the internet, and they take advantage
of ideas like residual connections and modern optimizers and many of the really good ideas
that have come out of the neural network revolution. So these are all reasons why
we wanted to test their applicability to phishing detection. Okay, so if you want to get into detail
about how transformers work, I'd recommend this blog post by Jay Alomar. It's where I cribbed this
screenshot here. I'm not going to get into the details of the series of matrix multiplications
that comprise summations and various linear algebra and operations that comprise the
transformer block. I just want to give a little bit of intuition here before passing the mic over
to Younghoo. The basic idea behind a transformer block, which is kind of a Lego block,
out of which you build a transformer-based neural network, is we're taking in a sequence of words.
This diagram shows a very simple example where we're just taking a sequence of two words. Typically
we've taken a larger window, like 512 words. We pass them into the block, and the way they
get passed in is not just as entries in a term vector matrix, but actually as vectors themselves.
The words get a vector representation. These are known as embeddings. We pass them both into the
transformer network. The first thing that the transformer network does to our input word
sequence is model the attention relationships between the words. In a two-word case, it's a
little bit harder to describe here, but basically if you had a sequence of 15 words,
for every word, the attention mechanism would compute how much
that word pertains to the other 14 words in the sentence. You'll see how that has a
relationship with the coreference resolution problem I talked about earlier. But it's just
intuitive that there's a graph of word relationships in a sentence, and self-attention
kind of models that in terms of which subjects pertain to which objects,
which pronouns pertain to which people, etc. There's an addition and normalization step
that happens when we've run this self-attention process a number of times
and typically we don't just do self-attention once. We have a number of what are called heads and
we run attention a number of times. We combine all that together, we do some non-linear transformation
on it, and then we wind up with a new embedding of your original sequence of the same dimensionality
as this original embedding, except that now thinking is encoded in this new representation
in the context in which it appears, in the context of machines. Machines is encoded in the context of
thinking. Then typically we stack these transformer blocks that we actually do.
We have another transformer block that then sort of refines the representation and we keep
going. Young will show that we use a number of transformer blocks in our phishing detection
work in a few minutes. Here's some intuition as to what comes out of the attention mechanism
in a typical transformer. So here's what a transformer block has decided it
sort of relates to in an input sentence. So here we have the input sentence,
the animal didn't cross the street because it was too tired.
And the strongest attentional relationship here goes to the animal, which is interesting
because one could ask whether or not it refers to the street or the animal here.
One can interpret the weight of the connection to the animal, meaning that the transformer block
has decided, in their quotes, that it pertains to the animal, which is really interesting.
So hopefully you get some intuition as to how powerful this attentional representation is and
how important it is in machine learning models, getting some what we might call understanding
over the language that they're analyzing. Okay, so I want to put the intuition together
around how transformers pertain to the work that YoungHu and I are presenting today.
So basically, what we're going to do in our phishing model is embed an email as a sequence
of embedded character sequence vectors. And then we're going to run that through a series of
transformer blocks, like what I just showed, that are going to create a very refined attentional
representation of the word sequences, and then produce these contextual embeddings that get at
the meaning of the words in the context in which they appear. And then finally, our network is
going to solve a classification task, say whether or not the email is a phishing email or not.
How all that magic works, how the network gets trained, which there's some tricks there that
are really cool. I'll leave to YoungHu, and hope a lot of this makes sense, and happy to
take questions about my piece of this presentation later at the end of the talk.
Thank you, Joshi. Let me continue the second part of our talk. The second part will include
our design decisions for CatBot and performance result.
CatBot is the name of our email model, context-aware tiny bot. The model size is tiny,
but it is mighty bot. Modern NLP models all have a nice and friendly name. For example,
Elmer was introduced in 2018. The model used bidirectional LSTMs to
generate contextualized word embeddings.
And then, same year later, Google researchers introduced bot and achieved the state-of-art
performance in many English-understanding problems. Next year, 2019, Baidu researchers
introduced the Orni, and the model achieved another state-of-art performance in many
Chinese-language-understanding problems. They are all popular characters from Sesame Street.
This year, 2020, we introduced CatBot to tackle email security problems.
Transformer-based NLP models are powerful, but they are complex and heavy.
And it is challenging to deploy heavy models for real-time applications.
So our first design goal is to convert the heavy model into a lightweight model,
so we can reduce the number of parameters and then we can improve inference speed.
We downsized a baseline model called DistribBot, which has six transformer blocks.
We take half of transformer blocks from pre-trained model and then replace missing
transformers with simple adapters. For example here, we take transformer 1, 3, 5,
and then we added two adapters. Also, we can take other number of transformer blocks.
This approach allows simply significantly reduced number of parameters.
The second goal is to improve the model performance by combining additional input.
Standard NLP models only accept text data as input. However, we can extract additional
features from email headers and we can use the additional input to our model.
So the text input will be the input to the embedding and then we can add additional input
to the classification header and we added additional dense layers in the classification
to combine the input from transformer block and another header-related input.
With the additional input, we improved our model's performance further.
Let me talk about the details of our trans adapters.
We inserted two adapters here and each adapter has quite a simple architecture.
Each adapter will have two dense layers and there is one non-linear activation unit in between
and we have a skip connection.
The dimensionality of a dense unit is the same as the output of transformer blocks.
And the two dense layers are initialized with near-zero values. So as initially, the adapters
will act as identity blocks, but however, they will gradually change the data from the lower
transformer block to upper transformer block and to minimize classification loss.
We also modified the standard fine-tuning method by using a partial fine-tuning method.
Standard fine-tuning involves updating all parameters jointly. However, for partial
fine-tuning, we only update upper blocks, but we fix low blocks. For example, here,
the low blocks embedding the transformer 1 and 3 are fixed, but we update adapter 1 and 2
and transformer 5 and classification header. This approach was to minimize forgetting
problems of learned presentations from low transformer blocks.
As mentioned earlier, we have two set of features. One from text data and another one from
email headers. We can use multiple email header builders, for example, from two cc-reply
builders to extract additional context information. And we consider the subject and the text as
context input. The first set of features are from email text content features.
We extract text data from subject and plain text body. If only html content is available,
then we can also extract plain text from html data using a html parser. For example,
this one is a simple html hyperlink, but we only extract visible text visit bank site as output.
And then from the extracted text, we remove less informative characters.
We remove digit and punctuation characters. The second example, there are many
hyphens, but the hyphens will be removed in this step. Otherwise, each hyphen will be
individual token. Finally, we select 120 tokens as input to the transformer blocks.
We use a sub-word tokenized called WordPiece. The WordPiece tokenized can overcome
some of the limitations from character-level or word-level tokens. Character-level tokens are
too fine-grained, so it is hard to recognize word boundaries and meaning of words.
And word-level tokens often have auto-vocabulary problems.
The sub-word token tokenized can split complex or uncommon words into sub-tokens.
For example, CAPBOT can be divided into simple CAP and BOT, and double HES is an indicator
for sub-words. Similarly, SOPHOS can be divided into three tokens, SO, PH, and OS.
The sub-word tokenized reduce number of unknown tokens in our email data.
With the selected tokens, the tokens will be the input to the embedding layer,
and then we have three transformer blocks. And each transformer block has 12 multi-head
attention layers. And each individual attention layer runs contextual relationship between tokens.
And the lighter diagram shows attention rate between
tokens. The transfer token has multiple attention rate for other tokens.
In our email data, we have many non-English emails, and also non-English emails can include
English words. Also, English emails can include non-English words.
The non-English emails account for 25% of our total P9 and malicious emails.
So, we needed to support a multilingual model which can recognize different languages.
How we can support multilingual emails? The solution is BOT comes with two versions,
English and multilingual BOT. The English version was pre-trained with the English text
datasets, including Wikipedia, and has 30,000 English tokens.
The multilingual version was pre-trained with large text datasets from more than 100 languages.
And this version has four times large vocabulary size, which is 120 tokens,
which will cover many Unicode characters and Unicode words.
So, we fine-tuned a multilingual BOT for our multilingual emails.
The second set of results from email headers. We can extract multiple indicators from
email header builders. For example, the first one, we check whether the emails are from
internal or external. We can compare the domain of recipient and sender. For example, here,
the domain name is a similar looking one, but actually there are extra ads.
So, this one we consider as external email. And then, we can also use external reply
by comparing the domain of from and reply to. Often, targeted phishing attackers use
a different domain for reply to. And also, we collect the size of the recipients and the size
of the carbon copy recipients as additional indicators. It is obvious that targeted
phishing attacks will have only single recipient. However, many user
phishing emails will have multiple recipients or carbon copy recipients.
Next, we will have a look at the performance of our CatBot.
Let's have a look at the performance of a CatBot.
We used a data set of 10 million benign samples.
And the data set also includes 350 phishing emails and 1000 BC emails.
We used time split to allocate 70% of samples as training and remaining 30% for test samples.
We included two baseline models to compare CatBot. The first one is DistilBot, which has
six transformer blocks. And another one is LSTM Long Short Memory. The model is a recurrent neural
network architecture, which also uses BERT's same embedding layer. We trained three models on a
GPU instance from AWS. And we assigned high sample weight for the
PEC samples and oversample minor class malicious samples to allocate balanced
samples in each mini-matching. To compare performance, we use R-Curves and L-Re-Under-Curve.
Also, we compare inference speed and the model size as key performance metrics.
These R-Curves compare our CatBot model with two baseline models. The top blue one is CatBot,
and the second one is DistilBot, and the bottom green one is LSTM.
Our model outperformed the two baseline models.
And our model achieved 0.82% positive rate at 0.1% false positive rate.
Next, we compare the performance when we removed adapters and context input rated layers.
The top one is CatBot, and the second orange one is when we removed adapters. And the bottom one
is when context input was removed from the CatBot.
We can see significant performance drop when we remove either adapter or context input,
which demonstrates we can improve performance by using additional adapters and context rated layers.
Next, we compare the performance of three models with targeted BC samples.
We assigned high sample rate for BC samples, and we achieved high performance for detecting
those BC samples and the CatBot above the two baseline models.
Next, we compare performance for phishing emails. We divide the phishing emails into two groups,
English and non-English emails. Our CatBot outperformed for English and non-English emails.
And our model was based on the multilingual bot, so we can see significant performance
when we use the model for detecting non-English emails, when it compares with simple LST model.
Next, we compare the inference speed. This bot has six transformer blocks, and the CatBot has
three transformer blocks, so we achieve two times speed up in inference time when we measure the
performance on a CPU machine. As the number of blocks decrease, the inference time can be reduced.
Next, model size.
For comparison, we divide the model size into two
parts, embedding and transformer blocks.
The DistilBot has six transformers and has 92 million parameters for embedding and 42 million
parameters for transformer blocks. DistilBot and the CatBot have six transformer blocks.
We reuse the same embedding parameters, but we reduce the number of parameters for
transformer blocks by 50%. In total, our model size is 85% of baseline model.
When we apply the same mechanism for English version,
the English CatBot will have 71% of parameters from baseline model.
Next, we will inspect how CatBot generates outputs.
We use a LIME method to interpret our predictions. LIME is a
local interpretable model ergodastic explanation method which can be applied to any black box
models. We can understand a model by perturbing the input and understanding how the predictions
change. The first LIME example is for a benign email. The prediction score for this email
is close to zero. And we highlight legitimate tokens with
blue color and malicious ones as orange one. And this one, we don't have any high-rated
tokens for malicious ones. Next, we have a BC sample. The model prediction score for
maliciousness is close to one. And the model recognizes transfer and urgent payment as
high-rated tokens. Next, we have another BC sample, which is related with gift card.
And the model prediction score is close to one and card and urgently.
The tokens are high-rated for this email. Next, we have two handcrafted social engineering emails.
They look quite different, but if you read the text carefully, they are asking the same wire transfer.
And the model predictions are close to one for both emails and the highlight tokens are payment
and swift or as soon as possible. These examples demonstrated our model's ability to
understand complex texts and conceptually similar emails can be identified.
In conclusion, our CatBot is a carefully re-architectural transform-based model.
With this architecture, we achieve both high-speed and high-accuracy in detecting
handicrafted social engineering email attacks. In the future, we want to apply the same
design decisions to new GPT-3 model. Thank you. Do you have any questions?
If you do have any questions, please head over to the Discord channel.
They are currently there answering questions in the DEF CON Discord channel
under aiv-talks-text.