We're gonna get started right away.
It is my pleasure to introduce an old friend of ours,
Ghida Liabari.
Thank you so much.
Thanks, everyone.
Thanks for your patience, first of all,
and thanks for coming to my presentation.
I'm going to talk about Yalda.
It's an automated tool that I wrote recently,
one of my recent projects,
and I thought to share it with you.
I work for Fidelity Cyber Security,
and I'm going to talk about Yalda.
So, we are going to talk about
having an introduction of Yalda,
how was it created, and what was the object of it.
Then, how could you use Yalda,
and what it could get used.
Then, we are going to have an overview of
architecture of Yalda,
a quick demonstration,
and a GitHub that you can go ahead and download it,
and start using it on your own files.
Using it is actually pretty much simple,
it's just going and clicking on a file
to configure and add some directories to it.
So, introducing Yalda.
The motivation was basically creating a tool
that analyzes files and gives you some indicators
that saves time for repeating the job
for each single file.
And also, it was just for fully understanding
what the file does,
and creating automated bulk intelligence collection.
And sort of being independent of other sources,
third parties.
So, that was the motivation.
So, that guy in the back, John Baminak,
he actually is my boss,
and he passed me a project to start with.
So, he started giving me some files,
massive files, I mean millions of files in directories,
and they were all compressed,
and they were in JSON format,
each one of them had a dictionary inside it,
and then the dictionary had information about the emails,
like each dictionary was corresponding
to a malicious email.
It has all the key and values, of course, for that one,
and it has some mime-encoded keys
that the values were there.
If you decode them,
then you would be able to download the files.
So, that is how Yalda was born.
I started analyzing them manually,
and then figured out that, yeah,
campaign from the same sort of campaigns,
although the format and everything
is a little bit different,
but the fingerprint and the way that they are introducing stuff
is almost the same.
And I figured out that it could get automated.
This is just like an example.
By just looking at it,
I found out that this just doesn't make sense.
This is malicious.
I mean, I would be able to just say by a glance
that this is malicious.
Look at the from.
It's coming from Russia,
and it's saying that USPS ground.
I mean, come on.
And so, the analyzing JSON file,
each dictionary,
this is just like the process of automation
that I would get the base 64 encoded MIME,
which was the key,
and then the value decoded it,
and getting a couple of files.
The most important one was the file
that I was getting that had the attachments.
And then, looking at the attachments,
this was, again, for a particular campaign,
it was almost the same.
Like, the domains were the same.
The domains were different,
but the structure was the same.
So, I started writing an automated tool
for extracting the information.
The steps was pretty much like following
whatever I'd done manually.
It was... yeah, it works at least.
So, I was getting the JSON MIME,
analyze keys, analyze from address,
analyze the body,
get base 64 encoded data, decode it,
download attachments, analyze files,
and extract domains.
So, that's how YALDA started working.
So, manual analysis,
non-process for automation,
automating it, YALDA,
and then adding more and more to it.
YALDA was born.
So, as I was extracting information,
then I started just talking to analysts
and then asking,
what do you guys do for getting more information?
And then I made YALDA somehow
that it does all of this stuff.
Automated Bulk Intelligence Collection,
it's a file scanner and analyzer,
extracts data,
cluster files,
and it categorizes files.
So, Automated Bulk Intelligence Collection Tool,
it has an automated process of analyzing files.
It applies intelligence in collected data
and clusters files based on the similarities.
It's a file scanner and analyzer.
YALDA scans and analyzes files.
It collects detailed information of the file.
About 20 indicators are being extracted
from each single file.
And it makes it possible...
How many of you guys are analysts here?
Okay, so it makes it possible for you guys,
instead of hours of doing sort of like analyzes,
you just run YALDA on the set of files,
let's say even like a million samples that you have,
and then it's going to analyze each single one of them
and it's going to give you some indicators,
like 20 indicators is going to be extracted.
Then based on whatever you are looking for
and whatever interests you,
it is going to extract the information
that is valuable for you guys.
So you can just like apply your filter
and you can extract the information.
It also extracts data,
like it collects embedded objects
and also it collects malicious URL and domains.
And it categorizes files.
There is like a flag and severity
that in combination they are being used.
So depending on the algorithm
that is being applied to the file
and the founding of the file,
a file could be categorized as clear
with a low severity like a 1 and 2,
suspicious with higher severity,
and malicious, of course, higher severity.
It also clusters data.
It gets the... if there is a malicious hash
that it detects,
it's going to cluster the strings
out of the file
and it's going to insert it
in a different collection.
Same applies if it is an executable file
that is malicious.
It is going to get the names of P sections,
convert it to SHA-1,
and in combination of the hash
and channel entropy is going to use it.
Now, what to do with Yalda
and when to use it.
So if you want to have a feed,
a collection of feeds that you could use it,
you could go ahead and just submit files to Yalda
and it could be used for extracting
MD5 feed, domain feed, URL feed,
all of them malicious.
Also, as I said,
there are 20 indicators
that are being extracted from each single file.
Depending on the file type,
some of them might be empty.
Like, for example,
if there is no P sections in it,
then it's going to be empty.
However, you are able to get all of these,
like MD5, SHA-1, SHA-256,
similar MD5.
If there are some malicious MD5s
that are similar to this particular file
that you are analyzing,
it is going to give you a list of similar MD5s.
File type, if there is any malicious URL
that has been extracted from the file,
it is going to be given to you.
Magic literal source,
VirusTotal information.
This is basically,
just gives you a paramount link
to VirusTotal page,
if it exists in VirusTotal
with the number of positive ABN genes
that are able to detect it.
You don't have to have the key
or use VirusTotal.
You can just disable it.
It's not mandatory.
Severity and flag are being used together.
Severity is like clear,
or it could be suspicious or malicious.
And then the flag,
it could be from 1 to 5.
1 stands for clear,
and 5 stands for very malicious.
So it just sort of gives you a scope
of how the file is analyzed.
File name, ingest time,
the time that it got analyzed
and got inserted in database.
Domain list, if it has any malicious domain.
URL list, I'm going to go through it in detail,
but the list of the functions
in URL rules that are matching
the particular file is also going
to be extracted in a list.
File path and the source
is going to be Yalda.
It is just like an indicator
to be able to collect the information
from database.
Embedded files, the list of objects
that are embedded in the file
also is going to be displayed to you.
PE sections in detail
is going to be there.
Parent MD5 and parent file path,
if the file is embedded
and is being analyzed,
then you are going to have a link
to the parent.
So let's say
that you are interested to
like you are analyzing the
obtained results and you are interested
to extract a PDF file
that has embedded objects in it.
It has also malicious domains
and it is
malicious with a stability of five.
So you could go ahead and apply a filter
and it's going to extract the data for you.
So just based on the indicators,
it is possible that you
select exactly what you are looking for.
It's a good source
for generating YARA rules
because it has detailed information about it
and if it is malicious, also it's going
to have strings out of it and PE sections
so you can use it as a good
source for writing your
YARA rules.
It's a
smart feed to Cocoa
Sandbox. Again, you don't want to submit
everything to Cocoa Sandbox, but if
there is a specific malicious
characteristic that you are interested in,
you can go ahead and start
feeding that category
to Sandbox
to see what is happening next.
YALDA
Architecture
Basically,
the architecture could be
categorized in four sections.
File,
you submit it,
then extract files,
it starts extracting files,
and then it starts analyzing and
scanning every single file
that has been extracted, and then
inserts data in database.
Extracting files is done
so mainly
what you need to have is
you need to have a directory
in your Linux box and dump
all of the subfolders and
files that you want, any type.
It could be compressed, it could be JSON,
it could be encoded
MIME, and
what it does, it goes through each single
one of them and it starts extracting
the files.
If it is
compressed file, it's going to uncompress it.
If it is having many
different subfolders, like what
my boss initially passed me, it's
going to walk through each single
folder and extract every
single file, and
again, if it is in
mail format, it's going to
download the mail attachments.
So you're going to get a list of smaller
files, and then for each file
it's going to apply foremost in it.
Now, when foremost
gets applied, it's going to have
sort of like children, embedded objects
in it. And not only it's going to
analyze the parent's file, but
it's going to analyze all the
children of the file, and each
one of them is going to be sent to the decoder
and it's going to get analyzed through
Yalda.
Decoding files.
Let me just put a pause
here and see,
like,
just want to ask you, how many of you
do automated process for
extracting information
of a file?
So the first thing that you would
do is manual
analysis. It's just like you have to
know what you are looking for,
and you have to know what is this malicious
file doing. And then you would need
to have a couple of samples to see what
is common between these.
The decoder part is actually written based
on the same thing here, that I
told you. So, based on
the file type, how Yalda
is designed, based on the file type, it's
going to apply,
it's going to send the file
to a set of decoders,
and then if there is a match with one
of the decoders or more, it's going to
just like flag it and go further
and see what could be extracted from it.
If there is any domain or
URL that could be matched, it's going to
extract it, and then it's going to flag the file.
Now,
the decoders, the way that they are written
is they,
as I said,
sort of like manual analyzer, detects
the malware, detects the campaign, and
starts analyzing them, one by one
sort of, and see what is the common
part that is being repeated.
When you know that part, then you can start
automation, and you can write the decoder
for it. That's how it's done, actually,
in Yalda.
Analyze known malicious samples
from fingerprints.
CVE2017
0199
Who heard of it?
Yeah, cool.
So, we know that it's
an RTF file with embedded
OLE object, and
it has a link,
like,
something like this, and then
analyzing all of them, we
can just extract the
comment section out of it,
because you see that it has a
link to download the embedded doc,
and what it does is actually,
as you see, this is the common part.
So if you start
parsing the file,
and write a regex
for it to see if
it is RTF file, go ahead and just
analyze it and see
if there is a pattern match
with this regex, then you could extract
the URL, just as simple as this.
This is, like, an analysis
that is done in one of the RTF files.
So, these are the information
that Yalda extracted.
The MD5, SHA1, SHA256,
size of file, magic
literal, file type, file name,
and then the severity is 5,
the highest severity, and it's
flagged as malicious. This is the
domain that has been extracted from this
particular file. There is no
piece section, of course. The source is
Yalda mining data. This is just an
indicator that I used.
And vtinfo,
if you want to have information,
this is, like, the AVN genes that are
able to detect it, and
paramelling to get detailed
information from VirusTotal.
URL list, it's empty because
I didn't apply any URL rules on it.
This is actually the dictionary
format of the output of Yalda,
and it's just, like, everything that I just told
you, it is going to be in dictionary format
and it's going to be inserted in database.
If you're interested to have it in JSON
format, it could be done easily, and you can
just have the information on Splunk.
When I talk
to analysts, usually they say
that, yeah, detecting a malware
is just like a chain of detection,
and usually it's just not
just one file, it's just a combination
of steps that is being done.
So it's really important to detect
the first chain, and then
start analysing it.
So there was
actually an email campaign that
was going on, and it was sending
PDF attachments, and
none of the AVN genes were able to detect
it. When it was
analysed, we found out that there is
a URL list embedded
in the PDF file, the URL
downloads JavaScript,
JavaScript has a link to a
URL, URL downloads
another executable file, so it's
just like a chain, right? But how
about if we just
detect the first chain, and just
understand that, okay, this is a
PDF file, and it is matching
with this particular chain.
Then we have the flag, right? Then we have
the flag that this is suspicious, at least.
And then we even can
extract the URL that is the first
URL that it's pointing.
So that we can just extract it,
have the information, and if you want, you can send it
to Kuku Sandbox for detailed information.
Applying
Yara rules. So I am not
giving you any Yara rules
myself, but Yalda is able to
apply
Yara rules on the given
files. There is just like
sort of like
enable, disable line
that you need to enable or disable it in the
config file, and then you would need to
pass some of the Yara rules
that interest you based
on your needs in a directory.
It goes to that directory if it is
enabled, and then applies every single
Yara rule to the file.
And if there is a match with one of the functions,
it's going to extract the file
and it's going to show you as a list
of the files that has been
extracted. So according
to the match, and according to the
number of functions that are
similar, it's going to get flagged as
malicious, suspicious, or clear
with appropriate
severity. Yalda
scoring is actually the brain in Yalda
because everything so far that
I told you, you should know
that there is a malicious file, or
there is a malicious campaign, and then
you have to write the decoder, so there are
none. It's not like
something new get detected. But
this method is somehow
based on similarities.
And when I
find, when actually Yalda finds a file
that is malicious, then it starts
clustering the strings.
So it applies clusters
on strings, and also
if it is executable and it's detected
as malicious, it's getting the
PE sections, the name
section of it, and then Shannon
entropy, and it clusters the information.
So it's something like this.
See?
The file
is detected as malicious. I'm applying
strings, and
get all of the strings out of it
and also a Unicode
string
out of it,
the Unicode strings, and then making a
list and cluster the information in
another collection.
There is
a file that I have no idea about.
Like, none of my decoders were able to
detect it, and none of the URLs were
able to detect it. Still, I apply the
strings on it. And now, based on
the similarities that I find with
these strings, the list of strings
found here, and the list of malicious
strings in my database, I
start scoring it.
How many files have been detected
malicious that are having similar
strings as the list of this file?
So if the score goes higher,
I would say that, yeah, it is suspicious.
It needs more
analysis.
So it's based on the similarities.
Same thing is applied for
the name in p-sections and
Shannon entropy. Name of p-sections,
if the file is executable,
and if it is malicious, I'm
getting it, and then I convert
it to SHA-1.
And also, the
entropy, the Shannon entropy, I'm just
collecting these two, and I'm using
an AND logic
for these two, and the malicious hash.
And all of this information
are being clustered somewhere.
Now it's just sort of like a scaling,
again, scaling of the file
that right now I'm seeing, and
see what are the similarities. If it is
executable, then what are the similarities
between these two? And
based on the similarities and the scoring
that I set a threshold, then
I'm going to flag it as suspicious.
Now, whitelisting.
Who
handles
feeds?
So you guys
know that how life is when you have
false positives, right?
And you start using all of
the possible scenarios
and all of the cases for whitelisting.
You're going to have manual whitelisting,
you're going to have a lot of different
sources, and still you'll have some false
positives, right?
To avoid such a thing in Yalda,
I started saying that, okay, fine,
I'm going to use the traditional way.
I'm going to use a log
file for
whitelisting. But at the same time,
how about just
clustering the good guys instead of
just clustering bad guys?
So how about whitelisting
the clear files, the ones that I
know that they are clear and they are not malicious?
And then extract that
information for
using it as whitelist.
So as Yalda is running
more samples, it's going to extract
more bad stuff
and good stuff at the same time, right?
So the whitelisting is getting
stronger as the detection of
malicious
hashes also gets
stronger.
So as I said, I'm just
applying strings on
clear files and
clustering it, and also I'm clustering
the PE section names and
Shannon entropy of executable
files that are known as clear.
And then I'm using this information
sort of to whitelist
for
having more
accurate results.
Let's have a demonstration.
So here I'm having three files in
the folder and
it starts analyzing the file.
It extracts the
files and then in this
case it's downloading the attachments
and you see that there are a list
of domain lists that they are all malicious
and it's extracting it. Also
it goes and analyzes
each single file, like in detail, if
there is any embedded object, it is going
to extract it. Like in this case,
it does have embedded objects, so it just
lists the list of embedded objects
here, like embedded files.
And not only it has the
detailed information of the file itself
but it also
analyzes the embedded
file in it and it gives you
information with a link to the parent.
And everything
is being inserted in the database,
as I mentioned.
So Yalda sort of like
minimizes false
positive or better
to be said, it has better correlation.
So the way
that we are doing Yalda scoring,
we get more accurate results
because as we are analyzing
more samples, we are getting
stronger results.
Same thing for automated
whitelisting, that we are
getting better results as we start
analyzing more samples.
And then this clustering
data, again, we are having better results
and then categorizing algorithm is
somehow that
I'm not saying that if I
don't know, like it doesn't match with
my decoders, then I'm not going
to say 100% it's malicious
but it has enough
high severity that you can select
and say that, okay, this is suspicious
and there is a flag going on, so I better
start analyzing this set of files
with this criteria, because you have 20
indicators that you can select out of it.
This is the GitHub, so you can
go ahead and start
downloading and using it.
It's in Fidelis GitHub.
So there
are three folders, like bin,
source, Yara rules. Yara rules is a
place that you place your
Yara rules if you want to. If not,
you don't have to worry about it.
Bin has all the
modules and functions, as you
see here. You can even go ahead
it's just like very straight format
and you can just go ahead and if you want
to add more decoders, you can do that.
And
the config file is something that I go
through it in detail. It's very
simple, it's just like a couple of lines that
you need to specify the directories in it
and when you want to run it, run
this guy, Yalda file analyzer.
So you would need to install the
required Python modules
and you would need
these are the Python modules that you would need.
And this is the
config file. You just go ahead and open it
and start
just adding the directories
that you want or enable or disable.
Like if you don't want to have virus
total information or if you don't want to have
sort of like Yara
rule to be applied to it, you can
say that zero, just like
disable it. If you want to
sort of like have a printout, you can have
debug one or you can just make it
zero. And of course, because I'm using
MongoDB, you would need to
have MongoDB
installed and just like specify
the criteria that you're looking for.
And to run it, you
go to source and you start running
its Yalda file analyzer
and it starts analyzing each single
file in detail and passes you the information.
I'd like to talk
to... thanks to
John Bamenak, my
boss, Hardik Modi
and Chad Robertson
and Jason Reeves. They
actually gave me really great feedback
for improving Yalda
and
really appreciate all of your feedback.
So if
you need to contact me or if you
have any questions, this is my email address
or Twitter.
Thank you so much.
If you have any questions or
feedback...