We'll get started right away.
It is my pleasure to introduce an old friend of ours,
Gita Zia-Bari.
Thank you so much.
Thanks, everyone.
Thanks for your patience, first of all,
and thanks for coming to my presentation.
I'm going to talk about Yalda.
It's an automated tool that I wrote recently,
one of my recent projects,
and I thought to share it with you.
I work for Fidelity Cyber Security,
and I'm going to talk about Yalda.
So we are going to talk about having an introduction of Yalda,
how was it created, and what was the object of it.
Then how could you use Yalda, and what it could get used.
Then we are going to have an overview of architecture of Yalda,
a quick demonstration,
and a GitHub that you can go ahead and download it
and start using it.
You can start using it on your own files,
and using it is actually pretty much simple.
It's just like going and just clicking on a file
to configure and add some directories to it.
So introducing Yalda.
The motivation was basically creating a tool
that analyzes files and gives you some indicators
that saves time
for
sort of like repeating the job for each single file.
And also, it was just like for sort of like fully understanding
what the file does,
and getting, creating automated bulk intelligence collection.
And sort of like being independent of other sources, third parties.
So that was the motivation.
So that guy in the back, John Baminat,
he actually is my boss,
and he passed me a project to start with.
So he sort of started giving me some sort of files,
in massive files,
I mean millions of files in directories.
And they were all compressed,
and they were in JSON format.
Each one of them, it had a dictionary inside it,
and then the dictionary had information about the emails.
And each dictionary has a dashed star on it,
generally was corresponding to a malicious email.
It has all the key and values, of course, for that one.
And it has some MIME-encoded keys that the values were there.
If you decode them, then you would
be able to download the files.
So that is how Yalda was born.
I started analyzing them manually,
and then figured out that, yeah, campaign
from the same sort of campaigns, although the format
and everything is a little bit different,
but the fingerprint and the way that they are introducing
stuff is almost the same.
And I figured out that it could get automated.
This is just like an example.
By just looking at it, I found out
that this just doesn't make sense.
This is malicious.
I mean, I would be able to just say by a glance
that this is malicious.
Look at the from.
It's coming from Russia, and it's saying that USPS ground.
I mean, come on.
And so the analyzing
JSON file, each dictionary, this is just
like the process of automation that I would get the base 64
encoded MIME, which was the key, and then the value,
decoded it, and getting a couple of files.
The most important one was the file
that I was getting that had the attachments.
And then looking at the attachments,
this was, again, for a particular campaign
that it was almost the same.
The domains were the same.
The domains were different, but the structure was the same.
So I started writing an automated tool
for extracting the information.
The steps was pretty much like following
whatever I'd done manually.
It was, yeah, it works at least.
So I was getting the JSON MIME, analyze keys,
analyze from address, analyze the body,
get basic 64 encoded data, decode it, download attachments,
analyze files.
and extract domains. So that's how Yalda started working. So manual analysis,
known process for automation, automating it, Yalda. And then adding more and more
to it. Yalda was born. So as I was extracting information then I started
just talking to analysts and then asking what do you guys do for getting more
information and then I made the Yalda somehow that it does all of this stuff
Automated bulk intelligence collection, it's a file scanner and analyzer, extract
data, cluster files and it categorizes files.
So automated bulk intelligence collection tool, it has an automated
process of analyzing files. It's applied intelligence in
collections. We would say it's used in recorduckle analysis and it's used in
data and clusters files based on the similarities. It's a file scanner and
analyzer. Yalda scans and analyzes files. It collects detailed information of the
file. About 20 indicators are being extracted from each single file and it
makes it possible. How many of you guys are analysts here? Okay, so it makes it
possible for you guys instead of hours of doing sort of like analyzes, you just
run Yalda on the set of files. Let's say even like a million samples that you
have and then it's going to analyze each single one of them and it's going to
give you some indicators. Like 20 indicators is going to be extracted. Then
based on whatever you are looking for and whatever interests you, it is going
to extract the information that is valuable for you guys. So you
can just like apply a filter and you can extract the information. It also
extracts data, like it collects embedded objects and also it collects malicious
URL and domains. And it categorizes files, there is like a flag and
severity that in combination they are being used. So depending on the
algorithm that is being applied to the file and the founding of the file and
A file could be categorized as clear with a low severity,
like a one and two, suspicious,
suspicious with higher severity,
and malicious, of course, higher severity.
It also clusters data.
It gets the, if there is any malicious hash that it detects,
it's going to cluster the strings out of the file
and it's going to insert it in a different collection.
Same applies if it is an executable file that is malicious.
It is going to get the names of P-sections,
convert it to SHA-1, and in combination of the hash
and Shannon entropy is going to use it.
Now, what to do with Yaldo and when to use it?
So if you want to have a feed, a collection of feeds
that you could use it, you could go ahead
and just, like, say,
submit files to Yaldo and it could be used
for extracting MD5 feed, domain feed, URL feed,
all of them malicious.
Also, as I said, like, there are 20 indicators
that are being extracted from each single file.
Depending on the file type, some of them might be empty.
Like, for example, if there is no P-sections in it,
then it's going to be empty.
However, you are able to get all of these,
like MD5, SHA-1, SHA-256, and so on.
Similar MD5.
If there are some malicious MD5s that
are similar to this particular file that you are analyzing,
it is going to give you a list of similar MD5s.
File type.
If there is any malicious URL that
has been extracted from the file,
it is going to be given to you.
Magic literal source of VirusTotal information,
this is basically just gives you a Parama link
to VirusTotal page if it exists in VirusTotal.
with the number of positive AV engines
that are able to detect it.
You don't have to have the key or use virus total,
you can just disable it, it's not mandatory.
Severity and flag are being used together.
Severity is like clear, or it could be suspicious
or malicious, and then the flag,
it could be from one to five.
One stands for clear and five stands for very malicious.
So it's just sort of like give you a scope
of how the file is analyzed.
File name, in this time, the time that it got analyzed
and got inserted in database.
Domain list, if it has any malicious domain.
Yara list, I'm going to go through it in detail,
but the list of the functions in Yara rules
that are matching the particular file
is also going to be extracted in a list.
File path, and the source is going to be Yalda.
It is just like as an indicator to be able
to connect to Yalda.
Collects the information from database.
Embedded files, the list of objects that are embedded
in the file also is going to be displayed to you.
PE sections in detail is going to be there.
Parent MD5 and parent file path,
if the file is embedded and is being analyzed,
then you are going to have a link to the parent.
So let's say that you are interested to,
like you are analyzing the obtained results
and you are interested to extract,
a PDF file that has embedded objects in it.
It has also malicious domains,
and it is malicious with a severity of five.
So you could go ahead and apply a filter
and it's going to extract the data for you.
So it's just based on the indicators.
It is possible that you select exactly
what you are looking for.
It's a good source for generating Yara rules
because it has detailed information about it.
And if it is malicious also,
it's going to have a strings out of it and PE sections.
So you can use it as a good source
for writing your Yara rules.
It's a smart feed to Cuckoo Sandbox.
Again, you don't want to submit everything to Cuckoo Sandbox,
but if there is a specific malicious characteristic
that you're interested,
you can go ahead and start feeding it that category
to Sandbox to see what is happening next.
Yalda architecture.
It's basically the architecture is,
could be categorized in file sections,
in four sections.
File, you submit it, then extract files.
It will start extracting files,
and then it starts analyzing and scanning
every single file that has been extracted,
and then inserts data in database.
Extracting files is done.
So mainly what you need to have is you have,
you need to have a directory in your Linux box
and dump all of the subfolders and files that you want.
Any type, it could be compressed, it could be JSON,
it could be encoded MIME.
And what it does, it goes through each single one of them
and it starts extracting the files.
If it is a compressed file, it's going to uncompress it.
If it is like having many different subfolders,
like what my boss initially passed me,
it's going to walk through each single folder
and extract every single file.
And again, if it has, if it is in mail format,
it's going to download the mail attachments.
So you're going to get a list of smaller files.
And then for each file, it's going to apply foremost in it.
Now, when foremost gets applied,
it's going to have sort of like children,
embedded objects in it.
And not only it's going to analyze the parent's file,
it's going to analyze all the children of the file.
And each one of them is going to be sent to the decoder
and it's going to get analyzed through Yalda.
Decoding files.
Let me just put a pause here and see,
just want to ask you,
how many of you do automated process
for extracting information of a file?
So the first thing that you would do is manual,
it's just like you have to know what you are looking for
and you have to know what is this malicious file doing.
And then you would need to have a couple of samples
to see what is common between these.
The decoder part is actually written
based on the same thing here that I told you.
So based on the file type, how Yalda is designed,
based on the file type, it's going to apply,
it's going to send the file to a set of decoders.
And then if there is a match with one of the decoders,
or more, it's going to just like flag it
and go further and see what could be extracted from it.
If there is any domain or URL that could be matched,
it's going to extract it
and then it's going to flag the file.
Now, the decoders, the way that they are written is,
they, as I said, like a sort of like manual analyzer,
detect the malware, detect the campaign,
and it starts analyzing them one by one, sort of,
and see what is the common part that is being repeated.
Then you know,
that part, then you can start automation
and you can write the decoder for it.
That's how it's done, actually, in Yalda.
Analyze non-malicious samples from fingerprints.
CVE-2017-0199.
Who heard of it?
Yeah, cool.
So, we know that it's an RTF file
with embedded OLE object, and it has a link,
like something like this.
And then analyzing all of them,
we can just like extract the comment section out of it,
because it will, like, you see that it has a link
to download the embedded doc.
And what it does is actually, as you see,
this is the comment part.
So, if you start sort of like parsing the file
and write a regex for it to see if it is RTF file,
go ahead and just like analyze it
and see if there is a pattern match with this regex.
Then you could extract the URL, just as simple as this.
This is like an analysis that is done in one of the RTF files.
So, these are the information that Yalda extracted.
The MD5 SHA-1 SHA-256, size of file,
magic literal, file type, file name.
And then the severity is 5, the highest severity,
and it's flagged as malicious.
This is the domain that has been extracted
from this particular file.
There is no P section, of course.
So, this is the domain that has been extracted
The source is Yalda mining data.
This is just an indicator that I used.
And VTINFO, if you want to have information,
this is like the avian genes that are able to detect it
and paramilling to get detailed information from VirusDota.
The URL list, it's empty because I didn't apply
any URL rules on it.
This is actually the dictionary format
of the output of Yalda.
And it's just like everything that I just told you.
It is going to be in dictionary format.
And it's going to be inserted in database.
If you're interested to have it in JSON format,
it could be done easily.
And you can just have the information on Splunk.
When I talk to analysts, usually they say that,
yeah, detecting a malware is just like a chain of detection.
And usually, it's just not just one file.
It's just combination of the steps that is being done.
So, it's really important to detect the first chain
and then start analyzing it.
So, there was actually an email campaign
that was going on and it was sending PDF attachments.
And none of the avian genes were able to detect it.
When it was analyzed, we found out that there is
a URL list embedded in the PDF file.
The URL downloads JavaScript.
JavaScript has a link to a URL.
URL downloads another executable file.
So, it's just like a chain, right?
But how about if you just like,
detect the first chain and just understand that,
okay, this is a PDF file
and it is matching with this particular chain.
Then we have the flag, right?
Then we have the flag that this is suspicious at least.
And then we even can extract the URL
that is the first URL that it's pointing.
So that we can just extract it, have the information,
and if you want, you can send it to Google Sandbox
for detailed information.
Applying URL rules.
I'm not giving you any URL rules myself,
but Yalda is able to apply URL rules on the given files.
There is just like, sort of like an enable, disable line
that you need to enable or disable it in the config file.
And then you would need to pass some of the URL rules
that interest you based on your needs in a directory.
It goes to that directory if it is enabled
and then applies every single URL rule to the file.
And if there is a match with one of the functions,
it's going to extract the file
and it's going to show you as a list
of the files that has been extracted.
So according to the match
and according to the number of functions that are similar,
it's going to get flagged as malicious, suspicious, or clear
with appropriate severity.
Yalda scoring is actually the brain in Yalda
because everything so far that I told you,
you should know that there is a malicious file
or there is a malicious campaign
and then you have to write the decoder.
So there are none.
It's not like something new get detected.
But this method is somehow based on similarities.
And when I find, when actually Yalda finds a file is malicious,
then it starts clustering the strings.
So it applies clusters on strings.
And also if it is executable and it's detected as malicious,
it's getting the PE sections,
the name section of it,
and then Shannon entropy.
And it clusters the information.
So it's something like this.
See?
The file is detected as malicious.
I'm applying strings
and get all of the strings out of it
and also a Unicode string out of it.
The Unicode strings
and then making a list
and cluster the information in another collection.
There is a file that I have no idea about.
Like none of my decoders were able to detect it
and none of the URLs were able to detect it.
Still I apply the strings on it.
And now based on the similarities
that I find with these strings,
the list of strings found here
and the list of malicious strings in my database,
I start scoring it.
How many files has been detected malicious
that are having similar strings
as the list of this file?
So if this score goes down,
if it goes higher,
I would say that yeah,
it is suspicious.
It needs more analysis.
So it's based on the similarities.
Same thing is applied for the name
in P sections and Shannon entropy.
Name of P sections,
if the file is executable
and if it is malicious,
I'm getting it
and then I convert it to SHA-1.
And also the entropy,
the Shannon entropy,
I'm just collecting these two
and I'm using an AND logic
for these two
and the malicious hash.
And all of this information
are being clustered somewhere.
Now it's just sort of like a scaling,
again,
scaling of the file
that right now I'm seeing
and see what are the similarities.
If it is executable,
then what are the similarities
between these two?
And based on the similarities
and the scoring that I set a threshold,
then I'm going to flag it as suspicious.
Now white listing.
Who handles feeds?
And yeah.
So you guys know that how life is
when you have false positives, right?
And you start using
all of the possible scenarios
and all of the cases for white listing.
You are going to have manual white listing.
You are going to have
a lot of different sources
and still you have some false positives, right?
So to avoid such a thing in Yalda,
I started saying that,
okay, fine,
I'm going to use the traditional way.
I'm going to use a log file
for white listing.
But at the same time,
how about just like clustering the good guys
instead of just clustering bad guys?
So how about white listing
the clear files,
the ones that I know that they are clear
and they are not malicious?
And then extract that information
for using it as white list.
So, you know,
as Yalda is running more samples,
it's going to extract more,
bad stuff and good stuff
at the same time, right?
So the white listing is getting stronger
as the detection of malicious hashes
also get stronger.
So as I said,
I'm just like applying a strings on clear file
and are clustering it.
And also I am clustering the PE section names
and Shannon entropy of executable files
that are known as clear.
And then I'm using this information
sort of to white list
and for having like more accurate results.
Let's have a demonstration.
So here I'm having three files in the folder
and it starts analyzing the file.
It extracts the files
and then in this case,
it's downloading the attachments
and you see that there are a list of domain list
that they are all malicious
and it's extracting it.
Also, it goes and analyze each single file
like in detail.
If there is any embedded object,
it is going to extract it.
Like in this case,
it does have embedded objects.
So it just like lists the list of embedded objects here,
like embedded files.
And not only it has the detailed information
of the file itself,
but it also analyzes the embedded file in it
and it gives you information
with a link to the parent.
And everything is being inserted
in the database as I mentioned.
So Yalda sort of like minimizes false positive
or better to be said,
it has better correlation.
So the way that we are doing Yalda scoring,
we get more accurate results
because as we are analyzing more samples,
we are getting like stronger results.
Same thing for automated white listing
that we are getting better results
as we start analyzing
more samples.
And then this clustering data,
again, we are having better results.
And then categorizing algorithm is somehow
that I'm not saying that if I don't know,
like it doesn't match with my decoders,
then I'm not going to say 100% it's malicious,
but it has enough high severity
that you can select and say that,
okay, this is suspicious
and there is a flag going on.
So I better start analyzing this set of files
with this criteria
because you have 20 indicators
that we can select out of it.
This is the GitHub.
So you can go ahead
and start downloading and using it.
It's in Fidelis GitHub.
So there are three folders
like bin, source, Yara rules.
Yara rules is a place
that you place your Yara rules if you want to.
If not, you don't have to.
You don't have to worry about it.
Bin has all the modules and functions,
as you see here.
You can even go ahead.
It's just like very straight format
and you can just go ahead
and if you want to add more decoders,
you can do that.
And the config file is something
that I go through it in detail.
It's very simple.
It's just like a couple of lines
that you need to specify the directories in it.
And when you want to run it,
run this guy, Yalda file analyzer.
You would need to install
the required Python modules
and you would need,
these are the Python modules
that you would need.
And this is the config file.
You just go ahead and open it
and start just adding the directories
that you want or enable or disable.
Like if you don't want to have
virus total information
or if you don't want to have
sort of like Yara rule
to be applied to it,
you can say that zero,
just like disable it.
If you want to sort of like have a printout,
you can have debug one
or you can just make it zero.
And of course,
because I'm using MongoDB,
you would need to have a MongoDB installed
and just like specify the criteria
that you're looking for.
And to run it,
you go to source
and you start running
its Yalda file analyzer
and it starts analyzing
each single file in detail
and passes you the information.
I'd like to talk to,
thanks to John
my boss,
Hardik Modi,
Chad Robertson
and Jason Reeves.
They actually gave me
really great feedbacks
for improving Yalda
and really appreciate
all of your feedbacks.
So if you need to contact me
or if you have any questions,
this is my email address
or Twitter.
And thank you so much.
If you have any questions
or feedback,
thank you.
Thank you.