WEBVTT 00:00.000 --> 00:01.660 We're gonna get started right away. 00:01.660 --> 00:05.940 It is my pleasure to introduce an old friend of ours, Ghida Liabari. 00:05.960 --> 00:08.780 Thank you so much. 00:08.780 --> 00:09.920 Thanks, everyone. 00:10.000 --> 00:14.640 Thanks for your patience, first of all, and thanks for coming to my presentation. 00:14.780 --> 00:17.300 I'm going to talk about Yalda. 00:17.520 --> 00:24.200 It's an automated tool that I wrote recently, one of my recent projects, and I thought to share it with you. 00:24.700 --> 00:31.520 I work for Fidelity Cyber Security, and I'm going to talk about Yalda. 00:31.560 --> 00:40.080 So, we are going to talk about having an introduction of Yalda, how was it created, and what was the object of it. 00:40.080 --> 00:44.780 Then, how could you use Yalda, and what it could get used. 00:44.900 --> 00:59.920 Then, we are going to have an overview of architecture of Yalda, a quick demonstration, and a GitHub that you can go ahead and download it, and start using it on your own files. 00:59.920 --> 01:09.200 Using it is actually pretty much simple, it's just going and clicking on a file to configure and add some directories to it. 01:11.840 --> 01:13.840 So, introducing Yalda. 01:16.680 --> 01:30.520 The motivation was basically creating a tool that analyzes files and gives you some indicators that saves time for repeating the job for each single file. 01:30.520 --> 01:42.880 And also, it was just for fully understanding what the file does, and creating automated bulk intelligence collection. 01:43.520 --> 01:47.980 And sort of being independent of other sources, third parties. 01:47.980 --> 01:50.720 So, that was the motivation. 01:53.020 --> 02:00.980 So, that guy in the back, John Baminak, he actually is my boss, and he passed me a project to start with. 02:00.980 --> 02:24.100 So, he started giving me some files, massive files, I mean millions of files in directories, and they were all compressed, and they were in JSON format, each one of them had a dictionary inside it, and then the dictionary had information about the emails, 02:24.840 --> 02:28.740 like each dictionary was corresponding to a malicious email. 02:28.740 --> 02:38.220 It has all the key and values, of course, for that one, and it has some mime-encoded keys that the values were there. 02:38.220 --> 02:41.220 If you decode them, then you would be able to download the files. 02:41.220 --> 02:43.480 So, that is how Yalda was born. 02:43.480 --> 03:01.920 I started analyzing them manually, and then figured out that, yeah, campaign from the same sort of campaigns, although the format and everything is a little bit different, but the fingerprint and the way that they are introducing stuff is almost the same. 03:01.920 --> 03:04.280 And I figured out that it could get automated. 03:04.280 --> 03:05.840 This is just like an example. 03:05.840 --> 03:09.700 By just looking at it, I found out that this just doesn't make sense. 03:09.700 --> 03:11.040 This is malicious. 03:11.100 --> 03:14.460 I mean, I would be able to just say by a glance that this is malicious. 03:14.460 --> 03:15.620 Look at the from. 03:15.700 --> 03:19.540 It's coming from Russia, and it's saying that USPS ground. 03:19.540 --> 03:20.560 I mean, come on. 03:23.020 --> 03:38.640 And so, the analyzing JSON file, each dictionary, this is just like the process of automation that I would get the base 64 encoded MIME, which was the key, and then the value decoded it, and getting a couple of files. 03:38.640 --> 03:43.660 The most important one was the file that I was getting that had the attachments. 03:43.700 --> 03:52.540 And then, looking at the attachments, this was, again, for a particular campaign, it was almost the same. 03:52.540 --> 03:54.240 Like, the domains were the same. 03:54.240 --> 03:57.000 The domains were different, but the structure was the same. 03:57.000 --> 04:01.560 So, I started writing an automated tool for extracting the information. 04:01.560 --> 04:06.790 The steps was pretty much like following whatever I'd done manually. 04:07.120 --> 04:11.760 It was... yeah, it works at least. 04:11.860 --> 04:25.940 So, I was getting the JSON MIME, analyze keys, analyze from address, analyze the body, get base 64 encoded data, decode it, download attachments, analyze files, and extract domains. 04:26.220 --> 04:31.360 So, that's how YALDA started working. 04:31.400 --> 04:38.960 So, manual analysis, non-process for automation, automating it, YALDA, and then adding more and more to it. 04:39.020 --> 04:40.500 YALDA was born. 04:41.060 --> 04:51.480 So, as I was extracting information, then I started just talking to analysts and then asking, what do you guys do for getting more information? 04:51.480 --> 04:56.580 And then I made YALDA somehow that it does all of this stuff. 04:56.580 --> 05:07.400 Automated Bulk Intelligence Collection, it's a file scanner and analyzer, extracts data, cluster files, and it categorizes files. 05:10.180 --> 05:17.540 So, Automated Bulk Intelligence Collection Tool, it has an automated process of analyzing files. 05:17.540 --> 05:25.200 It applies intelligence in collected data and clusters files based on the similarities. 05:26.480 --> 05:29.060 It's a file scanner and analyzer. 05:29.060 --> 05:31.940 YALDA scans and analyzes files. 05:31.940 --> 05:35.160 It collects detailed information of the file. 05:35.160 --> 05:40.200 About 20 indicators are being extracted from each single file. 05:40.200 --> 05:41.720 And it makes it possible... 05:41.720 --> 05:44.200 How many of you guys are analysts here? 05:44.940 --> 06:04.260 Okay, so it makes it possible for you guys, instead of hours of doing sort of like analyzes, you just run YALDA on the set of files, let's say even like a million samples that you have, and then it's going to analyze each single one of them and it's going to give you some indicators, 06:04.260 --> 06:06.860 like 20 indicators is going to be extracted. 06:06.860 --> 06:15.900 Then based on whatever you are looking for and whatever interests you, it is going to extract the information that is valuable for you guys. 06:15.900 --> 06:19.700 So you can just like apply your filter and you can extract the information. 06:21.900 --> 06:30.820 It also extracts data, like it collects embedded objects and also it collects malicious URL and domains. 06:32.960 --> 06:35.340 And it categorizes files. 06:35.340 --> 06:40.220 There is like a flag and severity that in combination they are being used. 06:40.220 --> 06:59.260 So depending on the algorithm that is being applied to the file and the founding of the file, a file could be categorized as clear with a low severity like a 1 and 2, suspicious with higher severity, and malicious, of course, higher severity. 06:59.680 --> 07:01.400 It also clusters data. 07:01.400 --> 07:13.490 It gets the... if there is a malicious hash that it detects, it's going to cluster the strings out of the file and it's going to insert it in a different collection. 07:13.880 --> 07:18.640 Same applies if it is an executable file that is malicious. 07:18.640 --> 07:29.530 It is going to get the names of P sections, convert it to SHA-1, and in combination of the hash and channel entropy is going to use it. 07:31.340 --> 07:35.510 Now, what to do with Yalda and when to use it. 07:35.510 --> 07:50.630 So if you want to have a feed, a collection of feeds that you could use it, you could go ahead and just submit files to Yalda and it could be used for extracting MD5 feed, domain feed, URL feed, all of them malicious. 07:51.770 --> 07:58.810 Also, as I said, there are 20 indicators that are being extracted from each single file. 07:58.810 --> 08:01.850 Depending on the file type, some of them might be empty. 08:01.850 --> 08:06.870 Like , for example, if there is no P sections in it, then it's going to be empty. 08:06.870 --> 08:14.570 However, you are able to get all of these, like MD5, SHA-1, SHA-256, similar MD5. 08:14.590 --> 08:25.250 If there are some malicious MD5s that are similar to this particular file that you are analyzing, it is going to give you a list of similar MD5s. 08:25.250 --> 08:31.950 File type, if there is any malicious URL that has been extracted from the file, it is going to be given to you. 08:31.950 --> 08:36.470 Magic literal source, VirusTotal information. 08:36.470 --> 08:47.550 This is basically, just gives you a paramount link to VirusTotal page, if it exists in VirusTotal with the number of positive ABN genes that are able to detect it. 08:47.590 --> 08:50.370 You don't have to have the key or use VirusTotal. 08:50.370 --> 08:51.490 You can just disable it. 08:51.490 --> 08:52.890 It's not mandatory. 08:53.070 --> 08:55.550 Severity and flag are being used together. 08:55.550 --> 09:01.930 Severity is like clear, or it could be suspicious or malicious. 09:01.930 --> 09:04.910 And then the flag, it could be from 1 to 5. 09:04.910 --> 09:08.850 1 stands for clear, and 5 stands for very malicious. 09:08.850 --> 09:13.470 So it just sort of gives you a scope of how the file is analyzed. 09:13.510 --> 09:20.930 File name, ingest time, the time that it got analyzed and got inserted in database. 09:20.930 --> 09:23.510 Domain list, if it has any malicious domain. 09:23.510 --> 09:34.350 URL list, I'm going to go through it in detail, but the list of the functions in URL rules that are matching the particular file is also going to be extracted in a list. 09:34.550 --> 09:37.610 File path and the source is going to be Yalda. 09:37.610 --> 09:43.310 It is just like an indicator to be able to collect the information from database. 09:43.310 --> 09:49.150 Embedded files, the list of objects that are embedded in the file also is going to be displayed to you. 09:49.150 --> 09:52.950 PE sections in detail is going to be there. 09:53.230 --> 10:01.170 Parent MD5 and parent file path, if the file is embedded and is being analyzed, then you are going to have a link to the parent. 10:02.810 --> 10:14.430 So let's say that you are interested to like you are analyzing the obtained results and you are interested to extract a PDF file that has embedded objects in it. 10:14.430 --> 10:19.890 It has also malicious domains and it is malicious with a stability of five. 10:19.890 --> 10:23.970 So you could go ahead and apply a filter and it's going to extract the data for you. 10:23.970 --> 10:30.470 So just based on the indicators, it is possible that you select exactly what you are looking for. 10:30.650 --> 10:45.530 It's a good source for generating YARA rules because it has detailed information about it and if it is malicious, also it's going to have strings out of it and PE sections so you can use it as a good source for writing your YARA rules. 10:47.170 --> 10:50.210 It's a smart feed to Cocoa Sandbox . 10:50.210 --> 11:05.730 Again, you don't want to submit everything to Cocoa Sandbox, but if there is a specific malicious characteristic that you are interested in, you can go ahead and start feeding that category to Sandbox to see what is happening next. 11:07.670 --> 11:17.210 YALDA Architecture Basically, the architecture could be categorized in four sections. 11:17.210 --> 11:31.870 File, you submit it, then extract files, it starts extracting files, and then it starts analyzing and scanning every single file that has been extracted, and then inserts data in database. 11:34.210 --> 11:47.690 Extracting files is done so mainly what you need to have is you need to have a directory in your Linux box and dump all of the subfolders and files that you want, any type. 11:47.690 --> 11:58.270 It could be compressed, it could be JSON, it could be encoded MIME, and what it does, it goes through each single one of them and it starts extracting the files. 11:58.950 --> 12:03.090 If it is compressed file, it's going to uncompress it. 12:03.090 --> 12:20.670 If it is having many different subfolders, like what my boss initially passed me, it's going to walk through each single folder and extract every single file, and again, if it is in mail format, it's going to download the mail attachments. 12:20.670 --> 12:27.430 So you're going to get a list of smaller files, and then for each file it's going to apply foremost in it. 12:27.470 --> 12:33.530 Now, when foremost gets applied, it's going to have sort of like children, embedded objects in it. 12:33.530 --> 12:47.710 And not only it's going to analyze the parent's file, but it's going to analyze all the children of the file, and each one of them is going to be sent to the decoder and it's going to get analyzed through Yalda. 12:49.070 --> 12:50.610 Decoding files. 12:50.990 --> 13:03.930 Let me just put a pause here and see, like, just want to ask you, how many of you do automated process for extracting information of a file? 13:05.130 --> 13:09.510 So the first thing that you would do is manual analysis. 13:09.510 --> 13:15.830 It's just like you have to know what you are looking for, and you have to know what is this malicious file doing. 13:15.830 --> 13:20.330 And then you would need to have a couple of samples to see what is common between these. 13:20.390 --> 13:25.290 The decoder part is actually written based on the same thing here, that I told you. 13:25.290 --> 13:44.570 So, based on the file type, how Yalda is designed, based on the file type, it's going to apply, it's going to send the file to a set of decoders, and then if there is a match with one of the decoders or more, it's going to just like flag it and go further and see what could be extracted from it. 13:44.570 --> 13:50.790 If there is any domain or URL that could be matched, it's going to extract it, and then it's going to flag the file. 13:51.050 --> 14:08.070 Now, the decoders, the way that they are written is they, as I said, sort of like manual analyzer, detects the malware, detects the campaign, and starts analyzing them, one by one sort of, and see what is the common part that is being repeated. 14:08.130 --> 14:12.770 When you know that part, then you can start automation, and you can write the decoder for it. 14:12.770 --> 14:14.910 That's how it's done, actually, in Yalda. 14:16.290 --> 14:20.090 Analyze known malicious samples from fingerprints. 14:21.610 --> 14:25.970 CVE2017 0199 Who heard of it? 14:26.350 --> 14:27.690 Yeah, cool. 14:27.690 --> 14:50.190 So, we know that it's an RTF file with embedded OLE object, and it has a link, like, something like this, and then analyzing all of them, we can just extract the comment section out of it, because you see that it has a link to download the embedded doc, 14:50.190 --> 14:54.190 and what it does is actually, as you see, this is the common part. 14:54.190 --> 15:12.550 So if you start parsing the file, and write a regex for it to see if it is RTF file, go ahead and just analyze it and see if there is a pattern match with this regex, then you could extract the URL, just as simple as this. 15:14.150 --> 15:18.510 This is, like, an analysis that is done in one of the RTF files. 15:18.510 --> 15:21.870 So, these are the information that Yalda extracted. 15:22.190 --> 15:33.150 The MD5, SHA1, SHA256, size of file, magic literal, file type, file name, and then the severity is 5, the highest severity, and it's flagged as malicious. 15:33.190 --> 15:37.130 This is the domain that has been extracted from this particular file. 15:37.130 --> 15:38.970 There is no piece section, of course. 15:38.970 --> 15:41.070 The source is Yalda mining data. 15:41.070 --> 15:43.430 This is just an indicator that I used. 15:43.830 --> 15:55.550 And vtinfo, if you want to have information, this is, like, the AVN genes that are able to detect it, and paramelling to get detailed information from VirusTotal. 15:55.550 --> 16:00.410 URL list, it's empty because I didn't apply any URL rules on it. 16:00.410 --> 16:10.610 This is actually the dictionary format of the output of Yalda, and it's just, like, everything that I just told you, it is going to be in dictionary format and it's going to be inserted in database. 16:10.650 --> 16:17.010 If you're interested to have it in JSON format, it could be done easily, and you can just have the information on Splunk. 16:19.290 --> 16:32.530 When I talk to analysts, usually they say that, yeah, detecting a malware is just like a chain of detection, and usually it's just not just one file, it's just a combination of steps that is being done. 16:32.530 --> 16:38.270 So it's really important to detect the first chain, and then start analysing it. 16:39.290 --> 16:49.270 So there was actually an email campaign that was going on, and it was sending PDF attachments, and none of the AVN genes were able to detect it. 16:49.270 --> 17:06.150 When it was analysed, we found out that there is a URL list embedded in the PDF file, the URL downloads JavaScript, JavaScript has a link to a URL , URL downloads another executable file, so it's just like a chain, right? 17:06.150 --> 17:16.550 But how about if we just detect the first chain, and just understand that, okay, this is a PDF file, and it is matching with this particular chain. 17:18.670 --> 17:20.310 Then we have the flag, right? 17:20.310 --> 17:23.030 Then we have the flag that this is suspicious, at least. 17:23.030 --> 17:29.010 And then we even can extract the URL that is the first URL that it's pointing. 17:29.010 --> 17:35.250 So that we can just extract it, have the information, and if you want, you can send it to Kuku Sandbox for detailed information. 17:35.990 --> 17:37.430 Applying Yara rules. 17:37.430 --> 17:47.350 So I am not giving you any Yara rules myself, but Yalda is able to apply Yara rules on the given files. 17:47.350 --> 18:02.930 There is just like sort of like enable, disable line that you need to enable or disable it in the config file, and then you would need to pass some of the Yara rules that interest you based on your needs in a directory. 18:02.930 --> 18:08.690 It goes to that directory if it is enabled, and then applies every single Yara rule to the file. 18:08.690 --> 18:17.570 And if there is a match with one of the functions, it's going to extract the file and it's going to show you as a list of the files that has been extracted. 18:18.230 --> 18:29.790 So according to the match, and according to the number of functions that are similar, it's going to get flagged as malicious, suspicious, or clear with appropriate severity. 18:30.250 --> 18:43.130 Yalda scoring is actually the brain in Yalda because everything so far that I told you, you should know that there is a malicious file, or there is a malicious campaign, and then you have to write the decoder, so there are none. 18:43.130 --> 18:46.390 It's not like something new get detected. 18:46.390 --> 18:50.430 But this method is somehow based on similarities. 18:50.710 --> 18:58.350 And when I find, when actually Yalda finds a file that is malicious, then it starts clustering the strings. 18:58.350 --> 19:13.090 So it applies clusters on strings, and also if it is executable and it's detected as malicious, it's getting the PE sections, the name section of it, and then Shannon entropy, and it clusters the information. 19:13.090 --> 19:14.390 So it's something like this. 19:14.390 --> 19:15.010 See? 19:15.950 --> 19:18.390 The file is detected as malicious. 19:18.390 --> 19:35.090 I'm applying strings, and get all of the strings out of it and also a Unicode string out of it, the Unicode strings, and then making a list and cluster the information in another collection. 19:36.190 --> 19:39.170 There is a file that I have no idea about. 19:39.170 --> 19:43.850 Like, none of my decoders were able to detect it , and none of the URLs were able to detect it. 19:43.850 --> 19:46.010 Still, I apply the strings on it. 19:46.010 --> 19:56.490 And now, based on the similarities that I find with these strings, the list of strings found here, and the list of malicious strings in my database, I start scoring it. 19:56.630 --> 20:03.110 How many files have been detected malicious that are having similar strings as the list of this file? 20:03.110 --> 20:07.050 So if the score goes higher, I would say that, yeah, it is suspicious. 20:07.050 --> 20:10.630 It needs more analysis. 20:10.630 --> 20:12.630 So it's based on the similarities. 20:12.630 --> 20:17.730 Same thing is applied for the name in p-sections and Shannon entropy. 20:17.730 --> 20:25.990 Name of p-sections, if the file is executable, and if it is malicious, I'm getting it, and then I convert it to SHA-1. 20:26.090 --> 20:36.910 And also, the entropy, the Shannon entropy, I'm just collecting these two, and I'm using an AND logic for these two, and the malicious hash. 20:36.910 --> 20:40.230 And all of this information are being clustered somewhere. 20:40.230 --> 20:47.730 Now it's just sort of like a scaling, again, scaling of the file that right now I'm seeing, and see what are the similarities. 20:47.730 --> 20:51.810 If it is executable, then what are the similarities between these two? 20:51.810 --> 20:58.630 And based on the similarities and the scoring that I set a threshold, then I'm going to flag it as suspicious. 21:00.510 --> 21:02.570 Now, whitelisting. 21:04.770 --> 21:08.450 Who handles feeds? 21:10.050 --> 21:15.030 So you guys know that how life is when you have false positives, right? 21:15.030 --> 21:21.550 And you start using all of the possible scenarios and all of the cases for whitelisting. 21:21.550 --> 21:28.370 You're going to have manual whitelisting, you're going to have a lot of different sources, and still you'll have some false positives, right? 21:28.750 --> 21:35.390 To avoid such a thing in Yalda, I started saying that, okay, fine, I'm going to use the traditional way. 21:35.390 --> 21:39.970 I'm going to use a log file for whitelisting. 21:39.970 --> 21:47.150 But at the same time, how about just clustering the good guys instead of just clustering bad guys? 21:47.150 --> 21:53.690 So how about whitelisting the clear files, the ones that I know that they are clear and they are not malicious? 21:53.790 --> 21:58.950 And then extract that information for using it as whitelist. 21:59.050 --> 22:07.090 So as Yalda is running more samples, it's going to extract more bad stuff and good stuff at the same time, right? 22:07.090 --> 22:16.290 So the whitelisting is getting stronger as the detection of malicious hashes also gets stronger. 22:16.830 --> 22:31.490 So as I said, I'm just applying strings on clear files and clustering it, and also I'm clustering the PE section names and Shannon entropy of executable files that are known as clear. 22:31.490 --> 22:40.890 And then I'm using this information sort of to whitelist for having more accurate results. 22:41.210 --> 22:43.410 Let's have a demonstration. 22:44.630 --> 22:51.370 So here I'm having three files in the folder and it starts analyzing the file. 22:51.370 --> 23:02.730 It extracts the files and then in this case it's downloading the attachments and you see that there are a list of domain lists that they are all malicious and it's extracting it. 23:02.730 --> 23:09.890 Also it goes and analyzes each single file, like in detail, if there is any embedded object, it is going to extract it. 23:09.890 --> 23:17.270 Like in this case, it does have embedded objects, so it just lists the list of embedded objects here, like embedded files. 23:17.310 --> 23:29.410 And not only it has the detailed information of the file itself but it also analyzes the embedded file in it and it gives you information with a link to the parent. 23:30.330 --> 23:34.170 And everything is being inserted in the database, as I mentioned. 23:35.070 --> 23:43.130 So Yalda sort of like minimizes false positive or better to be said, it has better correlation. 23:43.370 --> 23:55.370 So the way that we are doing Yalda scoring, we get more accurate results because as we are analyzing more samples, we are getting stronger results. 23:55.470 --> 24:03.550 Same thing for automated whitelisting, that we are getting better results as we start analyzing more samples. 24:03.570 --> 24:23.490 And then this clustering data, again, we are having better results and then categorizing algorithm is somehow that I'm not saying that if I don't know, like it doesn't match with my decoders, then I'm not going to say 100% it's malicious but it has enough high severity that you can select and say that, 24:23.490 --> 24:32.810 okay, this is suspicious and there is a flag going on, so I better start analyzing this set of files with this criteria, because you have 20 indicators that you can select out of it. 24:34.810 --> 24:40.750 This is the GitHub, so you can go ahead and start downloading and using it. 24:40.750 --> 24:42.430 It's in Fidelis GitHub. 24:51.480 --> 24:55.460 So there are three folders, like bin, source, Yara rules. 24:55.460 --> 24:59.780 Yara rules is a place that you place your Yara rules if you want to. 24:59.780 --> 25:02.100 If not, you don't have to worry about it. 25:02.300 --> 25:07.120 Bin has all the modules and functions, as you see here. 25:07.120 --> 25:14.620 You can even go ahead it's just like very straight format and you can just go ahead and if you want to add more decoders, you can do that. 25:15.160 --> 25:19.060 And the config file is something that I go through it in detail. 25:19.060 --> 25:28.480 It's very simple, it's just like a couple of lines that you need to specify the directories in it and when you want to run it, run this guy, Yalda file analyzer. 25:30.100 --> 25:38.600 So you would need to install the required Python modules and you would need these are the Python modules that you would need. 25:39.060 --> 25:40.840 And this is the config file. 25:40.840 --> 25:48.020 You just go ahead and open it and start just adding the directories that you want or enable or disable. 25:48.020 --> 25:59.700 Like if you don't want to have virus total information or if you don't want to have sort of like Yara rule to be applied to it, you can say that zero, just like disable it. 25:59.700 --> 26:05.320 If you want to sort of like have a printout, you can have debug one or you can just make it zero. 26:05.320 --> 26:14.580 And of course, because I'm using MongoDB, you would need to have MongoDB installed and just like specify the criteria that you're looking for. 26:15.200 --> 26:25.420 And to run it, you go to source and you start running its Yalda file analyzer and it starts analyzing each single file in detail and passes you the information. 26:25.420 --> 26:35.640 I'd like to talk to... thanks to John Bamenak, my boss, Hardik Modi and Chad Robertson and Jason Reeves. 26:35.640 --> 26:44.860 They actually gave me really great feedback for improving Yalda and really appreciate all of your feedback. 26:45.540 --> 26:52.000 So if you need to contact me or if you have any questions, this is my email address or Twitter. 26:53.520 --> 26:55.180 Thank you so much. 26:55.180 --> 26:57.960 If you have any questions or feedback...