[00:00.000 --> 00:01.660] We're gonna get started right away. [00:01.660 --> 00:04.820] It is my pleasure to introduce an old friend of ours, [00:04.820 --> 00:05.940] Ghida Liabari. [00:05.960 --> 00:08.780] Thank you so much. [00:08.780 --> 00:09.920] Thanks, everyone. [00:10.000 --> 00:12.040] Thanks for your patience, first of all, [00:12.040 --> 00:14.640] and thanks for coming to my presentation. [00:14.780 --> 00:17.300] I'm going to talk about Yalda. [00:17.520 --> 00:20.920] It's an automated tool that I wrote recently, [00:20.920 --> 00:22.200] one of my recent projects, [00:22.200 --> 00:24.200] and I thought to share it with you. [00:24.700 --> 00:27.560] I work for Fidelity Cyber Security, [00:29.500 --> 00:31.520] and I'm going to talk about Yalda. [00:31.560 --> 00:34.520] So, we are going to talk about [00:34.520 --> 00:36.920] having an introduction of Yalda, [00:36.920 --> 00:40.080] how was it created, and what was the object of it. [00:40.080 --> 00:42.500] Then, how could you use Yalda, [00:42.500 --> 00:44.780] and what it could get used. [00:44.900 --> 00:48.820] Then, we are going to have an overview of [00:48.820 --> 00:50.840] architecture of Yalda, [00:51.100 --> 00:52.800] a quick demonstration, [00:52.800 --> 00:56.000] and a GitHub that you can go ahead and download it, [00:56.000 --> 00:59.920] and start using it on your own files. [00:59.920 --> 01:02.720] Using it is actually pretty much simple, [01:02.720 --> 01:06.060] it's just going and clicking on a file [01:06.060 --> 01:09.200] to configure and add some directories to it. [01:11.840 --> 01:13.840] So, introducing Yalda. [01:16.680 --> 01:20.200] The motivation was basically creating a tool [01:20.200 --> 01:23.360] that analyzes files and gives you some indicators [01:23.360 --> 01:28.360] that saves time for repeating the job [01:28.360 --> 01:30.520] for each single file. [01:30.520 --> 01:35.660] And also, it was just for fully understanding [01:35.660 --> 01:37.860] what the file does, [01:37.860 --> 01:42.880] and creating automated bulk intelligence collection. [01:43.520 --> 01:46.880] And sort of being independent of other sources, [01:46.880 --> 01:47.980] third parties. [01:47.980 --> 01:50.720] So, that was the motivation. [01:53.020 --> 01:56.500] So, that guy in the back, John Baminak, [01:56.500 --> 01:58.000] he actually is my boss, [01:58.000 --> 02:00.980] and he passed me a project to start with. [02:00.980 --> 02:07.280] So, he started giving me some files, [02:07.280 --> 02:11.280] massive files, I mean millions of files in directories, [02:11.280 --> 02:12.960] and they were all compressed, [02:12.960 --> 02:15.280] and they were in JSON format, [02:15.280 --> 02:18.740] each one of them had a dictionary inside it, [02:18.740 --> 02:24.100] and then the dictionary had information about the emails, [02:24.840 --> 02:27.120] like each dictionary was corresponding [02:27.120 --> 02:28.740] to a malicious email. [02:28.740 --> 02:31.500] It has all the key and values, of course, for that one, [02:31.500 --> 02:35.100] and it has some mime-encoded keys [02:35.720 --> 02:38.220] that the values were there. [02:38.220 --> 02:39.140] If you decode them, [02:39.140 --> 02:41.220] then you would be able to download the files. [02:41.220 --> 02:43.480] So, that is how Yalda was born. [02:43.480 --> 02:45.800] I started analyzing them manually, [02:45.800 --> 02:47.960] and then figured out that, yeah, [02:47.960 --> 02:51.380] campaign from the same sort of campaigns, [02:51.380 --> 02:56.060] although the format and everything [02:56.060 --> 02:57.520] is a little bit different, [02:57.520 --> 03:00.580] but the fingerprint and the way that they are introducing stuff [03:00.580 --> 03:01.920] is almost the same. [03:01.920 --> 03:04.280] And I figured out that it could get automated. [03:04.280 --> 03:05.840] This is just like an example. [03:05.840 --> 03:07.160] By just looking at it, [03:07.160 --> 03:09.700] I found out that this just doesn't make sense. [03:09.700 --> 03:11.040] This is malicious. [03:11.100 --> 03:13.540] I mean, I would be able to just say by a glance [03:13.540 --> 03:14.460] that this is malicious. [03:14.460 --> 03:15.620] Look at the from. [03:15.700 --> 03:17.440] It's coming from Russia, [03:17.440 --> 03:19.540] and it's saying that USPS ground. [03:19.540 --> 03:20.560] I mean, come on. [03:23.020 --> 03:25.980] And so, the analyzing JSON file, [03:25.980 --> 03:27.080] each dictionary, [03:27.080 --> 03:29.400] this is just like the process of automation [03:29.400 --> 03:33.320] that I would get the base 64 encoded MIME, [03:33.320 --> 03:34.140] which was the key, [03:34.140 --> 03:36.060] and then the value decoded it, [03:36.060 --> 03:38.640] and getting a couple of files. [03:38.640 --> 03:40.760] The most important one was the file [03:40.760 --> 03:43.660] that I was getting that had the attachments. [03:43.700 --> 03:46.740] And then, looking at the attachments, [03:47.080 --> 03:50.540] this was, again, for a particular campaign, [03:50.540 --> 03:52.540] it was almost the same. [03:52.540 --> 03:54.240] Like, the domains were the same. [03:54.240 --> 03:55.380] The domains were different, [03:55.380 --> 03:57.000] but the structure was the same. [03:57.000 --> 03:59.640] So, I started writing an automated tool [03:59.640 --> 04:01.560] for extracting the information. [04:01.560 --> 04:04.440] The steps was pretty much like following [04:04.440 --> 04:06.790] whatever I'd done manually. [04:07.120 --> 04:11.760] It was... yeah, it works at least. [04:11.860 --> 04:14.480] So, I was getting the JSON MIME, [04:14.480 --> 04:17.540] analyze keys, analyze from address, [04:17.540 --> 04:19.280] analyze the body, [04:19.280 --> 04:21.700] get base 64 encoded data, decode it, [04:21.700 --> 04:24.140] download attachments, analyze files, [04:24.140 --> 04:25.940] and extract domains. [04:26.220 --> 04:31.360] So, that's how YALDA started working. [04:31.400 --> 04:32.960] So, manual analysis, [04:32.960 --> 04:34.840] non-process for automation, [04:34.840 --> 04:36.480] automating it, YALDA, [04:36.480 --> 04:38.960] and then adding more and more to it. [04:39.020 --> 04:40.500] YALDA was born. [04:41.060 --> 04:45.580] So, as I was extracting information, [04:45.580 --> 04:47.420] then I started just talking to analysts [04:47.420 --> 04:48.520] and then asking, [04:48.520 --> 04:51.480] what do you guys do for getting more information? [04:51.480 --> 04:54.240] And then I made YALDA somehow [04:54.240 --> 04:56.580] that it does all of this stuff. [04:56.580 --> 04:58.760] Automated Bulk Intelligence Collection, [04:58.760 --> 05:01.140] it's a file scanner and analyzer, [05:01.140 --> 05:03.260] extracts data, [05:03.260 --> 05:04.560] cluster files, [05:04.560 --> 05:07.400] and it categorizes files. [05:10.180 --> 05:13.080] So, Automated Bulk Intelligence Collection Tool, [05:13.080 --> 05:17.540] it has an automated process of analyzing files. [05:17.540 --> 05:21.100] It applies intelligence in collected data [05:21.100 --> 05:25.200] and clusters files based on the similarities. [05:26.480 --> 05:29.060] It's a file scanner and analyzer. [05:29.060 --> 05:31.940] YALDA scans and analyzes files. [05:31.940 --> 05:35.160] It collects detailed information of the file. [05:35.160 --> 05:37.820] About 20 indicators are being extracted [05:37.820 --> 05:40.200] from each single file. [05:40.200 --> 05:41.720] And it makes it possible... [05:41.720 --> 05:44.200] How many of you guys are analysts here? [05:44.940 --> 05:47.720] Okay, so it makes it possible for you guys, [05:47.720 --> 05:52.760] instead of hours of doing sort of like analyzes, [05:52.760 --> 05:55.320] you just run YALDA on the set of files, [05:55.320 --> 05:58.060] let's say even like a million samples that you have, [05:58.060 --> 06:00.780] and then it's going to analyze each single one of them [06:00.780 --> 06:04.260] and it's going to give you some indicators, [06:04.260 --> 06:06.860] like 20 indicators is going to be extracted. [06:06.860 --> 06:08.960] Then based on whatever you are looking for [06:08.960 --> 06:10.900] and whatever interests you, [06:10.900 --> 06:12.860] it is going to extract the information [06:12.860 --> 06:15.900] that is valuable for you guys. [06:15.900 --> 06:17.360] So you can just like apply your filter [06:17.360 --> 06:19.700] and you can extract the information. [06:21.900 --> 06:24.060] It also extracts data, [06:24.060 --> 06:25.960] like it collects embedded objects [06:25.960 --> 06:30.820] and also it collects malicious URL and domains. [06:32.960 --> 06:35.340] And it categorizes files. [06:35.340 --> 06:37.000] There is like a flag and severity [06:37.000 --> 06:40.220] that in combination they are being used. [06:40.220 --> 06:42.660] So depending on the algorithm [06:42.660 --> 06:44.200] that is being applied to the file [06:44.200 --> 06:46.400] and the founding of the file, [06:46.580 --> 06:49.100] a file could be categorized as clear [06:49.100 --> 06:51.960] with a low severity like a 1 and 2, [06:51.960 --> 06:56.460] suspicious with higher severity, [06:56.460 --> 06:59.260] and malicious, of course, higher severity. [06:59.680 --> 07:01.400] It also clusters data. [07:01.400 --> 07:05.160] It gets the... if there is a malicious hash [07:05.160 --> 07:06.280] that it detects, [07:06.280 --> 07:08.570] it's going to cluster the strings [07:08.820 --> 07:10.320] out of the file [07:10.320 --> 07:12.180] and it's going to insert it [07:12.180 --> 07:13.490] in a different collection. [07:13.880 --> 07:17.260] Same applies if it is an executable file [07:17.260 --> 07:18.640] that is malicious. [07:18.640 --> 07:22.960] It is going to get the names of P sections, [07:22.960 --> 07:24.260] convert it to SHA-1, [07:24.260 --> 07:26.860] and in combination of the hash [07:26.860 --> 07:29.530] and channel entropy is going to use it. [07:31.340 --> 07:33.840] Now, what to do with Yalda [07:33.840 --> 07:35.510] and when to use it. [07:35.510 --> 07:37.770] So if you want to have a feed, [07:37.950 --> 07:41.230] a collection of feeds that you could use it, [07:41.230 --> 07:44.150] you could go ahead and just submit files to Yalda [07:44.150 --> 07:46.550] and it could be used for extracting [07:46.550 --> 07:49.250] MD5 feed, domain feed, URL feed, [07:49.250 --> 07:50.630] all of them malicious. [07:51.770 --> 07:54.290] Also, as I said, [07:54.290 --> 07:55.650] there are 20 indicators [07:55.650 --> 07:58.810] that are being extracted from each single file. [07:58.810 --> 08:00.010] Depending on the file type, [08:00.010 --> 08:01.850] some of them might be empty. [08:01.850 --> 08:03.310] Like, for example, [08:03.310 --> 08:05.750] if there is no P sections in it, [08:05.750 --> 08:06.870] then it's going to be empty. [08:06.870 --> 08:10.190] However, you are able to get all of these, [08:10.190 --> 08:13.130] like MD5, SHA-1, SHA-256, [08:13.130 --> 08:14.570] similar MD5. [08:14.590 --> 08:18.350] If there are some malicious MD5s [08:18.350 --> 08:20.570] that are similar to this particular file [08:20.570 --> 08:21.930] that you are analyzing, [08:21.930 --> 08:25.250] it is going to give you a list of similar MD5s. [08:25.250 --> 08:27.910] File type, if there is any malicious URL [08:27.910 --> 08:30.110] that has been extracted from the file, [08:30.110 --> 08:31.950] it is going to be given to you. [08:31.950 --> 08:34.090] Magic literal source, [08:34.830 --> 08:36.470] VirusTotal information. [08:36.470 --> 08:37.930] This is basically, [08:37.930 --> 08:39.930] just gives you a paramount link [08:39.930 --> 08:41.590] to VirusTotal page, [08:41.590 --> 08:43.090] if it exists in VirusTotal [08:43.090 --> 08:45.650] with the number of positive ABN genes [08:45.650 --> 08:47.550] that are able to detect it. [08:47.590 --> 08:49.150] You don't have to have the key [08:49.150 --> 08:50.370] or use VirusTotal. [08:50.370 --> 08:51.490] You can just disable it. [08:51.490 --> 08:52.890] It's not mandatory. [08:53.070 --> 08:55.550] Severity and flag are being used together. [08:55.550 --> 08:58.150] Severity is like clear, [08:58.150 --> 09:01.930] or it could be suspicious or malicious. [09:01.930 --> 09:03.150] And then the flag, [09:03.150 --> 09:04.910] it could be from 1 to 5. [09:04.910 --> 09:06.310] 1 stands for clear, [09:06.310 --> 09:08.850] and 5 stands for very malicious. [09:08.850 --> 09:10.830] So it just sort of gives you a scope [09:10.830 --> 09:13.470] of how the file is analyzed. [09:13.510 --> 09:15.650] File name, ingest time, [09:15.650 --> 09:18.270] the time that it got analyzed [09:18.270 --> 09:20.930] and got inserted in database. [09:20.930 --> 09:23.510] Domain list, if it has any malicious domain. [09:23.510 --> 09:26.270] URL list, I'm going to go through it in detail, [09:26.270 --> 09:28.510] but the list of the functions [09:28.510 --> 09:30.190] in URL rules that are matching [09:30.190 --> 09:32.410] the particular file is also going [09:32.410 --> 09:34.350] to be extracted in a list. [09:34.550 --> 09:36.310] File path and the source [09:36.310 --> 09:37.610] is going to be Yalda. [09:37.610 --> 09:39.150] It is just like an indicator [09:39.150 --> 09:41.950] to be able to collect the information [09:41.950 --> 09:43.310] from database. [09:43.310 --> 09:45.550] Embedded files, the list of objects [09:45.550 --> 09:46.910] that are embedded in the file [09:46.910 --> 09:49.150] also is going to be displayed to you. [09:49.150 --> 09:51.490] PE sections in detail [09:51.490 --> 09:52.950] is going to be there. [09:53.230 --> 09:55.450] Parent MD5 and parent file path, [09:55.450 --> 09:56.910] if the file is embedded [09:56.910 --> 09:58.650] and is being analyzed, [09:58.650 --> 09:59.970] then you are going to have a link [09:59.970 --> 10:01.170] to the parent. [10:02.810 --> 10:04.090] So let's say [10:04.090 --> 10:05.670] that you are interested to [10:06.310 --> 10:08.030] like you are analyzing the [10:08.030 --> 10:09.810] obtained results and you are interested [10:09.810 --> 10:12.190] to extract a PDF file [10:12.190 --> 10:14.430] that has embedded objects in it. [10:14.430 --> 10:16.150] It has also malicious domains [10:16.150 --> 10:18.150] and it is [10:18.150 --> 10:19.890] malicious with a stability of five. [10:19.890 --> 10:21.530] So you could go ahead and apply a filter [10:21.530 --> 10:23.970] and it's going to extract the data for you. [10:23.970 --> 10:25.790] So just based on the indicators, [10:25.790 --> 10:27.530] it is possible that you [10:27.530 --> 10:30.470] select exactly what you are looking for. [10:30.650 --> 10:31.810] It's a good source [10:31.810 --> 10:33.650] for generating YARA rules [10:33.650 --> 10:35.890] because it has detailed information about it [10:35.890 --> 10:37.690] and if it is malicious, also it's going [10:37.690 --> 10:39.750] to have strings out of it and PE sections [10:39.750 --> 10:41.830] so you can use it as a good [10:41.830 --> 10:43.410] source for writing your [10:44.130 --> 10:45.530] YARA rules. [10:47.170 --> 10:48.450] It's a [10:48.450 --> 10:50.090] smart feed to Cocoa [10:50.090 --> 10:52.250] Sandbox. Again, you don't want to submit [10:52.250 --> 10:53.950] everything to Cocoa Sandbox, but if [10:54.190 --> 10:55.370] there is a specific malicious [10:55.370 --> 10:57.290] characteristic that you are interested in, [10:57.290 --> 10:59.390] you can go ahead and start [10:59.390 --> 11:01.250] feeding that category [11:01.250 --> 11:03.570] to Sandbox [11:03.570 --> 11:05.730] to see what is happening next. [11:07.670 --> 11:08.050] YALDA [11:08.050 --> 11:08.730] Architecture [11:10.710 --> 11:11.710] Basically, [11:11.710 --> 11:13.250] the architecture could be [11:13.250 --> 11:17.210] categorized in four sections. [11:17.210 --> 11:18.130] File, [11:18.130 --> 11:19.630] you submit it, [11:19.630 --> 11:21.750] then extract files, [11:21.750 --> 11:23.610] it starts extracting files, [11:23.610 --> 11:25.590] and then it starts analyzing and [11:25.590 --> 11:27.490] scanning every single file [11:27.490 --> 11:29.450] that has been extracted, and then [11:29.450 --> 11:31.870] inserts data in database. [11:34.210 --> 11:35.770] Extracting files is done [11:36.410 --> 11:37.550] so mainly [11:37.550 --> 11:38.970] what you need to have is [11:38.970 --> 11:40.810] you need to have a directory [11:41.590 --> 11:43.730] in your Linux box and dump [11:43.730 --> 11:45.470] all of the subfolders and [11:45.470 --> 11:47.690] files that you want, any type. [11:47.690 --> 11:49.590] It could be compressed, it could be JSON, [11:49.590 --> 11:51.130] it could be encoded [11:51.130 --> 11:53.030] MIME, and [11:53.030 --> 11:55.270] what it does, it goes through each single [11:55.270 --> 11:57.010] one of them and it starts extracting [11:57.010 --> 11:58.270] the files. [11:58.950 --> 12:00.270] If it is [12:00.930 --> 12:03.090] compressed file, it's going to uncompress it. [12:03.090 --> 12:05.070] If it is having many [12:05.070 --> 12:06.950] different subfolders, like what [12:06.950 --> 12:09.090] my boss initially passed me, it's [12:09.090 --> 12:11.010] going to walk through each single [12:11.010 --> 12:12.950] folder and extract every [12:12.950 --> 12:14.410] single file, and [12:14.910 --> 12:17.010] again, if it is in [12:17.010 --> 12:18.790] mail format, it's going to [12:18.790 --> 12:20.670] download the mail attachments. [12:20.670 --> 12:22.770] So you're going to get a list of smaller [12:22.770 --> 12:24.730] files, and then for each file [12:24.730 --> 12:27.430] it's going to apply foremost in it. [12:27.470 --> 12:28.710] Now, when foremost [12:28.710 --> 12:30.550] gets applied, it's going to have [12:30.550 --> 12:32.850] sort of like children, embedded objects [12:32.850 --> 12:35.110] in it. And not only it's going to [12:37.030 --> 12:38.750] analyze the parent's file, but [12:38.750 --> 12:40.710] it's going to analyze all the [12:40.710 --> 12:42.790] children of the file, and each [12:42.790 --> 12:44.810] one of them is going to be sent to the decoder [12:44.810 --> 12:46.770] and it's going to get analyzed through [12:46.770 --> 12:47.710] Yalda. [12:49.070 --> 12:50.610] Decoding files. [12:50.990 --> 12:52.870] Let me just put a pause [12:52.870 --> 12:55.190] here and see, [12:55.190 --> 12:56.170] like, [12:56.710 --> 12:58.750] just want to ask you, how many of you [12:58.750 --> 13:00.650] do automated process for [13:01.690 --> 13:02.730] extracting information [13:02.730 --> 13:03.930] of a file? [13:05.130 --> 13:06.790] So the first thing that you would [13:06.790 --> 13:08.530] do is manual [13:08.530 --> 13:10.750] analysis. It's just like you have to [13:10.750 --> 13:12.690] know what you are looking for, [13:12.690 --> 13:14.630] and you have to know what is this malicious [13:14.630 --> 13:16.530] file doing. And then you would need [13:16.530 --> 13:18.490] to have a couple of samples to see what [13:18.490 --> 13:20.330] is common between these. [13:20.390 --> 13:22.570] The decoder part is actually written based [13:22.570 --> 13:24.590] on the same thing here, that I [13:24.590 --> 13:26.530] told you. So, based on [13:26.530 --> 13:28.410] the file type, how Yalda [13:28.410 --> 13:30.550] is designed, based on the file type, it's [13:30.550 --> 13:32.990] going to apply, [13:32.990 --> 13:34.410] it's going to send the file [13:34.410 --> 13:36.350] to a set of decoders, [13:36.350 --> 13:38.470] and then if there is a match with one [13:38.470 --> 13:40.410] of the decoders or more, it's going to [13:40.410 --> 13:42.230] just like flag it and go further [13:42.230 --> 13:44.570] and see what could be extracted from it. [13:44.570 --> 13:46.230] If there is any domain or [13:46.230 --> 13:48.250] URL that could be matched, it's going to [13:48.250 --> 13:50.790] extract it, and then it's going to flag the file. [13:51.050 --> 13:52.470] Now, [13:52.470 --> 13:54.310] the decoders, the way that they are written [13:54.310 --> 13:56.230] is they, [13:56.930 --> 13:57.810] as I said, [13:58.430 --> 14:00.610] sort of like manual analyzer, detects [14:00.610 --> 14:02.310] the malware, detects the campaign, and [14:02.310 --> 14:04.490] starts analyzing them, one by one [14:04.490 --> 14:06.230] sort of, and see what is the common [14:06.230 --> 14:08.070] part that is being repeated. [14:08.130 --> 14:10.030] When you know that part, then you can start [14:10.030 --> 14:11.990] automation, and you can write the decoder [14:11.990 --> 14:13.970] for it. That's how it's done, actually, [14:13.970 --> 14:14.910] in Yalda. [14:16.290 --> 14:18.130] Analyze known malicious samples [14:18.130 --> 14:20.090] from fingerprints. [14:21.610 --> 14:22.210] CVE2017 [14:22.930 --> 14:23.530] 0199 [14:24.130 --> 14:25.970] Who heard of it? [14:26.350 --> 14:27.690] Yeah, cool. [14:27.690 --> 14:29.450] So, we know that it's [14:29.450 --> 14:32.010] an RTF file with embedded [14:32.010 --> 14:33.950] OLE object, and [14:33.950 --> 14:36.030] it has a link, [14:37.470 --> 14:38.150] like, [14:38.150 --> 14:39.950] something like this, and then [14:39.950 --> 14:42.030] analyzing all of them, we [14:42.030 --> 14:43.630] can just extract the [14:43.630 --> 14:46.050] comment section out of it, [14:46.050 --> 14:48.250] because you see that it has a [14:48.250 --> 14:50.430] link to download the embedded doc, [14:50.430 --> 14:52.470] and what it does is actually, [14:52.470 --> 14:54.310] as you see, this is the common part. [14:54.310 --> 14:56.030] So if you start [14:56.830 --> 14:59.290] parsing the file, [14:59.470 --> 15:00.390] and write a regex [15:00.390 --> 15:02.250] for it to see if [15:02.250 --> 15:03.850] it is RTF file, go ahead and just [15:04.590 --> 15:05.950] analyze it and see [15:05.950 --> 15:07.890] if there is a pattern match [15:07.890 --> 15:10.070] with this regex, then you could extract [15:10.070 --> 15:12.550] the URL, just as simple as this. [15:14.150 --> 15:16.090] This is, like, an analysis [15:16.090 --> 15:18.510] that is done in one of the RTF files. [15:18.510 --> 15:20.030] So, these are the information [15:20.030 --> 15:21.870] that Yalda extracted. [15:22.190 --> 15:24.530] The MD5, SHA1, SHA256, [15:24.530 --> 15:25.790] size of file, magic [15:25.790 --> 15:28.150] literal, file type, file name, [15:28.150 --> 15:29.910] and then the severity is 5, [15:29.910 --> 15:31.670] the highest severity, and it's [15:31.670 --> 15:33.670] flagged as malicious. This is the [15:33.670 --> 15:35.690] domain that has been extracted from this [15:35.690 --> 15:37.470] particular file. There is no [15:37.470 --> 15:39.810] piece section, of course. The source is [15:39.810 --> 15:41.650] Yalda mining data. This is just an [15:41.650 --> 15:43.430] indicator that I used. [15:43.830 --> 15:45.990] And vtinfo, [15:45.990 --> 15:47.910] if you want to have information, [15:47.910 --> 15:49.590] this is, like, the AVN genes that are [15:49.590 --> 15:50.950] able to detect it, and [15:51.750 --> 15:53.630] paramelling to get detailed [15:53.630 --> 15:55.550] information from VirusTotal. [15:55.550 --> 15:57.550] URL list, it's empty because [15:57.550 --> 16:00.410] I didn't apply any URL rules on it. [16:00.410 --> 16:02.250] This is actually the dictionary [16:02.250 --> 16:04.510] format of the output of Yalda, [16:04.510 --> 16:06.450] and it's just, like, everything that I just told [16:06.450 --> 16:08.430] you, it is going to be in dictionary format [16:08.430 --> 16:10.610] and it's going to be inserted in database. [16:10.650 --> 16:12.350] If you're interested to have it in JSON [16:12.350 --> 16:14.390] format, it could be done easily, and you can [16:14.390 --> 16:17.010] just have the information on Splunk. [16:19.290 --> 16:20.430] When I talk [16:20.430 --> 16:22.370] to analysts, usually they say [16:22.370 --> 16:24.530] that, yeah, detecting a malware [16:24.530 --> 16:26.830] is just like a chain of detection, [16:26.830 --> 16:28.410] and usually it's just not [16:28.410 --> 16:30.450] just one file, it's just a combination [16:30.450 --> 16:32.530] of steps that is being done. [16:32.530 --> 16:34.290] So it's really important to detect [16:34.290 --> 16:36.350] the first chain, and then [16:36.350 --> 16:38.270] start analysing it. [16:39.290 --> 16:40.430] So there was [16:40.430 --> 16:42.490] actually an email campaign that [16:42.490 --> 16:44.470] was going on, and it was sending [16:44.470 --> 16:46.410] PDF attachments, and [16:46.410 --> 16:48.610] none of the AVN genes were able to detect [16:48.610 --> 16:50.410] it. When it was [16:50.410 --> 16:52.650] analysed, we found out that there is [16:52.950 --> 16:54.330] a URL list embedded [16:54.330 --> 16:56.650] in the PDF file, the URL [16:56.650 --> 16:58.630] downloads JavaScript, [16:58.630 --> 17:01.090] JavaScript has a link to a [17:01.090 --> 17:03.470] URL, URL downloads [17:03.470 --> 17:04.850] another executable file, so it's [17:04.850 --> 17:06.930] just like a chain, right? But how [17:06.930 --> 17:08.930] about if we just [17:08.930 --> 17:11.070] detect the first chain, and just [17:11.070 --> 17:13.110] understand that, okay, this is a [17:13.110 --> 17:14.950] PDF file, and it is matching [17:14.950 --> 17:18.910] with this particular chain. [17:18.910 --> 17:20.830] Then we have the flag, right? Then we have [17:20.830 --> 17:23.290] the flag that this is suspicious, at least. [17:23.290 --> 17:24.670] And then we even can [17:24.670 --> 17:26.750] extract the URL that is the first [17:26.750 --> 17:29.010] URL that it's pointing. [17:29.010 --> 17:30.630] So that we can just extract it, [17:30.630 --> 17:32.770] have the information, and if you want, you can send it [17:32.770 --> 17:35.250] to Kuku Sandbox for detailed information. [17:35.990 --> 17:36.450] Applying [17:36.450 --> 17:38.490] Yara rules. So I am not [17:38.490 --> 17:40.570] giving you any Yara rules [17:40.570 --> 17:42.810] myself, but Yalda is able to [17:44.070 --> 17:44.630] apply [17:44.630 --> 17:46.630] Yara rules on the given [17:46.630 --> 17:48.270] files. There is just like [17:49.510 --> 17:50.450] sort of like [17:50.450 --> 17:52.650] enable, disable line [17:52.650 --> 17:54.630] that you need to enable or disable it in the [17:54.630 --> 17:56.690] config file, and then you would need to [17:56.690 --> 17:58.750] pass some of the Yara rules [17:58.750 --> 18:00.830] that interest you based [18:00.830 --> 18:02.930] on your needs in a directory. [18:02.930 --> 18:04.610] It goes to that directory if it is [18:04.610 --> 18:06.910] enabled, and then applies every single [18:06.910 --> 18:08.690] Yara rule to the file. [18:08.690 --> 18:11.010] And if there is a match with one of the functions, [18:11.010 --> 18:12.790] it's going to extract the file [18:12.790 --> 18:14.610] and it's going to show you as a list [18:14.610 --> 18:16.330] of the files that has been [18:16.330 --> 18:18.810] extracted. So according [18:18.810 --> 18:20.590] to the match, and according to the [18:20.590 --> 18:22.530] number of functions that are [18:22.530 --> 18:24.510] similar, it's going to get flagged as [18:24.510 --> 18:26.510] malicious, suspicious, or clear [18:26.510 --> 18:27.890] with appropriate [18:28.750 --> 18:30.590] severity. Yalda [18:30.590 --> 18:32.870] scoring is actually the brain in Yalda [18:32.870 --> 18:34.630] because everything so far that [18:34.630 --> 18:36.490] I told you, you should know [18:36.490 --> 18:38.490] that there is a malicious file, or [18:38.490 --> 18:40.250] there is a malicious campaign, and then [18:40.250 --> 18:42.530] you have to write the decoder, so there are [18:42.530 --> 18:43.830] none. It's not like [18:44.490 --> 18:46.650] something new get detected. But [18:46.650 --> 18:48.410] this method is somehow [18:48.410 --> 18:50.430] based on similarities. [18:50.710 --> 18:51.910] And when I [18:51.910 --> 18:54.570] find, when actually Yalda finds a file [18:54.570 --> 18:56.530] that is malicious, then it starts [18:56.530 --> 18:58.350] clustering the strings. [18:58.350 --> 19:00.250] So it applies clusters [19:00.250 --> 19:02.090] on strings, and also [19:02.090 --> 19:04.290] if it is executable and it's detected [19:04.290 --> 19:06.210] as malicious, it's getting the [19:06.210 --> 19:08.370] PE sections, the name [19:08.370 --> 19:10.310] section of it, and then Shannon [19:10.310 --> 19:13.090] entropy, and it clusters the information. [19:13.090 --> 19:14.390] So it's something like this. [19:14.390 --> 19:15.010] See? [19:15.950 --> 19:16.990] The file [19:16.990 --> 19:18.930] is detected as malicious. I'm applying [19:18.930 --> 19:20.750] strings, and [19:20.750 --> 19:23.050] get all of the strings out of it [19:23.050 --> 19:25.310] and also a Unicode [19:26.410 --> 19:26.930] string [19:26.930 --> 19:28.530] out of it, [19:28.530 --> 19:31.330] the Unicode strings, and then making a [19:31.330 --> 19:33.250] list and cluster the information in [19:34.050 --> 19:36.710] another collection. [19:36.710 --> 19:37.370] There is [19:37.370 --> 19:39.450] a file that I have no idea about. [19:39.450 --> 19:41.550] Like, none of my decoders were able to [19:41.550 --> 19:42.890] detect it, and none of the URLs were [19:42.890 --> 19:44.830] able to detect it. Still, I apply the [19:44.830 --> 19:46.930] strings on it. And now, based on [19:46.930 --> 19:48.830] the similarities that I find with [19:48.830 --> 19:50.650] these strings, the list of strings [19:50.650 --> 19:52.810] found here, and the list of malicious [19:52.810 --> 19:54.890] strings in my database, I [19:54.890 --> 19:56.490] start scoring it. [19:56.630 --> 19:58.510] How many files have been detected [19:58.510 --> 20:00.410] malicious that are having similar [20:00.410 --> 20:03.110] strings as the list of this file? [20:03.110 --> 20:04.910] So if the score goes higher, [20:04.910 --> 20:07.050] I would say that, yeah, it is suspicious. [20:07.050 --> 20:08.070] It needs more [20:09.630 --> 20:10.630] analysis. [20:10.630 --> 20:12.630] So it's based on the similarities. [20:12.630 --> 20:14.010] Same thing is applied for [20:14.610 --> 20:16.510] the name in p-sections and [20:16.510 --> 20:18.990] Shannon entropy. Name of p-sections, [20:18.990 --> 20:20.630] if the file is executable, [20:20.630 --> 20:22.510] and if it is malicious, I'm [20:22.510 --> 20:24.550] getting it, and then I convert [20:24.550 --> 20:25.990] it to SHA-1. [20:26.090 --> 20:28.310] And also, the [20:28.310 --> 20:30.510] entropy, the Shannon entropy, I'm just [20:30.510 --> 20:32.450] collecting these two, and I'm using [20:32.450 --> 20:34.270] an AND logic [20:34.270 --> 20:36.910] for these two, and the malicious hash. [20:36.910 --> 20:38.230] And all of this information [20:38.230 --> 20:40.230] are being clustered somewhere. [20:40.230 --> 20:42.130] Now it's just sort of like a scaling, [20:42.130 --> 20:44.090] again, scaling of the file [20:44.090 --> 20:46.150] that right now I'm seeing, and [20:46.150 --> 20:48.050] see what are the similarities. If it is [20:48.050 --> 20:50.270] executable, then what are the similarities [20:50.270 --> 20:52.110] between these two? And [20:52.110 --> 20:54.230] based on the similarities and the scoring [20:54.230 --> 20:56.130] that I set a threshold, then [20:56.130 --> 20:58.630] I'm going to flag it as suspicious. [21:00.510 --> 21:02.570] Now, whitelisting. [21:04.770 --> 21:05.290] Who [21:06.750 --> 21:07.250] handles [21:07.250 --> 21:08.450] feeds? [21:10.050 --> 21:11.290] So you guys [21:11.290 --> 21:13.410] know that how life is when you have [21:13.410 --> 21:15.030] false positives, right? [21:15.030 --> 21:17.110] And you start using all of [21:17.110 --> 21:19.090] the possible scenarios [21:19.090 --> 21:21.550] and all of the cases for whitelisting. [21:21.550 --> 21:23.710] You're going to have manual whitelisting, [21:23.710 --> 21:25.230] you're going to have a lot of different [21:25.230 --> 21:27.150] sources, and still you'll have some false [21:27.150 --> 21:28.370] positives, right? [21:28.750 --> 21:31.770] To avoid such a thing in Yalda, [21:31.770 --> 21:33.410] I started saying that, okay, fine, [21:33.410 --> 21:35.390] I'm going to use the traditional way. [21:35.390 --> 21:37.230] I'm going to use a log [21:37.230 --> 21:38.170] file for [21:39.170 --> 21:41.850] whitelisting. But at the same time, [21:41.850 --> 21:43.030] how about just [21:43.030 --> 21:45.290] clustering the good guys instead of [21:45.290 --> 21:47.150] just clustering bad guys? [21:47.150 --> 21:48.930] So how about whitelisting [21:48.930 --> 21:51.030] the clear files, the ones that I [21:51.030 --> 21:53.690] know that they are clear and they are not malicious? [21:53.790 --> 21:55.150] And then extract that [21:55.150 --> 21:56.330] information for [21:56.910 --> 21:58.950] using it as whitelist. [21:59.050 --> 22:00.930] So as Yalda is running [22:00.930 --> 22:02.970] more samples, it's going to extract [22:02.970 --> 22:04.610] more bad stuff [22:04.610 --> 22:07.090] and good stuff at the same time, right? [22:07.090 --> 22:08.730] So the whitelisting is getting [22:08.730 --> 22:11.210] stronger as the detection of [22:11.210 --> 22:12.450] malicious [22:12.450 --> 22:14.030] hashes also gets [22:14.930 --> 22:16.290] stronger. [22:16.830 --> 22:18.930] So as I said, I'm just [22:18.930 --> 22:20.710] applying strings on [22:20.710 --> 22:22.270] clear files and [22:22.930 --> 22:24.970] clustering it, and also I'm clustering [22:24.970 --> 22:26.790] the PE section names and [22:26.790 --> 22:29.050] Shannon entropy of executable [22:29.050 --> 22:31.490] files that are known as clear. [22:31.490 --> 22:33.390] And then I'm using this information [22:33.390 --> 22:34.830] sort of to whitelist [22:36.330 --> 22:37.010] for [22:37.010 --> 22:39.090] having more [22:39.090 --> 22:40.890] accurate results. [22:41.210 --> 22:43.410] Let's have a demonstration. [22:44.630 --> 22:47.130] So here I'm having three files in [22:47.130 --> 22:48.730] the folder and [22:48.730 --> 22:51.370] it starts analyzing the file. [22:51.370 --> 22:52.950] It extracts the [22:52.950 --> 22:55.190] files and then in this [22:55.190 --> 22:56.890] case it's downloading the attachments [22:56.890 --> 22:59.030] and you see that there are a list [22:59.030 --> 23:00.910] of domain lists that they are all malicious [23:00.910 --> 23:02.950] and it's extracting it. Also [23:02.950 --> 23:04.910] it goes and analyzes [23:04.910 --> 23:06.950] each single file, like in detail, if [23:06.950 --> 23:08.950] there is any embedded object, it is going [23:08.950 --> 23:10.930] to extract it. Like in this case, [23:10.930 --> 23:12.890] it does have embedded objects, so it just [23:12.890 --> 23:15.070] lists the list of embedded objects [23:15.070 --> 23:17.270] here, like embedded files. [23:17.310 --> 23:19.010] And not only it has the [23:19.010 --> 23:21.130] detailed information of the file itself [23:21.130 --> 23:22.570] but it also [23:23.130 --> 23:24.970] analyzes the embedded [23:24.970 --> 23:26.990] file in it and it gives you [23:26.990 --> 23:29.410] information with a link to the parent. [23:30.330 --> 23:31.170] And everything [23:31.170 --> 23:32.970] is being inserted in the database, [23:32.970 --> 23:34.170] as I mentioned. [23:35.070 --> 23:36.850] So Yalda sort of like [23:36.850 --> 23:38.850] minimizes false [23:38.850 --> 23:40.810] positive or better [23:40.810 --> 23:43.130] to be said, it has better correlation. [23:43.370 --> 23:44.870] So the way [23:44.870 --> 23:46.790] that we are doing Yalda scoring, [23:46.790 --> 23:48.690] we get more accurate results [23:48.690 --> 23:51.170] because as we are analyzing [23:51.170 --> 23:52.750] more samples, we are getting [23:52.750 --> 23:55.370] stronger results. [23:55.470 --> 23:57.030] Same thing for automated [23:57.030 --> 23:58.850] whitelisting, that we are [23:58.850 --> 24:00.950] getting better results as we start [24:01.630 --> 24:03.550] analyzing more samples. [24:03.570 --> 24:04.750] And then this clustering [24:04.750 --> 24:07.030] data, again, we are having better results [24:07.030 --> 24:09.030] and then categorizing algorithm is [24:09.030 --> 24:10.550] somehow that [24:10.550 --> 24:12.550] I'm not saying that if I [24:12.550 --> 24:14.850] don't know, like it doesn't match with [24:14.850 --> 24:16.950] my decoders, then I'm not going [24:16.950 --> 24:18.770] to say 100% it's malicious [24:18.770 --> 24:20.710] but it has enough [24:20.710 --> 24:22.690] high severity that you can select [24:22.690 --> 24:24.490] and say that, okay, this is suspicious [24:24.490 --> 24:26.870] and there is a flag going on, so I better [24:26.870 --> 24:28.690] start analyzing this set of files [24:28.690 --> 24:30.490] with this criteria, because you have 20 [24:30.490 --> 24:32.810] indicators that you can select out of it. [24:34.810 --> 24:36.790] This is the GitHub, so you can [24:36.790 --> 24:37.870] go ahead and start [24:38.970 --> 24:40.750] downloading and using it. [24:40.750 --> 24:42.430] It's in Fidelis GitHub. [24:51.480 --> 24:52.220] So there [24:52.220 --> 24:54.280] are three folders, like bin, [24:54.280 --> 24:56.220] source, Yara rules. Yara rules is a [24:56.220 --> 24:57.840] place that you place your [24:57.840 --> 25:00.100] Yara rules if you want to. If not, [25:00.100 --> 25:02.100] you don't have to worry about it. [25:02.300 --> 25:03.640] Bin has all the [25:04.120 --> 25:06.100] modules and functions, as you [25:06.100 --> 25:08.220] see here. You can even go ahead [25:08.220 --> 25:09.900] it's just like very straight format [25:09.900 --> 25:11.900] and you can just go ahead and if you want [25:11.900 --> 25:14.620] to add more decoders, you can do that. [25:15.160 --> 25:15.640] And [25:15.640 --> 25:17.880] the config file is something that I go [25:17.880 --> 25:19.740] through it in detail. It's very [25:19.740 --> 25:21.900] simple, it's just like a couple of lines that [25:21.900 --> 25:24.060] you need to specify the directories in it [25:24.060 --> 25:26.060] and when you want to run it, run [25:26.060 --> 25:28.480] this guy, Yalda file analyzer. [25:30.100 --> 25:32.060] So you would need to install the [25:32.060 --> 25:33.680] required Python modules [25:34.620 --> 25:36.060] and you would need [25:36.060 --> 25:38.600] these are the Python modules that you would need. [25:39.060 --> 25:40.080] And this is the [25:40.080 --> 25:42.160] config file. You just go ahead and open it [25:42.160 --> 25:42.920] and start [25:44.300 --> 25:46.000] just adding the directories [25:46.000 --> 25:48.020] that you want or enable or disable. [25:48.020 --> 25:49.840] Like if you don't want to have virus [25:49.840 --> 25:52.500] total information or if you don't want to have [25:52.500 --> 25:54.300] sort of like Yara [25:54.300 --> 25:56.360] rule to be applied to it, you can [25:56.360 --> 25:58.620] say that zero, just like [25:58.620 --> 26:00.760] disable it. If you want to [26:00.760 --> 26:02.640] sort of like have a printout, you can have [26:02.640 --> 26:04.620] debug one or you can just make it [26:04.620 --> 26:06.840] zero. And of course, because I'm using [26:06.840 --> 26:08.660] MongoDB, you would need to [26:08.660 --> 26:10.540] have MongoDB [26:10.540 --> 26:12.400] installed and just like specify [26:12.400 --> 26:14.580] the criteria that you're looking for. [26:15.200 --> 26:16.560] And to run it, you [26:16.560 --> 26:18.720] go to source and you start running [26:18.720 --> 26:20.420] its Yalda file analyzer [26:20.420 --> 26:22.360] and it starts analyzing each single [26:22.360 --> 26:25.420] file in detail and passes you the information. [26:25.420 --> 26:26.560] I'd like to talk [26:26.560 --> 26:28.460] to... thanks to [26:28.460 --> 26:30.360] John Bamenak, my [26:30.360 --> 26:32.140] boss, Hardik Modi [26:32.140 --> 26:34.280] and Chad Robertson [26:34.280 --> 26:35.940] and Jason Reeves. They [26:35.940 --> 26:38.540] actually gave me really great feedback [26:38.540 --> 26:40.200] for improving Yalda [26:40.200 --> 26:41.080] and [26:42.320 --> 26:44.860] really appreciate all of your feedback. [26:45.540 --> 26:46.520] So if [26:46.520 --> 26:48.420] you need to contact me or if you [26:48.420 --> 26:50.740] have any questions, this is my email address [26:50.740 --> 26:52.000] or Twitter. [26:53.520 --> 26:55.180] Thank you so much. [26:55.180 --> 26:56.480] If you have any questions or [26:57.480 --> 26:57.960] feedback...