1 00:00:00,000 --> 00:00:01,260 We'll get started right away. 2 00:00:01,580 --> 00:00:04,280 It is my pleasure to introduce an old friend of ours, 3 00:00:04,680 --> 00:00:05,400 Gita Zia-Bari. 4 00:00:07,400 --> 00:00:08,500 Thank you so much. 5 00:00:08,540 --> 00:00:09,300 Thanks, everyone. 6 00:00:09,860 --> 00:00:11,640 Thanks for your patience, first of all, 7 00:00:11,680 --> 00:00:14,000 and thanks for coming to my presentation. 8 00:00:14,700 --> 00:00:16,680 I'm going to talk about Yalda. 9 00:00:17,520 --> 00:00:20,560 It's an automated tool that I wrote recently, 10 00:00:20,720 --> 00:00:21,980 one of my recent projects, 11 00:00:22,060 --> 00:00:23,620 and I thought to share it with you. 12 00:00:24,640 --> 00:00:27,160 I work for Fidelity Cyber Security, 13 00:00:27,160 --> 00:00:31,320 and I'm going to talk about Yalda. 14 00:00:31,520 --> 00:00:36,660 So we are going to talk about having an introduction of Yalda, 15 00:00:36,820 --> 00:00:39,560 how was it created, and what was the object of it. 16 00:00:39,700 --> 00:00:44,000 Then how could you use Yalda, and what it could get used. 17 00:00:44,380 --> 00:00:50,240 Then we are going to have an overview of architecture of Yalda, 18 00:00:50,480 --> 00:00:51,880 a quick demonstration, 19 00:00:52,640 --> 00:00:55,720 and a GitHub that you can go ahead and download it 20 00:00:55,720 --> 00:00:57,140 and start using it. 21 00:00:57,160 --> 00:00:59,180 You can start using it on your own files, 22 00:00:59,460 --> 00:01:02,380 and using it is actually pretty much simple. 23 00:01:02,600 --> 00:01:06,000 It's just like going and just clicking on a file 24 00:01:06,000 --> 00:01:08,340 to configure and add some directories to it. 25 00:01:11,320 --> 00:01:12,940 So introducing Yalda. 26 00:01:16,620 --> 00:01:20,080 The motivation was basically creating a tool 27 00:01:20,080 --> 00:01:23,200 that analyzes files and gives you some indicators 28 00:01:24,080 --> 00:01:25,640 that saves time 29 00:01:25,930 --> 00:01:26,220 for 30 00:01:26,220 --> 00:01:29,680 sort of like repeating the job for each single file. 31 00:01:30,360 --> 00:01:35,460 And also, it was just like for sort of like fully understanding 32 00:01:35,460 --> 00:01:36,840 what the file does, 33 00:01:37,460 --> 00:01:42,260 and getting, creating automated bulk intelligence collection. 34 00:01:42,780 --> 00:01:47,480 And sort of like being independent of other sources, third parties. 35 00:01:47,760 --> 00:01:50,120 So that was the motivation. 36 00:01:53,040 --> 00:01:56,120 So that guy in the back, John Baminat, 37 00:01:56,220 --> 00:01:57,860 he actually is my boss, 38 00:01:57,960 --> 00:02:00,420 and he passed me a project to start with. 39 00:02:00,680 --> 00:02:06,140 So he sort of started giving me some sort of files, 40 00:02:06,740 --> 00:02:08,220 in massive files, 41 00:02:08,380 --> 00:02:10,960 I mean millions of files in directories. 42 00:02:11,060 --> 00:02:12,380 And they were all compressed, 43 00:02:12,640 --> 00:02:15,060 and they were in JSON format. 44 00:02:15,240 --> 00:02:18,560 Each one of them, it had a dictionary inside it, 45 00:02:18,640 --> 00:02:23,180 and then the dictionary had information about the emails. 46 00:02:23,500 --> 00:02:26,200 And each dictionary has a dashed star on it, 47 00:02:26,200 --> 00:02:28,240 generally was corresponding to a malicious email. 48 00:02:28,500 --> 00:02:31,200 It has all the key and values, of course, for that one. 49 00:02:31,380 --> 00:02:37,860 And it has some MIME-encoded keys that the values were there. 50 00:02:38,000 --> 00:02:39,600 If you decode them, then you would 51 00:02:39,600 --> 00:02:40,880 be able to download the files. 52 00:02:41,060 --> 00:02:43,160 So that is how Yalda was born. 53 00:02:43,340 --> 00:02:45,220 I started analyzing them manually, 54 00:02:45,460 --> 00:02:48,640 and then figured out that, yeah, campaign 55 00:02:48,640 --> 00:02:55,440 from the same sort of campaigns, although the format 56 00:02:55,440 --> 00:02:57,280 and everything is a little bit different, 57 00:02:57,420 --> 00:03:00,080 but the fingerprint and the way that they are introducing 58 00:03:00,080 --> 00:03:01,500 stuff is almost the same. 59 00:03:01,660 --> 00:03:03,840 And I figured out that it could get automated. 60 00:03:04,140 --> 00:03:05,560 This is just like an example. 61 00:03:05,760 --> 00:03:07,640 By just looking at it, I found out 62 00:03:07,640 --> 00:03:09,420 that this just doesn't make sense. 63 00:03:09,620 --> 00:03:10,480 This is malicious. 64 00:03:10,880 --> 00:03:13,480 I mean, I would be able to just say by a glance 65 00:03:13,480 --> 00:03:14,260 that this is malicious. 66 00:03:14,400 --> 00:03:15,100 Look at the from. 67 00:03:15,660 --> 00:03:19,260 It's coming from Russia, and it's saying that USPS ground. 68 00:03:19,460 --> 00:03:20,440 I mean, come on. 69 00:03:23,100 --> 00:03:24,920 And so the analyzing 70 00:03:24,920 --> 00:03:27,520 JSON file, each dictionary, this is just 71 00:03:27,520 --> 00:03:32,120 like the process of automation that I would get the base 64 72 00:03:32,120 --> 00:03:34,920 encoded MIME, which was the key, and then the value, 73 00:03:35,060 --> 00:03:38,200 decoded it, and getting a couple of files. 74 00:03:38,400 --> 00:03:40,740 The most important one was the file 75 00:03:40,740 --> 00:03:43,020 that I was getting that had the attachments. 76 00:03:43,360 --> 00:03:46,380 And then looking at the attachments, 77 00:03:46,700 --> 00:03:50,300 this was, again, for a particular campaign 78 00:03:50,300 --> 00:03:52,300 that it was almost the same. 79 00:03:52,440 --> 00:03:53,860 The domains were the same. 80 00:03:53,860 --> 00:03:56,660 The domains were different, but the structure was the same. 81 00:03:56,780 --> 00:03:59,560 So I started writing an automated tool 82 00:03:59,560 --> 00:04:01,220 for extracting the information. 83 00:04:01,520 --> 00:04:04,400 The steps was pretty much like following 84 00:04:04,400 --> 00:04:06,400 whatever I'd done manually. 85 00:04:07,100 --> 00:04:11,120 It was, yeah, it works at least. 86 00:04:12,100 --> 00:04:15,320 So I was getting the JSON MIME, analyze keys, 87 00:04:15,580 --> 00:04:18,700 analyze from address, analyze the body, 88 00:04:18,980 --> 00:04:22,260 get basic 64 encoded data, decode it, download attachments, 89 00:04:22,760 --> 00:04:23,800 analyze files. 90 00:04:23,800 --> 00:04:32,460 and extract domains. So that's how Yalda started working. So manual analysis, 91 00:04:32,820 --> 00:04:37,940 known process for automation, automating it, Yalda. And then adding more and more 92 00:04:37,940 --> 00:04:46,300 to it. Yalda was born. So as I was extracting information then I started 93 00:04:46,300 --> 00:04:50,620 just talking to analysts and then asking what do you guys do for getting more 94 00:04:50,620 --> 00:04:55,920 information and then I made the Yalda somehow that it does all of this stuff 95 00:04:55,920 --> 00:05:01,420 Automated bulk intelligence collection, it's a file scanner and analyzer, extract 96 00:05:01,420 --> 00:05:06,360 data, cluster files and it categorizes files. 97 00:05:09,580 --> 00:05:14,800 So automated bulk intelligence collection tool, it has an automated 98 00:05:14,800 --> 00:05:19,640 process of analyzing files. It's applied intelligence in 99 00:05:19,640 --> 00:05:20,600 collections. We would say it's used in recorduckle analysis and it's used in 100 00:05:20,620 --> 00:05:28,060 data and clusters files based on the similarities. It's a file scanner and 101 00:05:28,060 --> 00:05:34,360 analyzer. Yalda scans and analyzes files. It collects detailed information of the 102 00:05:34,360 --> 00:05:40,540 file. About 20 indicators are being extracted from each single file and it 103 00:05:40,540 --> 00:05:46,120 makes it possible. How many of you guys are analysts here? Okay, so it makes it 104 00:05:46,120 --> 00:05:53,100 possible for you guys instead of hours of doing sort of like analyzes, you just 105 00:05:53,100 --> 00:05:57,380 run Yalda on the set of files. Let's say even like a million samples that you 106 00:05:57,380 --> 00:06:01,400 have and then it's going to analyze each single one of them and it's going to 107 00:06:01,400 --> 00:06:06,920 give you some indicators. Like 20 indicators is going to be extracted. Then 108 00:06:06,920 --> 00:06:11,440 based on whatever you are looking for and whatever interests you, it is going 109 00:06:11,440 --> 00:06:16,080 to extract the information that is valuable for you guys. So you 110 00:06:16,080 --> 00:06:22,580 can just like apply a filter and you can extract the information. It also 111 00:06:22,580 --> 00:06:28,880 extracts data, like it collects embedded objects and also it collects malicious 112 00:06:28,880 --> 00:06:36,460 URL and domains. And it categorizes files, there is like a flag and 113 00:06:36,460 --> 00:06:42,100 severity that in combination they are being used. So depending on the 114 00:06:42,100 --> 00:06:46,060 algorithm that is being applied to the file and the founding of the file and 115 00:06:46,060 --> 00:06:50,200 A file could be categorized as clear with a low severity, 116 00:06:50,420 --> 00:06:52,380 like a one and two, suspicious, 117 00:06:53,140 --> 00:06:55,060 suspicious with higher severity, 118 00:06:56,300 --> 00:06:58,460 and malicious, of course, higher severity. 119 00:06:59,260 --> 00:07:01,020 It also clusters data. 120 00:07:01,400 --> 00:07:06,300 It gets the, if there is any malicious hash that it detects, 121 00:07:06,300 --> 00:07:10,260 it's going to cluster the strings out of the file 122 00:07:10,260 --> 00:07:13,020 and it's going to insert it in a different collection. 123 00:07:13,020 --> 00:07:18,220 Same applies if it is an executable file that is malicious. 124 00:07:18,580 --> 00:07:22,280 It is going to get the names of P-sections, 125 00:07:22,680 --> 00:07:26,800 convert it to SHA-1, and in combination of the hash 126 00:07:26,800 --> 00:07:29,160 and Shannon entropy is going to use it. 127 00:07:31,560 --> 00:07:35,080 Now, what to do with Yaldo and when to use it? 128 00:07:35,460 --> 00:07:39,120 So if you want to have a feed, a collection of feeds 129 00:07:39,120 --> 00:07:42,220 that you could use it, you could go ahead 130 00:07:42,220 --> 00:07:43,000 and just, like, say, 131 00:07:43,000 --> 00:07:45,580 submit files to Yaldo and it could be used 132 00:07:45,580 --> 00:07:49,100 for extracting MD5 feed, domain feed, URL feed, 133 00:07:49,220 --> 00:07:50,000 all of them malicious. 134 00:07:52,660 --> 00:07:55,540 Also, as I said, like, there are 20 indicators 135 00:07:55,540 --> 00:07:58,680 that are being extracted from each single file. 136 00:07:58,760 --> 00:08:01,420 Depending on the file type, some of them might be empty. 137 00:08:01,580 --> 00:08:05,680 Like, for example, if there is no P-sections in it, 138 00:08:05,700 --> 00:08:06,660 then it's going to be empty. 139 00:08:06,780 --> 00:08:09,920 However, you are able to get all of these, 140 00:08:09,940 --> 00:08:12,980 like MD5, SHA-1, SHA-256, and so on. 141 00:08:13,000 --> 00:08:14,160 Similar MD5. 142 00:08:14,460 --> 00:08:18,420 If there are some malicious MD5s that 143 00:08:18,420 --> 00:08:21,360 are similar to this particular file that you are analyzing, 144 00:08:21,680 --> 00:08:24,900 it is going to give you a list of similar MD5s. 145 00:08:24,920 --> 00:08:25,680 File type. 146 00:08:25,880 --> 00:08:28,240 If there is any malicious URL that 147 00:08:28,240 --> 00:08:30,000 has been extracted from the file, 148 00:08:30,120 --> 00:08:31,520 it is going to be given to you. 149 00:08:31,840 --> 00:08:35,600 Magic literal source of VirusTotal information, 150 00:08:36,120 --> 00:08:39,800 this is basically just gives you a Parama link 151 00:08:39,800 --> 00:08:42,980 to VirusTotal page if it exists in VirusTotal. 152 00:08:42,980 --> 00:08:45,480 with the number of positive AV engines 153 00:08:45,480 --> 00:08:46,900 that are able to detect it. 154 00:08:47,280 --> 00:08:50,180 You don't have to have the key or use virus total, 155 00:08:50,300 --> 00:08:52,200 you can just disable it, it's not mandatory. 156 00:08:53,120 --> 00:08:55,260 Severity and flag are being used together. 157 00:08:55,440 --> 00:09:00,560 Severity is like clear, or it could be suspicious 158 00:09:00,560 --> 00:09:02,980 or malicious, and then the flag, 159 00:09:03,000 --> 00:09:04,480 it could be from one to five. 160 00:09:04,820 --> 00:09:08,220 One stands for clear and five stands for very malicious. 161 00:09:08,660 --> 00:09:10,760 So it's just sort of like give you a scope 162 00:09:10,760 --> 00:09:12,780 of how the file is analyzed. 163 00:09:13,640 --> 00:09:18,140 File name, in this time, the time that it got analyzed 164 00:09:18,780 --> 00:09:20,240 and got inserted in database. 165 00:09:20,760 --> 00:09:23,120 Domain list, if it has any malicious domain. 166 00:09:23,340 --> 00:09:26,080 Yara list, I'm going to go through it in detail, 167 00:09:26,220 --> 00:09:29,360 but the list of the functions in Yara rules 168 00:09:29,360 --> 00:09:31,440 that are matching the particular file 169 00:09:31,440 --> 00:09:33,760 is also going to be extracted in a list. 170 00:09:34,440 --> 00:09:37,260 File path, and the source is going to be Yalda. 171 00:09:37,520 --> 00:09:39,700 It is just like as an indicator to be able 172 00:09:39,700 --> 00:09:40,740 to connect to Yalda. 173 00:09:40,760 --> 00:09:42,580 Collects the information from database. 174 00:09:43,160 --> 00:09:46,200 Embedded files, the list of objects that are embedded 175 00:09:46,200 --> 00:09:48,740 in the file also is going to be displayed to you. 176 00:09:49,060 --> 00:09:52,420 PE sections in detail is going to be there. 177 00:09:52,960 --> 00:09:55,200 Parent MD5 and parent file path, 178 00:09:55,380 --> 00:09:58,300 if the file is embedded and is being analyzed, 179 00:09:58,580 --> 00:10:00,560 then you are going to have a link to the parent. 180 00:10:02,860 --> 00:10:05,440 So let's say that you are interested to, 181 00:10:06,320 --> 00:10:08,740 like you are analyzing the obtained results 182 00:10:08,740 --> 00:10:10,600 and you are interested to extract, 183 00:10:10,600 --> 00:10:14,040 a PDF file that has embedded objects in it. 184 00:10:14,160 --> 00:10:15,880 It has also malicious domains, 185 00:10:16,260 --> 00:10:19,760 and it is malicious with a severity of five. 186 00:10:19,860 --> 00:10:21,480 So you could go ahead and apply a filter 187 00:10:21,480 --> 00:10:23,620 and it's going to extract the data for you. 188 00:10:23,760 --> 00:10:25,520 So it's just based on the indicators. 189 00:10:25,680 --> 00:10:28,700 It is possible that you select exactly 190 00:10:28,700 --> 00:10:29,860 what you are looking for. 191 00:10:30,440 --> 00:10:33,600 It's a good source for generating Yara rules 192 00:10:33,600 --> 00:10:35,880 because it has detailed information about it. 193 00:10:35,940 --> 00:10:37,260 And if it is malicious also, 194 00:10:37,340 --> 00:10:39,620 it's going to have a strings out of it and PE sections. 195 00:10:39,620 --> 00:10:42,060 So you can use it as a good source 196 00:10:42,060 --> 00:10:44,880 for writing your Yara rules. 197 00:10:47,080 --> 00:10:50,000 It's a smart feed to Cuckoo Sandbox. 198 00:10:50,180 --> 00:10:52,900 Again, you don't want to submit everything to Cuckoo Sandbox, 199 00:10:52,960 --> 00:10:55,800 but if there is a specific malicious characteristic 200 00:10:56,220 --> 00:10:57,040 that you're interested, 201 00:10:57,240 --> 00:11:01,080 you can go ahead and start feeding it that category 202 00:11:01,080 --> 00:11:05,060 to Sandbox to see what is happening next. 203 00:11:07,460 --> 00:11:08,340 Yalda architecture. 204 00:11:10,440 --> 00:11:12,720 It's basically the architecture is, 205 00:11:12,800 --> 00:11:15,000 could be categorized in file sections, 206 00:11:15,280 --> 00:11:16,480 in four sections. 207 00:11:16,920 --> 00:11:20,880 File, you submit it, then extract files. 208 00:11:21,280 --> 00:11:23,020 It will start extracting files, 209 00:11:23,260 --> 00:11:25,780 and then it starts analyzing and scanning 210 00:11:25,780 --> 00:11:28,500 every single file that has been extracted, 211 00:11:28,700 --> 00:11:30,960 and then inserts data in database. 212 00:11:34,080 --> 00:11:35,660 Extracting files is done. 213 00:11:36,640 --> 00:11:39,380 So mainly what you need to have is you have, 214 00:11:39,380 --> 00:11:42,980 you need to have a directory in your Linux box 215 00:11:42,980 --> 00:11:46,780 and dump all of the subfolders and files that you want. 216 00:11:46,880 --> 00:11:49,420 Any type, it could be compressed, it could be JSON, 217 00:11:49,560 --> 00:11:51,920 it could be encoded MIME. 218 00:11:52,420 --> 00:11:55,800 And what it does, it goes through each single one of them 219 00:11:55,800 --> 00:11:57,460 and it starts extracting the files. 220 00:11:59,040 --> 00:12:02,940 If it is a compressed file, it's going to uncompress it. 221 00:12:03,060 --> 00:12:06,260 If it is like having many different subfolders, 222 00:12:06,320 --> 00:12:08,800 like what my boss initially passed me, 223 00:12:08,800 --> 00:12:11,440 it's going to walk through each single folder 224 00:12:11,720 --> 00:12:13,540 and extract every single file. 225 00:12:13,800 --> 00:12:17,920 And again, if it has, if it is in mail format, 226 00:12:18,060 --> 00:12:20,120 it's going to download the mail attachments. 227 00:12:20,460 --> 00:12:23,080 So you're going to get a list of smaller files. 228 00:12:23,260 --> 00:12:26,920 And then for each file, it's going to apply foremost in it. 229 00:12:27,180 --> 00:12:29,280 Now, when foremost gets applied, 230 00:12:29,520 --> 00:12:31,560 it's going to have sort of like children, 231 00:12:31,700 --> 00:12:33,100 embedded objects in it. 232 00:12:33,240 --> 00:12:38,420 And not only it's going to analyze the parent's file, 233 00:12:38,800 --> 00:12:42,220 it's going to analyze all the children of the file. 234 00:12:42,340 --> 00:12:44,780 And each one of them is going to be sent to the decoder 235 00:12:44,780 --> 00:12:47,080 and it's going to get analyzed through Yalda. 236 00:12:49,040 --> 00:12:49,840 Decoding files. 237 00:12:50,760 --> 00:12:53,740 Let me just put a pause here and see, 238 00:12:56,320 --> 00:12:57,840 just want to ask you, 239 00:12:57,880 --> 00:12:59,920 how many of you do automated process 240 00:12:59,920 --> 00:13:03,380 for extracting information of a file? 241 00:13:04,960 --> 00:13:08,460 So the first thing that you would do is manual, 242 00:13:09,260 --> 00:13:12,560 it's just like you have to know what you are looking for 243 00:13:12,560 --> 00:13:15,400 and you have to know what is this malicious file doing. 244 00:13:15,600 --> 00:13:17,740 And then you would need to have a couple of samples 245 00:13:17,740 --> 00:13:19,800 to see what is common between these. 246 00:13:20,040 --> 00:13:22,100 The decoder part is actually written 247 00:13:22,100 --> 00:13:25,020 based on the same thing here that I told you. 248 00:13:25,200 --> 00:13:28,940 So based on the file type, how Yalda is designed, 249 00:13:29,080 --> 00:13:31,180 based on the file type, it's going to apply, 250 00:13:32,900 --> 00:13:35,960 it's going to send the file to a set of decoders. 251 00:13:36,160 --> 00:13:38,780 And then if there is a match with one of the decoders, 252 00:13:38,780 --> 00:13:41,320 or more, it's going to just like flag it 253 00:13:41,320 --> 00:13:44,380 and go further and see what could be extracted from it. 254 00:13:44,480 --> 00:13:47,480 If there is any domain or URL that could be matched, 255 00:13:47,660 --> 00:13:48,940 it's going to extract it 256 00:13:48,940 --> 00:13:50,280 and then it's going to flag the file. 257 00:13:51,740 --> 00:13:54,660 Now, the decoders, the way that they are written is, 258 00:13:56,280 --> 00:14:00,060 they, as I said, like a sort of like manual analyzer, 259 00:14:00,100 --> 00:14:02,000 detect the malware, detect the campaign, 260 00:14:02,180 --> 00:14:04,860 and it starts analyzing them one by one, sort of, 261 00:14:04,940 --> 00:14:07,620 and see what is the common part that is being repeated. 262 00:14:07,880 --> 00:14:08,600 Then you know, 263 00:14:08,780 --> 00:14:10,440 that part, then you can start automation 264 00:14:10,440 --> 00:14:12,400 and you can write the decoder for it. 265 00:14:12,520 --> 00:14:14,240 That's how it's done, actually, in Yalda. 266 00:14:16,380 --> 00:14:19,160 Analyze non-malicious samples from fingerprints. 267 00:14:21,040 --> 00:14:23,340 CVE-2017-0199. 268 00:14:24,160 --> 00:14:25,260 Who heard of it? 269 00:14:26,020 --> 00:14:27,120 Yeah, cool. 270 00:14:27,460 --> 00:14:31,260 So, we know that it's an RTF file 271 00:14:31,260 --> 00:14:34,860 with embedded OLE object, and it has a link, 272 00:14:37,400 --> 00:14:38,760 like something like this. 273 00:14:38,780 --> 00:14:41,600 And then analyzing all of them, 274 00:14:41,680 --> 00:14:45,820 we can just like extract the comment section out of it, 275 00:14:45,940 --> 00:14:48,180 because it will, like, you see that it has a link 276 00:14:48,180 --> 00:14:49,980 to download the embedded doc. 277 00:14:50,160 --> 00:14:52,780 And what it does is actually, as you see, 278 00:14:52,860 --> 00:14:53,700 this is the comment part. 279 00:14:53,900 --> 00:14:57,200 So, if you start sort of like parsing the file 280 00:14:58,340 --> 00:15:02,920 and write a regex for it to see if it is RTF file, 281 00:15:03,060 --> 00:15:05,500 go ahead and just like analyze it 282 00:15:05,500 --> 00:15:08,720 and see if there is a pattern match with this regex. 283 00:15:08,720 --> 00:15:11,980 Then you could extract the URL, just as simple as this. 284 00:15:14,060 --> 00:15:18,100 This is like an analysis that is done in one of the RTF files. 285 00:15:18,320 --> 00:15:21,160 So, these are the information that Yalda extracted. 286 00:15:21,820 --> 00:15:25,240 The MD5 SHA-1 SHA-256, size of file, 287 00:15:25,400 --> 00:15:27,460 magic literal, file type, file name. 288 00:15:27,980 --> 00:15:30,700 And then the severity is 5, the highest severity, 289 00:15:31,000 --> 00:15:32,540 and it's flagged as malicious. 290 00:15:32,800 --> 00:15:35,120 This is the domain that has been extracted 291 00:15:35,120 --> 00:15:36,780 from this particular file. 292 00:15:36,960 --> 00:15:38,440 There is no P section, of course. 293 00:15:38,480 --> 00:15:38,700 So, this is the domain that has been extracted 294 00:15:38,720 --> 00:15:40,840 The source is Yalda mining data. 295 00:15:41,000 --> 00:15:42,820 This is just an indicator that I used. 296 00:15:43,220 --> 00:15:47,380 And VTINFO, if you want to have information, 297 00:15:47,840 --> 00:15:50,520 this is like the avian genes that are able to detect it 298 00:15:50,520 --> 00:15:55,020 and paramilling to get detailed information from VirusDota. 299 00:15:55,240 --> 00:15:58,740 The URL list, it's empty because I didn't apply 300 00:15:58,740 --> 00:15:59,840 any URL rules on it. 301 00:16:00,640 --> 00:16:02,680 This is actually the dictionary format 302 00:16:02,680 --> 00:16:04,220 of the output of Yalda. 303 00:16:04,420 --> 00:16:06,600 And it's just like everything that I just told you. 304 00:16:06,660 --> 00:16:08,420 It is going to be in dictionary format. 305 00:16:08,420 --> 00:16:10,080 And it's going to be inserted in database. 306 00:16:10,380 --> 00:16:12,760 If you're interested to have it in JSON format, 307 00:16:12,880 --> 00:16:13,720 it could be done easily. 308 00:16:13,900 --> 00:16:16,440 And you can just have the information on Splunk. 309 00:16:19,080 --> 00:16:22,580 When I talk to analysts, usually they say that, 310 00:16:22,580 --> 00:16:26,200 yeah, detecting a malware is just like a chain of detection. 311 00:16:26,600 --> 00:16:29,280 And usually, it's just not just one file. 312 00:16:29,500 --> 00:16:32,120 It's just combination of the steps that is being done. 313 00:16:32,360 --> 00:16:35,220 So, it's really important to detect the first chain 314 00:16:35,220 --> 00:16:37,580 and then start analyzing it. 315 00:16:39,280 --> 00:16:41,740 So, there was actually an email campaign 316 00:16:41,740 --> 00:16:45,380 that was going on and it was sending PDF attachments. 317 00:16:45,900 --> 00:16:48,760 And none of the avian genes were able to detect it. 318 00:16:49,000 --> 00:16:52,580 When it was analyzed, we found out that there is 319 00:16:52,580 --> 00:16:55,380 a URL list embedded in the PDF file. 320 00:16:55,640 --> 00:16:57,800 The URL downloads JavaScript. 321 00:16:58,660 --> 00:17:01,060 JavaScript has a link to a URL. 322 00:17:01,600 --> 00:17:04,360 URL downloads another executable file. 323 00:17:04,500 --> 00:17:05,720 So, it's just like a chain, right? 324 00:17:05,960 --> 00:17:07,700 But how about if you just like, 325 00:17:08,420 --> 00:17:11,280 detect the first chain and just understand that, 326 00:17:11,360 --> 00:17:13,440 okay, this is a PDF file 327 00:17:13,440 --> 00:17:15,900 and it is matching with this particular chain. 328 00:17:18,300 --> 00:17:20,120 Then we have the flag, right? 329 00:17:20,280 --> 00:17:22,440 Then we have the flag that this is suspicious at least. 330 00:17:22,720 --> 00:17:25,700 And then we even can extract the URL 331 00:17:25,700 --> 00:17:28,500 that is the first URL that it's pointing. 332 00:17:28,760 --> 00:17:31,380 So that we can just extract it, have the information, 333 00:17:31,580 --> 00:17:33,540 and if you want, you can send it to Google Sandbox 334 00:17:33,540 --> 00:17:34,420 for detailed information. 335 00:17:35,880 --> 00:17:37,100 Applying URL rules. 336 00:17:38,420 --> 00:17:40,920 I'm not giving you any URL rules myself, 337 00:17:41,180 --> 00:17:46,920 but Yalda is able to apply URL rules on the given files. 338 00:17:47,220 --> 00:17:52,440 There is just like, sort of like an enable, disable line 339 00:17:52,440 --> 00:17:55,140 that you need to enable or disable it in the config file. 340 00:17:55,360 --> 00:17:58,720 And then you would need to pass some of the URL rules 341 00:17:58,720 --> 00:18:02,360 that interest you based on your needs in a directory. 342 00:18:02,740 --> 00:18:04,900 It goes to that directory if it is enabled 343 00:18:04,900 --> 00:18:08,260 and then applies every single URL rule to the file. 344 00:18:08,420 --> 00:18:10,740 And if there is a match with one of the functions, 345 00:18:10,940 --> 00:18:12,740 it's going to extract the file 346 00:18:12,740 --> 00:18:14,500 and it's going to show you as a list 347 00:18:14,500 --> 00:18:17,000 of the files that has been extracted. 348 00:18:17,840 --> 00:18:19,480 So according to the match 349 00:18:19,480 --> 00:18:22,800 and according to the number of functions that are similar, 350 00:18:22,920 --> 00:18:26,380 it's going to get flagged as malicious, suspicious, or clear 351 00:18:26,380 --> 00:18:28,960 with appropriate severity. 352 00:18:29,960 --> 00:18:32,800 Yalda scoring is actually the brain in Yalda 353 00:18:32,800 --> 00:18:35,300 because everything so far that I told you, 354 00:18:35,360 --> 00:18:38,140 you should know that there is a malicious file 355 00:18:38,140 --> 00:18:39,500 or there is a malicious campaign 356 00:18:39,500 --> 00:18:41,800 and then you have to write the decoder. 357 00:18:42,000 --> 00:18:42,800 So there are none. 358 00:18:43,000 --> 00:18:45,840 It's not like something new get detected. 359 00:18:46,160 --> 00:18:49,600 But this method is somehow based on similarities. 360 00:18:50,140 --> 00:18:55,220 And when I find, when actually Yalda finds a file is malicious, 361 00:18:55,420 --> 00:18:57,940 then it starts clustering the strings. 362 00:18:58,240 --> 00:19:00,820 So it applies clusters on strings. 363 00:19:01,420 --> 00:19:04,840 And also if it is executable and it's detected as malicious, 364 00:19:05,000 --> 00:19:07,040 it's getting the PE sections, 365 00:19:07,040 --> 00:19:09,080 the name section of it, 366 00:19:09,220 --> 00:19:10,660 and then Shannon entropy. 367 00:19:10,960 --> 00:19:12,260 And it clusters the information. 368 00:19:12,780 --> 00:19:14,100 So it's something like this. 369 00:19:14,260 --> 00:19:14,320 See? 370 00:19:15,680 --> 00:19:18,100 The file is detected as malicious. 371 00:19:18,300 --> 00:19:19,600 I'm applying strings 372 00:19:19,600 --> 00:19:23,020 and get all of the strings out of it 373 00:19:23,020 --> 00:19:27,460 and also a Unicode string out of it. 374 00:19:27,940 --> 00:19:29,680 The Unicode strings 375 00:19:29,680 --> 00:19:31,220 and then making a list 376 00:19:31,220 --> 00:19:34,380 and cluster the information in another collection. 377 00:19:35,820 --> 00:19:38,740 There is a file that I have no idea about. 378 00:19:39,020 --> 00:19:41,480 Like none of my decoders were able to detect it 379 00:19:41,480 --> 00:19:43,720 and none of the URLs were able to detect it. 380 00:19:43,800 --> 00:19:45,460 Still I apply the strings on it. 381 00:19:45,680 --> 00:19:47,620 And now based on the similarities 382 00:19:47,620 --> 00:19:49,360 that I find with these strings, 383 00:19:49,660 --> 00:19:51,180 the list of strings found here 384 00:19:51,180 --> 00:19:54,240 and the list of malicious strings in my database, 385 00:19:54,500 --> 00:19:55,880 I start scoring it. 386 00:19:56,320 --> 00:19:58,900 How many files has been detected malicious 387 00:19:58,900 --> 00:20:00,760 that are having similar strings 388 00:20:00,760 --> 00:20:02,620 as the list of this file? 389 00:20:02,860 --> 00:20:04,360 So if this score goes down, 390 00:20:04,380 --> 00:20:04,640 if it goes higher, 391 00:20:04,880 --> 00:20:05,740 I would say that yeah, 392 00:20:05,860 --> 00:20:06,700 it is suspicious. 393 00:20:07,000 --> 00:20:09,860 It needs more analysis. 394 00:20:10,420 --> 00:20:12,100 So it's based on the similarities. 395 00:20:12,460 --> 00:20:15,300 Same thing is applied for the name 396 00:20:15,300 --> 00:20:17,140 in P sections and Shannon entropy. 397 00:20:17,500 --> 00:20:18,720 Name of P sections, 398 00:20:18,960 --> 00:20:20,380 if the file is executable 399 00:20:20,380 --> 00:20:21,740 and if it is malicious, 400 00:20:21,980 --> 00:20:23,000 I'm getting it 401 00:20:23,000 --> 00:20:25,360 and then I convert it to SHA-1. 402 00:20:25,660 --> 00:20:28,780 And also the entropy, 403 00:20:28,960 --> 00:20:29,720 the Shannon entropy, 404 00:20:29,980 --> 00:20:31,640 I'm just collecting these two 405 00:20:31,640 --> 00:20:34,200 and I'm using an AND logic 406 00:20:34,380 --> 00:20:35,100 for these two 407 00:20:35,100 --> 00:20:36,560 and the malicious hash. 408 00:20:36,740 --> 00:20:37,960 And all of this information 409 00:20:37,960 --> 00:20:39,620 are being clustered somewhere. 410 00:20:39,920 --> 00:20:41,980 Now it's just sort of like a scaling, 411 00:20:42,060 --> 00:20:42,260 again, 412 00:20:42,500 --> 00:20:43,960 scaling of the file 413 00:20:43,960 --> 00:20:45,660 that right now I'm seeing 414 00:20:45,660 --> 00:20:47,300 and see what are the similarities. 415 00:20:47,300 --> 00:20:48,740 If it is executable, 416 00:20:48,820 --> 00:20:50,180 then what are the similarities 417 00:20:50,180 --> 00:20:51,320 between these two? 418 00:20:51,480 --> 00:20:53,180 And based on the similarities 419 00:20:53,180 --> 00:20:55,440 and the scoring that I set a threshold, 420 00:20:55,720 --> 00:20:57,980 then I'm going to flag it as suspicious. 421 00:21:00,400 --> 00:21:01,800 Now white listing. 422 00:21:06,240 --> 00:21:07,760 Who handles feeds? 423 00:21:08,180 --> 00:21:09,800 And yeah. 424 00:21:10,300 --> 00:21:12,400 So you guys know that how life is 425 00:21:12,400 --> 00:21:14,580 when you have false positives, right? 426 00:21:14,800 --> 00:21:16,060 And you start using 427 00:21:16,060 --> 00:21:18,880 all of the possible scenarios 428 00:21:18,880 --> 00:21:21,120 and all of the cases for white listing. 429 00:21:21,420 --> 00:21:23,520 You are going to have manual white listing. 430 00:21:23,680 --> 00:21:24,400 You are going to have 431 00:21:24,400 --> 00:21:25,540 a lot of different sources 432 00:21:25,540 --> 00:21:27,820 and still you have some false positives, right? 433 00:21:28,620 --> 00:21:31,380 So to avoid such a thing in Yalda, 434 00:21:31,540 --> 00:21:32,740 I started saying that, 435 00:21:32,800 --> 00:21:33,280 okay, fine, 436 00:21:33,280 --> 00:21:35,240 I'm going to use the traditional way. 437 00:21:35,320 --> 00:21:37,380 I'm going to use a log file 438 00:21:37,380 --> 00:21:39,540 for white listing. 439 00:21:39,780 --> 00:21:41,140 But at the same time, 440 00:21:41,860 --> 00:21:44,600 how about just like clustering the good guys 441 00:21:44,600 --> 00:21:46,480 instead of just clustering bad guys? 442 00:21:46,900 --> 00:21:48,620 So how about white listing 443 00:21:48,620 --> 00:21:50,020 the clear files, 444 00:21:50,200 --> 00:21:52,020 the ones that I know that they are clear 445 00:21:52,020 --> 00:21:53,020 and they are not malicious? 446 00:21:53,420 --> 00:21:55,440 And then extract that information 447 00:21:55,440 --> 00:21:58,320 for using it as white list. 448 00:21:58,540 --> 00:21:59,420 So, you know, 449 00:21:59,460 --> 00:22:01,560 as Yalda is running more samples, 450 00:22:01,780 --> 00:22:03,260 it's going to extract more, 451 00:22:03,280 --> 00:22:05,540 bad stuff and good stuff 452 00:22:05,540 --> 00:22:06,500 at the same time, right? 453 00:22:06,780 --> 00:22:09,040 So the white listing is getting stronger 454 00:22:09,040 --> 00:22:13,260 as the detection of malicious hashes 455 00:22:13,260 --> 00:22:15,460 also get stronger. 456 00:22:16,680 --> 00:22:18,280 So as I said, 457 00:22:18,440 --> 00:22:21,260 I'm just like applying a strings on clear file 458 00:22:21,260 --> 00:22:23,800 and are clustering it. 459 00:22:23,940 --> 00:22:26,440 And also I am clustering the PE section names 460 00:22:26,440 --> 00:22:29,240 and Shannon entropy of executable files 461 00:22:29,240 --> 00:22:31,020 that are known as clear. 462 00:22:31,240 --> 00:22:33,080 And then I'm using this information 463 00:22:33,280 --> 00:22:34,640 sort of to white list 464 00:22:35,100 --> 00:22:39,800 and for having like more accurate results. 465 00:22:41,220 --> 00:22:42,480 Let's have a demonstration. 466 00:22:44,640 --> 00:22:47,520 So here I'm having three files in the folder 467 00:22:48,100 --> 00:22:51,060 and it starts analyzing the file. 468 00:22:51,260 --> 00:22:54,000 It extracts the files 469 00:22:54,000 --> 00:22:55,400 and then in this case, 470 00:22:55,480 --> 00:22:56,780 it's downloading the attachments 471 00:22:56,780 --> 00:22:59,780 and you see that there are a list of domain list 472 00:22:59,780 --> 00:23:00,860 that they are all malicious 473 00:23:00,860 --> 00:23:02,240 and it's extracting it. 474 00:23:02,240 --> 00:23:05,820 Also, it goes and analyze each single file 475 00:23:05,820 --> 00:23:06,620 like in detail. 476 00:23:06,800 --> 00:23:08,160 If there is any embedded object, 477 00:23:08,400 --> 00:23:09,780 it is going to extract it. 478 00:23:09,880 --> 00:23:10,560 Like in this case, 479 00:23:10,660 --> 00:23:12,340 it does have embedded objects. 480 00:23:12,480 --> 00:23:15,320 So it just like lists the list of embedded objects here, 481 00:23:15,480 --> 00:23:16,640 like embedded files. 482 00:23:16,940 --> 00:23:19,620 And not only it has the detailed information 483 00:23:19,620 --> 00:23:20,880 of the file itself, 484 00:23:21,120 --> 00:23:25,760 but it also analyzes the embedded file in it 485 00:23:25,760 --> 00:23:27,300 and it gives you information 486 00:23:27,300 --> 00:23:28,780 with a link to the parent. 487 00:23:30,220 --> 00:23:32,020 And everything is being inserted 488 00:23:32,240 --> 00:23:33,520 in the database as I mentioned. 489 00:23:34,560 --> 00:23:39,300 So Yalda sort of like minimizes false positive 490 00:23:39,300 --> 00:23:41,320 or better to be said, 491 00:23:41,380 --> 00:23:42,400 it has better correlation. 492 00:23:42,800 --> 00:23:46,360 So the way that we are doing Yalda scoring, 493 00:23:46,640 --> 00:23:48,360 we get more accurate results 494 00:23:49,200 --> 00:23:51,780 because as we are analyzing more samples, 495 00:23:51,920 --> 00:23:54,440 we are getting like stronger results. 496 00:23:55,200 --> 00:23:57,740 Same thing for automated white listing 497 00:23:57,740 --> 00:23:59,920 that we are getting better results 498 00:23:59,920 --> 00:24:02,040 as we start analyzing 499 00:24:02,240 --> 00:24:02,880 more samples. 500 00:24:03,180 --> 00:24:05,100 And then this clustering data, 501 00:24:05,100 --> 00:24:06,900 again, we are having better results. 502 00:24:07,100 --> 00:24:09,320 And then categorizing algorithm is somehow 503 00:24:10,200 --> 00:24:13,540 that I'm not saying that if I don't know, 504 00:24:13,680 --> 00:24:15,580 like it doesn't match with my decoders, 505 00:24:15,700 --> 00:24:18,680 then I'm not going to say 100% it's malicious, 506 00:24:18,920 --> 00:24:21,460 but it has enough high severity 507 00:24:21,460 --> 00:24:23,240 that you can select and say that, 508 00:24:23,280 --> 00:24:24,420 okay, this is suspicious 509 00:24:24,420 --> 00:24:26,000 and there is a flag going on. 510 00:24:26,100 --> 00:24:28,620 So I better start analyzing this set of files 511 00:24:28,620 --> 00:24:29,500 with this criteria 512 00:24:29,500 --> 00:24:30,900 because you have 20 indicators 513 00:24:30,900 --> 00:24:32,340 that we can select out of it. 514 00:24:34,680 --> 00:24:35,860 This is the GitHub. 515 00:24:36,140 --> 00:24:37,140 So you can go ahead 516 00:24:37,140 --> 00:24:40,380 and start downloading and using it. 517 00:24:40,600 --> 00:24:41,880 It's in Fidelis GitHub. 518 00:24:51,240 --> 00:24:53,100 So there are three folders 519 00:24:53,100 --> 00:24:55,260 like bin, source, Yara rules. 520 00:24:55,400 --> 00:24:56,440 Yara rules is a place 521 00:24:56,440 --> 00:24:59,580 that you place your Yara rules if you want to. 522 00:24:59,640 --> 00:25:00,760 If not, you don't have to. 523 00:25:00,760 --> 00:25:01,540 You don't have to worry about it. 524 00:25:02,140 --> 00:25:05,000 Bin has all the modules and functions, 525 00:25:05,460 --> 00:25:06,540 as you see here. 526 00:25:06,920 --> 00:25:08,160 You can even go ahead. 527 00:25:08,280 --> 00:25:09,860 It's just like very straight format 528 00:25:09,860 --> 00:25:11,060 and you can just go ahead 529 00:25:11,060 --> 00:25:13,100 and if you want to add more decoders, 530 00:25:13,160 --> 00:25:13,940 you can do that. 531 00:25:14,880 --> 00:25:17,280 And the config file is something 532 00:25:17,280 --> 00:25:18,900 that I go through it in detail. 533 00:25:19,080 --> 00:25:20,140 It's very simple. 534 00:25:20,320 --> 00:25:21,580 It's just like a couple of lines 535 00:25:21,580 --> 00:25:24,020 that you need to specify the directories in it. 536 00:25:24,220 --> 00:25:25,760 And when you want to run it, 537 00:25:25,780 --> 00:25:27,920 run this guy, Yalda file analyzer. 538 00:25:30,760 --> 00:25:31,880 You would need to install 539 00:25:31,880 --> 00:25:33,460 the required Python modules 540 00:25:33,460 --> 00:25:35,980 and you would need, 541 00:25:36,100 --> 00:25:37,220 these are the Python modules 542 00:25:37,220 --> 00:25:38,040 that you would need. 543 00:25:38,920 --> 00:25:40,660 And this is the config file. 544 00:25:40,800 --> 00:25:42,100 You just go ahead and open it 545 00:25:42,100 --> 00:25:45,960 and start just adding the directories 546 00:25:45,960 --> 00:25:47,760 that you want or enable or disable. 547 00:25:47,880 --> 00:25:49,160 Like if you don't want to have 548 00:25:49,160 --> 00:25:50,360 virus total information 549 00:25:50,360 --> 00:25:52,140 or if you don't want to have 550 00:25:52,820 --> 00:25:54,980 sort of like Yara rule 551 00:25:54,980 --> 00:25:56,000 to be applied to it, 552 00:25:56,080 --> 00:25:57,780 you can say that zero, 553 00:25:57,960 --> 00:25:59,220 just like disable it. 554 00:25:59,220 --> 00:26:02,080 If you want to sort of like have a printout, 555 00:26:02,120 --> 00:26:03,180 you can have debug one 556 00:26:03,180 --> 00:26:04,840 or you can just make it zero. 557 00:26:05,040 --> 00:26:05,940 And of course, 558 00:26:05,940 --> 00:26:07,440 because I'm using MongoDB, 559 00:26:07,740 --> 00:26:10,940 you would need to have a MongoDB installed 560 00:26:10,940 --> 00:26:13,040 and just like specify the criteria 561 00:26:13,040 --> 00:26:14,060 that you're looking for. 562 00:26:14,780 --> 00:26:16,420 And to run it, 563 00:26:16,460 --> 00:26:17,220 you go to source 564 00:26:17,220 --> 00:26:18,640 and you start running 565 00:26:18,640 --> 00:26:20,400 its Yalda file analyzer 566 00:26:20,400 --> 00:26:21,480 and it starts analyzing 567 00:26:21,480 --> 00:26:23,060 each single file in detail 568 00:26:23,060 --> 00:26:24,520 and passes you the information. 569 00:26:25,020 --> 00:26:26,820 I'd like to talk to, 570 00:26:26,940 --> 00:26:29,200 thanks to John 571 00:26:29,220 --> 00:26:30,600 my boss, 572 00:26:30,860 --> 00:26:32,040 Hardik Modi, 573 00:26:32,920 --> 00:26:34,200 Chad Robertson 574 00:26:34,200 --> 00:26:35,400 and Jason Reeves. 575 00:26:35,540 --> 00:26:37,260 They actually gave me 576 00:26:37,260 --> 00:26:38,300 really great feedbacks 577 00:26:38,820 --> 00:26:40,120 for improving Yalda 578 00:26:40,120 --> 00:26:43,440 and really appreciate 579 00:26:43,440 --> 00:26:44,800 all of your feedbacks. 580 00:26:45,560 --> 00:26:47,880 So if you need to contact me 581 00:26:47,880 --> 00:26:49,020 or if you have any questions, 582 00:26:49,220 --> 00:26:50,460 this is my email address 583 00:26:50,460 --> 00:26:51,440 or Twitter. 584 00:26:53,120 --> 00:26:54,660 And thank you so much. 585 00:26:54,820 --> 00:26:56,100 If you have any questions 586 00:26:56,100 --> 00:26:56,780 or feedback, 587 00:26:57,120 --> 00:26:57,780 thank you. 588 00:26:59,220 --> 00:27:00,220 Thank you.