WEBVTT 00:01.240 --> 00:03.020 Okay, hi everyone. 00:03.400 --> 00:07.640 This is Zoltan and Hiram's... 00:09.800 --> 00:13.640 in conjunction with Microsoft and Cujo AI. 00:14.020 --> 00:24.740 They're launching a very interesting security competition which kind of relates back to some of the stuff we were talking about in opening remarks and I'll let them take it away. 00:51.600 --> 00:54.640 Sorry about the audio. 00:59.930 --> 01:01.950 I will get the audio working. 01:01.950 --> 01:03.270 I am sorry. 01:04.010 --> 01:06.650 Why is it not capturing the right thing? 01:22.320 --> 01:27.920 I'm Balazs and this is a project together with Hiram Anderson. 01:29.220 --> 01:48.280 We are going to talk about machine learning detection bypasses and in the past you probably have seen some researches where people modified existing images and the goal was to bypass machine learning classifiers. 01:48.280 --> 02:00.640 So for example for a human viewer the new image looks like the original one but for a machine learning classifier this looks like something different. 02:00.640 --> 02:06.000 Like it will believe that this looks like an ostrich. 02:06.320 --> 02:22.480 In this talk we are going to present machine learning bypasses when it comes to malicious software and there have been some interesting researches in the past regarding this topic as well. 02:22.580 --> 02:42.260 For example last year there was a new research published where people extracted strings from a known game executable, appended this to a known malware and it was able to bypass production machine learning model. 02:42.480 --> 02:55.520 I also did some research in the past and like four years ago you were able to bypass some machine learning models just by packing a sample with UPX. 02:57.020 --> 03:16.260 In order to advance the field of offensive and defensive machine learning based malware detection, last year we created a challenge where you had to download 50 working malware samples, you had to download three machine learning models with eight weights, 03:16.260 --> 03:29.460 modify the malware samples to evade detection by all models and if you were lucky and you had the most points you were able to win this nice GPU card. 03:30.180 --> 03:40.720 In total 70 people registered for this competition and at least 11 people were able to bypass at least one machine learning model. 03:41.120 --> 03:52.920 Congratulations to the winner William Fleschmann and I highly recommend you to check out his blog post on how he won this competition. 03:54.100 --> 03:58.880 There were some other write-ups and papers as well. 03:58.980 --> 04:02.320 I do recommend you to check out those as well. 04:02.320 --> 04:08.140 You can see one from Jakub and one from Fabrizio on the following links. 04:09.100 --> 04:15.380 When it comes to win this competition, multiple approaches were used. 04:15.380 --> 04:41.080 Some people started with a simple packer like the one I mentioned, but unfortunately some of the samples are already packed in a way and this means that if you use UPX or something similar it will not work anymore. 04:43.200 --> 04:49.480 Another great approach was to add new sections to the executable. 04:49.480 --> 05:00.760 For example, you can extract the end user license agreement resource from Microsoft Files and add it to the malware samples multiple times. 05:01.000 --> 05:13.740 This approach was really good at bypassing the detection for the ML models but unfortunately again this broke some of the malware binaries. 05:14.980 --> 05:39.360 Fun fact, if you just simply add sections to a malware sample, you might be able to bypass some antivirus detection because for performance reasons some AV engines check the number of sections before they evaluate the rest of the rules. 05:40.620 --> 05:49.400 At the end, the winning strategy was to just append random data to the end of the executable. 05:49.400 --> 05:51.690 This is called as an overlay. 05:52.280 --> 05:58.910 Even though this is a very simple strategy, it worked during last year's competition. 06:02.750 --> 06:12.800 This is also an easy way to bypass if the sample has any kind of self-protection, for example. 06:13.740 --> 06:20.660 Just by increasing the size of the sample, again, you might be able to bypass some antivirus engines. 06:20.660 --> 06:28.000 Again, they can have a file size in their rules. 06:31.220 --> 06:42.640 And one important thing, as you can see on the top right image, that's a visual representation of a malware. 06:42.640 --> 06:50.740 And I just appended some random strings to the end of this sample. 06:50.740 --> 07:03.820 And if you look at the green visual, you can clearly see how this changed the visual representation of the sample. 07:05.800 --> 07:11.620 For us, there were some key takeaways from last year's competition. 07:11.620 --> 07:17.820 For example, some of the machine learning models are way too academic, but not very effective in practice. 07:18.740 --> 07:26.100 Turned out it's not just us, but everybody thinks that the DF tool is awesome. 07:26.120 --> 07:31.880 This is a Python package you can use to modify binaries. 07:32.760 --> 07:38.500 And as it is the case with malware, it is always tricky to deal with them. 07:38.500 --> 07:45.720 For example, some of the samples do not reproduce the same indicators of compromise over time. 07:46.360 --> 07:55.640 This can be because, for example, the command and control server is down, and dealing with packed and protected samples can be hard sometimes. 07:57.740 --> 08:27.460 I also checked the SSDeepHashes of some of the samples, and it was interesting to see that whenever people added repeating patterns to the sample, for example, the same section, or they added the same overlay over and over again to the sample, then it created a repeating pattern in the SSDeepHash as well, 08:27.460 --> 08:36.340 which can be used for detecting a sample which uses machine learning evasion, for example. 08:36.840 --> 08:42.280 This year, we created the Defender and the Attacker challenge. 08:42.280 --> 08:59.680 In the Defender challenge, you had to create your own machine learning model and submit this to the competition in a Docker format, and in the Attacker challenge, now the machine learning models are not available for you, so this is now a black box challenge. 09:00.440 --> 09:12.160 And if you win this competition, either the Defender or the Attacker challenge, you can win some Azure credits for your machine learning research plans. 09:12.830 --> 09:15.540 The defensive track is already over. 09:15.540 --> 09:35.800 We received two submissions that passed the minimum requirements, and the offensive track already started, so I highly recommend you to go to our website, analsec.io, and check out what the competition is. 09:35.820 --> 09:58.700 In this year, we have used the malware families, and if you go to our website, review the terms of service, and then you can download the 50 provided malware samples, and after that, it is your time to modify the samples in order to evade the detection. 09:58.860 --> 10:08.020 And new to this year, you can use an API to check your samples or submit your samples. 10:08.240 --> 10:17.340 I also recommend you that you verify that the malware functionality remains the same in your local Windows box. 10:18.020 --> 10:42.000 Then, when you upload the zip files, or you can just upload the partial zip files, meaning that you only submit some of the samples and not all of them, you can receive one point for each bypassed machine learning model, which means that for every sample, 10:42.000 --> 10:44.800 you can get up to three points. 10:44.840 --> 10:48.760 And as usual, highest score wins. 10:48.880 --> 10:59.160 The details about this will be provided by Hyrum, and in order to claim your prize, you have to publish your solution. 10:59.840 --> 11:07.980 Please note that you have to keep the file names as it were in the original zip file. 11:07.980 --> 11:13.200 This helps us to track which file you modified originally. 11:14.400 --> 11:20.720 We also provide some additional tips and tricks you might use in this competition. 11:20.720 --> 11:29.220 Some of them may not make sense, but you can modify an executable in a lot of different ways. 11:29.220 --> 11:47.860 For example, you can add or remove signatures, change section names properties, modify the import or export tables, create TLS callbacks, change the PA header, fix or change the checksums, add, modify, or remove the version information, create new entry points, 11:47.860 --> 11:51.760 or just change some code or data in it. 11:53.040 --> 12:04.620 Still, it's not allowed to create droppers or self-extracting archives, because this will kind of defeat the purpose of the whole competition. 12:04.880 --> 12:13.770 And this year, keep in mind that multiple registration is against the rules, and it will result in immediate disqualification. 12:14.190 --> 12:29.570 Please do join our Slack channel, where you can discuss everything with us, and you can also discuss your progress with the other participants of this competition. 12:31.970 --> 12:46.550 Just a side note, the whole frontend was created in Python Flask admin, we are using Cloudflare, Nginx, and GUnicorn for scalability and performance reasons. 12:46.610 --> 13:01.690 There are some backend scripts running with Python, scheduled by Chrome, and as it was the case last year, we still use the VMware Sandbox to evaluate the samples. 13:02.170 --> 13:21.070 As mentioned, we already have an API, so if you want to check your sample against machine learning, against all the machine learning models, or just against one machine learning model, you can use the API just to do that. 13:21.310 --> 13:25.930 And also, you can use the API to get the results. 13:25.930 --> 13:40.610 And if you are satisfied with bypassing the machine learning models, you can upload your zip files and query the zip status and the sample statuses as well with the API keys. 13:40.690 --> 13:53.250 This is all I wanted to share with you guys, but please welcome Hiram, who will present you some other tips and tricks you can use to win this competition. 13:53.330 --> 13:54.630 Thank you. 13:56.640 --> 14:07.180 I'm going to describe to you the example solution in the machine learning security evasion competitions attacker challenge that has just begun. 14:07.400 --> 14:16.780 The models that you'll be attacking this year have been submitted by participants of the previous round in the defender challenge. 14:16.940 --> 14:22.960 Two of the models from the previous round have qualified to be included in this round. 14:22.960 --> 14:27.600 In addition, we have hosted our own model for you to attack. 14:27.600 --> 14:36.360 That model is trained on the Ember dataset and includes some basic capability to detect adversarial examples. 14:36.360 --> 14:44.220 The source code and model weights for this defended Ember model are provided on the competition's GitHub site. 14:44.880 --> 14:54.720 However, the remaining models are to you complete black boxes where you only get to observe the hard label predictions. 14:54.720 --> 15:01.120 That is a zero or a one for an output that you provide to the machine learning models. 15:01.980 --> 15:07.680 The final leaderboard ranking will be set by the following rank ordered criteria. 15:07.980 --> 15:20.500 First, the total number of evasions with one point for each of the three ML models times 50 malware samples, meaning that the maximum score is 150. 15:20.700 --> 15:30.280 Remember though that each evasive sample must reproduce its original functionality in a sandbox in order to be awarded a point. 15:30.280 --> 15:39.980 Functionality is verified only when you upload a zip file containing your candidate malware samples. 15:39.980 --> 15:47.580 It will not be verified when you merely query the machine learning models through the API. 15:47.940 --> 15:58.800 In the event of a tie for point number one, contestants will be ranked by the number of model queries used through the API. 15:59.000 --> 16:04.820 And lastly, the timestamp of your final zip upload would break any subsequent tie. 16:04.820 --> 16:14.760 More than likely though, we won't get to point number three, so you should feel incentivized to continue competing right up until the competition deadline. 16:15.500 --> 16:22.060 Even if you see a perfect score on the leaderboard, because you might achieve that same perfect score but do it more efficiently. 16:23.400 --> 16:27.540 So as a contestant, you can choose any strategy you'd like to compete. 16:27.540 --> 16:35.660 But to demonstrate one possible strategy, we have released some example code on the competition's GitHub site. 16:35.660 --> 16:41.700 You can find more information about the nitty-gritty details of this approach on the website. 16:41.700 --> 16:49.820 It essentially is using a discrete optimization technique over a space of functionality preserving file modifications. 16:50.040 --> 16:55.300 However, the general strategy might be more useful for you to adopt. 16:55.380 --> 17:09.180 The strategy consists of, is really simple, consists of doing a bunch of bulk work using an algorithm in part A and then kind of batting cleanup for manual manipulation of malware samples in part B. 17:09.180 --> 17:12.880 And I'm going to be describing and demoing the code for part A today. 17:13.100 --> 17:23.320 In it, because we'd like to be efficient in the number of queries against the hosted machine learning models, we'll actually break this attack into two parts. 17:23.320 --> 17:34.860 An offline attack where we use the defended Ember model for which we have code to kind of work out our strategy and generate initial malware samples. 17:34.860 --> 17:41.020 We hope that those seeds might evade some of the online models that are hosted. 17:41.520 --> 17:54.760 And then in the online attack, we'll take those initial seeds and the algorithm will further optimize and discover additional file modifications required to evade the online hosted models. 17:54.760 --> 18:24.480 Some tricks that we're using here include label smoothing, where we're converting the hard label outputs into a soft score by averaging four things, the three hard label outputs from the hosted machine learning models, as well as a local score from a local machine learning Ember model that will be used as a heuristic to kind of guide the optimization process. 18:25.520 --> 18:31.820 So as I demo this code, I want you to please be aware that this code writes malware to disk. 18:31.820 --> 18:37.140 So please do run this code only using a Linux VM. 18:38.040 --> 18:45.720 To begin, we initialize the attack by analyzing a connect collection of benign files. 18:45.720 --> 18:54.500 This init subcommand extracts elements of these benign files that will be later injected into the malware. 18:55.660 --> 19:02.640 To launch our offline attack, we'll run a local copy of the Ember model in the top window. 19:02.640 --> 19:11.800 Then in the bottom window, we'll use the run command, passing in malware samples that we downloaded after registering on the website. 19:11.980 --> 19:25.840 The tool will then write successful evasion attempts to pass one slash success that we've specified in the command line, and failed attempts to pass two slash failure folder. 19:26.020 --> 19:42.130 And also included in each output directory will be the history of file modifications that will be useful if we'd like to pick up to resume a failed attempt. 19:43.470 --> 20:00.830 So to demonstrate that, in a second pass of the offline attack, we'll start with the pass one failures and iterate on the optimization approach, again storing successes and failures to a pass two directory. 20:04.210 --> 20:22.220 So after doing that a number of times and having collected a bunch of candidate samples offline that evade the local defended Ember model, we'll then use those candidates as seeds for an online attack, which now counts against our API query usage. 20:22.560 --> 20:38.900 To do an online attack, simply use this tool with the dash dash online flag, and the optimization will then continue trying to find file modifications that will bypass all three of the hosted models. 20:41.200 --> 20:48.520 Of course, you want to do perhaps as many, you know, as many iterations as necessary, and the online version of this attack. 20:48.520 --> 21:07.780 But after you've done so, in a final pass of the online attack, you can now collect the successful samples into a zip file that you would then validate in a Windows 10 virtual machine, and then upload to the website for validation and leaderboard scoring. 21:08.400 --> 21:23.580 I want to point out that since there is a chance that by running this code, file modifications might break some of the samples, you should always run these samples in a Windows 10 sandbox before uploading to the competition website. 21:23.700 --> 21:30.180 Also note that zip file uploads contribute against your API query count. 21:30.200 --> 21:40.380 So it is to your benefit to double check your work and make sure that any files you upload are functional, so you don't have to redo that work and upload again. 21:40.500 --> 22:03.180 As a final note, kind of tricky that since the hosted models might be actually changing state and learning from the queries that you and others are giving them, there's a possibility that an evasive variant sample that you discovered along the way may no longer evade a model by the time that you upload your zip file. 22:03.180 --> 22:08.580 So be... I don't know that will be the case, but please be aware that that is a possibility. 22:09.020 --> 22:11.300 So with that, good luck on the competition. 22:12.220 --> 22:14.800 Visit the website at emailsecta.io. 22:14.800 --> 22:17.920 The competition will run for over six weeks. 22:18.540 --> 22:32.220 And those who are ranked first and second on the leaderboard will win our grand and first prizes respectively, so long as they publish their solution. 22:33.620 --> 22:42.020 And with that, I'd like to thank our sponsors, Microsoft and Cujo AI, with partners MRG Efitas and VMRay.