[00:00.720 --> 00:08.780] Oh, okay. Hi, welcome to our next talk. This one's by Gavin Kondyk, also known as GTKondyk on [00:08.780 --> 00:16.040] Twitter. He's a pre-circuited researcher, consultant, and has been super involved in [00:16.040 --> 00:24.060] the AI village community for a long time. He's an awesome guy, so I'll let him take it away. [00:27.770 --> 00:34.470] Hello, and welcome to Hacking with Skynet, How AI is Empowering Adversaries. [00:34.750 --> 00:39.850] My name is Gavin. I go by the pseudonym GTKondyk. I am a security researcher [00:39.850 --> 00:44.410] and senior security consultant. I'm very passionate about network security, [00:44.410 --> 00:49.570] both the attack and defense side. And through that passion, I run a project called NetSecExplained, [00:49.570 --> 00:55.110] which is a blog and YouTube channel where I explain intermediate and advanced level concepts [00:55.110 --> 01:02.870] in an easy-to-understand way. So, machine learning, let's get everybody on the same page. [01:03.210 --> 01:10.410] What is machine learning? Well, first, we have this big umbrella term of AI. And parts of AI [01:10.410 --> 01:16.350] are machine learning. Parts of AI are unrelated. So, we have rule-based artificial intelligence. [01:16.350 --> 01:22.750] We have classical image recognition. But inside of AI, we have this small subset called machine [01:22.750 --> 01:28.910] learning. And machine learning specifically allows us to take statistical analysis and [01:28.910 --> 01:34.710] draw inferences from data and perform data discovery techniques. And then out of machine [01:34.710 --> 01:40.570] learning, we also have deep learning, which is specifically deep neural networks. And this can [01:40.570 --> 01:45.750] be convolutional neural networks, which are used primarily for image recognition, or recurrent [01:45.750 --> 01:51.210] neural networks, which are used for cyclical pattern recognition, such as a heartbeat monitor. [01:53.610 --> 02:02.110] So, the burning question is, how is AI empowering adversaries? Or how can it empower adversaries [02:02.110 --> 02:09.110] moving forward? So, I see the age of AI today, or the age of machine learning today, [02:09.110 --> 02:13.270] we're at the very beginning of what it can do for the security industry on both the attack and the [02:13.270 --> 02:20.350] defense side. Now, if you remember back in the 80s and 90s, antivirus, intrusion detection, [02:20.350 --> 02:24.570] vulnerability scanners, these were all simple scripts that were written. And then over time, [02:24.570 --> 02:30.030] they grew into what are now multi-billion dollar industries. And so, I think that we're at the very [02:30.030 --> 02:37.550] beginning of what AI and machine learning can do for the security community. So, the way that I'm [02:37.550 --> 02:42.930] going to bring about this talk is really in two major parts. So, the first is to talk about [02:42.930 --> 02:49.450] offensive AI tools. Why do we want these? What's out there already? What are we seeing in the wild? [02:49.450 --> 02:54.390] How do we build them? And then adversarial machine learning, which is attacking the machine [02:54.390 --> 03:01.490] learning algorithms themselves. So, let's start with offensive AI tools. [03:04.370 --> 03:10.850] So, the first question is, why would we even want these? Well, offensive AI tools and AI tools in [03:10.850 --> 03:17.450] general will allow us to operate at a much faster speed and a much larger scale than we're currently [03:17.450 --> 03:22.510] able to do. And this is going to allow us to automate a lot of manual tasks. Now, we don't [03:22.510 --> 03:28.370] need to think about it as automating everything. But if there's something that requires or manages [03:28.370 --> 03:34.570] or deals with a lot of data, and these are things that we can apply a certain amount of fuzzy logic [03:34.570 --> 03:41.210] to, then this is a perfect candidate for automating with some form of AI or machine learning. [03:41.370 --> 03:45.690] And then, of course, this allows us to dynamically and intelligently explore our target attack [03:45.690 --> 03:52.330] surface and uncover hidden blind spots that come about some of the biases that we intrinsically have. [03:52.610 --> 03:57.050] And so, being able to statistically analyze some of this data or perform some machine learning [03:57.050 --> 04:00.710] operations, this will allow us to kind of uncover some of those blind spots [04:00.710 --> 04:09.710] and look for things that we don't already know. And then, how realistic is this? [04:09.710 --> 04:17.930] Well, back in 2016, DARPA kind of took on a large undertaking in their CyberGram challenge. [04:18.050 --> 04:23.250] In here, what they did was they required participants to develop some form of artificial [04:23.250 --> 04:30.710] intelligence that will perform a series of attack chain tasks. So, in this case, it was going to be [04:30.710 --> 04:37.050] vulnerability discovery, exploitation, patching, and then data theft. So, in the form of data [04:37.050 --> 04:42.250] theft, they have capture the flag. And then, as part of their major grand challenge, they had [04:43.730 --> 04:48.710] an attack, defend, capture the flag event that was air gapped. And you can see that in the picture [04:48.710 --> 04:56.230] kind of off to the side, where there's no human interaction whatsoever for the entire event. [04:56.230 --> 05:02.950] And everything had to be automated. So, this was a big reveal on a world stage that let us see [05:02.950 --> 05:07.790] what the power of artificial intelligence is, especially when it comes to a number of these [05:07.790 --> 05:16.170] tasks. And then, where are we seeing applications of AI already? So, the current proof of concepts [05:16.170 --> 05:24.930] we can see for social engineering. We can see in defense detection and evasion. There are some [05:24.930 --> 05:29.250] researchers that are identifying intrusion detection systems and sandbox detection systems [05:29.250 --> 05:36.350] so that attacks can be stealthier and break out of various systems and bypass existing [05:36.350 --> 05:42.790] defense measures. Evaluating data leaks. So, large data sets such as password dumps, [05:42.790 --> 05:51.010] subdomains, things of that nature, websites, applications, configuration files. Automated [05:51.010 --> 05:56.790] network exploitation. So, this will be going from standard user to domain admin, automating that [05:56.790 --> 06:04.680] process, as well as normal internal and external network pen tests. And then, software exploitation. [06:05.630 --> 06:12.170] And so, we'll see this with AFL, which is a software fuzzer that will utilize its own custom [06:12.170 --> 06:20.390] compiler, put in certain key points in the software itself, and then it allows the system to fuzz and [06:20.390 --> 06:26.830] track where through the application user input is being provided and how that can reach potentially [06:26.830 --> 06:32.790] vulnerable functions. So, let's start with social engineering. Social engineering, there's a couple [06:32.790 --> 06:38.970] things that are in the works right now that will allow us to improve our social engineering [06:38.970 --> 06:46.290] capabilities and improve attackers' abilities to exploit corporations and systems. So, in this case, [06:46.290 --> 06:52.070] we have an NVIDIA project called StyleGAN. StyleGAN, what it does is generate human faces. [06:52.710 --> 06:58.970] So, all four of the images up at the top you see are actually generated by StyleGAN. And then, just [06:58.970 --> 07:03.610] to kind of show you that there's no tricks up my sleeve, on the bottom row, we have StyleGAN [07:03.610 --> 07:10.950] generated cats. And you can see here in the bottom right corner, this abomination that kind of looks [07:10.950 --> 07:19.410] like a cat, but is really messed up and difficult to see, difficult to really recognize for a human. [07:19.410 --> 07:25.530] And this is kind of the imperfect model generating a cat-like image. But the level of detail that [07:25.530 --> 07:28.790] these can go into, if you look at the cat over here on the left, just kind of sitting on the [07:28.790 --> 07:34.350] carpet, you can see the individual hair, the whiskers, the texture of the carpet underneath. [07:34.950 --> 07:41.810] And so, these are really difficult to tell whether or not they are legitimate faces or cats, or if [07:41.810 --> 07:46.450] these are machine learning generated. One of the telltale signs, if you look here in the top right [07:46.450 --> 07:51.250] corner, this gentleman, it seems like there's a picket fence behind him, but it's a little warped [07:51.250 --> 07:57.370] in the back. You also notice that all of the images have a blurred background. And if you [07:57.370 --> 08:02.850] look really carefully, all of the eyes point directly at you. So, it doesn't matter what [08:02.850 --> 08:09.070] direction they're faced, the eyes are pointing straight. So, there's already some technologies [08:09.530 --> 08:13.910] and some research being performed to kind of identify what is a legitimate human face and [08:13.910 --> 08:19.410] what is a generated human face. Now, why is this important? How can we use this for social [08:19.410 --> 08:25.950] engineering? Well, for several types of social engineering campaigns, what's pretty common to do [08:25.950 --> 08:34.250] is go to the 10th page of a Google image search, grab somebody's stock photo or profile photo, [08:34.250 --> 08:38.950] and then you can paste that up as your own LinkedIn profile photo and say, okay, I am now [08:38.950 --> 08:46.370] Becky. I'm not a 32-year-old male. My name is Becky. I'm 27. I'm starting my career and I want [08:46.370 --> 08:51.950] to ask somebody, hey, as a fellow woman, how did you get started? And then I can use that to build [08:51.950 --> 08:57.830] rapport, have my pretext, and perform social engineering tasks to try and exploit employees, [08:57.830 --> 09:07.090] maybe in the HR department, sales, wherever. And so, instead of grabbing that image and throwing [09:07.090 --> 09:12.490] it up as a pretext where an analyst can go in, grab that same image, do a reverse Google image [09:12.490 --> 09:18.610] search, see that it's a stock photo, call me out, realize that I'm fake, what I can do is use [09:18.610 --> 09:24.570] something like StyleGAN to generate my own face. So, why steal a person when I can just make a [09:24.570 --> 09:31.950] whole new person? But there needs to be a little bit more legitimacy to my pretext. So, this is [09:31.950 --> 09:38.410] where GPT-2 and Grover come into play. GPT-2 and Grover, they operate very similarly. These are [09:38.410 --> 09:44.370] automated generation of long-form content. So, these can be used to generate emails, [09:44.370 --> 09:51.550] blog posts, articles, anything of that nature, with very minimal effort from the attacker or [09:51.550 --> 09:58.110] the adversary in that respect. And so, unfortunately, the website that was used to generate [09:58.110 --> 10:04.610] these has been taken offline, but the GPT-2 model is still something that you can download, put on [10:04.610 --> 10:11.410] your own systems, and generate long-form content. So, if I was developing a pretext and I can use [10:11.410 --> 10:19.690] this to write phishing emails that could get past a spam filter, or that look distinctly unique, [10:19.690 --> 10:23.750] so that I'm not sending the same email to multiple people within an environment, [10:24.370 --> 10:29.910] or I can use this to write blog posts and long-form content, and because any sort of [10:29.910 --> 10:39.010] Google image search or... what is that called? Plagiarism detection, it will see that I am the [10:39.010 --> 10:44.210] only source of truth, therefore I must be a real person, therefore my website must be legitimate. [10:44.950 --> 10:50.790] And so, this will kind of add to the pretext. And then on top of that, long-form content Google [10:50.790 --> 10:56.370] really enjoys, and so that can give us some Google ranking to add a little bit more credibility to our [10:56.370 --> 11:04.130] initial pretext. So, this is one of the scarier ones that I see as being at the very beginning [11:04.130 --> 11:09.710] of the research, and I'm curious to see where this will be in the next several years. We have [11:09.710 --> 11:17.450] Lyrebird and Tacotron. Lyrebird and Tacotron, these are text-to-speech that are based on human voices [11:18.130 --> 11:24.970] and use those human voices to generate conversation bits. And so, we can use this, [11:24.970 --> 11:30.790] I foresee, in the phishing campaigns where we're actually calling people or leaving voicemails. [11:30.790 --> 11:34.950] So, it's a lot less automated sounding, a little less robocall, [11:34.950 --> 11:38.730] and then you just add a little bit of compression and it sounds like you just have a really bad [11:38.730 --> 11:44.090] connection. So, I have a couple audio clips. I want you to hear them. So, these first two, [11:44.090 --> 11:49.670] I have the original voice and then the second one is the synthetic voice based off that original [11:49.670 --> 11:57.150] voice. So, let's go ahead and take a listen. In another moment, down went Alice after it. [11:57.150 --> 12:01.330] Never once did she consider how in the world she was to get out again. [12:03.310 --> 12:09.090] Awesome. And then the synthetic voice built off of that sample. My voice might be generated by [12:09.250 --> 12:14.290] a computer, but I think it sounds pretty human. I don't know exactly how they made it, but I'm [12:14.290 --> 12:20.110] really impressed. So, what I'm imagining is that as these models get better, what we can do is [12:20.110 --> 12:26.590] take interviews of CEOs and VPs of large corporations, grab their audio, isolate it, [12:26.590 --> 12:31.930] and then put it into a model like this so that we can have our own version of synthetic text to [12:31.930 --> 12:39.890] speech using their voice. So, we can call throughout the company and impersonate these [12:39.890 --> 12:46.010] individuals in order to cause some sort of social engineering action. Now, this last one is really [12:46.010 --> 12:51.250] interesting. This is Tacotron. This is a Google project. And I have the text over here on the [12:51.250 --> 12:55.630] right, but what I want you to do while you're listening to this is pay attention to the breaths. [12:55.630 --> 13:01.330] As humans, we tend to have filler words like um and uh, and then in between sentences and [13:01.330 --> 13:05.670] commas, we usually have a bit of a pause or a breath. And so, the Tacotron model actually [13:05.670 --> 13:11.490] picked that up. This is fully synthetic voice. Just take a listen. Only the photograph really [13:11.490 --> 13:16.350] showed how much time had passed. Ten years ago, there had been lots of pictures of what looked [13:16.350 --> 13:20.890] like a large pink beach ball wearing different colored bonnets. But Dudley Dursley was no longer [13:21.050 --> 13:25.190] a baby, and now the photograph showed a large blond boy riding his first bicycle [13:25.750 --> 13:30.390] on a carousel at the fair playing a computer game with his father being hugged and kissed [13:30.390 --> 13:40.930] by his mother by his mother. That's pretty scary. So, these projects are still underway. Liarbird [13:40.930 --> 13:48.090] is actually advertising itself as a transcription or reverse transcription service, which is just [13:48.090 --> 13:53.450] text-to-speech. In their original pay model that they had pushed out, currently they're in closed [13:53.450 --> 13:59.610] beta, but in their original pay model that they pushed out, you can transcribe three hours of [13:59.610 --> 14:05.650] audio pretty cheaply. And then if you're so inclined to pay, you can transcribe obviously [14:05.650 --> 14:11.130] more. But if a malicious user or somebody who's maliciously inclined, they can just grab those [14:11.130 --> 14:16.290] voice samples, throw them into Liarbird or Tacotron, or just use the voice samples that [14:16.290 --> 14:21.690] are already in there and generate their own adversarial examples. [14:24.720 --> 14:30.480] So, next we get into sandbox detection. And so, this is the defense detection and evasion [14:31.100 --> 14:36.020] part of the system. So, we have these researchers over at Silent Brick Security, and what they did [14:36.020 --> 14:42.640] was decide that with many different types of sandbox, what you can do is take the process list [14:42.640 --> 14:48.220] as data itself. And so, based on the process list, the number of processes, [14:48.220 --> 14:52.740] the number of users on a system, you can generate and decide whether or not you're in a sandbox. [14:52.740 --> 14:57.140] And then you can take it a step further and decide what kind of sandbox you're in, [14:57.140 --> 15:06.120] whether you're in FireEye or Cuckoo sandbox. So, what the researchers did was kind of classify, [15:06.120 --> 15:11.940] they took a number of basic Windows systems and they took a number of sandboxes and just kind of [15:11.940 --> 15:16.600] identified how many processes there are, how many users there are. Let's take a look at what malware [15:16.600 --> 15:22.720] can identify. And so, when we have a low process count, you see over here in column A, we have 33. [15:22.720 --> 15:29.500] In column F, we have 34. And then we have a high user count, such as, again, A and F, we have 4 [15:29.500 --> 15:38.320] and 4. And the process count per user, 8.25, 8.5. It gives us a very high probability that it is a [15:38.320 --> 15:44.220] sandbox. And so, we can see that with our host total score. And then based off that total score, [15:44.220 --> 15:50.780] we can set a threshold. In this case, it looks like they set the host average score as their [15:50.780 --> 15:55.280] threshold. So, anything below the average score is considered a sandbox. Anything above that [15:55.280 --> 16:01.160] is considered a legitimate system. And so, they can have malware that completely shuts down or [16:01.160 --> 16:07.160] erases itself if it identifies that it's in a sandbox, or it can use targeted evasion techniques [16:07.160 --> 16:12.880] per sandbox. So, if I'm in FireEye, these are the things that I'll do to get past it. If I'm in [16:12.880 --> 16:18.100] Cuckoo, these are the things that I'll do to get past it. So, it's a really interesting article, [16:18.100 --> 16:25.260] highly recommend taking a look at that. And then for automated network exploitation, [16:25.260 --> 16:30.300] we have systems like DeepExploit and Goethon. Goethon and DeepExploit, they both operate very [16:30.300 --> 16:36.020] similarly. The way that these work is they will go out, they will perform an nmap scan based on [16:36.020 --> 16:42.100] the results of that nmap scan, if it sees common services that it knows are exploitable or things [16:42.100 --> 16:47.400] that it would like to kind of take to that next level. For example, port 80 or 443, it's web [16:47.400 --> 16:52.380] traffic. So, it's going to go ahead and run Scrappy to go and crawl that system. I'm sure they could [16:52.380 --> 16:58.840] add other tools such as Go Buster, which will go out and enumerate a lot of the files or directories [16:58.840 --> 17:03.400] or URLs that are on that system. Try and fingerprint the server. You want to see if it's running [17:03.400 --> 17:12.120] Nginx, Apache, PHP, Java, things of that nature. And then it performs additional content exploration [17:12.120 --> 17:17.740] and its own Google hacking, which is kind of cool. So, that will go out, do a little bit of research, [17:17.740 --> 17:22.920] see if there's any sort of exploitable scripts, things of that nature. And it takes all of this [17:22.920 --> 17:31.040] data and it processes it using some statistical techniques and then it uses Metasploit as a [17:31.040 --> 17:36.260] targeted exploitation engine and it goes and tries to exploit those things. So, if it sees [17:36.260 --> 17:41.120] certain versions of JBoss or Jenkins, it will go and try and target that with maybe [17:41.680 --> 17:49.340] password spray or any sort of Metasploit module. Tomcat, same thing. So, this is kind of cool [17:49.340 --> 17:55.060] as a proof of concept of automating the entire process of network discovery all the way up to [17:55.060 --> 18:02.620] exploitation and in some cases post-exploitation. One of my favorite while I was researching this [18:02.620 --> 18:08.940] is Deathstar. And this is my favorite just kind of in the way that it is assembled. So, Deathstar, [18:08.940 --> 18:14.160] it's built on PowerShell Empire. And the way that it works is it starts, it kind of covers its [18:14.160 --> 18:20.560] baseline and it is designed to go from standard user all the way up to domain administrator. [18:20.560 --> 18:26.140] And so, it charts and maps attack paths all the way to that location. And so, it starts at its [18:26.140 --> 18:30.780] baseline and it says, am I already a domain administrator? If so, cancel out. And we see [18:30.780 --> 18:36.460] that's the path off to the left. If I have domain administrator credentials, all right, cool. Log [18:36.460 --> 18:41.420] in. And that charts off the path to the right. But if it's neither of those, then it goes straight [18:41.420 --> 18:47.040] down the middle and it says, okay, let's spawn a new agent. Let's go ahead and identify where is [18:47.040 --> 18:52.660] the domain administrators and where is the domain controllers. And so, it runs over here to the [18:52.660 --> 18:58.160] right. We can see get net domain controller, get net group member, invoke user hunter. And so, [18:58.160 --> 19:04.660] this is going to enumerate the domain controllers, enumerate the domain admins, and enumerate the [19:04.660 --> 19:09.440] users that it can on the systems that it can. And you can see it goes through this entire tree [19:09.440 --> 19:13.660] kind of off here to the bottom left. It identifies and says, hey, is this a vulnerable, [19:13.660 --> 19:18.560] known vulnerable version of Windows, for example, Windows 7? If yes, let's run Mimikatz, [19:18.560 --> 19:24.040] try and do Mimikatz, secret stump, things of that nature, collect credentials, and then we can kind [19:24.040 --> 19:29.420] of start back over at the top. Let's go ahead and enumerate the running processes. Do we see any [19:29.420 --> 19:35.440] processes that are already being ran by other users that we don't have in our inventory? If so, [19:35.440 --> 19:39.120] let's add those users to our inventory and try and do some lateral movement. [19:39.580 --> 19:44.380] And so, in this case, it'll use psinject, and then it'll go ahead and spawn a new instance. [19:44.380 --> 19:50.540] And so, it goes through this iterative process all the way until it reaches domain admin credentials. [19:51.300 --> 19:57.260] And so, if you've ever used a system like Bloodhound, where you'll generate an attack [19:57.260 --> 20:02.640] graph and say, okay, this user is part of this group, which has these members, which this member [20:02.640 --> 20:07.360] is also part of domain admins. And so, this is the attack path that you want to go in order to [20:07.360 --> 20:13.500] become domain admin. And so, Deathstar kind of takes that idea a step further and actually tries [20:13.500 --> 20:22.720] to navigate that attack path in an automated fashion. So, that's all I have for the machine [20:22.720 --> 20:27.060] learning tools that already exist, some of the proof of concepts. There's a lot more information [20:27.060 --> 20:31.760] that I wanted to cover that I wouldn't be able to cover in such a short amount of time. So, [20:31.760 --> 20:37.940] at the end of this presentation, I have a link to my slide deck, which has a lot more detail [20:37.940 --> 20:44.280] and a lot more tools and a lot more techniques in it. But in the meantime, I kind of want to [20:44.280 --> 20:49.460] answer the question of how do I go about creating my own machine learning tools [20:50.360 --> 20:58.180] or AI tools? What should I look for? How do I automate that? What is that process? And so, [20:58.180 --> 21:04.100] honestly, my advice would be to identify somewhere that either utilizes a lot of data [21:04.660 --> 21:10.900] or identify somewhere that uses a... it's a common process, but it uses some sort of fuzzy logic, [21:10.900 --> 21:15.060] right? Something that's kind of hard to script, hard to write a signature for, [21:15.060 --> 21:20.400] but with AI and machine learning, you can use some of that fuzzy logic to generate some heuristics [21:20.400 --> 21:25.660] and it can perform based off of that. So, in the case of some of these, what we can do is say, [21:25.660 --> 21:32.060] all right, NMAP scans, everybody does NMAP scans, and then you have to manually look at them, [21:32.060 --> 21:37.180] and then based off of that, you have to follow up and go over here. Instead, like I showed with [21:37.180 --> 21:43.720] Goython, is you can just use the NMAP scan and then automate the process of, okay, let's enumerate [21:43.720 --> 21:47.540] and fingerprint these services, and then let's see if any of these services are already exploitable [21:47.540 --> 21:54.240] using Metasploit, or let's go with next steps. I identified a web service, let's run web analysis [21:54.240 --> 22:01.820] techniques. So, Go Buster, Crawler, any sort of things like that. So, that's kind of the thing [22:01.820 --> 22:06.300] that you want to keep an eye on if you want to develop your own machine learning tools. And [22:06.300 --> 22:11.920] there's more that are coming out all of the time, and it's really interesting and a really fast-paced [22:12.440 --> 22:17.280] area, and it's just ripe with research. So, go ahead and play around with some of that. [22:19.200 --> 22:23.260] So, the next bit that I want to get into is adversarial machine learning, [22:23.260 --> 22:28.360] and this is attacking the machine learning algorithms themselves. So, adversarial machine [22:28.360 --> 22:33.900] learning, there's really three ways to go about attacking a machine learning model. [22:34.160 --> 22:39.780] So, the first way we have is model evasion. This is to take what is clearly a malicious example [22:40.300 --> 22:46.500] and get the model to consider it to be benign or normal or standard to bypass that filter. [22:46.500 --> 22:50.700] So, this would be taking a spam message and getting past a spam filter, or this would be [22:50.700 --> 22:56.780] taking a piece of malware and getting it past antivirus or anti-malware solutions. [22:57.420 --> 23:05.160] Model poisoning is where we add our own data into the training data set. Now, machine learning [23:05.160 --> 23:13.680] models specifically are... they're done learning as soon as they're done being trained. To get [23:13.680 --> 23:18.420] around this, there are some models that are called online systems, and so what they'll do is they'll [23:18.980 --> 23:24.860] take data that has been classified by itself already. So, say I send it a piece of data, [23:24.860 --> 23:28.680] like a piece of malware, it looks at that, says, okay, bam, that's malware, [23:28.680 --> 23:34.820] now that goes into my training set, and I learn from that malware. So, data poisoning allows us [23:34.820 --> 23:42.580] to slowly inject certain pieces of data so that we push that threshold a little off to the side [23:42.580 --> 23:49.240] into what is more acceptable, and then we target where that threshold has been pushed. [23:49.680 --> 23:55.160] And we'll see an example of that in a bit. Data leakage. Data leakage is actually [23:56.360 --> 24:03.560] a combination of two smaller bits. So, one is stealing the model itself. So, for example, [24:03.560 --> 24:08.820] say you're an IoT engineering company and you develop an IoT system that utilizes some form [24:08.820 --> 24:15.000] of machine learning. What I can do is buy that device and then reverse engineer it, [24:15.000 --> 24:20.300] and now I have your model, and so I can use that model or train another model based off of that [24:20.300 --> 24:23.900] instead of having to go through the hundreds of thousands of dollars that you went through [24:23.900 --> 24:30.660] to develop that in the first place. And so, this is where the model itself is easy to be grabbed. [24:30.660 --> 24:39.440] There's also data leakage, which is I can identify what data you use to train the model. [24:39.760 --> 24:43.240] And so, this is really damaging, especially for healthcare industry or [24:45.320 --> 24:50.860] HIPAA-protected information. So, if I can identify any sort of clients or [24:52.120 --> 24:57.280] patients that you have, any sort of their PII, then that's really damaging. [24:59.760 --> 25:05.080] So, when it comes to model testing, there's really two approaches. You have a white box where you [25:05.080 --> 25:08.140] know what the model is, you know how the hyperparameters are set, you know everything [25:08.140 --> 25:13.280] that's involved in the building of this, and then you target that model, much like you would do in [25:13.280 --> 25:17.820] software where you already have all of the code involved. And then you have a black box, which is [25:17.820 --> 25:24.000] just the model itself. You aren't able to peek into the way that the model operates, but what you [25:24.000 --> 25:27.260] can do is treat it like an oracle. So, if you've ever done any sort of cryptography, you know, [25:27.260 --> 25:34.280] exercises, you have an oracle, you ask yes-no questions, or things of that nature, and then [25:34.280 --> 25:40.020] you can generate what is called a surrogate model. So, based off of the yes-no questions that I asked [25:40.020 --> 25:45.760] the black box model, I generated my own model, and then based off of that model, I can perform attacks [25:46.620 --> 25:52.020] and that those attacks should be able to impact the original black box itself. [25:52.660 --> 25:57.780] Now, the way that this operates is under a principle called attack transferability. [25:57.900 --> 26:03.160] This is a documented phenomenon, and the way that attack transferability works is, [26:03.160 --> 26:08.460] it doesn't matter what model or the underlying architecture that these models are designed with, [26:08.460 --> 26:13.600] an attack on one model can also be an attack on another model. So, if I have a deep learning [26:13.600 --> 26:18.300] model over here, and I find adversarial examples that I can use to attack this model, [26:18.300 --> 26:23.740] then here is a random forest, a logistic regression, SVM, or anything else, any other [26:23.740 --> 26:30.220] machine learning algorithm or architecture, that those same adversarial examples will be able to [26:30.220 --> 26:37.100] impact that. And then, of course, the attack surface. So, where are the machine learning [26:37.100 --> 26:43.980] models themselves vulnerable? So, machine learning, as it works, is you have a physical object, [26:43.980 --> 26:48.120] and then you take that physical object and you try to digitize it. So, say you have a picture [26:48.120 --> 26:52.840] of a panda or a car, you take a photograph of it, now it's digitized, and you can make [26:52.840 --> 26:58.200] information off of that. You take that digitized information, you run your pre-processing as you [26:58.200 --> 27:02.540] would per the normal machine learning process, and then you put that into the machine learning [27:02.540 --> 27:09.460] model as part of its input features, it presents some output, usually a probability or a number [27:09.460 --> 27:16.180] of some sort, and then a decision is made based off of that action. So, for example, if I am 67% [27:16.180 --> 27:21.320] confident that this is malware, then I will probably flag it as malware. If I'm only 42% [27:21.320 --> 27:27.200] confident, then I'll just let it continue to operate. So, where can we attack things? Well, [27:27.200 --> 27:33.620] one, we can actually impact the physical object itself. So, down below, I have the network [27:33.620 --> 27:39.020] construction detection system as kind of an example that follows this generic machine learning model [27:39.020 --> 27:44.160] path. So, we have the attack traffic. We can actually, as attackers, modify the attack traffic [27:44.160 --> 27:49.380] in such a way that as it goes through the pipeline of the machine learning pipeline, [27:50.040 --> 27:58.040] it will modify the behaviors at different points. So, I can slow down my attack traffic, [27:58.040 --> 28:03.000] I can use my attack traffic for multiple hosts so that it's a little bit harder to detect. [28:03.680 --> 28:09.560] And then that attack traffic is then digitized. The digitization process, we don't really have [28:09.660 --> 28:17.080] a lot of impact over as attackers. It's not realistic for us to be able to affect TCP dump. [28:19.220 --> 28:23.880] There's really famous examples. For example, you may have come across this if you ever looked at [28:23.880 --> 28:28.880] adversarial machine learning, a picture of a panda, and you modify some pixels, [28:28.880 --> 28:35.260] and now it thinks it's a gibbon or a bird or some sort. But if we have the ability to modify pixels, [28:35.260 --> 28:42.120] why don't we just give it a picture of a bird at that point? And then more to the point, [28:42.120 --> 28:46.480] if we can modify pictures, if you have something like a self-driving car, it has a camera, [28:46.480 --> 28:51.200] how do we interface with that camera in an adversarial perspective? We would already have [28:51.200 --> 28:54.860] to have access to the system, which at that point we can just shut it down or do whatever. [28:54.860 --> 29:02.820] It's physical access. So, we're not actually able to interface with the digitization of the object, [29:02.820 --> 29:08.200] but what we can do is change the way that the attack traffic is modified. [29:08.200 --> 29:11.620] We can change the input features, the input vectors, the machine learning model, especially [29:11.620 --> 29:16.920] with the model poisoning. We can change the way that the data is represented. [29:18.460 --> 29:22.620] So, in the case of the network intrusion detection system, we have the packet metadata, [29:22.620 --> 29:28.520] and then it spits out an attack probability, and then off of that it makes a decision. [29:28.840 --> 29:32.340] And in between the machine learning model and the decision-making process, you got to remember [29:32.340 --> 29:37.540] this is just software, and you can approach it like any other piece of software that you're [29:37.540 --> 29:41.440] pen testing or trying to hack. So, you have the machine learning model in it, and then [29:41.440 --> 29:45.840] all wrapped around it is a nice piece of software that probably has some of its own vulnerabilities [29:46.360 --> 29:54.840] involved. So, model invasion. Model evasion, not invasion, sorry. [29:55.680 --> 29:59.600] So, we're going to try and hide in the blind spots. Now, we have this massive theoretical [29:59.600 --> 30:05.660] space. This is all of the possible pieces of malware that could possibly exist, but we can't [30:05.660 --> 30:11.080] collect all of that, realistically speaking. Antivirus companies can't collect all of it, [30:11.080 --> 30:15.520] and so what they have is a subset, and this is considered the training space. So, this is what [30:15.520 --> 30:21.140] they are realistically able to collect. So, in their training space, they perform their training, [30:21.140 --> 30:25.960] and then off of a small piece of that that's already clean labeled data, they perform their [30:25.960 --> 30:32.040] testing to make sure that the model operates properly. Well, part of that testing space, [30:32.040 --> 30:35.820] kind of right there off the sliver, this is the adversarial space. This is where we live. [30:35.820 --> 30:42.020] This is where we are able to generate adversarial examples that target that model specifically. [30:42.440 --> 30:46.760] And so, this is kind of what we want to play around with and hide in the blind spots, [30:46.760 --> 30:52.240] so to speak. So, a couple really good examples is, in this case, multinomial [30:52.240 --> 30:57.860] Naive Bayesian spam filters, or just any of the early spam filters that used some rudimentary [30:57.860 --> 31:02.360] forms of machine learning, how they would operate is they would look at keywords. So, [31:02.360 --> 31:09.920] if you had something like, buy Viagra now, the probability that the word Viagra shows up in a [31:09.920 --> 31:15.160] spam email is much higher than the probability of the word Viagra showing up in a regular email. [31:15.160 --> 31:19.200] And so, based off of these keywords, it does its own statistical analysis and says, [31:19.200 --> 31:25.680] I am 99% sure this is spam. Well, the interesting thing about emails is that [31:25.680 --> 31:34.440] emails are usually represented as HTML pages or web pages. And so, as malicious actors, [31:34.440 --> 31:39.700] what we can do is actually add in a comment. So, in this case, we can add in the entire Wikipedia [31:39.700 --> 31:45.200] page article for a horse. And so, we have a bunch of bad words, but we have even more good words. [31:45.200 --> 31:51.060] What this lets us do is kind of skip past the spam filter by showing, hey, we have a lot more [31:51.060 --> 31:56.960] good words than bad words, so we must be good, right? It says, yeah, absolutely. I am 99% sure [31:56.960 --> 32:03.280] you are not spam. And the thing is that the way that this is presented to users is as an HTML [32:03.280 --> 32:08.520] web page. And so, the comment itself is not actually rendered for the users. So, the users [32:08.520 --> 32:16.620] still see the original spam email as though it was unmodified. And so, this is kind of a real [32:16.620 --> 32:23.680] world example on some of the spam filters. A really good example of this is by Skylight Cyber. [32:23.680 --> 32:29.200] The researchers over there were actually able to bypass the silence machine learning antivirus [32:30.960 --> 32:36.620] in this article, Silence, I Kill You. Really good article. Highly recommend you take a look at it. [32:36.620 --> 32:42.060] We can see some familiar malware samples here. So, we have CoinMiner, Emotet, Zeus, [32:42.880 --> 32:48.980] and we see the original scores that they were actually able to pull out of the silence antivirus, [32:48.980 --> 32:57.920] negative 826, negative 923, negative 997. So, these are really bad, really bad, flagged immediately. [32:58.980 --> 33:04.920] So, what the researchers over at Skylight Cyber did was they took the silence antivirus, [33:04.920 --> 33:09.920] reverse engineered it, and they were able to pull out the scores based on some of the [33:09.920 --> 33:14.740] syscalls and other key features that silence looked at for malware. [33:15.940 --> 33:22.120] One of the curious things is that they identified certain things that were whitelisted. Now, [33:22.120 --> 33:27.820] why would silence whitelist certain things, even though the standard machine learning process would [33:27.820 --> 33:32.220] actually identify it as potentially malware? Well, the reason why is because when you think [33:32.220 --> 33:37.720] about malware, think about how malware operates, right? So, maybe it asks for access to your webcam [33:37.720 --> 33:45.360] or your microphone, or it tries to resize or open up new windows. Maybe it makes calls out to the [33:45.360 --> 33:53.280] internet. Well, what's another thing that does that? Chrome, Firefox, Internet Explorer. They [33:53.280 --> 33:59.360] operate very similar to some modern malware. In this case, what they found was a video game. [33:59.820 --> 34:04.640] I'm not sure which video game it is. My money is on Fortnite or League of Legends. [34:05.240 --> 34:10.520] But the video games that they found were certain whitelisted keywords. So, what they did was took [34:10.520 --> 34:17.660] these clearly bad malware and they just added these keywords to the end of the malware. So, [34:17.660 --> 34:23.340] it didn't change the way that the malware operated, but it changed the way that silence looked at it [34:23.340 --> 34:27.440] and said, okay, you have a bunch of bad boy points, but you have even more good boy points. [34:27.440 --> 34:33.020] So, we're going to give you some good boy scores. So, in this case, CoinMiner had 884. [34:33.120 --> 34:38.460] My favorite is looking at Zeus, negative 997, one of the worst in this table, [34:38.460 --> 34:41.960] all the way up to positive 997, which is one of the best in this table. [34:42.680 --> 34:50.140] And so, it shows that model evasion techniques are very useful in a number of different ways. [34:52.100 --> 34:55.680] So, how do we defend against this? Well, the first and probably one of the better [34:55.680 --> 35:02.220] is adversarial training. Adversarial training kind of comes from the idea of chaos engineering. [35:02.220 --> 35:06.600] Netflix is a really big proponent of chaos engineering, where you already know something [35:06.600 --> 35:11.500] bad is going to happen, so you might as well account for those bad things to happen. So, [35:11.500 --> 35:15.360] in this case, you train it with adversarial examples. You generate your own adversarial [35:15.360 --> 35:20.120] examples and you train the model based off of that to kind of harden and make it a little bit [35:20.120 --> 35:27.060] more robust. Again, if it's not a robust model, it's not a good model. Chaos engineering as a [35:27.060 --> 35:31.820] whole, the way that it operates is you already know something is going to go bad. So, you have [35:31.820 --> 35:37.260] Netflix, for example. Netflix, they know that their systems, they need to have that uptime. [35:37.260 --> 35:40.280] And so, they know that it's possible one of their systems goes down. They want to make sure [35:40.280 --> 35:45.540] their redundancy operates properly. So, you purposefully go into production and you break [35:45.540 --> 35:53.360] things. And then you develop processes and policies around knowing things are going to [35:53.360 --> 35:58.000] break. So, in some cases, you'll have the CIO walk into a data center and just start unplugging [35:58.000 --> 36:05.180] servers. So, adversarial training is a really good way to kind of counteract this. Another [36:05.180 --> 36:11.840] slightly more technical way is called defensive distillation. Defensive distillation is a [36:11.840 --> 36:16.520] smoothing algorithm that kind of smooths the curve. So, instead of having, okay, malware, [36:16.520 --> 36:21.900] malware, malware, malware, nope, it's okay. You can't have sharp edges like that. So, what it [36:21.900 --> 36:27.220] does is it smooths it. So, instead of going straight down, it just kind of goes in a downward [36:27.220 --> 36:34.680] trend. And so, it's a little bit harder, a little bit more robust way of analyzing and dealing with [36:34.680 --> 36:41.400] the adversarial examples. And then the last example would be monotonic classification. [36:41.400 --> 36:46.100] Which means that the graph grows in one direction. So, we see here in the image off to the right, [36:46.100 --> 36:53.560] the non-monotonic example, we see that it's getting worse and worse and worse as far as a [36:53.560 --> 36:57.620] keyword. So, it goes worse and worse and worse across that threshold. Now, it's considered bad, [36:57.980 --> 37:02.700] it's spam, it's malware, it's whatever. We're ready to throw it out, but we're still analyzing [37:02.700 --> 37:06.680] it. And so, it's crossed that threshold, but wait, no, it's getting better and better and better. [37:06.680 --> 37:12.780] We just hit that pad of keywords that are really positive. And so, we went from good to bad, [37:12.780 --> 37:18.080] all the way back up to good. Monotonic classification means that we don't count the [37:18.080 --> 37:23.280] good words at all, we say, we measure your badness. How many bad words do you have? [37:23.560 --> 37:27.680] And so, in this case, on the bottom example, we have the monotonic classification, which says, [37:27.680 --> 37:31.420] okay, we have some bad words, we have some bad words, up, we all went all the way up to the top, [37:31.420 --> 37:36.620] and then we hit all of the padded good words. And it doesn't matter, you're still bad, [37:36.620 --> 37:41.780] Jesus is not going to save you, you're done. And so, monotonic classification is a really [37:41.780 --> 37:46.500] powerful technique to deal with these model evasion techniques. [37:48.480 --> 37:56.600] Next, we have model poisoning. Model poisoning almost exclusively works on online learning [37:56.600 --> 38:01.940] systems. So, as adversaries, we need to find a way to get adversarial examples into the training [38:01.940 --> 38:10.240] space. And so, this is where systems will grab data that it's already classified a certain way, [38:10.240 --> 38:13.760] and add it into the training space. And so, we're just trying to push that barrier [38:15.040 --> 38:20.080] past a certain threshold where we can actually operate safely as adversaries without it detecting [38:20.080 --> 38:29.380] us as such. So, some real world examples. Tay AI, if you're not already familiar, [38:29.380 --> 38:37.460] I highly recommend checking it out. Tay AI was a Microsoft Twitter chat bot, and it was designed [38:37.460 --> 38:46.000] to learn from real conversations by real people. Well, 4chan, of course, gets a hold of this, and [38:46.000 --> 38:54.580] they decide, how much does it learn? How much can we teach it? And about 16 hours after coming online, [38:54.580 --> 39:01.280] it was taken offline permanently, because within that time, 4chan was able to get it to say, [39:01.280 --> 39:08.500] Hitler did nothing wrong, and racial slurs. If you want more information on Tay AI, and I highly [39:08.500 --> 39:13.660] recommend checking this out, it's called The People's Chatbot by the Internet Historian on [39:13.660 --> 39:20.400] YouTube. Really funny guy, really great story, and it's just incredible to think of how many [39:20.400 --> 39:27.240] data scientists and developers and people behind this project put time in, put effort, put work, [39:27.240 --> 39:35.880] and it was shut down by a bunch of kids on 4chan. The other one is Jacobian Map Saliency [39:35.880 --> 39:43.440] Attack, and this kind of goes into the pixel manipulation. So, what you do is you grab a [39:43.440 --> 39:48.540] couple pixels, and you modify them, or you tweak them a little bit, or you tweak the data a little [39:48.540 --> 39:54.240] bit, doesn't necessarily need to be image data, and you pass that on as a certain class. [39:54.880 --> 39:58.880] And so, as the model is learning from this data, learning from this class, it starts identifying [39:58.880 --> 40:05.720] these tweaks, and then you can implement those tweaks in an adversarial example. So, in this [40:05.720 --> 40:12.180] case, we have a 70 kilometer per hour sign, we add in these little tweaks, and it latches onto those [40:12.180 --> 40:22.120] tweaks and says, okay, you're a 30 kilometer per hour sign. And so, imagine the kind of chaos that [40:22.120 --> 40:30.280] that can create. Again, these tweaks are difficult to implement in real world examples. This is also [40:30.280 --> 40:36.980] difficult to pull off simply because it relies on an online learning system, something that learns [40:36.980 --> 40:41.680] from the data that you've already presented it. But these are things that researchers have been [40:41.680 --> 40:45.600] able to identify and actually use to attack machine learning systems. [40:47.440 --> 40:53.540] So, how do we defend against poisoning attacks? Basically, it comes down to, you can still use a [40:53.540 --> 41:00.040] lot of the data, but what you want to do is add your own little smoothing filter. So, in this case, [41:00.040 --> 41:06.620] have longer periods of time between retraining. So, what that will do is kind of narrow the window, [41:06.620 --> 41:13.600] not narrow the window, it will expand the window that an attacker would need to generate adversarial [41:13.600 --> 41:19.690] examples for. And then they would also have to compete for votes against legitimate traffic. [41:20.240 --> 41:27.280] So, it's kind of like, if you look at Bitcoin, you need 51% of the vote in order to kind of [41:27.280 --> 41:36.060] take over cryptocurrency networks. So, this is kind of their own example of doing this. [41:36.060 --> 41:41.180] You can also analyze longer periods of data. Again, it broadens the window that you're looking [41:41.180 --> 41:45.060] at. And so, the adversaries, the attackers are going to have to generate more traffic, [41:45.060 --> 41:50.680] more examples in a longer period of time, which is difficult to do. And then you just want to [41:50.680 --> 41:54.640] generally minimize the impact of adversarial training examples. So, whether you have a human [41:54.640 --> 42:00.960] in loop that's manually looking at some of these examples, or maybe you're taking a small sample [42:00.960 --> 42:06.240] of the stuff that you're seeing and only allowing that to affect the model very... [42:08.420 --> 42:15.060] a little bit. So, it's just kind of a general rule of thumb. It's just minimize the impact, [42:15.060 --> 42:21.900] come to the understanding that people will try and poison the well, account for that. [42:23.460 --> 42:29.420] And then, of course, we have data leakage. So, data leakage, this is usually when, [42:29.420 --> 42:34.160] if we're trying to steal data, not steal the model, but steal data itself, this is usually [42:34.160 --> 42:40.440] when the models are trained to be a little too good. So, this image down below, you see aerial [42:40.440 --> 42:45.320] photographs, generated map, and then an aerial reconstruction. And so, articles at the time, [42:45.320 --> 42:50.300] this is a project by Google, articles at the time were kind of grandstanding and saying, [42:50.300 --> 42:56.900] oh my god, the Google AI found a way to cheat and generate aerial photographs. [42:57.460 --> 43:02.640] It was able to embed its own data. From the machine learning perspective or data science [43:02.640 --> 43:06.360] perspective, this is actually called overfitting. It's a very common problem. It's nothing [43:06.360 --> 43:14.140] spectacular. It's just, you didn't train your model right. And so, especially with overfit [43:14.140 --> 43:18.040] models, you have the training data in the model itself, or the model is able to generate that [43:18.040 --> 43:23.440] training data. And so, as I was saying earlier, with things like credit card information, PCI [43:24.000 --> 43:28.480] protected information, personally identifiable information, public health, or not public, [43:28.480 --> 43:34.160] sorry, personal health information, or HIPAA protected data, things of that nature, [43:34.160 --> 43:41.680] it can be very damaging for a machine learning model to leak real training data. And so, [43:41.680 --> 43:48.500] it's important to kind of evaluate and make sure that the models are not overfit, are not underfit, [43:48.500 --> 43:53.700] just hidden just right, or that the data is being preprocessed in such a way that it's [43:53.780 --> 43:59.760] a one directional. So, in some cases, what they'll do is generate a hash of certain features. For [43:59.760 --> 44:03.800] example, instead of putting in a credit card number, they generate a hash of the credit [44:03.800 --> 44:12.140] card number. And so, it's not reversible in any meaningful way. And then, of course, model theft, [44:12.140 --> 44:18.340] model theft by competitors. There's a technique called federated learning, which is you encrypt [44:18.340 --> 44:26.180] the machine learning model itself so that the model has to be used in that encrypted form. You [44:26.180 --> 44:32.620] can't steal the model, you can't decrypt it. It's a one way kind of thing, or a one way street in [44:32.620 --> 44:38.680] that scenario. And so, this will allow like IoT devices to be deployed with machine learning [44:38.680 --> 44:43.920] models. And it will prevent competitors or attackers, malicious actors from reversing it, [44:43.920 --> 44:46.820] stealing that model and building their own product off of it. [44:49.420 --> 44:53.760] And then, one last thing that I kind of want to cover is adversarial stickers. So, [44:53.760 --> 44:58.720] you've probably seen adversarial stickers. They look like melted crayons, in my opinion. [45:00.980 --> 45:05.700] But adversarial stickers, they have a bit of a problem. This is considered a brittle attack, [45:05.700 --> 45:11.680] and it's not very repeatable. And the reason why is because it doesn't have that attack transfer [45:11.680 --> 45:19.980] ability that I brought up earlier. So, in this case, kind of off to the bottom right, [45:19.980 --> 45:24.440] the stop sign, this is one of the first articles I was talking about adversarial stickers, and they [45:24.440 --> 45:31.960] were trying to modify a stop sign in such a way that the machine learning model didn't recognize [45:31.960 --> 45:36.360] it as a stop sign anymore, recognized it as something else. And so, this could be really [45:36.360 --> 45:42.560] damaging for self-driving vehicles. One of the problems is that this paper ran into some [45:42.560 --> 45:48.400] repeatability problems. And the reason why is because this attack is so brittle, they were [45:48.400 --> 45:54.080] able to convince their model, but they were not able to convince, or people who were trying to [45:54.080 --> 46:00.980] replicate this were not able to convince other models. So, it's not very useful. You can't just [46:00.980 --> 46:09.440] slap an adversarial sticker onto a sign and hope that you're driving a self-driving vehicle off [46:09.440 --> 46:15.480] into a wall or something of that nature. And then off here to the left is kind of the same idea with [46:15.480 --> 46:22.360] facial recognition. So, this researcher here said, okay, I'm going to set some constraints. I only [46:22.360 --> 46:28.300] want the pixels or the image around my eyes in the shape of glasses to be modified so that I look like [46:28.300 --> 46:35.860] this actress. Honestly, I don't remember who this is. I apologize. But I'm a man. I want to look like [46:35.860 --> 46:41.560] this actress per a facial recognition model. And so, he was successfully able to do that. [46:41.760 --> 46:49.380] The problem is that these glasses are designed for his face and the rest of the features of his face. [46:49.420 --> 46:54.440] And so, if I or many of you put on these glasses, it'll either look like somebody else or it won't [46:54.440 --> 47:01.740] recognize it as an adversarial example and it'll just recognize us as people, the same people, [47:01.740 --> 47:06.620] just wearing stupid glasses. So, attack transferability, adversarial stickers don't [47:06.620 --> 47:11.800] really have it. It's not a really good attack. It's very brittle. And I just kind of wanted to [47:11.800 --> 47:20.140] bring that up. For more information on this, I highly recommend DEF CON 2018 AI Village. Sven [47:20.140 --> 47:25.940] has a really good 30-minute talk on adversarial stickers and kind of the challenge with them. [47:25.940 --> 47:33.420] And then, for the facial recognition specifically, I know that Rich Horang in AI Village 2019 has a [47:33.420 --> 47:39.580] really good talk talking about facial recognition and trying to bypass that. [47:39.580 --> 47:47.180] So, to recap, how has AI empowered attackers? Well, it allows us to operate at a speed and scale [47:47.180 --> 47:53.460] like never before. We can operate at machine speed. And then, what we are seeing is the very [47:53.460 --> 47:58.540] beginning of what AI can bring to the offensive security space. So, we've already seen some [47:58.540 --> 48:04.940] adversarial attacks in the wild. I anticipate us seeing more as more researchers are coming out. [48:05.120 --> 48:10.180] AI and machine learning is kind of the hot new buzzword that everybody's latching onto. So, [48:10.180 --> 48:15.480] CEOs and decision makers are grabbing the AI-powered or machine learning-powered [48:15.480 --> 48:21.000] detection systems, defense systems. And it's kind of become a bit of a buzzword [48:22.180 --> 48:27.260] where you say that in a lot of common companies, especially in places like DEF CON, and they tell [48:27.260 --> 48:31.980] you to go talk to the Bitcoin folks and blockchain and all that fun stuff. There's no silver bullet, [48:31.980 --> 48:37.880] but we are seeing the very beginning of what is possible. And I anticipate, and I'm really hoping [48:37.880 --> 48:45.820] in the next several years, that we see a much larger impact of this AI versus AI kind of warfare [48:47.480 --> 48:53.720] that should be coming about. But anyways, I appreciate it. Thank you for your time. Thanks [48:53.720 --> 48:58.160] for sticking around this long. You can go ahead and email me if you have any questions. I'm also [48:58.160 --> 49:04.300] going to be in the chat. And then, for the slides, I have that are much more detailed are on [49:04.300 --> 49:10.820] Slideshare, and you can find them at slideshare.net slash GTKalandai. So that is my presentation. I [49:10.820 --> 49:16.100] hope you learned something, and thank you for sticking around. Bye!