WEBVTT 00:00.720 --> 00:02.020 Oh, okay. 00:02.020 --> 00:04.080 Hi, welcome to our next talk. 00:04.080 --> 00:09.960 This one's by Gavin Kondyk, also known as GTKondyk on Twitter. 00:10.000 --> 00:19.160 He's a pre-circuited researcher, consultant, and has been super involved in the AI village community for a long time. 00:19.320 --> 00:24.060 He's an awesome guy, so I'll let him take it away. 00:27.770 --> 00:34.470 Hello, and welcome to Hacking with Skynet, How AI is Empowering Adversaries. 00:34.750 --> 00:36.010 My name is Gavin. 00:36.010 --> 00:38.430 I go by the pseudonym GTKondyk. 00:38.430 --> 00:41.970 I am a security researcher and senior security consultant. 00:41.970 --> 00:46.330 I'm very passionate about network security, both the attack and defense side. 00:46.330 --> 00:57.270 And through that passion, I run a project called NetSecExplained, which is a blog and YouTube channel where I explain intermediate and advanced level concepts in an easy-to-understand way. 00:58.970 --> 01:02.870 So, machine learning, let's get everybody on the same page. 01:03.210 --> 01:04.590 What is machine learning? 01:04.590 --> 01:09.070 Well, first, we have this big umbrella term of AI. 01:09.190 --> 01:11.470 And parts of AI are machine learning. 01:11.470 --> 01:13.290 Parts of AI are unrelated. 01:13.290 --> 01:16.350 So, we have rule-based artificial intelligence. 01:16.350 --> 01:18.930 We have classical image recognition. 01:19.530 --> 01:23.770 But inside of AI, we have this small subset called machine learning. 01:23.770 --> 01:33.530 And machine learning specifically allows us to take statistical analysis and draw inferences from data and perform data discovery techniques. 01:33.590 --> 01:39.970 And then out of machine learning, we also have deep learning, which is specifically deep neural networks. 01:39.970 --> 01:51.210 And this can be convolutional neural networks, which are used primarily for image recognition, or recurrent neural networks, which are used for cyclical pattern recognition, such as a heartbeat monitor. 01:53.610 --> 02:00.050 So, the burning question is, how is AI empowering adversaries? 02:00.050 --> 02:03.810 Or how can it empower adversaries moving forward? 02:04.170 --> 02:14.450 So, I see the age of AI today, or the age of machine learning today, we're at the very beginning of what it can do for the security industry on both the attack and the defense side. 02:14.530 --> 02:23.770 Now, if you remember back in the 80s and 90s, antivirus, intrusion detection, vulnerability scanners, these were all simple scripts that were written. 02:23.770 --> 02:28.510 And then over time, they grew into what are now multi-billion dollar industries. 02:28.710 --> 02:34.630 And so, I think that we're at the very beginning of what AI and machine learning can do for the security community. 02:36.410 --> 02:41.230 So, the way that I'm going to bring about this talk is really in two major parts. 02:41.230 --> 02:45.030 So, the first is to talk about offensive AI tools. 02:45.090 --> 02:46.510 Why do we want these? 02:46.510 --> 02:47.790 What's out there already? 02:47.790 --> 02:49.450 What are we seeing in the wild? 02:49.450 --> 02:51.050 How do we build them? 02:51.050 --> 02:56.430 And then adversarial machine learning, which is attacking the machine learning algorithms themselves. 02:56.930 --> 03:01.490 So, let's start with offensive AI tools. 03:04.370 --> 03:08.270 So, the first question is, why would we even want these? 03:08.410 --> 03:18.470 Well, offensive AI tools and AI tools in general will allow us to operate at a much faster speed and a much larger scale than we're currently able to do. 03:18.570 --> 03:21.950 And this is going to allow us to automate a lot of manual tasks. 03:22.050 --> 03:25.210 Now, we don't need to think about it as automating everything. 03:25.210 --> 03:41.210 But if there's something that requires or manages or deals with a lot of data, and these are things that we can apply a certain amount of fuzzy logic to, then this is a perfect candidate for automating with some form of AI or machine learning. 03:41.370 --> 03:52.330 And then, of course, this allows us to dynamically and intelligently explore our target attack surface and uncover hidden blind spots that come about some of the biases that we intrinsically have. 03:52.610 --> 04:05.570 And so, being able to statistically analyze some of this data or perform some machine learning operations, this will allow us to kind of uncover some of those blind spots and look for things that we don't already know. 04:07.030 --> 04:09.710 And then, how realistic is this? 04:09.710 --> 04:17.930 Well, back in 2016, DARPA kind of took on a large undertaking in their CyberGram challenge. 04:18.050 --> 04:29.270 In here, what they did was they required participants to develop some form of artificial intelligence that will perform a series of attack chain tasks. 04:29.270 --> 04:36.110 So, in this case, it was going to be vulnerability discovery, exploitation, patching, and then data theft. 04:36.110 --> 04:39.450 So, in the form of data theft, they have capture the flag. 04:39.790 --> 04:47.710 And then, as part of their major grand challenge, they had an attack, defend, capture the flag event that was air gapped. 04:47.710 --> 04:56.230 And you can see that in the picture kind of off to the side, where there's no human interaction whatsoever for the entire event. 04:56.230 --> 04:58.230 And everything had to be automated. 04:58.790 --> 05:09.010 So, this was a big reveal on a world stage that let us see what the power of artificial intelligence is, especially when it comes to a number of these tasks. 05:10.270 --> 05:14.150 And then, where are we seeing applications of AI already? 05:14.150 --> 05:19.010 So, the current proof of concepts we can see for social engineering. 05:19.170 --> 05:24.370 We can see in defense detection and evasion. 05:24.390 --> 05:38.690 There are some researchers that are identifying intrusion detection systems and sandbox detection systems so that attacks can be stealthier and break out of various systems and bypass existing defense measures. 05:38.930 --> 05:40.450 Evaluating data leaks. 05:40.450 --> 05:50.310 So, large data sets such as password dumps, subdomains, things of that nature, websites, applications, configuration files. 05:50.550 --> 05:52.430 Automated network exploitation. 05:52.430 --> 06:02.730 So, this will be going from standard user to domain admin, automating that process, as well as normal internal and external network pen tests. 06:03.290 --> 06:04.680 And then, software exploitation. 06:05.630 --> 06:28.430 And so, we'll see this with AFL, which is a software fuzzer that will utilize its own custom compiler, put in certain key points in the software itself, and then it allows the system to fuzz and track where through the application user input is being provided and how that can reach potentially vulnerable functions. 06:29.810 --> 06:31.790 So, let's start with social engineering. 06:31.790 --> 06:45.390 Social engineering, there's a couple things that are in the works right now that will allow us to improve our social engineering capabilities and improve attackers' abilities to exploit corporations and systems. 06:45.530 --> 06:48.270 So, in this case, we have an NVIDIA project called StyleGAN. 06:48.330 --> 06:52.070 StyleGAN, what it does is generate human faces. 06:52.710 --> 06:57.910 So, all four of the images up at the top you see are actually generated by StyleGAN. 06:58.270 --> 07:05.290 And then, just to kind of show you that there's no tricks up my sleeve, on the bottom row, we have StyleGAN generated cats. 07:05.290 --> 07:19.410 And you can see here in the bottom right corner, this abomination that kind of looks like a cat, but is really messed up and difficult to see, difficult to really recognize for a human. 07:19.410 --> 07:24.190 And this is kind of the imperfect model generating a cat-like image. 07:24.190 --> 07:34.350 But the level of detail that these can go into, if you look at the cat over here on the left, just kind of sitting on the carpet, you can see the individual hair, the whiskers, the texture of the carpet underneath. 07:34.950 --> 07:43.610 And so, these are really difficult to tell whether or not they are legitimate faces or cats, or if these are machine learning generated. 07:43.630 --> 07:52.250 One of the telltale signs, if you look here in the top right corner, this gentleman, it seems like there's a picket fence behind him, but it's a little warped in the back. 07:52.250 --> 07:56.450 You also notice that all of the images have a blurred background. 07:56.670 --> 08:01.810 And if you look really carefully, all of the eyes point directly at you. 08:01.810 --> 08:06.890 So, it doesn't matter what direction they're faced, the eyes are pointing straight. 08:07.610 --> 08:15.830 So, there's already some technologies and some research being performed to kind of identify what is a legitimate human face and what is a generated human face. 08:16.530 --> 08:18.370 Now, why is this important? 08:18.370 --> 08:20.290 How can we use this for social engineering? 08:20.670 --> 08:37.230 Well, for several types of social engineering campaigns, what's pretty common to do is go to the 10th page of a Google image search, grab somebody's stock photo or profile photo, and then you can paste that up as your own LinkedIn profile photo and say, 08:37.230 --> 08:39.490 okay, I am now Becky. 08:39.490 --> 08:41.410 I'm not a 32-year-old male. 08:41.410 --> 08:42.450 My name is Becky. 08:42.590 --> 08:44.270 I'm 27. 08:44.270 --> 08:50.790 I'm starting my career and I want to ask somebody, hey, as a fellow woman, how did you get started? 08:50.790 --> 09:01.370 And then I can use that to build rapport, have my pretext, and perform social engineering tasks to try and exploit employees, maybe in the HR department, sales, wherever. 09:02.010 --> 09:21.990 And so, instead of grabbing that image and throwing it up as a pretext where an analyst can go in, grab that same image, do a reverse Google image search, see that it's a stock photo, call me out, realize that I'm fake, what I can do is use something like StyleGAN to generate my own face. 09:21.990 --> 09:25.650 So, why steal a person when I can just make a whole new person? 09:26.250 --> 09:30.810 But there needs to be a little bit more legitimacy to my pretext. 09:31.050 --> 09:34.170 So, this is where GPT-2 and Grover come into play. 09:34.170 --> 09:36.590 GPT-2 and Grover, they operate very similarly. 09:36.590 --> 09:41.610 These are automated generation of long-form content. 09:42.050 --> 09:54.590 So, these can be used to generate emails, blog posts, articles, anything of that nature, with very minimal effort from the attacker or the adversary in that respect. 09:55.210 --> 10:09.310 And so, unfortunately, the website that was used to generate these has been taken offline, but the GPT-2 model is still something that you can download, put on your own systems, and generate long-form content. 10:09.310 --> 10:28.450 So, if I was developing a pretext and I can use this to write phishing emails that could get past a spam filter, or that look distinctly unique, so that I'm not sending the same email to multiple people within an environment, or I can use this to write blog posts and long-form content, 10:28.450 --> 10:34.930 and because any sort of Google image search or... what is that called? 10:36.010 --> 10:44.210 Plagiarism detection, it will see that I am the only source of truth, therefore I must be a real person, therefore my website must be legitimate. 10:44.950 --> 10:47.790 And so, this will kind of add to the pretext. 10:47.790 --> 10:57.930 And then on top of that, long-form content Google really enjoys, and so that can give us some Google ranking to add a little bit more credibility to our initial pretext. 10:59.950 --> 11:09.110 So, this is one of the scarier ones that I see as being at the very beginning of the research, and I'm curious to see where this will be in the next several years. 11:09.190 --> 11:11.630 We have Lyrebird and Tacotron. 11:11.630 --> 11:23.610 Lyrebird and Tacotron, these are text-to-speech that are based on human voices and use those human voices to generate conversation bits. 11:23.610 --> 11:30.790 And so, we can use this, I foresee, in the phishing campaigns where we're actually calling people or leaving voicemails. 11:30.790 --> 11:39.610 So, it's a lot less automated sounding, a little less robocall, and then you just add a little bit of compression and it sounds like you just have a really bad connection. 11:40.290 --> 11:42.110 So, I have a couple audio clips. 11:42.110 --> 11:43.110 I want you to hear them. 11:43.110 --> 11:50.570 So, these first two, I have the original voice and then the second one is the synthetic voice based off that original voice. 11:50.590 --> 11:53.170 So, let's go ahead and take a listen. 11:53.890 --> 11:57.150 In another moment, down went Alice after it. 11:57.150 --> 12:01.330 Never once did she consider how in the world she was to get out again. 12:03.310 --> 12:04.090 Awesome. 12:04.090 --> 12:07.350 And then the synthetic voice built off of that sample. 12:07.350 --> 12:12.150 My voice might be generated by a computer, but I think it sounds pretty human. 12:12.230 --> 12:15.610 I don't know exactly how they made it, but I'm really impressed. 12:16.170 --> 12:33.710 So, what I'm imagining is that as these models get better, what we can do is take interviews of CEOs and VPs of large corporations, grab their audio, isolate it, and then put it into a model like this so that we can have our own version of synthetic text to speech using their voice. 12:33.750 --> 12:44.390 So, we can call throughout the company and impersonate these individuals in order to cause some sort of social engineering action. 12:44.990 --> 12:46.810 Now, this last one is really interesting. 12:47.050 --> 12:48.030 This is Tacotron. 12:48.030 --> 12:49.490 This is a Google project. 12:49.630 --> 12:55.630 And I have the text over here on the right, but what I want you to do while you're listening to this is pay attention to the breaths. 12:55.630 --> 13:04.170 As humans, we tend to have filler words like um and uh, and then in between sentences and commas, we usually have a bit of a pause or a breath. 13:04.170 --> 13:06.630 And so, the Tacotron model actually picked that up. 13:06.630 --> 13:08.150 This is fully synthetic voice. 13:08.150 --> 13:09.450 Just take a listen. 13:10.150 --> 13:13.430 Only the photograph really showed how much time had passed. 13:13.430 --> 13:19.470 Ten years ago, there had been lots of pictures of what looked like a large pink beach ball wearing different colored bonnets. 13:19.470 --> 13:32.610 But Dudley Dursley was no longer a baby, and now the photograph showed a large blond boy riding his first bicycle on a carousel at the fair playing a computer game with his father being hugged and kissed by his mother by his mother. 13:34.930 --> 13:36.750 That's pretty scary. 13:38.170 --> 13:40.510 So, these projects are still underway. 13:40.530 --> 13:49.290 Liarbird is actually advertising itself as a transcription or reverse transcription service, which is just text-to-speech. 13:49.490 --> 14:02.270 In their original pay model that they had pushed out, currently they're in closed beta, but in their original pay model that they pushed out, you can transcribe three hours of audio pretty cheaply. 14:02.690 --> 14:06.510 And then if you're so inclined to pay, you can transcribe obviously more. 14:06.510 --> 14:21.690 But if a malicious user or somebody who's maliciously inclined, they can just grab those voice samples, throw them into Liarbird or Tacotron, or just use the voice samples that are already in there and generate their own adversarial examples. 14:24.720 --> 14:28.080 So, next we get into sandbox detection. 14:28.080 --> 14:32.900 And so, this is the defense detection and evasion part of the system. 14:32.900 --> 14:44.800 So, we have these researchers over at Silent Brick Security, and what they did was decide that with many different types of sandbox, what you can do is take the process list as data itself. 14:45.600 --> 14:52.740 And so, based on the process list, the number of processes, the number of users on a system, you can generate and decide whether or not you're in a sandbox. 14:52.740 --> 15:00.280 And then you can take it a step further and decide what kind of sandbox you're in, whether you're in FireEye or Cuckoo sandbox. 15:01.440 --> 15:15.440 So, what the researchers did was kind of classify, they took a number of basic Windows systems and they took a number of sandboxes and just kind of identified how many processes there are, how many users there are. 15:15.440 --> 15:17.840 Let's take a look at what malware can identify. 15:18.140 --> 15:22.720 And so, when we have a low process count, you see over here in column A, we have 33. 15:22.720 --> 15:24.420 In column F, we have 34. 15:24.720 --> 15:30.440 And then we have a high user count, such as, again, A and F, we have 4 and 4. 15:30.660 --> 15:34.720 And the process count per user, 8.25, 8.5. 15:34.940 --> 15:39.080 It gives us a very high probability that it is a sandbox. 15:39.560 --> 15:42.320 And so, we can see that with our host total score. 15:42.440 --> 15:45.880 And then based off that total score, we can set a threshold. 15:45.880 --> 15:51.320 In this case, it looks like they set the host average score as their threshold. 15:51.320 --> 15:54.460 So, anything below the average score is considered a sandbox. 15:54.460 --> 15:58.020 Anything above that is considered a legitimate system. 15:58.060 --> 16:08.460 And so, they can have malware that completely shuts down or erases itself if it identifies that it's in a sandbox, or it can use targeted evasion techniques per sandbox. 16:08.460 --> 16:11.980 So, if I'm in FireEye, these are the things that I'll do to get past it. 16:11.980 --> 16:16.660 If I'm in Cuckoo, these are the things that I'll do to get past it. 16:16.880 --> 16:20.180 So, it's a really interesting article, highly recommend taking a look at that. 16:22.540 --> 16:28.340 And then for automated network exploitation, we have systems like DeepExploit and Goethon. 16:28.340 --> 16:31.140 Goethon and DeepExploit, they both operate very similarly. 16:31.580 --> 16:45.360 The way that these work is they will go out, they will perform an nmap scan based on the results of that nmap scan, if it sees common services that it knows are exploitable or things that it would like to kind of take to that next level. 16:45.380 --> 16:48.220 For example, port 80 or 443, it's web traffic. 16:48.420 --> 16:51.920 So, it's going to go ahead and run Scrappy to go and crawl that system. 16:51.920 --> 17:01.040 I'm sure they could add other tools such as Go Buster, which will go out and enumerate a lot of the files or directories or URLs that are on that system. 17:01.260 --> 17:02.680 Try and fingerprint the server. 17:02.680 --> 17:09.600 You want to see if it's running Nginx, Apache, PHP, Java, things of that nature. 17:10.000 --> 17:15.340 And then it performs additional content exploration and its own Google hacking, which is kind of cool. 17:15.580 --> 17:22.100 So, that will go out, do a little bit of research, see if there's any sort of exploitable scripts, things of that nature. 17:22.100 --> 17:35.480 And it takes all of this data and it processes it using some statistical techniques and then it uses Metasploit as a targeted exploitation engine and it goes and tries to exploit those things. 17:35.480 --> 17:45.900 So, if it sees certain versions of JBoss or Jenkins, it will go and try and target that with maybe password spray or any sort of Metasploit module. 17:46.080 --> 17:47.660 Tomcat, same thing. 17:48.160 --> 17:58.040 So, this is kind of cool as a proof of concept of automating the entire process of network discovery all the way up to exploitation and in some cases post-exploitation. 17:59.980 --> 18:03.980 One of my favorite while I was researching this is Deathstar. 18:03.980 --> 18:07.440 And this is my favorite just kind of in the way that it is assembled. 18:07.940 --> 18:10.840 So, Deathstar, it's built on PowerShell Empire. 18:11.260 --> 18:20.560 And the way that it works is it starts, it kind of covers its baseline and it is designed to go from standard user all the way up to domain administrator. 18:20.560 --> 18:25.400 And so, it charts and maps attack paths all the way to that location. 18:25.400 --> 18:28.720 And so, it starts at its baseline and it says, am I already a domain administrator? 18:28.720 --> 18:30.560 If so, cancel out. 18:30.560 --> 18:32.860 And we see that's the path off to the left. 18:33.140 --> 18:36.280 If I have domain administrator credentials, all right, cool. 18:36.280 --> 18:37.220 Log in. 18:37.520 --> 18:39.540 And that charts off the path to the right. 18:39.720 --> 18:44.240 But if it's neither of those, then it goes straight down the middle and it says, okay, let's spawn a new agent. 18:44.640 --> 18:50.840 Let's go ahead and identify where is the domain administrators and where is the domain controllers. 18:50.840 --> 18:52.900 And so, it runs over here to the right. 18:52.900 --> 18:57.980 We can see get net domain controller, get net group member, invoke user hunter. 18:58.000 --> 19:07.200 And so, this is going to enumerate the domain controllers, enumerate the domain admins, and enumerate the users that it can on the systems that it can. 19:07.700 --> 19:11.360 And you can see it goes through this entire tree kind of off here to the bottom left. 19:11.360 --> 19:17.120 It identifies and says, hey, is this a vulnerable, known vulnerable version of Windows, for example, Windows 7? 19:17.120 --> 19:25.740 If yes, let's run Mimikatz, try and do Mimikatz, secret stump, things of that nature, collect credentials, and then we can kind of start back over at the top. 19:26.400 --> 19:28.740 Let's go ahead and enumerate the running processes. 19:28.740 --> 19:34.660 Do we see any processes that are already being ran by other users that we don't have in our inventory? 19:34.660 --> 19:39.120 If so, let's add those users to our inventory and try and do some lateral movement. 19:39.580 --> 19:44.380 And so, in this case, it'll use psinject, and then it'll go ahead and spawn a new instance. 19:44.380 --> 19:50.540 And so, it goes through this iterative process all the way until it reaches domain admin credentials. 19:51.300 --> 20:04.420 And so, if you've ever used a system like Bloodhound, where you'll generate an attack graph and say, okay, this user is part of this group, which has these members, which this member is also part of domain admins. 20:04.420 --> 20:08.740 And so, this is the attack path that you want to go in order to become domain admin. 20:08.880 --> 20:16.940 And so, Deathstar kind of takes that idea a step further and actually tries to navigate that attack path in an automated fashion. 20:18.120 --> 20:25.700 So, that's all I have for the machine learning tools that already exist, some of the proof of concepts. 20:25.700 --> 20:31.420 There's a lot more information that I wanted to cover that I wouldn't be able to cover in such a short amount of time. 20:31.420 --> 20:42.060 So, at the end of this presentation, I have a link to my slide deck, which has a lot more detail and a lot more tools and a lot more techniques in it. 20:42.500 --> 20:51.960 But in the meantime, I kind of want to answer the question of how do I go about creating my own machine learning tools or AI tools? 20:51.960 --> 20:53.640 What should I look for? 20:53.640 --> 20:54.840 How do I automate that? 20:54.840 --> 20:57.360 What is that process? 20:57.360 --> 21:11.380 And so, honestly, my advice would be to identify somewhere that either utilizes a lot of data or identify somewhere that uses a... it's a common process, but it uses some sort of fuzzy logic, right? 21:11.380 --> 21:22.520 Something that's kind of hard to script, hard to write a signature for, but with AI and machine learning, you can use some of that fuzzy logic to generate some heuristics and it can perform based off of that. 21:23.160 --> 21:34.980 So, in the case of some of these, what we can do is say, all right, NMAP scans, everybody does NMAP scans, and then you have to manually look at them, and then based off of that, you have to follow up and go over here. 21:35.380 --> 21:50.660 Instead, like I showed with Goython, is you can just use the NMAP scan and then automate the process of, okay, let's enumerate and fingerprint these services, and then let's see if any of these services are already exploitable using Metasploit, or let's go with next steps. 21:50.660 --> 21:55.060 I identified a web service, let's run web analysis techniques. 21:55.140 --> 21:59.580 So, Go Buster, Crawler, any sort of things like that. 22:00.520 --> 22:06.120 So, that's kind of the thing that you want to keep an eye on if you want to develop your own machine learning tools. 22:06.120 --> 22:15.240 And there's more that are coming out all of the time, and it's really interesting and a really fast-paced area, and it's just ripe with research. 22:15.240 --> 22:17.280 So, go ahead and play around with some of that. 22:19.200 --> 22:26.640 So, the next bit that I want to get into is adversarial machine learning, and this is attacking the machine learning algorithms themselves. 22:27.420 --> 22:33.900 So, adversarial machine learning, there's really three ways to go about attacking a machine learning model. 22:34.160 --> 22:36.800 So, the first way we have is model evasion. 22:36.800 --> 22:46.500 This is to take what is clearly a malicious example and get the model to consider it to be benign or normal or standard to bypass that filter. 22:46.500 --> 22:56.780 So, this would be taking a spam message and getting past a spam filter, or this would be taking a piece of malware and getting it past antivirus or anti-malware solutions. 22:57.420 --> 23:04.360 Model poisoning is where we add our own data into the training data set. 23:04.560 --> 23:13.360 Now, machine learning models specifically are... they're done learning as soon as they're done being trained. 23:13.380 --> 23:22.120 To get around this, there are some models that are called online systems, and so what they'll do is they'll take data that has been classified by itself already. 23:22.120 --> 23:32.820 So, say I send it a piece of data, like a piece of malware, it looks at that, says, okay, bam, that's malware, now that goes into my training set, and I learn from that malware. 23:33.580 --> 23:49.240 So, data poisoning allows us to slowly inject certain pieces of data so that we push that threshold a little off to the side into what is more acceptable, and then we target where that threshold has been pushed. 23:49.680 --> 23:52.740 And we'll see an example of that in a bit. 23:53.320 --> 23:54.120 Data leakage. 23:54.120 --> 23:58.660 Data leakage is actually a combination of two smaller bits. 23:58.660 --> 24:02.520 So, one is stealing the model itself. 24:02.660 --> 24:10.320 So, for example, say you're an IoT engineering company and you develop an IoT system that utilizes some form of machine learning. 24:11.620 --> 24:25.840 What I can do is buy that device and then reverse engineer it, and now I have your model, and so I can use that model or train another model based off of that instead of having to go through the hundreds of thousands of dollars that you went through to develop that in the first place. 24:27.400 --> 24:30.660 And so, this is where the model itself is easy to be grabbed. 24:30.660 --> 24:39.440 There's also data leakage, which is I can identify what data you use to train the model. 24:39.760 --> 24:47.540 And so, this is really damaging, especially for healthcare industry or HIPAA-protected information. 24:47.800 --> 24:57.280 So, if I can identify any sort of clients or patients that you have, any sort of their PII, then that's really damaging. 24:59.760 --> 25:03.080 So, when it comes to model testing, there's really two approaches. 25:03.480 --> 25:15.840 You have a white box where you know what the model is, you know how the hyperparameters are set, you know everything that's involved in the building of this, and then you target that model, much like you would do in software where you already have all of the code involved. 25:16.380 --> 25:19.520 And then you have a black box, which is just the model itself. 25:19.520 --> 25:26.040 You aren't able to peek into the way that the model operates, but what you can do is treat it like an oracle. 25:26.100 --> 25:37.240 So, if you've ever done any sort of cryptography, you know, exercises, you have an oracle, you ask yes-no questions, or things of that nature, and then you can generate what is called a surrogate model. 25:37.240 --> 25:52.020 So, based off of the yes-no questions that I asked the black box model, I generated my own model, and then based off of that model, I can perform attacks and that those attacks should be able to impact the original black box itself. 25:52.660 --> 25:57.780 Now, the way that this operates is under a principle called attack transferability. 25:57.900 --> 26:12.580 This is a documented phenomenon, and the way that attack transferability works is, it doesn't matter what model or the underlying architecture that these models are designed with, an attack on one model can also be an attack on another model. 26:12.580 --> 26:31.520 So, if I have a deep learning model over here, and I find adversarial examples that I can use to attack this model, then here is a random forest, a logistic regression, SVM, or anything else, any other machine learning algorithm or architecture, that those same adversarial examples will be able to impact that. 26:33.620 --> 26:35.600 And then, of course, the attack surface. 26:35.600 --> 26:39.320 So, where are the machine learning models themselves vulnerable? 26:39.960 --> 26:47.300 So, machine learning, as it works, is you have a physical object, and then you take that physical object and you try to digitize it. 26:47.300 --> 26:54.880 So, say you have a picture of a panda or a car, you take a photograph of it, now it's digitized, and you can make information off of that. 26:54.880 --> 27:10.380 You take that digitized information, you run your pre-processing as you would per the normal machine learning process, and then you put that into the machine learning model as part of its input features, it presents some output, usually a probability or a number of some sort, 27:10.380 --> 27:13.340 and then a decision is made based off of that action. 27:13.440 --> 27:20.060 So, for example, if I am 67% confident that this is malware, then I will probably flag it as malware. 27:20.140 --> 27:24.500 If I'm only 42% confident, then I'll just let it continue to operate. 27:25.120 --> 27:26.820 So, where can we attack things? 27:26.820 --> 27:30.980 Well, one, we can actually impact the physical object itself. 27:31.840 --> 27:39.960 So, down below, I have the network construction detection system as kind of an example that follows this generic machine learning model path. 27:39.960 --> 27:41.320 So, we have the attack traffic. 27:41.320 --> 27:54.180 We can actually, as attackers, modify the attack traffic in such a way that as it goes through the pipeline of the machine learning pipeline, it will modify the behaviors at different points. 27:55.360 --> 28:03.000 So, I can slow down my attack traffic, I can use my attack traffic for multiple hosts so that it's a little bit harder to detect. 28:03.680 --> 28:07.100 And then that attack traffic is then digitized. 28:07.100 --> 28:12.100 The digitization process, we don't really have a lot of impact over as attackers. 28:12.100 --> 28:17.080 It's not realistic for us to be able to affect TCP dump. 28:19.220 --> 28:21.320 There's really famous examples. 28:21.520 --> 28:32.620 For example, you may have come across this if you ever looked at adversarial machine learning, a picture of a panda, and you modify some pixels, and now it thinks it's a gibbon or a bird or some sort. 28:33.100 --> 28:40.200 But if we have the ability to modify pixels, why don't we just give it a picture of a bird at that point? 28:40.380 --> 28:50.380 And then more to the point, if we can modify pictures, if you have something like a self-driving car, it has a camera, how do we interface with that camera in an adversarial perspective? 28:50.380 --> 28:54.860 We would already have to have access to the system, which at that point we can just shut it down or do whatever. 28:54.860 --> 28:56.200 It's physical access. 28:57.620 --> 29:08.200 So, we're not actually able to interface with the digitization of the object, but what we can do is change the way that the attack traffic is modified. 29:08.200 --> 29:13.120 We can change the input features, the input vectors, the machine learning model, especially with the model poisoning. 29:14.080 --> 29:16.920 We can change the way that the data is represented. 29:18.460 --> 29:28.520 So, in the case of the network intrusion detection system, we have the packet metadata, and then it spits out an attack probability, and then off of that it makes a decision. 29:28.840 --> 29:39.320 And in between the machine learning model and the decision-making process, you got to remember this is just software, and you can approach it like any other piece of software that you're pen testing or trying to hack. 29:39.320 --> 29:47.240 So, you have the machine learning model in it, and then all wrapped around it is a nice piece of software that probably has some of its own vulnerabilities involved. 29:49.420 --> 29:51.420 So, model invasion. 29:52.220 --> 29:54.840 Model evasion, not invasion, sorry. 29:55.680 --> 29:57.740 So, we're going to try and hide in the blind spots. 29:57.740 --> 30:00.060 Now, we have this massive theoretical space. 30:00.060 --> 30:08.100 This is all of the possible pieces of malware that could possibly exist, but we can't collect all of that, realistically speaking. 30:08.620 --> 30:15.020 Antivirus companies can't collect all of it, and so what they have is a subset, and this is considered the training space. 30:15.020 --> 30:18.560 So, this is what they are realistically able to collect. 30:18.860 --> 30:29.140 So, in their training space, they perform their training, and then off of a small piece of that that's already clean labeled data, they perform their testing to make sure that the model operates properly. 30:29.780 --> 30:34.780 Well, part of that testing space, kind of right there off the sliver, this is the adversarial space. 30:34.780 --> 30:35.820 This is where we live. 30:35.820 --> 30:42.020 This is where we are able to generate adversarial examples that target that model specifically. 30:42.440 --> 30:47.800 And so, this is kind of what we want to play around with and hide in the blind spots, so to speak. 30:48.300 --> 31:02.160 So, a couple really good examples is, in this case, multinomial Naive Bayesian spam filters, or just any of the early spam filters that used some rudimentary forms of machine learning, how they would operate is they would look at keywords. 31:02.160 --> 31:14.960 So, if you had something like, buy Viagra now, the probability that the word Viagra shows up in a spam email is much higher than the probability of the word Viagra showing up in a regular email. 31:15.080 --> 31:22.200 And so, based off of these keywords, it does its own statistical analysis and says, I am 99% sure this is spam. 31:23.180 --> 31:30.140 Well, the interesting thing about emails is that emails are usually represented as HTML pages or web pages. 31:30.800 --> 31:37.120 And so, as malicious actors, what we can do is actually add in a comment. 31:37.540 --> 31:41.400 So, in this case, we can add in the entire Wikipedia page article for a horse. 31:41.400 --> 31:45.200 And so, we have a bunch of bad words, but we have even more good words. 31:45.200 --> 31:53.500 What this lets us do is kind of skip past the spam filter by showing, hey, we have a lot more good words than bad words, so we must be good, right? 31:53.500 --> 31:55.080 It says, yeah, absolutely. 31:55.440 --> 31:58.540 I am 99% sure you are not spam. 31:59.400 --> 32:03.960 And the thing is that the way that this is presented to users is as an HTML web page. 32:03.960 --> 32:08.040 And so, the comment itself is not actually rendered for the users. 32:08.040 --> 32:12.860 So, the users still see the original spam email as though it was unmodified. 32:13.920 --> 32:18.760 And so, this is kind of a real world example on some of the spam filters. 32:19.420 --> 32:23.680 A really good example of this is by Skylight Cyber. 32:23.680 --> 32:33.440 The researchers over there were actually able to bypass the silence machine learning antivirus in this article, Silence, I Kill You. 32:33.440 --> 32:34.620 Really good article. 32:34.620 --> 32:36.620 Highly recommend you take a look at it. 32:36.620 --> 32:38.900 We can see some familiar malware samples here. 32:38.900 --> 32:54.320 So, we have CoinMiner, Emotet, Zeus, and we see the original scores that they were actually able to pull out of the silence antivirus, negative 826, negative 923, negative 997. 32:54.420 --> 32:57.920 So, these are really bad, really bad, flagged immediately. 32:58.980 --> 33:14.740 So, what the researchers over at Skylight Cyber did was they took the silence antivirus, reverse engineered it, and they were able to pull out the scores based on some of the syscalls and other key features that silence looked at for malware. 33:15.940 --> 33:21.620 One of the curious things is that they identified certain things that were whitelisted. 33:21.840 --> 33:30.540 Now, why would silence whitelist certain things, even though the standard machine learning process would actually identify it as potentially malware? 33:30.780 --> 33:35.860 Well, the reason why is because when you think about malware, think about how malware operates, right? 33:35.860 --> 33:43.800 So, maybe it asks for access to your webcam or your microphone, or it tries to resize or open up new windows. 33:44.100 --> 33:46.060 Maybe it makes calls out to the internet. 33:46.060 --> 33:48.960 Well, what's another thing that does that? 33:49.520 --> 33:53.020 Chrome, Firefox, Internet Explorer. 33:53.080 --> 33:55.640 They operate very similar to some modern malware. 33:55.920 --> 33:59.360 In this case, what they found was a video game. 33:59.820 --> 34:01.680 I'm not sure which video game it is. 34:01.680 --> 34:04.640 My money is on Fortnite or League of Legends. 34:05.240 --> 34:09.160 But the video games that they found were certain whitelisted keywords. 34:09.640 --> 34:17.500 So, what they did was took these clearly bad malware and they just added these keywords to the end of the malware. 34:17.540 --> 34:27.440 So, it didn't change the way that the malware operated, but it changed the way that silence looked at it and said, okay, you have a bunch of bad boy points, but you have even more good boy points. 34:27.440 --> 34:30.460 So, we're going to give you some good boy scores. 34:30.500 --> 34:33.020 So, in this case, CoinMiner had 884. 34:33.120 --> 34:41.960 My favorite is looking at Zeus, negative 997, one of the worst in this table, all the way up to positive 997, which is one of the best in this table. 34:42.680 --> 34:50.140 And so, it shows that model evasion techniques are very useful in a number of different ways. 34:52.100 --> 34:53.920 So, how do we defend against this? 34:53.920 --> 34:57.960 Well, the first and probably one of the better is adversarial training. 34:58.040 --> 35:02.220 Adversarial training kind of comes from the idea of chaos engineering. 35:02.220 --> 35:11.300 Netflix is a really big proponent of chaos engineering, where you already know something bad is going to happen, so you might as well account for those bad things to happen. 35:11.300 --> 35:14.200 So, in this case, you train it with adversarial examples. 35:14.200 --> 35:20.920 You generate your own adversarial examples and you train the model based off of that to kind of harden and make it a little bit more robust. 35:21.720 --> 35:25.020 Again, if it's not a robust model, it's not a good model. 35:25.940 --> 35:31.240 Chaos engineering as a whole , the way that it operates is you already know something is going to go bad. 35:31.240 --> 35:32.520 So, you have Netflix , for example. 35:32.520 --> 35:37.220 Netflix, they know that their systems, they need to have that uptime. 35:37.220 --> 35:39.680 And so, they know that it's possible one of their systems goes down. 35:39.680 --> 35:42.060 They want to make sure their redundancy operates properly. 35:42.320 --> 35:46.380 So, you purposefully go into production and you break things. 35:46.380 --> 35:54.200 And then you develop processes and policies around knowing things are going to break. 35:54.200 --> 35:59.000 So, in some cases, you'll have the CIO walk into a data center and just start unplugging servers. 36:00.660 --> 36:04.500 So, adversarial training is a really good way to kind of counteract this. 36:04.680 --> 36:08.940 Another slightly more technical way is called defensive distillation. 36:08.940 --> 36:14.640 Defensive distillation is a smoothing algorithm that kind of smooths the curve. 36:14.640 --> 36:18.720 So, instead of having, okay, malware, malware, malware, malware, nope, it's okay. 36:19.400 --> 36:21.620 You can't have sharp edges like that. 36:21.620 --> 36:23.040 So, what it does is it smooths it. 36:23.040 --> 36:27.920 So, instead of going straight down, it just kind of goes in a downward trend. 36:27.920 --> 36:36.940 And so, it's a little bit harder, a little bit more robust way of analyzing and dealing with the adversarial examples. 36:37.600 --> 36:41.400 And then the last example would be monotonic classification. 36:41.400 --> 36:44.020 Which means that the graph grows in one direction. 36:44.480 --> 36:53.840 So, we see here in the image off to the right, the non-monotonic example, we see that it's getting worse and worse and worse as far as a keyword . 36:53.840 --> 36:56.040 So, it goes worse and worse and worse across that threshold. 36:56.040 --> 36:59.780 Now, it's considered bad, it's spam, it's malware, it's whatever. 37:00.340 --> 37:02.700 We're ready to throw it out, but we're still analyzing it. 37:02.700 --> 37:06.680 And so, it's crossed that threshold, but wait, no, it's getting better and better and better. 37:06.680 --> 37:10.880 We just hit that pad of keywords that are really positive. 37:11.040 --> 37:14.120 And so, we went from good to bad, all the way back up to good. 37:15.260 --> 37:21.540 Monotonic classification means that we don't count the good words at all, we say, we measure your badness. 37:21.540 --> 37:23.280 How many bad words do you have? 37:23.560 --> 37:34.600 And so, in this case, on the bottom example, we have the monotonic classification, which says, okay, we have some bad words, we have some bad words, up, we all went all the way up to the top, and then we hit all of the padded good words. 37:34.860 --> 37:39.680 And it doesn't matter, you're still bad, Jesus is not going to save you, you're done. 37:40.280 --> 37:46.500 And so, monotonic classification is a really powerful technique to deal with these model evasion techniques. 37:48.480 --> 37:50.880 Next, we have model poisoning. 37:50.960 --> 37:57.540 Model poisoning almost exclusively works on online learning systems. 37:57.540 --> 38:02.860 So, as adversaries, we need to find a way to get adversarial examples into the training space. 38:03.180 --> 38:12.060 And so, this is where systems will grab data that it's already classified a certain way, and add it into the training space. 38:12.060 --> 38:21.960 And so, we're just trying to push that barrier past a certain threshold where we can actually operate safely as adversaries without it detecting us as such. 38:22.840 --> 38:24.720 So, some real world examples. 38:25.180 --> 38:32.460 Tay AI, if you're not already familiar, I highly recommend checking it out. 38:32.460 --> 38:41.280 Tay AI was a Microsoft Twitter chat bot, and it was designed to learn from real conversations by real people. 38:41.960 --> 38:49.840 Well, 4chan, of course, gets a hold of this, and they decide, how much does it learn? 38:49.840 --> 38:51.700 How much can we teach it? 38:51.880 --> 39:05.240 And about 16 hours after coming online, it was taken offline permanently, because within that time, 4chan was able to get it to say, Hitler did nothing wrong, and racial slurs. 39:05.960 --> 39:14.600 If you want more information on Tay AI, and I highly recommend checking this out, it's called The People's Chatbot by the Internet Historian on YouTube. 39:14.640 --> 39:32.700 Really funny guy, really great story, and it's just incredible to think of how many data scientists and developers and people behind this project put time in, put effort, put work, and it was shut down by a bunch of kids on 4chan. 39:33.800 --> 39:40.180 The other one is Jacobian Map Saliency Attack, and this kind of goes into the pixel manipulation. 39:40.460 --> 39:54.240 So, what you do is you grab a couple pixels, and you modify them, or you tweak them a little bit, or you tweak the data a little bit, doesn't necessarily need to be image data, and you pass that on as a certain class. 39:54.880 --> 40:05.220 And so, as the model is learning from this data, learning from this class, it starts identifying these tweaks, and then you can implement those tweaks in an adversarial example. 40:05.360 --> 40:18.120 So, in this case, we have a 70 kilometer per hour sign, we add in these little tweaks, and it latches onto those tweaks and says, okay, you're a 30 kilometer per hour sign. 40:18.380 --> 40:23.260 And so, imagine the kind of chaos that that can create. 40:23.260 --> 40:29.780 Again, these tweaks are difficult to implement in real world examples. 40:29.780 --> 40:39.140 This is also difficult to pull off simply because it relies on an online learning system, something that learns from the data that you've already presented it. 40:39.200 --> 40:45.600 But these are things that researchers have been able to identify and actually use to attack machine learning systems. 40:47.440 --> 40:50.400 So, how do we defend against poisoning attacks? 40:50.780 --> 40:58.640 Basically, it comes down to, you can still use a lot of the data, but what you want to do is add your own little smoothing filter. 40:58.640 --> 41:03.100 So, in this case, have longer periods of time between retraining. 41:03.340 --> 41:14.960 So, what that will do is kind of narrow the window, not narrow the window, it will expand the window that an attacker would need to generate adversarial examples for. 41:14.960 --> 41:19.690 And then they would also have to compete for votes against legitimate traffic. 41:20.240 --> 41:32.480 So, it's kind of like, if you look at Bitcoin, you need 51% of the vote in order to kind of take over cryptocurrency networks. 41:32.840 --> 41:36.060 So, this is kind of their own example of doing this. 41:36.060 --> 41:38.880 You can also analyze longer periods of data. 41:38.880 --> 41:41.460 Again, it broadens the window that you're looking at. 41:41.460 --> 41:48.940 And so, the adversaries, the attackers are going to have to generate more traffic, more examples in a longer period of time, which is difficult to do. 41:49.340 --> 41:53.640 And then you just want to generally minimize the impact of adversarial training examples. 41:53.640 --> 42:06.240 So, whether you have a human in loop that's manually looking at some of these examples, or maybe you're taking a small sample of the stuff that you're seeing and only allowing that to affect the model very... 42:08.420 --> 42:09.460 a little bit. 42:10.980 --> 42:13.360 So, it's just kind of a general rule of thumb. 42:13.360 --> 42:21.900 It's just minimize the impact, come to the understanding that people will try and poison the well, account for that. 42:23.460 --> 42:25.320 And then, of course, we have data leakage. 42:25.320 --> 42:36.780 So, data leakage, this is usually when, if we're trying to steal data, not steal the model, but steal data itself, this is usually when the models are trained to be a little too good. 42:37.920 --> 42:43.440 So, this image down below, you see aerial photographs, generated map, and then an aerial reconstruction. 42:43.500 --> 42:56.900 And so, articles at the time, this is a project by Google, articles at the time were kind of grandstanding and saying, oh my god, the Google AI found a way to cheat and generate aerial photographs. 42:57.460 --> 42:59.840 It was able to embed its own data. 43:00.340 --> 43:04.780 From the machine learning perspective or data science perspective, this is actually called overfitting. 43:04.780 --> 43:06.040 It's a very common problem. 43:06.040 --> 43:07.280 It's nothing spectacular. 43:07.280 --> 43:10.700 It's just, you didn't train your model right. 43:11.240 --> 43:18.920 And so, especially with overfit models, you have the training data in the model itself, or the model is able to generate that training data. 43:18.920 --> 43:34.160 And so, as I was saying earlier, with things like credit card information, PCI protected information, personally identifiable information, public health, or not public, sorry, personal health information, or HIPAA protected data, things of that nature, 43:34.160 --> 43:40.800 it can be very damaging for a machine learning model to leak real training data. 43:40.900 --> 43:54.940 And so, it's important to kind of evaluate and make sure that the models are not overfit, are not underfit, just hidden just right, or that the data is being preprocessed in such a way that it's a one directional. 43:55.560 --> 43:59.660 So, in some cases, what they'll do is generate a hash of certain features. 43:59.660 --> 44:04.780 For example, instead of putting in a credit card number, they generate a hash of the credit card number. 44:04.980 --> 44:08.580 And so, it's not reversible in any meaningful way. 44:10.460 --> 44:13.640 And then, of course, model theft, model theft by competitors. 44:13.900 --> 44:26.120 There's a technique called federated learning, which is you encrypt the machine learning model itself so that the model has to be used in that encrypted form. 44:26.120 --> 44:28.500 You can't steal the model, you can't decrypt it. 44:28.500 --> 44:34.100 It's a one way kind of thing, or a one way street in that scenario. 44:34.340 --> 44:39.340 And so, this will allow like IoT devices to be deployed with machine learning models. 44:39.340 --> 44:46.820 And it will prevent competitors or attackers, malicious actors from reversing it, stealing that model and building their own product off of it. 44:49.420 --> 44:53.560 And then, one last thing that I kind of want to cover is adversarial stickers. 44:53.560 --> 44:55.600 So, you've probably seen adversarial stickers. 44:55.600 --> 44:58.720 They look like melted crayons, in my opinion. 45:00.980 --> 45:04.100 But adversarial stickers, they have a bit of a problem. 45:04.100 --> 45:08.240 This is considered a brittle attack, and it's not very repeatable. 45:08.240 --> 45:14.700 And the reason why is because it doesn't have that attack transfer ability that I brought up earlier. 45:15.500 --> 45:34.100 So, in this case, kind of off to the bottom right, the stop sign, this is one of the first articles I was talking about adversarial stickers, and they were trying to modify a stop sign in such a way that the machine learning model didn't recognize it as a stop sign anymore, 45:34.100 --> 45:35.440 recognized it as something else. 45:35.440 --> 45:38.580 And so, this could be really damaging for self-driving vehicles. 45:38.880 --> 45:44.360 One of the problems is that this paper ran into some repeatability problems. 45:44.880 --> 45:57.600 And the reason why is because this attack is so brittle, they were able to convince their model, but they were not able to convince, or people who were trying to replicate this were not able to convince other models. 45:58.340 --> 46:00.500 So, it's not very useful. 46:00.500 --> 46:12.840 You can't just slap an adversarial sticker onto a sign and hope that you're driving a self-driving vehicle off into a wall or something of that nature. 46:12.900 --> 46:16.440 And then off here to the left is kind of the same idea with facial recognition. 46:16.440 --> 46:22.060 So, this researcher here said, okay, I'm going to set some constraints. 46:22.060 --> 46:29.600 I only want the pixels or the image around my eyes in the shape of glasses to be modified so that I look like this actress. 46:29.980 --> 46:31.720 Honestly, I don't remember who this is. 46:31.720 --> 46:32.700 I apologize. 46:33.180 --> 46:35.340 But I'm a man. 46:35.340 --> 46:39.440 I want to look like this actress per a facial recognition model. 46:39.440 --> 46:41.560 And so, he was successfully able to do that. 46:41.760 --> 46:49.380 The problem is that these glasses are designed for his face and the rest of the features of his face. 46:49.420 --> 47:03.400 And so, if I or many of you put on these glasses, it'll either look like somebody else or it won't recognize it as an adversarial example and it'll just recognize us as people, the same people, just wearing stupid glasses. 47:04.060 --> 47:07.620 So, attack transferability, adversarial stickers don't really have it. 47:07.620 --> 47:10.000 It's not a really good attack. 47:10.000 --> 47:11.140 It's very brittle. 47:11.140 --> 47:12.740 And I just kind of wanted to bring that up. 47:12.740 --> 47:19.920 For more information on this, I highly recommend DEF CON 2018 AI Village. 47:19.920 --> 47:25.940 Sven has a really good 30-minute talk on adversarial stickers and kind of the challenge with them. 47:25.940 --> 47:37.600 And then, for the facial recognition specifically, I know that Rich Horang in AI Village 2019 has a really good talk talking about facial recognition and trying to bypass that. 47:38.940 --> 47:44.580 So , to recap, how has AI empowered attackers? 47:44.620 --> 47:48.220 Well, it allows us to operate at a speed and scale like never before. 47:48.220 --> 47:50.800 We can operate at machine speed. 47:51.260 --> 47:57.340 And then, what we are seeing is the very beginning of what AI can bring to the offensive security space. 47:57.480 --> 48:00.380 So, we've already seen some adversarial attacks in the wild. 48:00.380 --> 48:04.940 I anticipate us seeing more as more researchers are coming out. 48:05.120 --> 48:09.520 AI and machine learning is kind of the hot new buzzword that everybody's latching onto. 48:09.520 --> 48:18.100 So, CEOs and decision makers are grabbing the AI-powered or machine learning-powered detection systems, defense systems. 48:18.460 --> 48:30.420 And it's kind of become a bit of a buzzword where you say that in a lot of common companies, especially in places like DEF CON, and they tell you to go talk to the Bitcoin folks and blockchain and all that fun stuff. 48:30.940 --> 48:35.100 There's no silver bullet, but we are seeing the very beginning of what is possible. 48:35.100 --> 48:49.520 And I anticipate, and I'm really hoping in the next several years, that we see a much larger impact of this AI versus AI kind of warfare that should be coming about. 48:50.500 --> 48:52.520 But anyways, I appreciate it. 48:52.520 --> 48:53.560 Thank you for your time. 48:53.560 --> 48:55.320 Thanks for sticking around this long. 48:55.440 --> 48:57.840 You can go ahead and email me if you have any questions. 48:57.840 --> 48:59.380 I'm also going to be in the chat. 48:59.600 --> 49:08.180 And then, for the slides, I have that are much more detailed are on Slideshare, and you can find them at slideshare.net slash GTKalandai. 49:08.220 --> 49:10.720 So that is my presentation. 49:10.720 --> 49:14.700 I hope you learned something, and thank you for sticking around. 49:14.880 --> 49:16.100 Bye!