1
00:00:00,720 --> 00:00:02,020
Oh, okay.

2
00:00:02,020 --> 00:00:04,080
Hi, welcome to our next talk.

3
00:00:04,080 --> 00:00:09,960
This one's by Gavin Kondyk, also known as GTKondyk on Twitter.

4
00:00:10,000 --> 00:00:19,160
He's a pre-circuited researcher, consultant, and has been super involved in the AI village community for a long time.

5
00:00:19,320 --> 00:00:24,060
He's an awesome guy, so I'll let him take it away.

6
00:00:27,770 --> 00:00:34,470
Hello, and welcome to Hacking with Skynet, How AI is Empowering Adversaries.

7
00:00:34,750 --> 00:00:36,010
My name is Gavin.

8
00:00:36,010 --> 00:00:38,430
I go by the pseudonym GTKondyk.

9
00:00:38,430 --> 00:00:41,970
I am a security researcher and senior security consultant.

10
00:00:41,970 --> 00:00:46,330
I'm very passionate about network security, both the attack and defense side.

11
00:00:46,330 --> 00:00:57,270
And through that passion, I run a project called NetSecExplained, which is a blog and YouTube channel where I explain intermediate and advanced level concepts in an easy-to-understand way.

12
00:00:58,970 --> 00:01:02,870
So, machine learning, let's get everybody on the same page.

13
00:01:03,210 --> 00:01:04,590
What is machine learning?

14
00:01:04,590 --> 00:01:09,070
Well, first, we have this big umbrella term of AI.

15
00:01:09,190 --> 00:01:11,470
And parts of AI are machine learning.

16
00:01:11,470 --> 00:01:13,290
Parts of AI are unrelated.

17
00:01:13,290 --> 00:01:16,350
So, we have rule-based artificial intelligence.

18
00:01:16,350 --> 00:01:18,930
We have classical image recognition.

19
00:01:19,530 --> 00:01:23,770
But inside of AI, we have this small subset called machine learning.

20
00:01:23,770 --> 00:01:33,530
And machine learning specifically allows us to take statistical analysis and draw inferences from data and perform data discovery techniques.

21
00:01:33,590 --> 00:01:39,970
And then out of machine learning, we also have deep learning, which is specifically deep neural networks.

22
00:01:39,970 --> 00:01:51,210
And this can be convolutional neural networks, which are used primarily for image recognition, or recurrent neural networks, which are used for cyclical pattern recognition, such as a heartbeat monitor.

23
00:01:53,610 --> 00:02:00,050
So, the burning question is, how is AI empowering adversaries?

24
00:02:00,050 --> 00:02:03,810
Or how can it empower adversaries moving forward?

25
00:02:04,170 --> 00:02:14,450
So, I see the age of AI today, or the age of machine learning today, we're at the very beginning of what it can do for the security industry on both the attack and the defense side.

26
00:02:14,530 --> 00:02:23,770
Now, if you remember back in the 80s and 90s, antivirus, intrusion detection, vulnerability scanners, these were all simple scripts that were written.

27
00:02:23,770 --> 00:02:28,510
And then over time, they grew into what are now multi-billion dollar industries.

28
00:02:28,710 --> 00:02:34,630
And so, I think that we're at the very beginning of what AI and machine learning can do for the security community.

29
00:02:36,410 --> 00:02:41,230
So, the way that I'm going to bring about this talk is really in two major parts.

30
00:02:41,230 --> 00:02:45,030
So, the first is to talk about offensive AI tools.

31
00:02:45,090 --> 00:02:46,510
Why do we want these?

32
00:02:46,510 --> 00:02:47,790
What's out there already?

33
00:02:47,790 --> 00:02:49,450
What are we seeing in the wild?

34
00:02:49,450 --> 00:02:51,050
How do we build them?

35
00:02:51,050 --> 00:02:56,430
And then adversarial machine learning, which is attacking the machine learning algorithms themselves.

36
00:02:56,930 --> 00:03:01,490
So, let's start with offensive AI tools.

37
00:03:04,370 --> 00:03:08,270
So, the first question is, why would we even want these?

38
00:03:08,410 --> 00:03:18,470
Well, offensive AI tools and AI tools in general will allow us to operate at a much faster speed and a much larger scale than we're currently able to do.

39
00:03:18,570 --> 00:03:21,950
And this is going to allow us to automate a lot of manual tasks.

40
00:03:22,050 --> 00:03:25,210
Now, we don't need to think about it as automating everything.

41
00:03:25,210 --> 00:03:41,210
But if there's something that requires or manages or deals with a lot of data, and these are things that we can apply a certain amount of fuzzy logic to, then this is a perfect candidate for automating with some form of AI or machine learning.

42
00:03:41,370 --> 00:03:52,330
And then, of course, this allows us to dynamically and intelligently explore our target attack surface and uncover hidden blind spots that come about some of the biases that we intrinsically have.

43
00:03:52,610 --> 00:04:05,570
And so, being able to statistically analyze some of this data or perform some machine learning operations, this will allow us to kind of uncover some of those blind spots and look for things that we don't already know.

44
00:04:07,030 --> 00:04:09,710
And then, how realistic is this?

45
00:04:09,710 --> 00:04:17,930
Well, back in 2016, DARPA kind of took on a large undertaking in their CyberGram challenge.

46
00:04:18,050 --> 00:04:29,270
In here, what they did was they required participants to develop some form of artificial intelligence that will perform a series of attack chain tasks.

47
00:04:29,270 --> 00:04:36,110
So, in this case, it was going to be vulnerability discovery, exploitation, patching, and then data theft.

48
00:04:36,110 --> 00:04:39,450
So, in the form of data theft, they have capture the flag.

49
00:04:39,790 --> 00:04:47,710
And then, as part of their major grand challenge, they had an attack, defend, capture the flag event that was air gapped.

50
00:04:47,710 --> 00:04:56,230
And you can see that in the picture kind of off to the side, where there's no human interaction whatsoever for the entire event.

51
00:04:56,230 --> 00:04:58,230
And everything had to be automated.

52
00:04:58,790 --> 00:05:09,010
So, this was a big reveal on a world stage that let us see what the power of artificial intelligence is, especially when it comes to a number of these tasks.

53
00:05:10,270 --> 00:05:14,150
And then, where are we seeing applications of AI already?

54
00:05:14,150 --> 00:05:19,010
So, the current proof of concepts we can see for social engineering.

55
00:05:19,170 --> 00:05:24,370
We can see in defense detection and evasion.

56
00:05:24,390 --> 00:05:38,690
There are some researchers that are identifying intrusion detection systems and sandbox detection systems so that attacks can be stealthier and break out of various systems and bypass existing defense measures.

57
00:05:38,930 --> 00:05:40,450
Evaluating data leaks.

58
00:05:40,450 --> 00:05:50,310
So, large data sets such as password dumps, subdomains, things of that nature, websites, applications, configuration files.

59
00:05:50,550 --> 00:05:52,430
Automated network exploitation.

60
00:05:52,430 --> 00:06:02,730
So, this will be going from standard user to domain admin, automating that process, as well as normal internal and external network pen tests.

61
00:06:03,290 --> 00:06:04,680
And then, software exploitation.

62
00:06:05,630 --> 00:06:28,430
And so, we'll see this with AFL, which is a software fuzzer that will utilize its own custom compiler, put in certain key points in the software itself, and then it allows the system to fuzz and track where through the application user input is being provided and how that can reach potentially vulnerable functions.

63
00:06:29,810 --> 00:06:31,790
So, let's start with social engineering.

64
00:06:31,790 --> 00:06:45,390
Social engineering, there's a couple things that are in the works right now that will allow us to improve our social engineering capabilities and improve attackers' abilities to exploit corporations and systems.

65
00:06:45,530 --> 00:06:48,270
So, in this case, we have an NVIDIA project called StyleGAN.

66
00:06:48,330 --> 00:06:52,070
StyleGAN, what it does is generate human faces.

67
00:06:52,710 --> 00:06:57,910
So, all four of the images up at the top you see are actually generated by StyleGAN.

68
00:06:58,270 --> 00:07:05,290
And then, just to kind of show you that there's no tricks up my sleeve, on the bottom row, we have StyleGAN generated cats.

69
00:07:05,290 --> 00:07:19,410
And you can see here in the bottom right corner, this abomination that kind of looks like a cat, but is really messed up and difficult to see, difficult to really recognize for a human.

70
00:07:19,410 --> 00:07:24,190
And this is kind of the imperfect model generating a cat-like image.

71
00:07:24,190 --> 00:07:34,350
But the level of detail that these can go into, if you look at the cat over here on the left, just kind of sitting on the carpet, you can see the individual hair, the whiskers, the texture of the carpet underneath.

72
00:07:34,950 --> 00:07:43,610
And so, these are really difficult to tell whether or not they are legitimate faces or cats, or if these are machine learning generated.

73
00:07:43,630 --> 00:07:52,250
One of the telltale signs, if you look here in the top right corner, this gentleman, it seems like there's a picket fence behind him, but it's a little warped in the back.

74
00:07:52,250 --> 00:07:56,450
You also notice that all of the images have a blurred background.

75
00:07:56,670 --> 00:08:01,810
And if you look really carefully, all of the eyes point directly at you.

76
00:08:01,810 --> 00:08:06,890
So, it doesn't matter what direction they're faced, the eyes are pointing straight.

77
00:08:07,610 --> 00:08:15,830
So, there's already some technologies and some research being performed to kind of identify what is a legitimate human face and what is a generated human face.

78
00:08:16,530 --> 00:08:18,370
Now, why is this important?

79
00:08:18,370 --> 00:08:20,290
How can we use this for social engineering?

80
00:08:20,670 --> 00:08:37,230
Well, for several types of social engineering campaigns, what's pretty common to do is go to the 10th page of a Google image search, grab somebody's stock photo or profile photo, and then you can paste that up as your own LinkedIn profile photo and say,

81
00:08:37,230 --> 00:08:39,490
okay, I am now Becky.

82
00:08:39,490 --> 00:08:41,410
I'm not a 32-year-old male.

83
00:08:41,410 --> 00:08:42,450
My name is Becky.

84
00:08:42,590 --> 00:08:44,270
I'm 27.

85
00:08:44,270 --> 00:08:50,790
I'm starting my career and I want to ask somebody, hey, as a fellow woman, how did you get started?

86
00:08:50,790 --> 00:09:01,370
And then I can use that to build rapport, have my pretext, and perform social engineering tasks to try and exploit employees, maybe in the HR department, sales, wherever.

87
00:09:02,010 --> 00:09:21,990
And so, instead of grabbing that image and throwing it up as a pretext where an analyst can go in, grab that same image, do a reverse Google image search, see that it's a stock photo, call me out, realize that I'm fake, what I can do is use something like StyleGAN to generate my own face.

88
00:09:21,990 --> 00:09:25,650
So, why steal a person when I can just make a whole new person?

89
00:09:26,250 --> 00:09:30,810
But there needs to be a little bit more legitimacy to my pretext.

90
00:09:31,050 --> 00:09:34,170
So, this is where GPT-2 and Grover come into play.

91
00:09:34,170 --> 00:09:36,590
GPT-2 and Grover, they operate very similarly.

92
00:09:36,590 --> 00:09:41,610
These are automated generation of long-form content.

93
00:09:42,050 --> 00:09:54,590
So, these can be used to generate emails, blog posts, articles, anything of that nature, with very minimal effort from the attacker or the adversary in that respect.

94
00:09:55,210 --> 00:10:09,310
And so, unfortunately, the website that was used to generate these has been taken offline, but the GPT-2 model is still something that you can download, put on your own systems, and generate long-form content.

95
00:10:09,310 --> 00:10:28,450
So, if I was developing a pretext and I can use this to write phishing emails that could get past a spam filter, or that look distinctly unique, so that I'm not sending the same email to multiple people within an environment, or I can use this to write blog posts and long-form content,

96
00:10:28,450 --> 00:10:34,930
and because any sort of Google image search or... what is that called?

97
00:10:36,010 --> 00:10:44,210
Plagiarism detection, it will see that I am the only source of truth, therefore I must be a real person, therefore my website must be legitimate.

98
00:10:44,950 --> 00:10:47,790
And so, this will kind of add to the pretext.

99
00:10:47,790 --> 00:10:57,930
And then on top of that, long-form content Google really enjoys, and so that can give us some Google ranking to add a little bit more credibility to our initial pretext.

100
00:10:59,950 --> 00:11:09,110
So, this is one of the scarier ones that I see as being at the very beginning of the research, and I'm curious to see where this will be in the next several years.

101
00:11:09,190 --> 00:11:11,630
We have Lyrebird and Tacotron.

102
00:11:11,630 --> 00:11:23,610
Lyrebird and Tacotron, these are text-to-speech that are based on human voices and use those human voices to generate conversation bits.

103
00:11:23,610 --> 00:11:30,790
And so, we can use this, I foresee, in the phishing campaigns where we're actually calling people or leaving voicemails.

104
00:11:30,790 --> 00:11:39,610
So, it's a lot less automated sounding, a little less robocall, and then you just add a little bit of compression and it sounds like you just have a really bad connection.

105
00:11:40,290 --> 00:11:42,110
So, I have a couple audio clips.

106
00:11:42,110 --> 00:11:43,110
I want you to hear them.

107
00:11:43,110 --> 00:11:50,570
So, these first two, I have the original voice and then the second one is the synthetic voice based off that original voice.

108
00:11:50,590 --> 00:11:53,170
So, let's go ahead and take a listen.

109
00:11:53,890 --> 00:11:57,150
In another moment, down went Alice after it.

110
00:11:57,150 --> 00:12:01,330
Never once did she consider how in the world she was to get out again.

111
00:12:03,310 --> 00:12:04,090
Awesome.

112
00:12:04,090 --> 00:12:07,350
And then the synthetic voice built off of that sample.

113
00:12:07,350 --> 00:12:12,150
My voice might be generated by a computer, but I think it sounds pretty human.

114
00:12:12,230 --> 00:12:15,610
I don't know exactly how they made it, but I'm really impressed.

115
00:12:16,170 --> 00:12:33,710
So, what I'm imagining is that as these models get better, what we can do is take interviews of CEOs and VPs of large corporations, grab their audio, isolate it, and then put it into a model like this so that we can have our own version of synthetic text to speech using their voice.

116
00:12:33,750 --> 00:12:44,390
So, we can call throughout the company and impersonate these individuals in order to cause some sort of social engineering action.

117
00:12:44,990 --> 00:12:46,810
Now, this last one is really interesting.

118
00:12:47,050 --> 00:12:48,030
This is Tacotron.

119
00:12:48,030 --> 00:12:49,490
This is a Google project.

120
00:12:49,630 --> 00:12:55,630
And I have the text over here on the right, but what I want you to do while you're listening to this is pay attention to the breaths.

121
00:12:55,630 --> 00:13:04,170
As humans, we tend to have filler words like um and uh, and then in between sentences and commas, we usually have a bit of a pause or a breath.

122
00:13:04,170 --> 00:13:06,630
And so, the Tacotron model actually picked that up.

123
00:13:06,630 --> 00:13:08,150
This is fully synthetic voice.

124
00:13:08,150 --> 00:13:09,450
Just take a listen.

125
00:13:10,150 --> 00:13:13,430
Only the photograph really showed how much time had passed.

126
00:13:13,430 --> 00:13:19,470
Ten years ago, there had been lots of pictures of what looked like a large pink beach ball wearing different colored bonnets.

127
00:13:19,470 --> 00:13:32,610
But Dudley Dursley was no longer a baby, and now the photograph showed a large blond boy riding his first bicycle on a carousel at the fair playing a computer game with his father being hugged and kissed by his mother by his mother.

128
00:13:34,930 --> 00:13:36,750
That's pretty scary.

129
00:13:38,170 --> 00:13:40,510
So, these projects are still underway.

130
00:13:40,530 --> 00:13:49,290
Liarbird is actually advertising itself as a transcription or reverse transcription service, which is just text-to-speech.

131
00:13:49,490 --> 00:14:02,270
In their original pay model that they had pushed out, currently they're in closed beta, but in their original pay model that they pushed out, you can transcribe three hours of audio pretty cheaply.

132
00:14:02,690 --> 00:14:06,510
And then if you're so inclined to pay, you can transcribe obviously more.

133
00:14:06,510 --> 00:14:21,690
But if a malicious user or somebody who's maliciously inclined, they can just grab those voice samples, throw them into Liarbird or Tacotron, or just use the voice samples that are already in there and generate their own adversarial examples.

134
00:14:24,720 --> 00:14:28,080
So, next we get into sandbox detection.

135
00:14:28,080 --> 00:14:32,900
And so, this is the defense detection and evasion part of the system.

136
00:14:32,900 --> 00:14:44,800
So, we have these researchers over at Silent Brick Security, and what they did was decide that with many different types of sandbox, what you can do is take the process list as data itself.

137
00:14:45,600 --> 00:14:52,740
And so, based on the process list, the number of processes, the number of users on a system, you can generate and decide whether or not you're in a sandbox.

138
00:14:52,740 --> 00:15:00,280
And then you can take it a step further and decide what kind of sandbox you're in, whether you're in FireEye or Cuckoo sandbox.

139
00:15:01,440 --> 00:15:15,440
So, what the researchers did was kind of classify, they took a number of basic Windows systems and they took a number of sandboxes and just kind of identified how many processes there are, how many users there are.

140
00:15:15,440 --> 00:15:17,840
Let's take a look at what malware can identify.

141
00:15:18,140 --> 00:15:22,720
And so, when we have a low process count, you see over here in column A, we have 33.

142
00:15:22,720 --> 00:15:24,420
In column F, we have 34.

143
00:15:24,720 --> 00:15:30,440
And then we have a high user count, such as, again, A and F, we have 4 and 4.

144
00:15:30,660 --> 00:15:34,720
And the process count per user, 8.25, 8.5.

145
00:15:34,940 --> 00:15:39,080
It gives us a very high probability that it is a sandbox.

146
00:15:39,560 --> 00:15:42,320
And so, we can see that with our host total score.

147
00:15:42,440 --> 00:15:45,880
And then based off that total score, we can set a threshold.

148
00:15:45,880 --> 00:15:51,320
In this case, it looks like they set the host average score as their threshold.

149
00:15:51,320 --> 00:15:54,460
So, anything below the average score is considered a sandbox.

150
00:15:54,460 --> 00:15:58,020
Anything above that is considered a legitimate system.

151
00:15:58,060 --> 00:16:08,460
And so, they can have malware that completely shuts down or erases itself if it identifies that it's in a sandbox, or it can use targeted evasion techniques per sandbox.

152
00:16:08,460 --> 00:16:11,980
So, if I'm in FireEye, these are the things that I'll do to get past it.

153
00:16:11,980 --> 00:16:16,660
If I'm in Cuckoo, these are the things that I'll do to get past it.

154
00:16:16,880 --> 00:16:20,180
So, it's a really interesting article, highly recommend taking a look at that.

155
00:16:22,540 --> 00:16:28,340
And then for automated network exploitation, we have systems like DeepExploit and Goethon.

156
00:16:28,340 --> 00:16:31,140
Goethon and DeepExploit, they both operate very similarly.

157
00:16:31,580 --> 00:16:45,360
The way that these work is they will go out, they will perform an nmap scan based on the results of that nmap scan, if it sees common services that it knows are exploitable or things that it would like to kind of take to that next level.

158
00:16:45,380 --> 00:16:48,220
For example, port 80 or 443, it's web traffic.

159
00:16:48,420 --> 00:16:51,920
So, it's going to go ahead and run Scrappy to go and crawl that system.

160
00:16:51,920 --> 00:17:01,040
I'm sure they could add other tools such as Go Buster, which will go out and enumerate a lot of the files or directories or URLs that are on that system.

161
00:17:01,260 --> 00:17:02,680
Try and fingerprint the server.

162
00:17:02,680 --> 00:17:09,600
You want to see if it's running Nginx, Apache, PHP, Java, things of that nature.

163
00:17:10,000 --> 00:17:15,340
And then it performs additional content exploration and its own Google hacking, which is kind of cool.

164
00:17:15,580 --> 00:17:22,100
So, that will go out, do a little bit of research, see if there's any sort of exploitable scripts, things of that nature.

165
00:17:22,100 --> 00:17:35,480
And it takes all of this data and it processes it using some statistical techniques and then it uses Metasploit as a targeted exploitation engine and it goes and tries to exploit those things.

166
00:17:35,480 --> 00:17:45,900
So, if it sees certain versions of JBoss or Jenkins, it will go and try and target that with maybe password spray or any sort of Metasploit module.

167
00:17:46,080 --> 00:17:47,660
Tomcat, same thing.

168
00:17:48,160 --> 00:17:58,040
So, this is kind of cool as a proof of concept of automating the entire process of network discovery all the way up to exploitation and in some cases post-exploitation.

169
00:17:59,980 --> 00:18:03,980
One of my favorite while I was researching this is Deathstar.

170
00:18:03,980 --> 00:18:07,440
And this is my favorite just kind of in the way that it is assembled.

171
00:18:07,940 --> 00:18:10,840
So, Deathstar, it's built on PowerShell Empire.

172
00:18:11,260 --> 00:18:20,560
And the way that it works is it starts, it kind of covers its baseline and it is designed to go from standard user all the way up to domain administrator.

173
00:18:20,560 --> 00:18:25,400
And so, it charts and maps attack paths all the way to that location.

174
00:18:25,400 --> 00:18:28,720
And so, it starts at its baseline and it says, am I already a domain administrator?

175
00:18:28,720 --> 00:18:30,560
If so, cancel out.

176
00:18:30,560 --> 00:18:32,860
And we see that's the path off to the left.

177
00:18:33,140 --> 00:18:36,280
If I have domain administrator credentials, all right, cool.

178
00:18:36,280 --> 00:18:37,220
Log in.

179
00:18:37,520 --> 00:18:39,540
And that charts off the path to the right.

180
00:18:39,720 --> 00:18:44,240
But if it's neither of those, then it goes straight down the middle and it says, okay, let's spawn a new agent.

181
00:18:44,640 --> 00:18:50,840
Let's go ahead and identify where is the domain administrators and where is the domain controllers.

182
00:18:50,840 --> 00:18:52,900
And so, it runs over here to the right.

183
00:18:52,900 --> 00:18:57,980
We can see get net domain controller, get net group member, invoke user hunter.

184
00:18:58,000 --> 00:19:07,200
And so, this is going to enumerate the domain controllers, enumerate the domain admins, and enumerate the users that it can on the systems that it can.

185
00:19:07,700 --> 00:19:11,360
And you can see it goes through this entire tree kind of off here to the bottom left.

186
00:19:11,360 --> 00:19:17,120
It identifies and says, hey, is this a vulnerable, known vulnerable version of Windows, for example, Windows 7?

187
00:19:17,120 --> 00:19:25,740
If yes, let's run Mimikatz, try and do Mimikatz, secret stump, things of that nature, collect credentials, and then we can kind of start back over at the top.

188
00:19:26,400 --> 00:19:28,740
Let's go ahead and enumerate the running processes.

189
00:19:28,740 --> 00:19:34,660
Do we see any processes that are already being ran by other users that we don't have in our inventory?

190
00:19:34,660 --> 00:19:39,120
If so, let's add those users to our inventory and try and do some lateral movement.

191
00:19:39,580 --> 00:19:44,380
And so, in this case, it'll use psinject, and then it'll go ahead and spawn a new instance.

192
00:19:44,380 --> 00:19:50,540
And so, it goes through this iterative process all the way until it reaches domain admin credentials.

193
00:19:51,300 --> 00:20:04,420
And so, if you've ever used a system like Bloodhound, where you'll generate an attack graph and say, okay, this user is part of this group, which has these members, which this member is also part of domain admins.

194
00:20:04,420 --> 00:20:08,740
And so, this is the attack path that you want to go in order to become domain admin.

195
00:20:08,880 --> 00:20:16,940
And so, Deathstar kind of takes that idea a step further and actually tries to navigate that attack path in an automated fashion.

196
00:20:18,120 --> 00:20:25,700
So, that's all I have for the machine learning tools that already exist, some of the proof of concepts.

197
00:20:25,700 --> 00:20:31,420
There's a lot more information that I wanted to cover that I wouldn't be able to cover in such a short amount of time.

198
00:20:31,420 --> 00:20:42,060
So, at the end of this presentation, I have a link to my slide deck, which has a lot more detail and a lot more tools and a lot more techniques in it.

199
00:20:42,500 --> 00:20:51,960
But in the meantime, I kind of want to answer the question of how do I go about creating my own machine learning tools or AI tools?

200
00:20:51,960 --> 00:20:53,640
What should I look for?

201
00:20:53,640 --> 00:20:54,840
How do I automate that?

202
00:20:54,840 --> 00:20:57,360
What is that process?

203
00:20:57,360 --> 00:21:11,380
And so, honestly, my advice would be to identify somewhere that either utilizes a lot of data or identify somewhere that uses a... it's a common process, but it uses some sort of fuzzy logic, right?

204
00:21:11,380 --> 00:21:22,520
Something that's kind of hard to script, hard to write a signature for, but with AI and machine learning, you can use some of that fuzzy logic to generate some heuristics and it can perform based off of that.

205
00:21:23,160 --> 00:21:34,980
So, in the case of some of these, what we can do is say, all right, NMAP scans, everybody does NMAP scans, and then you have to manually look at them, and then based off of that, you have to follow up and go over here.

206
00:21:35,380 --> 00:21:50,660
Instead, like I showed with Goython, is you can just use the NMAP scan and then automate the process of, okay, let's enumerate and fingerprint these services, and then let's see if any of these services are already exploitable using Metasploit, or let's go with next steps.

207
00:21:50,660 --> 00:21:55,060
I identified a web service, let's run web analysis techniques.

208
00:21:55,140 --> 00:21:59,580
So, Go Buster, Crawler, any sort of things like that.

209
00:22:00,520 --> 00:22:06,120
So, that's kind of the thing that you want to keep an eye on if you want to develop your own machine learning tools.

210
00:22:06,120 --> 00:22:15,240
And there's more that are coming out all of the time, and it's really interesting and a really fast-paced area, and it's just ripe with research.

211
00:22:15,240 --> 00:22:17,280
So, go ahead and play around with some of that.

212
00:22:19,200 --> 00:22:26,640
So, the next bit that I want to get into is adversarial machine learning, and this is attacking the machine learning algorithms themselves.

213
00:22:27,420 --> 00:22:33,900
So, adversarial machine learning, there's really three ways to go about attacking a machine learning model.

214
00:22:34,160 --> 00:22:36,800
So, the first way we have is model evasion.

215
00:22:36,800 --> 00:22:46,500
This is to take what is clearly a malicious example and get the model to consider it to be benign or normal or standard to bypass that filter.

216
00:22:46,500 --> 00:22:56,780
So, this would be taking a spam message and getting past a spam filter, or this would be taking a piece of malware and getting it past antivirus or anti-malware solutions.

217
00:22:57,420 --> 00:23:04,360
Model poisoning is where we add our own data into the training data set.

218
00:23:04,560 --> 00:23:13,360
Now, machine learning models specifically are... they're done learning as soon as they're done being trained.

219
00:23:13,380 --> 00:23:22,120
To get around this, there are some models that are called online systems, and so what they'll do is they'll take data that has been classified by itself already.

220
00:23:22,120 --> 00:23:32,820
So, say I send it a piece of data, like a piece of malware, it looks at that, says, okay, bam, that's malware, now that goes into my training set, and I learn from that malware.

221
00:23:33,580 --> 00:23:49,240
So, data poisoning allows us to slowly inject certain pieces of data so that we push that threshold a little off to the side into what is more acceptable, and then we target where that threshold has been pushed.

222
00:23:49,680 --> 00:23:52,740
And we'll see an example of that in a bit.

223
00:23:53,320 --> 00:23:54,120
Data leakage.

224
00:23:54,120 --> 00:23:58,660
Data leakage is actually a combination of two smaller bits.

225
00:23:58,660 --> 00:24:02,520
So, one is stealing the model itself.

226
00:24:02,660 --> 00:24:10,320
So, for example, say you're an IoT engineering company and you develop an IoT system that utilizes some form of machine learning.

227
00:24:11,620 --> 00:24:25,840
What I can do is buy that device and then reverse engineer it, and now I have your model, and so I can use that model or train another model based off of that instead of having to go through the hundreds of thousands of dollars that you went through to develop that in the first place.

228
00:24:27,400 --> 00:24:30,660
And so, this is where the model itself is easy to be grabbed.

229
00:24:30,660 --> 00:24:39,440
There's also data leakage, which is I can identify what data you use to train the model.

230
00:24:39,760 --> 00:24:47,540
And so, this is really damaging, especially for healthcare industry or HIPAA-protected information.

231
00:24:47,800 --> 00:24:57,280
So, if I can identify any sort of clients or patients that you have, any sort of their PII, then that's really damaging.

232
00:24:59,760 --> 00:25:03,080
So, when it comes to model testing, there's really two approaches.

233
00:25:03,480 --> 00:25:15,840
You have a white box where you know what the model is, you know how the hyperparameters are set, you know everything that's involved in the building of this, and then you target that model, much like you would do in software where you already have all of the code involved.

234
00:25:16,380 --> 00:25:19,520
And then you have a black box, which is just the model itself.

235
00:25:19,520 --> 00:25:26,040
You aren't able to peek into the way that the model operates, but what you can do is treat it like an oracle.

236
00:25:26,100 --> 00:25:37,240
So, if you've ever done any sort of cryptography, you know, exercises, you have an oracle, you ask yes-no questions, or things of that nature, and then you can generate what is called a surrogate model.

237
00:25:37,240 --> 00:25:52,020
So, based off of the yes-no questions that I asked the black box model, I generated my own model, and then based off of that model, I can perform attacks and that those attacks should be able to impact the original black box itself.

238
00:25:52,660 --> 00:25:57,780
Now, the way that this operates is under a principle called attack transferability.

239
00:25:57,900 --> 00:26:12,580
This is a documented phenomenon, and the way that attack transferability works is, it doesn't matter what model or the underlying architecture that these models are designed with, an attack on one model can also be an attack on another model.

240
00:26:12,580 --> 00:26:31,520
So, if I have a deep learning model over here, and I find adversarial examples that I can use to attack this model, then here is a random forest, a logistic regression, SVM, or anything else, any other machine learning algorithm or architecture, that those same adversarial examples will be able to impact that.

241
00:26:33,620 --> 00:26:35,600
And then, of course, the attack surface.

242
00:26:35,600 --> 00:26:39,320
So, where are the machine learning models themselves vulnerable?

243
00:26:39,960 --> 00:26:47,300
So, machine learning, as it works, is you have a physical object, and then you take that physical object and you try to digitize it.

244
00:26:47,300 --> 00:26:54,880
So, say you have a picture of a panda or a car, you take a photograph of it, now it's digitized, and you can make information off of that.

245
00:26:54,880 --> 00:27:10,380
You take that digitized information, you run your pre-processing as you would per the normal machine learning process, and then you put that into the machine learning model as part of its input features, it presents some output, usually a probability or a number of some sort,

246
00:27:10,380 --> 00:27:13,340
and then a decision is made based off of that action.

247
00:27:13,440 --> 00:27:20,060
So, for example, if I am 67% confident that this is malware, then I will probably flag it as malware.

248
00:27:20,140 --> 00:27:24,500
If I'm only 42% confident, then I'll just let it continue to operate.

249
00:27:25,120 --> 00:27:26,820
So, where can we attack things?

250
00:27:26,820 --> 00:27:30,980
Well, one, we can actually impact the physical object itself.

251
00:27:31,840 --> 00:27:39,960
So, down below, I have the network construction detection system as kind of an example that follows this generic machine learning model path.

252
00:27:39,960 --> 00:27:41,320
So, we have the attack traffic.

253
00:27:41,320 --> 00:27:54,180
We can actually, as attackers, modify the attack traffic in such a way that as it goes through the pipeline of the machine learning pipeline, it will modify the behaviors at different points.

254
00:27:55,360 --> 00:28:03,000
So, I can slow down my attack traffic, I can use my attack traffic for multiple hosts so that it's a little bit harder to detect.

255
00:28:03,680 --> 00:28:07,100
And then that attack traffic is then digitized.

256
00:28:07,100 --> 00:28:12,100
The digitization process, we don't really have a lot of impact over as attackers.

257
00:28:12,100 --> 00:28:17,080
It's not realistic for us to be able to affect TCP dump.

258
00:28:19,220 --> 00:28:21,320
There's really famous examples.

259
00:28:21,520 --> 00:28:32,620
For example, you may have come across this if you ever looked at adversarial machine learning, a picture of a panda, and you modify some pixels, and now it thinks it's a gibbon or a bird or some sort.

260
00:28:33,100 --> 00:28:40,200
But if we have the ability to modify pixels, why don't we just give it a picture of a bird at that point?

261
00:28:40,380 --> 00:28:50,380
And then more to the point, if we can modify pictures, if you have something like a self-driving car, it has a camera, how do we interface with that camera in an adversarial perspective?

262
00:28:50,380 --> 00:28:54,860
We would already have to have access to the system, which at that point we can just shut it down or do whatever.

263
00:28:54,860 --> 00:28:56,200
It's physical access.

264
00:28:57,620 --> 00:29:08,200
So, we're not actually able to interface with the digitization of the object, but what we can do is change the way that the attack traffic is modified.

265
00:29:08,200 --> 00:29:13,120
We can change the input features, the input vectors, the machine learning model, especially with the model poisoning.

266
00:29:14,080 --> 00:29:16,920
We can change the way that the data is represented.

267
00:29:18,460 --> 00:29:28,520
So, in the case of the network intrusion detection system, we have the packet metadata, and then it spits out an attack probability, and then off of that it makes a decision.

268
00:29:28,840 --> 00:29:39,320
And in between the machine learning model and the decision-making process, you got to remember this is just software, and you can approach it like any other piece of software that you're pen testing or trying to hack.

269
00:29:39,320 --> 00:29:47,240
So, you have the machine learning model in it, and then all wrapped around it is a nice piece of software that probably has some of its own vulnerabilities involved.

270
00:29:49,420 --> 00:29:51,420
So, model invasion.

271
00:29:52,220 --> 00:29:54,840
Model evasion, not invasion, sorry.

272
00:29:55,680 --> 00:29:57,740
So, we're going to try and hide in the blind spots.

273
00:29:57,740 --> 00:30:00,060
Now, we have this massive theoretical space.

274
00:30:00,060 --> 00:30:08,100
This is all of the possible pieces of malware that could possibly exist, but we can't collect all of that, realistically speaking.

275
00:30:08,620 --> 00:30:15,020
Antivirus companies can't collect all of it, and so what they have is a subset, and this is considered the training space.

276
00:30:15,020 --> 00:30:18,560
So, this is what they are realistically able to collect.

277
00:30:18,860 --> 00:30:29,140
So, in their training space, they perform their training, and then off of a small piece of that that's already clean labeled data, they perform their testing to make sure that the model operates properly.

278
00:30:29,780 --> 00:30:34,780
Well, part of that testing space, kind of right there off the sliver, this is the adversarial space.

279
00:30:34,780 --> 00:30:35,820
This is where we live.

280
00:30:35,820 --> 00:30:42,020
This is where we are able to generate adversarial examples that target that model specifically.

281
00:30:42,440 --> 00:30:47,800
And so, this is kind of what we want to play around with and hide in the blind spots, so to speak.

282
00:30:48,300 --> 00:31:02,160
So, a couple really good examples is, in this case, multinomial Naive Bayesian spam filters, or just any of the early spam filters that used some rudimentary forms of machine learning, how they would operate is they would look at keywords.

283
00:31:02,160 --> 00:31:14,960
So, if you had something like, buy Viagra now, the probability that the word Viagra shows up in a spam email is much higher than the probability of the word Viagra showing up in a regular email.

284
00:31:15,080 --> 00:31:22,200
And so, based off of these keywords, it does its own statistical analysis and says, I am 99% sure this is spam.

285
00:31:23,180 --> 00:31:30,140
Well, the interesting thing about emails is that emails are usually represented as HTML pages or web pages.

286
00:31:30,800 --> 00:31:37,120
And so, as malicious actors, what we can do is actually add in a comment.

287
00:31:37,540 --> 00:31:41,400
So, in this case, we can add in the entire Wikipedia page article for a horse.

288
00:31:41,400 --> 00:31:45,200
And so, we have a bunch of bad words, but we have even more good words.

289
00:31:45,200 --> 00:31:53,500
What this lets us do is kind of skip past the spam filter by showing, hey, we have a lot more good words than bad words, so we must be good, right?

290
00:31:53,500 --> 00:31:55,080
It says, yeah, absolutely.

291
00:31:55,440 --> 00:31:58,540
I am 99% sure you are not spam.

292
00:31:59,400 --> 00:32:03,960
And the thing is that the way that this is presented to users is as an HTML web page.

293
00:32:03,960 --> 00:32:08,040
And so, the comment itself is not actually rendered for the users.

294
00:32:08,040 --> 00:32:12,860
So, the users still see the original spam email as though it was unmodified.

295
00:32:13,920 --> 00:32:18,760
And so, this is kind of a real world example on some of the spam filters.

296
00:32:19,420 --> 00:32:23,680
A really good example of this is by Skylight Cyber.

297
00:32:23,680 --> 00:32:33,440
The researchers over there were actually able to bypass the silence machine learning antivirus in this article, Silence, I Kill You.

298
00:32:33,440 --> 00:32:34,620
Really good article.

299
00:32:34,620 --> 00:32:36,620
Highly recommend you take a look at it.

300
00:32:36,620 --> 00:32:38,900
We can see some familiar malware samples here.

301
00:32:38,900 --> 00:32:54,320
So, we have CoinMiner, Emotet, Zeus, and we see the original scores that they were actually able to pull out of the silence antivirus, negative 826, negative 923, negative 997.

302
00:32:54,420 --> 00:32:57,920
So, these are really bad, really bad, flagged immediately.

303
00:32:58,980 --> 00:33:14,740
So, what the researchers over at Skylight Cyber did was they took the silence antivirus, reverse engineered it, and they were able to pull out the scores based on some of the syscalls and other key features that silence looked at for malware.

304
00:33:15,940 --> 00:33:21,620
One of the curious things is that they identified certain things that were whitelisted.

305
00:33:21,840 --> 00:33:30,540
Now, why would silence whitelist certain things, even though the standard machine learning process would actually identify it as potentially malware?

306
00:33:30,780 --> 00:33:35,860
Well, the reason why is because when you think about malware, think about how malware operates, right?

307
00:33:35,860 --> 00:33:43,800
So, maybe it asks for access to your webcam or your microphone, or it tries to resize or open up new windows.

308
00:33:44,100 --> 00:33:46,060
Maybe it makes calls out to the internet.

309
00:33:46,060 --> 00:33:48,960
Well, what's another thing that does that?

310
00:33:49,520 --> 00:33:53,020
Chrome, Firefox, Internet Explorer.

311
00:33:53,080 --> 00:33:55,640
They operate very similar to some modern malware.

312
00:33:55,920 --> 00:33:59,360
In this case, what they found was a video game.

313
00:33:59,820 --> 00:34:01,680
I'm not sure which video game it is.

314
00:34:01,680 --> 00:34:04,640
My money is on Fortnite or League of Legends.

315
00:34:05,240 --> 00:34:09,160
But the video games that they found were certain whitelisted keywords.

316
00:34:09,640 --> 00:34:17,500
So, what they did was took these clearly bad malware and they just added these keywords to the end of the malware.

317
00:34:17,540 --> 00:34:27,440
So, it didn't change the way that the malware operated, but it changed the way that silence looked at it and said, okay, you have a bunch of bad boy points, but you have even more good boy points.

318
00:34:27,440 --> 00:34:30,460
So, we're going to give you some good boy scores.

319
00:34:30,500 --> 00:34:33,020
So, in this case, CoinMiner had 884.

320
00:34:33,120 --> 00:34:41,960
My favorite is looking at Zeus, negative 997, one of the worst in this table, all the way up to positive 997, which is one of the best in this table.

321
00:34:42,680 --> 00:34:50,140
And so, it shows that model evasion techniques are very useful in a number of different ways.

322
00:34:52,100 --> 00:34:53,920
So, how do we defend against this?

323
00:34:53,920 --> 00:34:57,960
Well, the first and probably one of the better is adversarial training.

324
00:34:58,040 --> 00:35:02,220
Adversarial training kind of comes from the idea of chaos engineering.

325
00:35:02,220 --> 00:35:11,300
Netflix is a really big proponent of chaos engineering, where you already know something bad is going to happen, so you might as well account for those bad things to happen.

326
00:35:11,300 --> 00:35:14,200
So, in this case, you train it with adversarial examples.

327
00:35:14,200 --> 00:35:20,920
You generate your own adversarial examples and you train the model based off of that to kind of harden and make it a little bit more robust.

328
00:35:21,720 --> 00:35:25,020
Again, if it's not a robust model, it's not a good model.

329
00:35:25,940 --> 00:35:31,240
Chaos engineering as a whole , the way that it operates is you already know something is going to go bad.

330
00:35:31,240 --> 00:35:32,520
So, you have Netflix , for example.

331
00:35:32,520 --> 00:35:37,220
Netflix, they know that their systems, they need to have that uptime.

332
00:35:37,220 --> 00:35:39,680
And so, they know that it's possible one of their systems goes down.

333
00:35:39,680 --> 00:35:42,060
They want to make sure their redundancy operates properly.

334
00:35:42,320 --> 00:35:46,380
So, you purposefully go into production and you break things.

335
00:35:46,380 --> 00:35:54,200
And then you develop processes and policies around knowing things are going to break.

336
00:35:54,200 --> 00:35:59,000
So, in some cases, you'll have the CIO walk into a data center and just start unplugging servers.

337
00:36:00,660 --> 00:36:04,500
So, adversarial training is a really good way to kind of counteract this.

338
00:36:04,680 --> 00:36:08,940
Another slightly more technical way is called defensive distillation.

339
00:36:08,940 --> 00:36:14,640
Defensive distillation is a smoothing algorithm that kind of smooths the curve.

340
00:36:14,640 --> 00:36:18,720
So, instead of having, okay, malware, malware, malware, malware, nope, it's okay.

341
00:36:19,400 --> 00:36:21,620
You can't have sharp edges like that.

342
00:36:21,620 --> 00:36:23,040
So, what it does is it smooths it.

343
00:36:23,040 --> 00:36:27,920
So, instead of going straight down, it just kind of goes in a downward trend.

344
00:36:27,920 --> 00:36:36,940
And so, it's a little bit harder, a little bit more robust way of analyzing and dealing with the adversarial examples.

345
00:36:37,600 --> 00:36:41,400
And then the last example would be monotonic classification.

346
00:36:41,400 --> 00:36:44,020
Which means that the graph grows in one direction.

347
00:36:44,480 --> 00:36:53,840
So, we see here in the image off to the right, the non-monotonic example, we see that it's getting worse and worse and worse as far as a keyword .

348
00:36:53,840 --> 00:36:56,040
So, it goes worse and worse and worse across that threshold.

349
00:36:56,040 --> 00:36:59,780
Now, it's considered bad, it's spam, it's malware, it's whatever.

350
00:37:00,340 --> 00:37:02,700
We're ready to throw it out, but we're still analyzing it.

351
00:37:02,700 --> 00:37:06,680
And so, it's crossed that threshold, but wait, no, it's getting better and better and better.

352
00:37:06,680 --> 00:37:10,880
We just hit that pad of keywords that are really positive.

353
00:37:11,040 --> 00:37:14,120
And so, we went from good to bad, all the way back up to good.

354
00:37:15,260 --> 00:37:21,540
Monotonic classification means that we don't count the good words at all, we say, we measure your badness.

355
00:37:21,540 --> 00:37:23,280
How many bad words do you have?

356
00:37:23,560 --> 00:37:34,600
And so, in this case, on the bottom example, we have the monotonic classification, which says, okay, we have some bad words, we have some bad words, up, we all went all the way up to the top, and then we hit all of the padded good words.

357
00:37:34,860 --> 00:37:39,680
And it doesn't matter, you're still bad, Jesus is not going to save you, you're done.

358
00:37:40,280 --> 00:37:46,500
And so, monotonic classification is a really powerful technique to deal with these model evasion techniques.

359
00:37:48,480 --> 00:37:50,880
Next, we have model poisoning.

360
00:37:50,960 --> 00:37:57,540
Model poisoning almost exclusively works on online learning systems.

361
00:37:57,540 --> 00:38:02,860
So, as adversaries, we need to find a way to get adversarial examples into the training space.

362
00:38:03,180 --> 00:38:12,060
And so, this is where systems will grab data that it's already classified a certain way, and add it into the training space.

363
00:38:12,060 --> 00:38:21,960
And so, we're just trying to push that barrier past a certain threshold where we can actually operate safely as adversaries without it detecting us as such.

364
00:38:22,840 --> 00:38:24,720
So, some real world examples.

365
00:38:25,180 --> 00:38:32,460
Tay AI, if you're not already familiar, I highly recommend checking it out.

366
00:38:32,460 --> 00:38:41,280
Tay AI was a Microsoft Twitter chat bot, and it was designed to learn from real conversations by real people.

367
00:38:41,960 --> 00:38:49,840
Well, 4chan, of course, gets a hold of this, and they decide, how much does it learn?

368
00:38:49,840 --> 00:38:51,700
How much can we teach it?

369
00:38:51,880 --> 00:39:05,240
And about 16 hours after coming online, it was taken offline permanently, because within that time, 4chan was able to get it to say, Hitler did nothing wrong, and racial slurs.

370
00:39:05,960 --> 00:39:14,600
If you want more information on Tay AI, and I highly recommend checking this out, it's called The People's Chatbot by the Internet Historian on YouTube.

371
00:39:14,640 --> 00:39:32,700
Really funny guy, really great story, and it's just incredible to think of how many data scientists and developers and people behind this project put time in, put effort, put work, and it was shut down by a bunch of kids on 4chan.

372
00:39:33,800 --> 00:39:40,180
The other one is Jacobian Map Saliency Attack, and this kind of goes into the pixel manipulation.

373
00:39:40,460 --> 00:39:54,240
So, what you do is you grab a couple pixels, and you modify them, or you tweak them a little bit, or you tweak the data a little bit, doesn't necessarily need to be image data, and you pass that on as a certain class.

374
00:39:54,880 --> 00:40:05,220
And so, as the model is learning from this data, learning from this class, it starts identifying these tweaks, and then you can implement those tweaks in an adversarial example.

375
00:40:05,360 --> 00:40:18,120
So, in this case, we have a 70 kilometer per hour sign, we add in these little tweaks, and it latches onto those tweaks and says, okay, you're a 30 kilometer per hour sign.

376
00:40:18,380 --> 00:40:23,260
And so, imagine the kind of chaos that that can create.

377
00:40:23,260 --> 00:40:29,780
Again, these tweaks are difficult to implement in real world examples.

378
00:40:29,780 --> 00:40:39,140
This is also difficult to pull off simply because it relies on an online learning system, something that learns from the data that you've already presented it.

379
00:40:39,200 --> 00:40:45,600
But these are things that researchers have been able to identify and actually use to attack machine learning systems.

380
00:40:47,440 --> 00:40:50,400
So, how do we defend against poisoning attacks?

381
00:40:50,780 --> 00:40:58,640
Basically, it comes down to, you can still use a lot of the data, but what you want to do is add your own little smoothing filter.

382
00:40:58,640 --> 00:41:03,100
So, in this case, have longer periods of time between retraining.

383
00:41:03,340 --> 00:41:14,960
So, what that will do is kind of narrow the window, not narrow the window, it will expand the window that an attacker would need to generate adversarial examples for.

384
00:41:14,960 --> 00:41:19,690
And then they would also have to compete for votes against legitimate traffic.

385
00:41:20,240 --> 00:41:32,480
So, it's kind of like, if you look at Bitcoin, you need 51% of the vote in order to kind of take over cryptocurrency networks.

386
00:41:32,840 --> 00:41:36,060
So, this is kind of their own example of doing this.

387
00:41:36,060 --> 00:41:38,880
You can also analyze longer periods of data.

388
00:41:38,880 --> 00:41:41,460
Again, it broadens the window that you're looking at.

389
00:41:41,460 --> 00:41:48,940
And so, the adversaries, the attackers are going to have to generate more traffic, more examples in a longer period of time, which is difficult to do.

390
00:41:49,340 --> 00:41:53,640
And then you just want to generally minimize the impact of adversarial training examples.

391
00:41:53,640 --> 00:42:06,240
So, whether you have a human in loop that's manually looking at some of these examples, or maybe you're taking a small sample of the stuff that you're seeing and only allowing that to affect the model very...

392
00:42:08,420 --> 00:42:09,460
a little bit.

393
00:42:10,980 --> 00:42:13,360
So, it's just kind of a general rule of thumb.

394
00:42:13,360 --> 00:42:21,900
It's just minimize the impact, come to the understanding that people will try and poison the well, account for that.

395
00:42:23,460 --> 00:42:25,320
And then, of course, we have data leakage.

396
00:42:25,320 --> 00:42:36,780
So, data leakage, this is usually when, if we're trying to steal data, not steal the model, but steal data itself, this is usually when the models are trained to be a little too good.

397
00:42:37,920 --> 00:42:43,440
So, this image down below, you see aerial photographs, generated map, and then an aerial reconstruction.

398
00:42:43,500 --> 00:42:56,900
And so, articles at the time, this is a project by Google, articles at the time were kind of grandstanding and saying, oh my god, the Google AI found a way to cheat and generate aerial photographs.

399
00:42:57,460 --> 00:42:59,840
It was able to embed its own data.

400
00:43:00,340 --> 00:43:04,780
From the machine learning perspective or data science perspective, this is actually called overfitting.

401
00:43:04,780 --> 00:43:06,040
It's a very common problem.

402
00:43:06,040 --> 00:43:07,280
It's nothing spectacular.

403
00:43:07,280 --> 00:43:10,700
It's just, you didn't train your model right.

404
00:43:11,240 --> 00:43:18,920
And so, especially with overfit models, you have the training data in the model itself, or the model is able to generate that training data.

405
00:43:18,920 --> 00:43:34,160
And so, as I was saying earlier, with things like credit card information, PCI protected information, personally identifiable information, public health, or not public, sorry, personal health information, or HIPAA protected data, things of that nature,

406
00:43:34,160 --> 00:43:40,800
it can be very damaging for a machine learning model to leak real training data.

407
00:43:40,900 --> 00:43:54,940
And so, it's important to kind of evaluate and make sure that the models are not overfit, are not underfit, just hidden just right, or that the data is being preprocessed in such a way that it's a one directional.

408
00:43:55,560 --> 00:43:59,660
So, in some cases, what they'll do is generate a hash of certain features.

409
00:43:59,660 --> 00:44:04,780
For example, instead of putting in a credit card number, they generate a hash of the credit card number.

410
00:44:04,980 --> 00:44:08,580
And so, it's not reversible in any meaningful way.

411
00:44:10,460 --> 00:44:13,640
And then, of course, model theft, model theft by competitors.

412
00:44:13,900 --> 00:44:26,120
There's a technique called federated learning, which is you encrypt the machine learning model itself so that the model has to be used in that encrypted form.

413
00:44:26,120 --> 00:44:28,500
You can't steal the model, you can't decrypt it.

414
00:44:28,500 --> 00:44:34,100
It's a one way kind of thing, or a one way street in that scenario.

415
00:44:34,340 --> 00:44:39,340
And so, this will allow like IoT devices to be deployed with machine learning models.

416
00:44:39,340 --> 00:44:46,820
And it will prevent competitors or attackers, malicious actors from reversing it, stealing that model and building their own product off of it.

417
00:44:49,420 --> 00:44:53,560
And then, one last thing that I kind of want to cover is adversarial stickers.

418
00:44:53,560 --> 00:44:55,600
So, you've probably seen adversarial stickers.

419
00:44:55,600 --> 00:44:58,720
They look like melted crayons, in my opinion.

420
00:45:00,980 --> 00:45:04,100
But adversarial stickers, they have a bit of a problem.

421
00:45:04,100 --> 00:45:08,240
This is considered a brittle attack, and it's not very repeatable.

422
00:45:08,240 --> 00:45:14,700
And the reason why is because it doesn't have that attack transfer ability that I brought up earlier.

423
00:45:15,500 --> 00:45:34,100
So, in this case, kind of off to the bottom right, the stop sign, this is one of the first articles I was talking about adversarial stickers, and they were trying to modify a stop sign in such a way that the machine learning model didn't recognize it as a stop sign anymore,

424
00:45:34,100 --> 00:45:35,440
recognized it as something else.

425
00:45:35,440 --> 00:45:38,580
And so, this could be really damaging for self-driving vehicles.

426
00:45:38,880 --> 00:45:44,360
One of the problems is that this paper ran into some repeatability problems.

427
00:45:44,880 --> 00:45:57,600
And the reason why is because this attack is so brittle, they were able to convince their model, but they were not able to convince, or people who were trying to replicate this were not able to convince other models.

428
00:45:58,340 --> 00:46:00,500
So, it's not very useful.

429
00:46:00,500 --> 00:46:12,840
You can't just slap an adversarial sticker onto a sign and hope that you're driving a self-driving vehicle off into a wall or something of that nature.

430
00:46:12,900 --> 00:46:16,440
And then off here to the left is kind of the same idea with facial recognition.

431
00:46:16,440 --> 00:46:22,060
So, this researcher here said, okay, I'm going to set some constraints.

432
00:46:22,060 --> 00:46:29,600
I only want the pixels or the image around my eyes in the shape of glasses to be modified so that I look like this actress.

433
00:46:29,980 --> 00:46:31,720
Honestly, I don't remember who this is.

434
00:46:31,720 --> 00:46:32,700
I apologize.

435
00:46:33,180 --> 00:46:35,340
But I'm a man.

436
00:46:35,340 --> 00:46:39,440
I want to look like this actress per a facial recognition model.

437
00:46:39,440 --> 00:46:41,560
And so, he was successfully able to do that.

438
00:46:41,760 --> 00:46:49,380
The problem is that these glasses are designed for his face and the rest of the features of his face.

439
00:46:49,420 --> 00:47:03,400
And so, if I or many of you put on these glasses, it'll either look like somebody else or it won't recognize it as an adversarial example and it'll just recognize us as people, the same people, just wearing stupid glasses.

440
00:47:04,060 --> 00:47:07,620
So, attack transferability, adversarial stickers don't really have it.

441
00:47:07,620 --> 00:47:10,000
It's not a really good attack.

442
00:47:10,000 --> 00:47:11,140
It's very brittle.

443
00:47:11,140 --> 00:47:12,740
And I just kind of wanted to bring that up.

444
00:47:12,740 --> 00:47:19,920
For more information on this, I highly recommend DEF CON 2018 AI Village.

445
00:47:19,920 --> 00:47:25,940
Sven has a really good 30-minute talk on adversarial stickers and kind of the challenge with them.

446
00:47:25,940 --> 00:47:37,600
And then, for the facial recognition specifically, I know that Rich Horang in AI Village 2019 has a really good talk talking about facial recognition and trying to bypass that.

447
00:47:38,940 --> 00:47:44,580
So , to recap, how has AI empowered attackers?

448
00:47:44,620 --> 00:47:48,220
Well, it allows us to operate at a speed and scale like never before.

449
00:47:48,220 --> 00:47:50,800
We can operate at machine speed.

450
00:47:51,260 --> 00:47:57,340
And then, what we are seeing is the very beginning of what AI can bring to the offensive security space.

451
00:47:57,480 --> 00:48:00,380
So, we've already seen some adversarial attacks in the wild.

452
00:48:00,380 --> 00:48:04,940
I anticipate us seeing more as more researchers are coming out.

453
00:48:05,120 --> 00:48:09,520
AI and machine learning is kind of the hot new buzzword that everybody's latching onto.

454
00:48:09,520 --> 00:48:18,100
So, CEOs and decision makers are grabbing the AI-powered or machine learning-powered detection systems, defense systems.

455
00:48:18,460 --> 00:48:30,420
And it's kind of become a bit of a buzzword where you say that in a lot of common companies, especially in places like DEF CON, and they tell you to go talk to the Bitcoin folks and blockchain and all that fun stuff.

456
00:48:30,940 --> 00:48:35,100
There's no silver bullet, but we are seeing the very beginning of what is possible.

457
00:48:35,100 --> 00:48:49,520
And I anticipate, and I'm really hoping in the next several years, that we see a much larger impact of this AI versus AI kind of warfare that should be coming about.

458
00:48:50,500 --> 00:48:52,520
But anyways, I appreciate it.

459
00:48:52,520 --> 00:48:53,560
Thank you for your time.

460
00:48:53,560 --> 00:48:55,320
Thanks for sticking around this long.

461
00:48:55,440 --> 00:48:57,840
You can go ahead and email me if you have any questions.

462
00:48:57,840 --> 00:48:59,380
I'm also going to be in the chat.

463
00:48:59,600 --> 00:49:08,180
And then, for the slides, I have that are much more detailed are on Slideshare, and you can find them at slideshare.net slash GTKalandai.

464
00:49:08,220 --> 00:49:10,720
So that is my presentation.

465
00:49:10,720 --> 00:49:14,700
I hope you learned something, and thank you for sticking around.

466
00:49:14,880 --> 00:49:16,100
Bye!