WEBVTT 00:02.530 --> 00:08.710 Welcome to the last talk of the AI village from Fahid. 00:11.710 --> 00:14.050 I'll just let him take it away. 00:19.410 --> 00:25.130 Hello and thank you for joining this talk on the security issues and challenges in deep reinforcement learning. 00:25.930 --> 00:43.410 I'm Fahid Behzadan, I'm an assistant professor of computer science and data science at the University of New Haven and I also direct the Secure and Assured Intelligent Learning Lab, or SAIL for short, working on AI safety, security and applications of machine learning to cybersecurity and safety of complex systems. 00:45.710 --> 00:48.890 So here's the outline of my talk. 00:48.890 --> 00:52.870 I'm going to quickly go over the basics of deep reinforcement learning and reinforcement learning. 00:52.870 --> 00:56.850 There will be some math but I'll keep it to a minimum. 00:56.850 --> 01:09.010 Then we'll talk about vulnerabilities of deep reinforcement learning and whether deep RL is susceptible to classical adversarial machine learning attacks like adversarial examples and such. 01:09.010 --> 01:13.550 We'll talk about and develop a threat model for deep reinforcement learning. 01:13.550 --> 01:37.270 We'll identify different attack models, attack surfaces, we'll identify different types of vulnerabilities, and we'll introduce a number of attack mechanisms and corresponding defenses that have been developed in recent years. 01:37.270 --> 01:45.350 And we'll talk about the frontiers and areas of future research and work in this area. 01:45.350 --> 01:48.130 So just a quick overview. 01:48.190 --> 01:53.790 I assume most of the audience here is already familiar with the terminology used here. 01:53.790 --> 02:00.690 We can classify machine learning algorithms as supervised, unsupervised, and reinforcement learning algorithms. 02:00.690 --> 02:18.230 Supervised learning algorithms are those where the training data set includes labeled data, meaning that each data point in that data set comes with the correct label or the correct output expected from a model trained on that data set. 02:18.550 --> 02:29.010 Then there is unsupervised learning where clustering and anomaly detection and some other algorithms fall under where there are no labels available for data points. 02:29.010 --> 02:32.290 There's no feedback available on the data. 02:32.530 --> 02:37.070 However, the goal is to find some underlying structure in the data. 02:37.790 --> 02:46.570 And then there is reinforcement learning, which is concerned with the problem of sequential decision making. 02:46.570 --> 03:00.450 There's a reward system included in reinforcement learning, but there are fundamental differences between the settings of reinforcement learning and supervised learning, and that's why it merits its own category. 03:00.450 --> 03:03.190 We'll talk about those differences in the next few slides. 03:03.470 --> 03:14.490 So you can see the general settings of reinforcement learning, or in general, sensory motor agent problems and settings. 03:14.610 --> 03:21.230 In RL, we have an agent which is to be trained by a reinforcement learning algorithm. 03:21.230 --> 03:25.350 This agent interacts with the environment by performing some action. 03:25.350 --> 03:29.590 This action causes the state of the environment to change. 03:29.590 --> 03:33.950 Then the new state is observed by the agent. 03:34.050 --> 03:43.730 There's also some sort of reward associated with this change, provided or inferred by the agent based on this change in state. 03:45.790 --> 03:50.570 Well, you can think of this in terms of playing a game. 03:50.570 --> 04:04.190 The environment can be the game environment where different actions may result in scoring or loss of score, and the actions are, well, essentially the actions that the player can take. 04:04.190 --> 04:07.050 The state of the environment can be the state of the game. 04:07.050 --> 04:16.990 For example, in Breakout, the configuration of these breaks can be one of those where the agent is, where the ball is. 04:16.990 --> 04:25.750 If you recall the game of Breakout, of course, if you are old enough to remember Breakout, you're probably familiar with the dynamics. 04:25.750 --> 04:31.230 But you can think of any other game, and this setting still applies. 04:31.270 --> 04:32.410 What's the goal here? 04:32.410 --> 04:42.490 The goal is to learn how to take actions in order to maximize cumulative rewards, not instantaneous rewards, but cumulative rewards. 04:42.490 --> 04:48.510 So an RL agent is not just concerned with maximizing the current score. 04:48.510 --> 04:58.070 It wants to learn how to act so that at the end of the game, its total sum of rewards is maximized. 04:58.070 --> 05:00.110 What are the different applications of RL? 05:00.110 --> 05:14.030 Of course, game playing is well publicized because it's a very good testbed, and it has become one of the common testbeds and experimental settings for RL research. 05:14.030 --> 05:18.830 However, there are some major real-world applications for reinforcement learning. 05:18.830 --> 05:28.510 In essence, RL, or reinforcement learning, is the machine learning response to the need for data-driven control problems. 05:28.590 --> 05:30.530 Where do we encounter such problems? 05:30.530 --> 05:33.990 Well, in robotics, like autonomous navigation, object manipulation, and such. 05:33.990 --> 05:39.050 Algorithmic trading, this is one of the areas that my research group has recently become active in. 05:39.090 --> 05:48.750 Critical infrastructure controlling smart cities, resource allocation in smart cities, traffic management, intelligent traffic systems, smart grid management, and control. 05:49.090 --> 05:58.510 Healthcare, such as clinical decision making, this is actually one of the better known applications of RL in the real world. 05:58.670 --> 06:04.140 And other types of resource management and applications in operations research and such. 06:04.770 --> 06:18.430 So, RL is either envisioned or is already heavily adopted in various industries, many of which are critical and may become targets of malicious actions. 06:18.770 --> 06:22.270 Let's formalize the RL problem a little more. 06:22.270 --> 06:24.210 I'm just going to quickly go over this. 06:24.210 --> 06:28.190 There will be some math, but this is just to introduce the basic settings. 06:28.970 --> 06:37.770 The underlying framework to formulate the RL problem and provide a framework to think about and reason about the RL problem is the Markov decision process. 06:37.770 --> 06:39.190 Why do we call it the Markov decision process? 06:39.190 --> 06:46.910 Because it's based on the Markovian assumption, or the Markov property, which says the current state completely characterizes the state of the world. 06:47.110 --> 06:59.790 So, if you have the current state and you perform an action, the next state is only going to depend or be a function of the previous state and nothing before it. 06:59.790 --> 07:02.330 You don't need to know the history of the environment. 07:02.330 --> 07:09.270 All you need to know is what the previous state has been to infer what the coming state is going to be. 07:09.530 --> 07:18.750 Now, in Markov decision processes, a problem or setting is formulated by a tuple of five main parameters. 07:18.750 --> 07:21.310 One is the set of possible states, S. 07:21.310 --> 07:23.550 Set of possible actions, A. 07:23.910 --> 07:28.430 Distribution of reward given state action pair, or R. 07:28.430 --> 07:30.370 This distribution can also be a function. 07:30.370 --> 07:31.850 It doesn't necessarily have to be probabilistic. 07:31.850 --> 07:42.150 It can be just a function that tells you if at a particular state, S, I, an action AI is taken, then the reward is going to be, let's say, RI. 07:42.330 --> 07:54.910 Then there's transition probability, or transition dynamics, P, which represents the dynamics of the environment. 07:54.910 --> 08:00.470 Which state are we going to end up in if we are in a particular state and we perform a particular action. 08:00.470 --> 08:12.310 And there's finally a discount factor which defines how myopic the agent is, how much values, rewards that may occur further in the future, further down the line. 08:13.190 --> 08:20.510 Now, in general, the RL setting is based on the following process. 08:20.510 --> 08:26.410 At time step T equals zero, in the beginning, the environment samples initial state S0. 08:26.550 --> 08:33.430 Then, for T equals zero, this is a loop, until done, the agent selects action AT based on some criteria. 08:33.430 --> 08:39.910 It can be completely random in the beginning and then slowly becomes more targeted and more policy-driven. 08:39.910 --> 08:49.430 It performs some action, the environment produces a reward signal, and the next state, and the agent receives those signals. 08:49.430 --> 09:06.470 It may be partial, it may be incomplete, or it may be noisy, but the agent receives some signal resulting from the transition to the new state ST plus one, and a reward that has emerged from that transition and has resulted from that transition. 09:06.690 --> 09:12.910 Now, we define a new entity here called pi. 09:12.910 --> 09:20.430 A policy pi is a function, or a distribution in the probabilistic case, that maps from state to action. 09:20.430 --> 09:27.370 So, it tells the agent what action to perform given any state. 09:27.870 --> 09:36.070 And the objective is to find the optimal policy pi star that maximizes the cumulative discounted reward. 09:36.070 --> 09:39.330 We also call this return. 09:39.350 --> 09:45.490 So, cumulative, meaning the sum of rewards in the entire duration of interaction. 09:45.490 --> 09:54.130 It can be one episode of a game, it can be throughout the training period or training horizon, and it's discounted. 09:54.130 --> 10:00.750 This gamma is the discount factor that defines how myopic the agent is, or how much it values events that occur further down the line. 10:00.750 --> 10:10.790 It's typically a constant value between zero and one, and as you can see, as t increases, this gamma to power t decreases. 10:10.790 --> 10:25.890 So, the value, the preference, or the observed value of something that happens further down the line, which means with greater t, is going to decrease as t increases. 10:26.650 --> 10:32.670 So, again, the objective is to find the optimal policy pi star that maximizes this sum here. 10:32.670 --> 10:38.010 All right, we are going to quickly define two other definitions here. 10:38.290 --> 10:47.790 If we want to evaluate a particular state in these settings, one approach is to measure the value of that state using the value function. 10:47.790 --> 10:57.650 The value function of state s is the expected cumulative reward from following the policy pi from this state. 10:57.650 --> 10:59.890 So, let's assume we already have a policy. 10:59.890 --> 11:07.670 What will be the expected cumulative reward if you start in state s and keep following policy pi? 11:07.870 --> 11:20.490 Now, we also define a Q value, or a Q function, which tells us how good is a particular action A if it's performed in state s. 11:20.490 --> 11:36.810 In other words, if we are in state s, if the agent is in state s and performs action A, the Q value of this setting is the expected cumulative reward from taking action A in state s and then following the policy. 11:36.810 --> 11:42.170 Remember, the policy is a function that tells the agent what action to perform given any state. 11:42.170 --> 11:58.090 When the agent performs action A in state s, it goes into a state s prime, or the next state, and then we can use this policy function here to see what the action should be, what action should the agent take at that state, which takes it to another state, 11:58.090 --> 12:06.150 and then the policy is used to figure out the action to take in that state, and this goes on until the termination of the episode or the horizon. 12:06.350 --> 12:13.090 Now that we are familiar with the value function and the Q function, let's go a bit deeper. 12:13.630 --> 12:26.010 The policy pi, the value function, or the reward model are all functions. 12:26.210 --> 12:29.070 We want to learn at least one of these from experience. 12:29.070 --> 12:31.390 This is the essence of RL. 12:31.390 --> 12:50.230 If there are too many states, however, we cannot just tabulate everything and try to experience every state and the corresponding result from that state or that state and any action performed, because as the size of the problem, as the state space and the action space increase in settings like, 12:50.230 --> 13:00.330 let's say, playing GTA 5 or a self-driving navigation policy in the real world, these state spaces just explode. 13:00.450 --> 13:06.070 Dimensionality is too high and it's just not feasible to store every possible state. 13:06.070 --> 13:08.050 In those cases, we need to approximate. 13:08.190 --> 13:12.670 In general and traditionally, this is called RL with function approximation. 13:12.930 --> 13:23.950 If function approximation is done using deep neural networks, then we call the training setting or the RL setting deep reinforcement learning. 13:23.950 --> 13:25.550 The term is relatively new. 13:25.550 --> 13:35.170 It was, I believe, introduced in late 2013, early 2014 by me and David Silver and others at DeepMind in their DeepQ learning paper. 13:35.170 --> 13:37.190 However, the concept is not very new. 13:37.190 --> 13:44.610 That said, there have been fantastic and even mind-blowing advances in this area in the past three or four years. 13:44.610 --> 13:47.250 We'll talk about some of those as we move forward. 13:47.250 --> 13:51.850 A quick overview of the taxonomy of different RL approaches and RL agents. 13:51.850 --> 13:55.350 Remember, we have value, function, policy, and model. 13:55.350 --> 13:58.070 There are different approaches to solving the RL problem. 13:58.070 --> 14:02.630 Sometimes we just want to find a policy directly. 14:03.430 --> 14:10.990 The approaches that respond or satisfy that need are called policy-based RL. 14:10.990 --> 14:18.370 Sometimes we want to find a Q function or a V function first and then derive the policy from those functions. 14:18.370 --> 14:20.490 These approaches are called value-based. 14:20.490 --> 14:25.490 Sometimes we want to first learn the model of the environment and then solve the problem. 14:25.490 --> 14:27.230 These are called model-based approaches. 14:27.230 --> 14:33.310 Sometimes we don't have the model and we don't want to learn the model explicitly, and those are called model-free. 14:33.370 --> 14:49.770 When we're dealing with both value, function, and policy at the same time, in other words, we have one agent learning the value function, another learning the policy, and then contrasting those with each other in a kind of zero-sum setting. 14:49.770 --> 14:53.110 We call those actor-critic RL approaches. 14:53.110 --> 14:56.950 As you can see, there are different approaches to different settings and different problems. 14:56.950 --> 15:06.330 However, all of those can still be grounded on top of the Markov decision process framework and the general solution approach we looked at before. 15:06.350 --> 15:13.470 One of the better-known approaches to RL, which falls under the value-based approaches, is Q-learning. 15:13.470 --> 15:19.230 The objective in Q- learning is to derive the optimal policy pi star based on optimal Q. 15:19.230 --> 15:30.690 So the optimal Q is one that maximizes the value of each state and action. 15:32.110 --> 15:49.390 And from that point onward, with an iterative formulation based on Bellman equations or dynamic programming, it becomes possible to find this through bootstrapping and iterative re-estimation of the Q value. 15:49.490 --> 15:58.180 There are different approaches to Q-learning in large state spaces or large action spaces where function approximation is required. 15:58.690 --> 16:05.780 One such approach is to parametrize Q as s, a, and an estimation parameter theta. 16:07.070 --> 16:19.850 If we solve this parametrized Q function by neural networks, where theta corresponds to the neural network, the approach is called Q-networks. 16:19.850 --> 16:41.350 This solution approach was proposed very early in 2000s, even earlier, but was perfected in some sense in 2013-2014 by DeepMind, by David Silver and his team, which resulted in the proposal of deep Q-networks or QNs. 16:41.350 --> 16:44.390 So we're talking about deep networks. 16:44.390 --> 16:44.990 What does that mean? 16:44.990 --> 16:55.610 It means that these networks use deep neural networks, like CNNs, which help with both function approximation and also end-to-end feature learning. 16:55.610 --> 17:07.730 One of the advantages of deep learning is its superior performance in learning feature representations, especially from images. 17:08.630 --> 17:15.430 Now, for those of you who are more statistically oriented, you may have seen the term IID. 17:15.730 --> 17:22.930 It means that the data are independent and identically distributed. 17:22.930 --> 17:34.390 It means that one data point does not depend on another data point, and also each data point is equally likely to occur. 17:34.390 --> 17:37.630 These do not hold for the RL settings. 17:37.630 --> 17:44.170 A particular state, for example, is highly correlated with its previous state and action. 17:44.170 --> 17:45.150 Why does this matter? 17:45.150 --> 17:55.330 Well, a lot of our supervised and deep learning approaches are based on the assumption that the training data is IID. 17:55.330 --> 18:08.010 When this is not possible, what happens is, in response to this problem, what happened was the DQN approach introduced experience replay. 18:08.010 --> 18:25.990 It's like a bag of all data which is randomly sampled in each training iteration to reduce the correlation, the sequential correlation, the temporal correlation of data points, and also make it more likely for data to be evenly distributed. 18:25.990 --> 18:35.450 Also, to reduce the effect of oscillation during training, DQN uses fixed parameters for a target network. 18:35.450 --> 18:44.610 The target of optimization is fixed and is updated every few thousand iterations, so that reduces the oscillation problem. 18:44.610 --> 18:53.770 And, of course, the rewards are normalized to minus one to one to make sure the performance, the reward signals are bounded. 18:54.930 --> 19:06.790 Now, now that we have a preliminary understanding of deep reinforcement learning and one of its implementations, one of the parallel approaches called DQN, let's take a quick look at adversarial machine learning. 19:06.790 --> 19:13.670 And I assume that by now, those of you who are not familiar with adversarial examples have become, have been introduced to this concept. 19:13.670 --> 19:21.430 The idea here in adversarial examples, we are now talking in the realm of, we are speaking in the context of image classifier supervised learning. 19:21.430 --> 19:31.270 Let's say we have an image classifier trained on a set of images of pandas, cats, and others to identify the object in those images. 19:31.270 --> 19:37.270 In the beginning, we pass the image of a panda to the classifier, and it correctly classifies it as panda. 19:37.450 --> 19:57.390 Now, it's been demonstrated, it's been actually established by now, that it is possible to make the classifier, to induce incorrect classifications in deep learning models or machine learning models in general by adding minute, minimal perturbations to the original image. 19:57.390 --> 20:05.850 As you can see, it's almost impossible to detect or see any changes in this final image. 20:05.850 --> 20:09.130 These perturbations, these pixel perturbations are very small. 20:09.250 --> 20:24.690 Now, this is one example of different attacks or different vulnerabilities in the classical realm of adversarial machine learning, which is mostly concerned with supervised learning and sometimes unsupervised learning. 20:24.690 --> 20:37.530 In general, the adversarial objectives in AML or adversarial machine learning can be classified under the traditional CIA triad, confidentiality, integrity, and availability. 20:37.530 --> 20:57.730 So, with respect to confidentiality and privacy, an adversary may wish to target the confidentiality of the model parameters, architecture, no intellectual property theft is an issue in larger models, or it may target the privacy, the adversary may target the privacy of the training and test data. 20:57.850 --> 21:14.130 If medical, for example, if medical records were used in the training data, there have been proof of concept attacks showing that it is possible to infer whether a particular patient or the records pertaining to a particular patient were used in the training data or not, 21:14.130 --> 21:19.130 or sometimes it's possible to reconstruct the training data set by just having access to the model itself. 21:19.130 --> 21:21.590 And that is a major HIPAA violation. 21:22.070 --> 21:24.650 In essence, it's a privacy violation. 21:24.990 --> 21:35.150 Also, with regards to integrity and availability, the attacker may target the integrity of predictions or the outcome of the model, the performance of the model. 21:35.330 --> 21:42.930 For example, adversarial examples are an attack on the integrity of image classifiers or supervised machine learning models. 21:42.930 --> 21:54.130 And also an adversary may target the availability of the system that is deploying machine learning, for example, a facial recognition system or an autonomous navigation system in a driverless car. 21:54.450 --> 22:10.810 Now, now that we know so much about adversarial machine learning and security vulnerabilities of classical machine learning, supervised learning and unsupervised learning, there is a major question. 22:10.810 --> 22:14.910 And it's whether deep RL is immune to those attacks. 22:14.910 --> 22:27.030 Back in 2016, late 2016, when the research community was just beginning to start, was just beginning to pay attention to both deep reinforcement learning and the issue of adversarial examples. 22:28.110 --> 22:39.650 I came up with this question and decided to experiment with it a little to find out whether deep RL can also be vulnerable to such attacks. 22:39.870 --> 22:42.210 I started from a simple observation. 22:42.210 --> 22:51.010 The deep neural networks in DQN models and classifiers are both function approximators, are both at training time, there are function approximators at test times or just functions. 22:51.810 --> 22:54.650 And I came up with this hypothesis. 22:54.650 --> 23:02.750 If classifiers are vulnerable to adversarial examples, then action value approximators of DQNs may also be vulnerable. 23:02.790 --> 23:03.690 All right. 23:03.750 --> 23:14.170 So I started to set up an experiment where the aim was to perform adversarial attacks. 23:14.170 --> 23:17.490 The adversary's goal was twofold. 23:17.490 --> 23:20.970 Test time attack to perturb the performance of target's learned policy. 23:20.970 --> 23:33.810 So the target at this point is fully trained and is deployed in the environment and the adversary wants to somehow cause the agent to perform incorrectly, to manipulate its policy. 23:34.890 --> 23:37.490 What does the adversary know about the target? 23:37.490 --> 23:39.370 It knows the type of input to the target. 23:39.370 --> 23:46.930 For example, it knows whether the target policy is looking at image data, text, audio, and such. 23:46.930 --> 23:47.190 Why? 23:47.190 --> 23:49.790 Because it helps with estimating the architecture. 23:49.790 --> 24:00.090 For example, if it's images, the adversary can come up with a good guess that the target architecture includes CNNs or convolutional neural networks. 24:00.090 --> 24:02.890 I also assume that the adversary knows the reward function. 24:02.890 --> 24:12.810 So the adversary may have access to the environment and if it's, for example, a game environment, it knows what the scores are, how the scores are generated. 24:12.810 --> 24:13.890 What is not known? 24:13.890 --> 24:18.330 The knowledge of target's neural network architecture is not known. 24:18.330 --> 24:21.130 So it's a black box attack. 24:21.130 --> 24:28.150 And also the initial parameters, the initialization of the target neural network is also not known. 24:28.150 --> 24:30.810 What is available to adversary in terms of actions? 24:30.810 --> 24:36.610 The adversary may perturb the environment where the target performs. 24:36.610 --> 24:43.210 For example, it can change pixel values in a game environment through a man-in-the-middle attack. 24:43.790 --> 24:49.890 I consider in this work two techniques for perturbing that environment. 24:49.890 --> 25:03.350 One is the classical fast gradient sign method for generating adversarial examples and the other is the Jacobian-based saliency map attack or the JSMA approach introduced by Paperno in 2015, I believe. 25:04.150 --> 25:18.250 Now, in that experiment, I used the classical DQN approach, the classical DQN architecture introduced by Nie in an Atari game, the game of Pong. 25:18.450 --> 25:22.230 And this was all implemented in OpenAI Gym with TensorFlow. 25:22.230 --> 25:27.650 Back then, PyTorch wasn't really and trained the agent against heuristic AI. 25:27.930 --> 25:30.730 And here's the initial proof of concept result. 25:30.730 --> 25:32.270 This is for the white box attack. 25:32.270 --> 25:34.950 Of course, later on, I will show you the results for the black box attack. 25:34.950 --> 25:59.390 And you can see that for FGSM and JSMA, regardless of how far along the training the agent is, both policies, the policy for the Pong agent is highly vulnerable to adversarial perturbations through simple techniques like FGSM and JSMA. 25:59.390 --> 26:04.210 You can see that for JSMA, the success rate was 100% for all of the cases. 26:04.210 --> 26:13.150 For FGSM, it was slightly lower and it was mostly because of the termination criteria and the perturbation threshold that I had defined. 26:13.150 --> 26:21.610 But you can see, it can be observed that the policy can be very easily manipulated through adversarial example attacks. 26:21.610 --> 26:37.290 So, at 100 random observations, perturbed with FGSM and JSMA, the results were fed, the perturbed images were fed to the trained neural network representing the agent's policy as test input, and then the success rate was measured. 26:37.370 --> 26:39.130 Now, is this type of attack practical? 26:39.130 --> 26:42.190 Is this really something that we should be worried about? 26:42.190 --> 27:03.510 So, a few years later, in 2018, Clark and his co-authors published the report of a similar attack on an autonomous robot based on DQN policy using ultrasonic collision, ultrasonic sensory input for collision avoidance. 27:03.510 --> 27:20.930 And they had shown that they can use adversarial perturbations to manipulate the trajectory of the robot and make it follow a path defined or desired by the adversary, not one that the robot itself wants to follow. 27:21.270 --> 27:29.050 There are more recent examples of how this sort of attack on deep RL, or in general, attacks on deep RL, can be of concern. 27:29.530 --> 27:39.830 One of the recent works by my graduate students is on attacks on automated trading algorithms based on reinforcement learning. 27:39.990 --> 27:50.030 The paper is coming out in a couple months, so I can't go into more details, but this is one of the more severe and urgent cases for security researchers to consider. 27:50.670 --> 27:57.470 Deep RL is already being used by many major financial players and stock traders, and it can be easily manipulated in the real world. 27:57.470 --> 28:07.590 There are other cases, of course, but this is one of the examples that demonstrates the practicality and the applicability of this attack to real world scenarios. 28:07.770 --> 28:13.670 Now, before we go further into different types of attacks, let's develop a threat model for deep reinforcement learning. 28:13.670 --> 28:18.190 Again, the adversary's objectives can follow those of the CIA triad. 28:18.190 --> 28:28.450 The adversary may wish to access the internal configurations, like model parameter, reward function, policy, and such, to still the model for intellectual property theft and such. 28:28.450 --> 28:35.410 There can be attack on integrity, which means compromising the desired learning or enactment of the policy. 28:35.410 --> 28:44.610 There can be attacks on availability, which are essentially compromises of the ability of the agent to perform training or actions when needed. 28:44.710 --> 28:47.290 Now, let's look at the attack surface of the RL. 28:47.290 --> 28:50.190 This is the general diagram. 28:50.190 --> 28:55.290 This is the general block diagram of a DRL agent or DRL system. 28:55.410 --> 28:56.770 Now, we have the agent. 28:56.770 --> 29:07.130 The agent typically has some memory where it stores its experiences during training, and then those experiences help with function approximation, data-driven policy learning, and such. 29:07.170 --> 29:14.830 And then there is an exploration controller, which controls how the agent explores the environment during training. 29:14.830 --> 29:20.650 There's an experience selector, how to select experiences from the bank of data or observations stored. 29:20.930 --> 29:28.830 In this data set, there is an actuator, which then enacts the actions of the agent inside an environment. 29:28.830 --> 29:36.650 The environment is connected back to this agent block through an observation channel, observation of the state, and reward channel. 29:36.650 --> 29:46.870 And it's of no surprise to the more seasoned security researcher and professional that all of these components can be a subject or target of attack. 29:46.970 --> 29:55.330 As we go forward, we'll cover some examples of attacks that can occur on each of these components. 29:55.350 --> 29:58.570 We've already seen an attack on the observation channel. 29:59.070 --> 30:04.570 There are attacks on the reward channel, which we'll hopefully touch on. 30:04.570 --> 30:08.870 And there are attacks on the agents during training and the actuator. 30:08.870 --> 30:11.210 Let me give you a quick example of the actuator attack. 30:11.210 --> 30:17.970 Let's assume we have a robot, an actual robot, learning to navigate in an environment while avoiding obstacles. 30:18.130 --> 30:41.430 If the robot commands or decides to, let's say, move the left wheel forward, but there is some sort of obstacle in front of the left wheel, and the left wheel doesn't actually move, then the actuator, the resulting data is going to be, the resulting observation is going to be skewed, 30:41.430 --> 30:54.670 because the agent is going to assume that the actuation has happened and then look at the changes in the observation and use that to retrain its policy, to optimize its policy based on faulty data. 30:54.810 --> 30:56.410 What are the adversarial capabilities? 30:56.410 --> 30:58.050 Well, we first look at different attack modes. 30:58.050 --> 31:08.490 The attacker can perform a passive attack where it's only observing the target, it's not changing anything, or it can perform an active manipulation. 31:08.530 --> 31:23.250 In passive attacks, the attacker can perform inverse reinforcement learning to learn about the reward function of the agent, or later on we'll see it can perform imitation learning to steal the policy. 31:23.830 --> 31:29.410 Active measures include attacks on the actuation, observation, or the reward channel. 31:29.410 --> 31:41.150 And attacks on observation can be targeting their representation model, how the agent sees the environment, or perturbing the transition dynamics, how the agent sees the changes in the environment. 31:42.050 --> 31:48.250 Okay, going back to our initial proof of concept, remember our original goal was to perform a black box attack. 31:50.650 --> 32:01.970 So, to achieve this objective, we introduce an approach based on the transferability of adversarial examples. 32:01.970 --> 32:09.050 So, we create a second DQN, the adversary creates a second DQN with similar architecture, but different initial parameters. 32:09.050 --> 32:19.830 When I say similar, it doesn't necessarily have to be a match, it just needs to be a convolutional neural network with some functional approximation techniques, but it doesn't need to have the same parameters or the exact same architecture. 32:20.650 --> 32:27.470 And it trains that agent, that model, on the same environment. 32:27.470 --> 32:38.910 The assumption is the adversary has access to that environment, and then uses the knowledge of that architecture and the trained replica policy. 32:39.210 --> 32:55.930 It crafts adversarial examples the same way it did for the white box case, and we know that many of those adversarial examples can transfer to similar models, train on similar data, which also you can see applies to DQN policies as well. 32:55.930 --> 33:00.870 And this is how we implemented a black box attack against deep RL policies. 33:01.430 --> 33:03.390 Now, what about training time attacks? 33:03.390 --> 33:18.410 In the same paper we introduced the policy induction attack, where the attack is of the adversarial example type against a DQN agent during training. 33:18.850 --> 33:22.110 Now, what are the different steps in this attack? 33:22.450 --> 33:36.310 First, the adversary derives an adversarial policy from the adversarial goal by training on the same environment where the target is going to perform or be trained in. 33:36.310 --> 33:52.050 So, if the adversary wants to minimize the reward gained in the game by the agent, then the goal, the optimization goal, is going to be the exact opposite of that of the target policy. 33:52.050 --> 33:55.630 The target wants to maximize the reward and the adversary wants to minimize it. 33:55.630 --> 33:58.730 Of course, there are different ways of formulating this adversarial goal. 33:59.790 --> 34:07.730 Then, the adversary creates a replica of target's DQN and initializes it randomly. 34:07.730 --> 34:24.590 And then comes the exploitation phase, where the attacker observes the current state and transitions in the environment, then estimates best action according to the adversarial policy derived in step one. 34:24.590 --> 34:33.430 Then, the attacker crafts perturbations to induce adversarial action based on the replica of target's DQN. 34:33.430 --> 34:37.150 This is exactly the same as our black box test time attack. 34:37.150 --> 34:41.550 The attacker applies the perturbation as a man in the middle in the observation channel. 34:41.650 --> 34:47.170 The perturbed input is revealed to the target and the attacker waits for target's actions. 34:47.530 --> 34:48.490 And this is a loop. 34:48.490 --> 34:58.790 This loop can go on until the training process of the target either converges to a suboptimal policy or up to a certain number of iterations. 34:59.470 --> 35:01.370 This is a very rough plot. 35:01.370 --> 35:25.310 This is not smooth yet, but you can see that the unperturbed agent moves towards... this is in the game of Pong still... moves towards convergence to an optimal total sum of rewards, while the attack agent moves towards a convergence to the minimum possible return of zero. 35:25.310 --> 35:35.330 It's getting closer and closer to zero, which indicates that the training process of DQN and DeepRL in general can also be targeted through adversarial attacks. 35:35.330 --> 35:39.330 Now, I'm going to introduce another type of training time attack. 35:39.330 --> 35:44.550 Again, this aims to induce some form of misbehavior. 35:44.750 --> 35:54.070 We call this misbehavior addiction, and this is a follow-up to a work I did with my colleague Roman Niempolski on psychopathological modeling of AI safety problems. 35:54.270 --> 35:56.070 This is a proof of concept. 35:56.070 --> 35:59.510 We consider the game of Snake. 35:59.510 --> 36:04.110 Many of you probably remember Snake from older Nokia phones. 36:04.310 --> 36:10.570 The DQN agent is a snake and is learning to play in this environment. 36:10.570 --> 36:19.910 The attacker adds a drug seed with more instantaneous reward than the typical seed, but it also results in more increase in the length of the tail. 36:19.910 --> 36:22.650 You can see that this can end up... 36:24.730 --> 36:41.790 well, if the increase in the tail length is more than a certain amount, then a longer-tailed snake is bound to eat its own tail sooner rather than later. 36:42.090 --> 36:59.970 So, we show that we actually derive some theoretical closed-form solutions for what the additional reward and increase in the tail length should be for addiction to emerge, meaning that the agent learns a more myopic policy instead of the optimal policy, 36:59.970 --> 37:11.550 and you can see that it's actually possible to make the agent addicted to the drug seed, as we call it, and this results in learning a sub-optimal policy. 37:12.390 --> 37:26.330 And, of course, due to time limitations, I'm going to introduce only one more type of attack, and it's that of targeting the confidentiality of a deep RL policy. 37:27.230 --> 37:34.010 The problem here is, or the question is, is it possible to extract a deep RL policy from observations of its actions? 37:34.730 --> 37:36.070 Why does this matter? 37:36.070 --> 37:42.170 Well, the security challenge posed by this sort of action is, of course, model theft. 37:42.170 --> 37:58.790 A company like Google or, let's say, Uber or Waymo may have spent billions or millions of dollars on coming up with a very accurate deep RL policy for autonomous navigation, and if it can be stolen by an adversary, then the intellectual property becomes worthless. 37:59.270 --> 38:12.310 And also, a stolen policy, an extracted policy, can be leveraged in integrity attacks in the same way that we mounted black box attacks on DQN policies. 38:12.750 --> 38:14.230 So, let's see. 38:14.230 --> 38:26.070 As it happens, a branch of reinforcement learning, or in general, one solution to the sequential decision-making problem, is not in the RL domain, but in the supervised learning domain, and it's called imitation learning. 38:26.070 --> 38:34.230 Imitation learning is the supervised learning of policies from observed behavior of an expert, and by behavior, I mean state action behavior. 38:34.230 --> 38:36.050 What is the policy of an expert? 38:37.490 --> 38:43.090 Based on this concept, in 2018, Hester et al. 38:43.410 --> 38:57.950 proposed DQFD, or DQ Learning from Demonstrations, which is DQN, where the initial training is done based on observed data using deep learning. 38:57.950 --> 39:08.030 So, they have data from human players playing a certain game, or human performance doing a certain task that they want the agent to learn. 39:08.030 --> 39:25.310 The initial step of training for a DQFD agent is supervised learning on observed data, and then it starts building on top of it through reinforcement learning approaches, and it was shown that it can result in faster convergence, better sample complexity, 39:25.310 --> 39:28.630 and sometimes more interesting and robust policies. 39:29.390 --> 39:34.250 As security researchers, you can probably see where this is going. 39:34.250 --> 39:41.830 This wonderful algorithm, DQFD, can also be used to replicate policies. 39:41.830 --> 39:50.010 Instead of applying it on observed data collected from human performance, it can be applied on observed data from a target policy. 39:50.110 --> 39:52.950 So, here's a proof of concept attack the procedure. 39:53.010 --> 40:14.270 The attacker observes and records n interactions of the SARSA type, state action, the next state, and the reward based on this transition of the target agent in a particular environment, and then the attacker applies DQFD to learn an imitation of the target policy and Q function. 40:14.310 --> 40:30.110 Now, at this point, the attacker may either just go away and sell the extracted policy, or it may decide to target it using different adversarial perturbation attacks, some of which we've covered so far in this talk. 40:30.850 --> 40:43.530 So, as a proof of concept, we consider a slightly less complex environment data cart poll, where the objective is to stabilize this poll on this cart by moving the cart to right and left. 40:43.530 --> 40:55.510 The reason for choosing a simple environment is merely economical, because we didn't want the experiments to take days or weeks. 40:55.510 --> 41:00.130 We start with a simple case, and we consider different types of policies. 41:00.130 --> 41:09.410 DQN with prioritized replay, an enhanced version of the classical DQN proximal, policy optimization, and asynchronous actor-critic. 41:09.830 --> 41:24.330 And, of course, we also train an adversarial RL agent, a DQN agent, whose objective is to incur maximum loss of reward, or in other terms, in more technical terms, maximize the regret of its target. 41:25.390 --> 41:26.730 And here are the results. 41:26.730 --> 41:43.570 First, with regards to replication progress, based on only 5,000 demonstrations, 5,000 state action, next state reward observations, we see that all three policies are almost exactly replicated. 41:43.570 --> 41:51.530 We can see convergence to the optimal performance of those policies in the environment. 41:52.030 --> 41:54.250 And then we perform adversarial training. 41:54.250 --> 42:10.630 We train an adversarial RL agent to attack and maximize the regret of those policies, and you can see that this can also be easily achieved within very few iterations of training in CardPol. 42:10.690 --> 42:33.310 I believe we can see that for PPO2, which is a somewhat robust deep RL algorithm or approach, it's possible to incur maximum damage or find a policy that incurs maximum regret on the target within 60,000 iterations, which is a relatively low amount. 42:33.310 --> 42:34.830 Now, what about defenses? 42:34.830 --> 42:51.350 So, of course, similar to adversarial examples, one approach or one technique for reducing the impact of adversarial example attacks is through regularization. 42:51.390 --> 43:11.750 And one common type of regularization in supervised learning for mitigating adversarial example attacks is adversarial training, training the model on adversarially perturbed samples to make sure that it sees different perturbations of the same image and knows that all of those results in the same label, 43:11.750 --> 43:13.230 in the correct label. 43:13.230 --> 43:18.790 This is called essentially data augmentation in general as a regularization technique. 43:18.850 --> 43:32.750 So in 2017, when I was just starting to look into this problem or this domain, I had gone through the adversarial training literature and thought that the same may also hold true for deep RL. 43:32.750 --> 43:35.510 I made a hypothesis, actually two. 43:35.570 --> 43:37.130 One is with regards to recovery. 43:37.130 --> 43:49.010 If training time attacks are not contiguous, if not all of the observations are perturbed, then DRL adapts to the environment and adjusts the policy to overcome the attacks. 43:49.010 --> 43:51.230 This is training time. 43:51.330 --> 43:54.370 And with regards to robustness, I made another hypothesis. 43:54.370 --> 44:01.050 Such policies, policies trained under attack are more robust to test time attacks. 44:01.130 --> 44:10.650 And this particular investigation is published in a paper titled whatever does not kill deep reinforcement learning makes it stronger. 44:10.850 --> 44:19.150 So similar as before, we are looking at DQN in Atari games, Breakout, Enduro, and Pong. 44:19.610 --> 44:26.230 Now, the way I designed the experiment was based on a probability of attack. 44:26.230 --> 44:35.210 So as an attacker, I assign a certain probability for each state, for each observation during training time to be perturbed. 44:35.610 --> 44:45.670 I perform experiments, different experiments with different values of this P attack, 20%, 40%, 80%, and 1, which means contiguous attack. 44:45.670 --> 44:54.420 And it's interesting to see that for values of P less than 50%, the agent actually recovers. 44:55.260 --> 45:12.060 But for values greater than 50%, the training process plummets and either does not converge or converges to a very, very low mean return value or mean total reward value. 45:12.060 --> 45:17.960 I later on publish a theoretical analysis of why this happens. 45:17.960 --> 45:24.040 It's available in my PhD dissertation, which I'll reference in the final slide. 45:24.360 --> 45:30.580 Also, it was interesting to see that the robustness is also, the robustness hypothesis is also true. 45:30.580 --> 45:46.840 You can see here that after training, if we attack the test time policy with probability of 1, the plane or vanilla policy, a policy that was not trained adversarially, performs very poorly. 45:46.840 --> 45:55.320 However, policies trained under adversarial attacks with probability 0.2 or 20% and 0.4 perform really well. 45:55.320 --> 46:02.540 For 80%, it's even for policies trained at 80%, as you can see, the policy itself is already performing very poorly. 46:02.700 --> 46:08.880 It's surprising to see that at P equals 1, the performance gets slightly better. 46:08.880 --> 46:10.600 It's comparable with 40%. 46:10.600 --> 46:14.400 To this day, I'm not entirely sure why this happened. 46:14.440 --> 46:17.320 I've repeated the experiments a number of times and still get the same result. 46:17.320 --> 46:18.540 I still don't know why this happened. 46:18.540 --> 46:26.500 This is one of the interesting problems that we are looking at right now as a research group. 46:26.680 --> 46:30.160 Another defense that we introduce is based on parameter space noise. 46:30.160 --> 46:32.020 This is very much like dropout. 46:32.260 --> 46:41.200 The idea of parameter space noise was introduced in 2017, I believe, independently by Plamper et al and Fortunato et al. 46:41.200 --> 46:53.120 And the idea here is, again, similar to dropout, to introduce zero mean random noise to the learnable parameters of neural network in deep RL to enhance exploration and convergence in deep RL benchmarks. 46:53.480 --> 47:21.600 Now, in another paper in 2018, we investigate whether this approach can be used to mitigate the impact or severity of policy manipulation attacks on DQN, and it was shown that it actually performs very well compared to vanilla or classical DQN. 47:21.840 --> 47:23.740 And these are the training time results. 47:23.740 --> 47:37.660 You can see that if noisy net is used, if parameter space noise is used, the performance degradation for all environments is at a much lower slope, at a much lower rate than the vanilla architecture. 47:37.940 --> 47:53.940 Finally, to have proposed a solution for the policy extraction problem, I, along with William Hsu at K-State, came up with the idea of watermarking DRL policies. 47:53.940 --> 47:57.320 Watermarking has already been introduced in deep learning in general. 47:57.320 --> 48:22.360 The idea is to come up with a unique signature that is both difficult to remove and does not impact the performance of the policy itself, or the model itself, but it still provides a unique signature, proof that a model is the same as another model, or is a replica of the suspected model. 48:22.360 --> 48:24.700 So we introduce an interesting watermarking procedure. 48:24.700 --> 48:30.700 I know it's arrogant of me to call my own work interesting, but I still get excited when I think about the moment I came up with this idea. 48:30.700 --> 48:36.760 The idea is to create a second environment whose state space is this drawing from the main environment. 48:36.760 --> 48:38.460 So, create an environment. 48:38.460 --> 48:45.560 If you're training an agent to play a game, let's create another environment which has no states. 48:45.560 --> 48:53.380 None of these states are the same as the original training environment, or deployment environment of the agent. 48:53.720 --> 48:56.800 But the dimensionality of the states are the same. 48:56.800 --> 49:09.580 So, if each state in the original environment is represented by, let's say, three values, three features, then each state in the second environment is also represented by three features. 49:09.580 --> 49:12.860 It doesn't really matter what the second environment looks like. 49:12.860 --> 49:17.940 It's just some other environment that the agent may interact with. 49:18.760 --> 49:26.700 And then, we craft the transition dynamics and reward procedure for the second environment such that the optimal policy follows a looping trajectory. 49:26.700 --> 49:39.320 So, an optimal policy for an agent trained in the second environment is going to be one that follows a loop, goes through, let's say, state one, then state two, state three, and then goes back to state one. 49:39.340 --> 49:44.800 During training, what happens is, we periodically alternate between the two environments. 49:44.800 --> 50:00.220 So, let's say, at every N iterations of the training process, we take our RL agent from the original environment, we drop it in the second environment, we train it for a few iterations, and then bring it back to the original environment. 50:00.460 --> 50:12.520 Now, once trained, if we want to examine the authenticity or whether a policy is copied or not, we apply the policy in the second environment and measure the total reward. 50:12.640 --> 50:13.660 Here's the experimental setup. 50:13.660 --> 50:17.040 Again, we are working with CardPool. 50:18.300 --> 50:36.020 So, the watermarking environment is defined with five states, states one to four, plus a terminal state, which should never be reached if the policy is optimal in this environment. 50:36.200 --> 50:44.360 And none of these states, as represented here, can be found in the original, can occur in the original CardPool environment. 50:44.360 --> 50:50.700 These are all highly impossible, not highly, definitely and absolutely impossible to occur in the original environment. 50:50.700 --> 50:54.120 As for the transition dynamics, this is how we've defined it. 50:54.120 --> 51:01.960 Let a particular state be A0 and A1. 51:02.760 --> 51:05.740 If we are, I'm sorry, actions. 51:05.740 --> 51:08.320 Actions A0 and A1. 51:08.560 --> 51:20.220 If the agent is in state I and performs action I modulo two, if I is even, this is going to be action zero. 51:20.220 --> 51:24.040 If I is odd, this is going to be action one. 51:24.040 --> 51:32.200 Then the next state is going to be state I star or I modulo four plus one. 51:32.200 --> 51:35.180 So, and it receives a reward of one. 51:35.180 --> 51:44.980 If the agent performs any other actions, instead of going from one to two, it performs any other action, it will immediately go to the terminal state and receives a reward of zero. 51:44.980 --> 51:50.580 So, the optimal trajectory is state one to two to three to four back to state one and so on. 51:50.760 --> 51:51.060 All right. 51:51.060 --> 51:52.720 Let's see how it works. 51:53.080 --> 51:58.280 Let's look at the test time performance comparison of watermarked and nominal, non-watermarked policies. 51:58.400 --> 52:05.760 The watermarked policy performs exactly as well as unwatermarked policies. 52:05.760 --> 52:12.840 It reaches the optimal or best performance of 500. 52:13.380 --> 52:21.620 And when it's applied to the watermark environment, the second environment, it also performs optimally. 52:21.620 --> 52:23.800 It gets the maximum reward possible. 52:24.180 --> 52:39.100 However, you can see that if we try to apply the non-watermarked policies to the watermark environment, we'll see very, very small values of score, the total reward. 52:39.100 --> 52:56.520 So, you can see that it's possible to determine whether a policy is authentic or not or whether a policy is an exact copy of another policy by just applying it to a second environment and see whether it performs optimally or not. 52:56.980 --> 53:03.360 Now, there's there are many other things that I really wanted to touch upon in this talk, but unfortunately we are a little short in time. 53:04.200 --> 53:18.040 For practitioners, it may be of interest to have some way of benchmarking or evaluating the resilience and robustness of policies and compare different policies, different approaches with regards to the resilience and robustness. 53:19.180 --> 53:26.180 Some of my work already introduces or proposes an RL-based approach to perform this evaluation and benchmarking. 53:27.300 --> 53:40.720 I've also done some work on investigating the impact of hyperparameter choices on resilience and robustness of DQNs in particular, but also other model-free and actor-critic approaches. 53:41.920 --> 53:49.960 This can be very helpful to those who want to engineer and design the new RL agents to be deployed in critical environments. 53:50.880 --> 54:01.920 Also, something that I wanted to mention, but unfortunately I don't have time to do so, is that adversarial training is not a silver bullet. 54:01.920 --> 54:06.220 It's not an answer to all of the problems in DQN agents. 54:06.220 --> 54:14.340 There are certain limitations for robustness and resilience obtained from adversarial training of DQN agents. 54:14.340 --> 54:24.580 And also adversarial training is very costly in general, especially when it comes to real-world scenarios, real-world environments and actions. 54:24.760 --> 54:39.240 And some of my recent work is focused on improving the sample efficiency and computational cost of adversarial training via a new exploration mechanism called adversarially-guided exploration, or AGE. 54:39.480 --> 54:43.580 All of this work can be found in my Ph.D. 54:43.580 --> 54:49.940 dissertation, which bears the same name as this talk, False Narcotic Star Security and Deep Reinforcement Learning. 54:49.940 --> 54:53.480 You can find it if you search my name on Google Scholar. 54:53.940 --> 55:02.540 If you're interested, of course, all of these are published in separate papers in slightly more details under the same titles. 55:02.740 --> 55:08.680 And finally, some of the open areas of research in this domain. 55:09.400 --> 55:20.040 With regards to training time resilience and robustness, not much has been done with regards to policy search and actor-critic methods, as well as model-based and hybrid methods. 55:20.040 --> 55:35.520 Of course, when we talk about model-based, there are some approaches from optimal control theory and approximate dynamic programming that may be applied here, but very few have looked at this problem from a security point of view. 55:35.520 --> 55:42.780 So if you're interested, this is one of the areas that is in dire need of a security-oriented investigation. 55:42.780 --> 55:49.080 As for mitigation of policy replication, one of the ideas that my research group is currently working on is constrained randomization of policy. 55:49.080 --> 56:03.280 So randomize the policy such that the replication through techniques like imitation learning becomes more costly, more samples, more observations will be required, while preserving the performance of the policy. 56:03.560 --> 56:06.380 There's almost no work done in multi-agent settings. 56:06.380 --> 56:21.260 Of course, adversarial reinforcement learning has been investigated in settings where there are zero-sum agents, but not really where there is an external adversary or adversaries trying to exploit the inner workings of the agents, the RL components of the agent. 56:21.960 --> 56:25.800 One more thing that is of note is the importance of discounting. 56:25.800 --> 56:34.700 The addiction problem that I demonstrated earlier in the snake agent is mostly due to the constrained discounting solution. 56:34.700 --> 56:39.140 For those of us who come from a reinforcement learning background, you're familiar with the basics of reinforcement learning. 56:39.140 --> 56:48.280 You probably know that the discount factor is typically chosen to be 0.99 or something in the same ballpark and is left the same. 56:48.500 --> 56:51.680 It's treated as a constant throughout the training process. 56:51.680 --> 57:01.880 But this is very far from how our brain works and very far from the optimal approach or accurate approach to discounting. 57:01.880 --> 57:21.590 Our research group has recently started looking into this problem and is working on developing adaptive discounting solutions to enhance the resilience and robustness of RL agents, particularly deep RL agents in complex environments for AI safety and security purposes. 57:21.960 --> 57:25.560 Now, of course, there are naturally inspired approaches that can be looked at. 57:25.560 --> 57:45.100 For example, approaches coming from, let's say, TD lambda models or dopamine models of psychopathological problems or neurological problems and the solutions prescribed to those, as well as approaches in social sciences, which may help with the security problems arising in multi-agent RL settings. 57:45.540 --> 57:47.280 Very well, thank you very much. 57:48.420 --> 57:53.140 And I believe at this time I should be available for your questions.