start	end	text
2530	8710	Welcome to the last talk of the AI village from Fahid.
11710	14050	I'll just let him take it away.
19410	23750	Hello and thank you for joining this talk on the security issues and challenges in deep
23750	29190	reinforcement learning. I'm Fahid Behzadan, I'm an assistant professor of computer science and
29190	32550	data science at the University of New Haven and I also direct the Secure and Assured
32550	38330	Intelligent Learning Lab, or SAIL for short, working on AI safety, security and applications
38330	43410	of machine learning to cybersecurity and safety of complex systems.
45710	51310	So here's the outline of my talk. I'm going to quickly go over the basics of deep reinforcement
51310	56850	learning and reinforcement learning. There will be some math but I'll keep it to a minimum.
56850	61310	Then we'll talk about vulnerabilities of deep reinforcement learning and whether
61310	68070	deep RL is susceptible to classical adversarial machine learning attacks like adversarial examples
68070	73650	and such. We'll talk about and develop a threat model for deep reinforcement learning. We'll
73650	82590	identify different attack models, attack surfaces, we'll identify different types of
82590	92590	vulnerabilities, and we'll introduce a number of attack mechanisms and corresponding defenses
92590	101750	that have been developed in recent years. And we'll talk about the frontiers and areas of future
101750	110230	research and work in this area. So just a quick overview. I assume most of the audience here is
110230	117030	already familiar with the terminology used here. We can classify machine learning algorithms as
117030	121530	supervised, unsupervised, and reinforcement learning algorithms. Supervised learning
121530	127450	algorithms are those where the training data set includes labeled data, meaning that
128590	135390	each data point in that data set comes with the correct label or the correct output expected from
135650	142690	a model trained on that data set. Then there is unsupervised learning where clustering
142690	147910	and anomaly detection and some other algorithms fall under where there are no labels available
147910	154670	for data points. There's no feedback available on the data. However, the goal is to find some
154670	162870	underlying structure in the data. And then there is reinforcement learning, which is concerned with
163350	170130	the problem of sequential decision making. There's a reward system included in reinforcement learning,
170130	175550	but there are fundamental differences between the settings of reinforcement learning and supervised
176070	181410	learning, and that's why it merits its own category. We'll talk about those differences
181410	188750	in the next few slides. So you can see the general settings of reinforcement learning,
188750	198910	or in general, sensory motor agent problems and settings. In RL, we have an agent which is to be
198910	203870	trained by a reinforcement learning algorithm. This agent interacts with the environment by
203870	210770	performing some action. This action causes the state of the environment to change. Then the new
210770	218110	state is observed by the agent. There's also some sort of reward associated with this change,
218810	223730	provided or inferred by the agent based on this change in state.
225790	232570	Well, you can think of this in terms of playing a game. The environment can be the game environment
232570	241750	where different actions may result in scoring or loss of score, and the actions are, well,
241750	246370	essentially the actions that the player can take. The state of the environment can be the state of
246370	254530	the game. For example, in Breakout, the configuration of these breaks can be one of
254530	259010	those where the agent is, where the ball is. If you recall the game of Breakout, of course, if you
259010	266070	are old enough to remember Breakout, you're probably familiar with the dynamics. But you can
266070	273710	think of any other game, and this setting still applies. What's the goal here? The goal is to
273710	280570	learn how to take actions in order to maximize cumulative rewards, not instantaneous rewards,
280570	288510	but cumulative rewards. So an RL agent is not just concerned with maximizing the current score.
288510	298070	It wants to learn how to act so that at the end of the game, its total sum of rewards is maximized.
298070	305490	What are the different applications of RL? Of course, game playing is well publicized because
305490	311370	it's a very good testbed, and it has become one of the common testbeds and experimental
312250	318230	settings for RL research. However, there are some major real-world applications for reinforcement
318230	323290	learning. In essence, RL, or reinforcement learning, is the machine learning response to
324010	330730	the need for data-driven control problems. Where do we encounter such problems? Well,
330730	334850	in robotics, like autonomous navigation, object manipulation, and such. Algorithmic trading,
334850	339970	this is one of the areas that my research group has recently become active in. Critical infrastructure
339970	344530	controlling smart cities, resource allocation in smart cities, traffic management, intelligent
344530	350630	traffic systems, smart grid management, and control. Healthcare, such as clinical decision
350630	358510	making, this is actually one of the better known applications of RL in the real world.
358670	364140	And other types of resource management and applications in operations research and such.
364770	373430	So, RL is either envisioned or is already heavily adopted in various industries,
373430	380250	many of which are critical and may become targets of malicious actions. Let's formalize
380250	385330	the RL problem a little more. I'm just going to quickly go over this. There will be some math,
385330	392010	but this is just to introduce the basic settings. The underlying framework to formulate the RL
392010	397030	problem and provide a framework to think about and reason about the RL problem is the Markov
397030	400850	decision process. Why do we call it the Markov decision process? Because it's based on the
400850	405470	Markovian assumption, or the Markov property, which says the current state completely characterizes
405470	412050	the state of the world. So, if you have the current state and you perform an action,
412050	419410	the next state is only going to depend or be a function of the previous state and nothing
419410	423350	before it. You don't need to know the history of the environment. All you need to know is what the
423350	431130	previous state has been to infer what the coming state is going to be. Now, in Markov decision
431130	439490	processes, a problem or setting is formulated by a tuple of five main parameters. One is the
439490	446830	set of possible states, S. Set of possible actions, A. Distribution of reward given state
446830	451090	action pair, or R. This distribution can also be a function. It doesn't necessarily have to
451090	456270	be probabilistic. It can be just a function that tells you if at a particular state, S,
456270	463410	I, an action AI is taken, then the reward is going to be, let's say, RI. Then there's transition
463410	474910	probability, or transition dynamics, P, which represents the dynamics of the environment.
474910	479810	Which state are we going to end up in if we are in a particular state and we perform a particular
479810	486050	action. And there's finally a discount factor which defines how myopic the agent is, how much
486050	496090	values, rewards that may occur further in the future, further down the line. Now, in general,
496090	503210	the RL setting is based on the following process. At time step T equals zero, in the beginning,
503210	510690	the environment samples initial state S0. Then, for T equals zero, this is a loop, until done,
510690	514910	the agent selects action AT based on some criteria. It can be completely random in the beginning and
514910	521730	then slowly becomes more targeted and more policy-driven. It performs some action,
521730	529430	the environment produces a reward signal, and the next state, and the agent receives those signals.
529430	535370	It may be partial, it may be incomplete, or it may be noisy, but the agent receives some signal
536390	544070	resulting from the transition to the new state ST plus one, and a reward that has emerged from
544070	551270	that transition and has resulted from that transition. Now, we define a new entity here
551270	557770	called pi. A policy pi is a function, or a distribution in the probabilistic case, that
557770	567370	maps from state to action. So, it tells the agent what action to perform given any state.
567870	573730	And the objective is to find the optimal policy pi star that maximizes the cumulative
573730	582270	discounted reward. We also call this return. So, cumulative, meaning the sum of rewards
582270	587850	in the entire duration of interaction. It can be one episode of a game, it can be throughout the
588430	595390	training period or training horizon, and it's discounted. This gamma is the discount factor
595390	600750	that defines how myopic the agent is, or how much it values events that occur further down the line.
600750	608070	It's typically a constant value between zero and one, and as you can see, as t increases,
608070	618770	this gamma to power t decreases. So, the value, the preference, or the observed value of something
618770	625890	that happens further down the line, which means with greater t, is going to decrease as t increases.
626650	632670	So, again, the objective is to find the optimal policy pi star that maximizes this sum here.
632670	639510	All right, we are going to quickly define two other definitions here. If we want to evaluate
639670	647010	a particular state in these settings, one approach is to measure the value of that state using the
647010	654170	value function. The value function of state s is the expected cumulative reward from following
654170	661430	the policy pi from this state. So, let's assume we already have a policy. What will be the expected
661430	669750	cumulative reward if you start in state s and keep following policy pi? Now, we also define
670310	679210	a Q value, or a Q function, which tells us how good is a particular action A if it's performed
679210	686650	in state s. In other words, if we are in state s, if the agent is in state s and performs action A,
686650	694990	the Q value of this setting is the expected cumulative reward from taking action A in state
694990	700530	s and then following the policy. Remember, the policy is a function that tells the agent what
700530	705910	action to perform given any state. When the agent performs action A in state s, it goes into a state
705910	713250	s prime, or the next state, and then we can use this policy function here to see what the action
713250	718090	should be, what action should the agent take at that state, which takes it to another state,
718090	722070	and then the policy is used to figure out the action to take in that state, and this goes on
722070	727650	until the termination of the episode or the horizon. Now that we are familiar with the value
727650	743230	function and the Q function, let's go a bit deeper. The policy pi, the value function,
743230	749070	or the reward model are all functions. We want to learn at least one of these from experience.
749070	755510	This is the essence of RL. If there are too many states, however, we cannot just
756670	762830	tabulate everything and try to experience every state and the corresponding result from that state
762830	767630	or that state and any action performed, because as the size of the problem, as the state space
767630	774070	and the action space increase in settings like, let's say, playing GTA 5 or a self-driving
774830	781090	navigation policy in the real world, these state spaces just explode. Dimensionality
781630	786710	is too high and it's just not feasible to store every possible state. In those cases,
786710	791190	we need to approximate. In general and traditionally, this is called RL with
791190	797610	function approximation. If function approximation is done using deep neural networks,
797610	804450	then we call the training setting or the RL setting deep reinforcement learning. The term
804450	811450	is relatively new. It was, I believe, introduced in late 2013, early 2014 by me and David Silver
811450	817190	and others at DeepMind in their DeepQ learning paper. However, the concept is not very new.
817190	823630	That said, there have been fantastic and even mind-blowing advances in this area in the past
823630	828690	three or four years. We'll talk about some of those as we move forward. A quick overview of
828690	834250	the taxonomy of different RL approaches and RL agents. Remember, we have value, function, policy,
834250	839690	and model. There are different approaches to solving the RL problem. Sometimes we just want to
840430	850990	find a policy directly. The approaches that respond or satisfy that need are called policy-based RL.
850990	856390	Sometimes we want to find a Q function or a V function first and then derive the policy
856390	862430	from those functions. These approaches are called value-based. Sometimes we want to first learn the
862430	867230	model of the environment and then solve the problem. These are called model-based approaches.
867230	870620	Sometimes we don't have the model and we don't want to learn the model explicitly,
871250	877150	and those are called model-free. When we're dealing with both value, function, and policy at the same
877150	881730	time, in other words, we have one agent learning the value function, another learning the policy,
881730	890910	and then contrasting those with each other in a kind of zero-sum setting. We call those actor-critic
891970	896430	RL approaches. As you can see, there are different approaches to different settings and different
896430	901910	problems. However, all of those can still be grounded on top of the Markov decision process
901910	907830	framework and the general solution approach we looked at before. One of the better-known
907830	915090	approaches to RL, which falls under the value-based approaches, is Q-learning. The objective in Q-
915090	922170	learning is to derive the optimal policy pi star based on optimal Q. So the optimal Q is one that
922170	935170	maximizes the value of each state and action. And from that point onward, with an iterative
935170	941890	formulation based on Bellman equations or dynamic programming, it becomes possible to find this
941890	950790	through bootstrapping and iterative re-estimation of the Q value. There are different approaches to
950790	958180	Q-learning in large state spaces or large action spaces where function approximation is required.
958690	965780	One such approach is to parametrize Q as s, a, and an estimation parameter theta.
967070	975170	If we solve this parametrized Q function by neural networks, where theta corresponds to the
975170	983450	neural network, the approach is called Q-networks. This solution approach was proposed
984950	992770	very early in 2000s, even earlier, but was perfected in some sense in 2013-2014 by
992770	1000250	DeepMind, by David Silver and his team, which resulted in the proposal of deep Q-networks or
1000250	1007270	QNs. So we're talking about deep networks. What does that mean? It means that these networks use
1007270	1013010	deep neural networks, like CNNs, which help with both function approximation
1013690	1021810	and also end-to-end feature learning. One of the advantages of deep learning is its
1021810	1027730	superior performance in learning feature representations, especially from images.
1028630	1035430	Now, for those of you who are more statistically oriented, you may have seen the term IID.
1035730	1042070	It means that the data are independent and identically
1042070	1050770	distributed. It means that one data point does not depend on another data point, and also
1050770	1057630	each data point is equally likely to occur. These do not hold for the RL settings.
1057630	1064170	A particular state, for example, is highly correlated with its previous state and action.
1064170	1069730	Why does this matter? Well, a lot of our supervised and deep learning approaches
1070570	1078550	are based on the assumption that the training data is IID. When this is not possible,
1078550	1086850	what happens is, in response to this problem, what happened was the DQN approach introduced
1086850	1092810	experience replay. It's like a bag of all data which is randomly sampled in each training
1092810	1098530	iteration to reduce the correlation, the sequential correlation, the temporal correlation
1098530	1106830	of data points, and also make it more likely for data to be evenly distributed. Also, to
1107490	1115450	reduce the effect of oscillation during training, DQN uses fixed parameters for a target network.
1115450	1122790	The target of optimization is fixed and is updated every few thousand iterations,
1122790	1126270	so that reduces the oscillation problem. And, of course, the rewards are normalized to
1126270	1133770	minus one to one to make sure the performance, the reward signals are bounded.
1134930	1140630	Now, now that we have a preliminary understanding of deep reinforcement learning and one of its
1140630	1146130	implementations, one of the parallel approaches called DQN, let's take a quick look at adversarial
1146130	1150090	machine learning. And I assume that by now, those of you who are not familiar with adversarial
1150090	1155810	examples have become, have been introduced to this concept. The idea here in adversarial examples,
1155810	1160590	we are now talking in the realm of, we are speaking in the context of image classifier
1160590	1167050	supervised learning. Let's say we have an image classifier trained on a set of images of pandas,
1167050	1173290	cats, and others to identify the object in those images. In the beginning, we pass the image of a
1173290	1179590	panda to the classifier, and it correctly classifies it as panda. Now, it's been demonstrated,
1179590	1187490	it's been actually established by now, that it is possible to make the classifier,
1187490	1191030	to induce incorrect classifications in deep learning models or machine learning models in
1191030	1198590	general by adding minute, minimal perturbations to the original image. As you can see,
1198590	1206710	it's almost impossible to detect or see any changes in this final image. These perturbations,
1206710	1215890	these pixel perturbations are very small. Now, this is one example of different attacks or
1215890	1220170	different vulnerabilities in the classical realm of adversarial machine learning, which is mostly
1220170	1225870	concerned with supervised learning and sometimes unsupervised learning. In general, the adversarial
1225870	1234350	objectives in AML or adversarial machine learning can be classified under the traditional CIA
1234970	1239010	triad, confidentiality, integrity, and availability. So, with respect to confidentiality
1239010	1245670	and privacy, an adversary may wish to target the confidentiality of the model parameters,
1245670	1253650	architecture, no intellectual property theft is an issue in larger models, or it may target the
1253650	1259310	privacy, the adversary may target the privacy of the training and test data. If medical, for example,
1259310	1264450	if medical records were used in the training data, there have been proof of concept attacks showing
1264450	1270990	that it is possible to infer whether a particular patient or the records pertaining to a particular
1270990	1276290	patient were used in the training data or not, or sometimes it's possible to reconstruct the
1276290	1281590	training data set by just having access to the model itself. And that is a major HIPAA violation.
1282070	1288750	In essence, it's a privacy violation. Also, with regards to integrity and availability,
1288750	1293490	the attacker may target the integrity of predictions or the outcome of the model,
1293490	1298810	the performance of the model. For example, adversarial examples are an attack on the
1298810	1304090	integrity of image classifiers or supervised machine learning models. And also an adversary
1304090	1308790	may target the availability of the system that is deploying machine learning, for example,
1308870	1315170	a facial recognition system or an autonomous navigation system in a driverless car. Now,
1316130	1323230	now that we know so much about adversarial machine learning and security vulnerabilities
1323230	1327250	of classical machine learning, supervised learning and unsupervised learning,
1328050	1334910	there is a major question. And it's whether deep RL is immune to those attacks.
1334910	1340590	Back in 2016, late 2016, when the research community was just beginning to start,
1340590	1345790	was just beginning to pay attention to both deep reinforcement learning and the issue of
1345790	1355450	adversarial examples. I came up with this question and decided to experiment with it a little to
1355450	1362210	find out whether deep RL can also be vulnerable to such attacks. I started from a simple observation.
1362210	1366790	The deep neural networks in DQN models and classifiers are both function approximators,
1366790	1371010	are both at training time, there are function approximators at test times or just functions.
1371810	1378230	And I came up with this hypothesis. If classifiers are vulnerable to adversarial examples,
1378230	1385250	then action value approximators of DQNs may also be vulnerable. All right. So I started to
1385450	1395230	set up an experiment where the aim was to perform adversarial attacks. The adversary's goal
1395850	1401110	was twofold. Test time attack to perturb the performance of target's learned policy. So
1401110	1405670	the target at this point is fully trained and is deployed in the environment and the adversary
1405670	1413810	wants to somehow cause the agent to perform incorrectly, to manipulate its policy.
1414890	1419450	What does the adversary know about the target? It knows the type of input to the target. For
1419450	1426930	example, it knows whether the target policy is looking at image data, text, audio, and such.
1426930	1431050	Why? Because it helps with estimating the architecture. For example, if it's images,
1431050	1438690	the adversary can come up with a good guess that the target architecture includes CNNs
1438690	1442890	or convolutional neural networks. I also assume that the adversary knows the reward function.
1442890	1447210	So the adversary may have access to the environment and if it's, for example, a game
1447210	1454130	environment, it knows what the scores are, how the scores are generated. What is not known? The
1454130	1461710	knowledge of target's neural network architecture is not known. So it's a black box attack. And also
1461710	1468150	the initial parameters, the initialization of the target neural network is also not known.
1468150	1471910	What is available to adversary in terms of actions? The adversary may perturb
1472550	1479610	the environment where the target performs. For example, it can change pixel values in a game
1479610	1488570	environment through a man-in-the-middle attack. I consider in this work two techniques for
1488570	1493370	perturbing that environment. One is the classical fast gradient sign method for generating adversarial
1493370	1497830	examples and the other is the Jacobian-based saliency map attack or the JSMA approach
1497830	1509490	introduced by Paperno in 2015, I believe. Now, in that experiment, I used the classical DQN
1509490	1518250	approach, the classical DQN architecture introduced by Nie in an Atari game, the game of Pong.
1518450	1523350	And this was all implemented in OpenAI Gym with TensorFlow. Back then, PyTorch wasn't really
1523350	1530730	and trained the agent against heuristic AI. And here's the initial proof of concept result.
1530730	1533930	This is for the white box attack. Of course, later on, I will show you the results for the
1533930	1544870	black box attack. And you can see that for FGSM and JSMA, regardless of how far along the training
1544870	1554230	the agent is, both policies, the policy for the Pong agent is highly vulnerable
1554230	1559630	to adversarial perturbations through simple techniques like FGSM and JSMA. You can see
1559630	1565870	that for JSMA, the success rate was 100% for all of the cases. For FGSM, it was slightly lower
1566540	1572150	and it was mostly because of the termination criteria and the perturbation threshold that
1572150	1578750	I had defined. But you can see, it can be observed that the policy can be very easily
1578750	1583690	manipulated through adversarial example attacks. So, at 100 random observations,
1583690	1588810	perturbed with FGSM and JSMA, the results were fed, the perturbed images were fed to
1588810	1595790	the trained neural network representing the agent's policy as test input, and then the success
1595790	1600930	rate was measured. Now, is this type of attack practical? Is this really something that we should
1600930	1611490	be worried about? So, a few years later, in 2018, Clark and his co-authors published the report of
1611990	1619770	a similar attack on an autonomous robot based on DQN policy using ultrasonic collision,
1619770	1626150	ultrasonic sensory input for collision avoidance. And they had shown that they can use adversarial
1626150	1636670	perturbations to manipulate the trajectory of the robot and make it follow a path defined or desired
1636670	1643030	by the adversary, not one that the robot itself wants to follow. There are more recent examples
1643030	1649050	of how this sort of attack on deep RL, or in general, attacks on deep RL, can be of concern.
1649530	1658130	One of the recent works by my graduate students is on attacks on automated trading algorithms
1658130	1662750	based on reinforcement learning. The paper is coming out in a couple months, so I can't go
1662750	1668630	into more details, but this is one of the more severe and urgent cases for security researchers
1668630	1674910	to consider. Deep RL is already being used by many major financial players and stock traders,
1674910	1678950	and it can be easily manipulated in the real world. There are other cases, of course,
1678950	1685750	but this is one of the examples that demonstrates the practicality and the applicability of this
1685750	1690370	attack to real world scenarios. Now, before we go further into different types of attacks,
1690370	1695490	let's develop a threat model for deep reinforcement learning. Again, the adversary's objectives can
1695490	1700890	follow those of the CIA triad. The adversary may wish to access the internal configurations,
1700890	1706510	like model parameter, reward function, policy, and such, to still the model for intellectual
1706510	1712270	property theft and such. There can be attack on integrity, which means compromising the desired
1712270	1718510	learning or enactment of the policy. There can be attacks on availability, which are essentially
1718510	1725010	compromises of the ability of the agent to perform training or actions when needed. Now,
1725010	1730930	let's look at the attack surface of the RL. This is the general diagram. This is the general block
1730930	1738790	diagram of a DRL agent or DRL system. Now, we have the agent. The agent typically has some memory
1738790	1743450	where it stores its experiences during training, and then those experiences help with function
1743450	1750770	approximation, data-driven policy learning, and such. And then there is an exploration controller,
1750770	1755210	which controls how the agent explores the environment during training. There's an
1755210	1760650	experience selector, how to select experiences from the bank of data or observations stored.
1760930	1767990	In this data set, there is an actuator, which then enacts the actions of the agent inside an
1767990	1773970	environment. The environment is connected back to this agent block through an observation channel,
1773970	1779390	observation of the state, and reward channel. And it's of no surprise to the more seasoned
1779390	1785910	security researcher and professional that all of these components can be a subject or target of
1785910	1794590	attack. As we go forward, we'll cover some examples of attacks that can occur on each of
1794590	1800190	these components. We've already seen an attack on the observation channel. There are attacks on the
1800190	1807930	reward channel, which we'll hopefully touch on. And there are attacks on the agents during training
1807930	1812290	and the actuator. Let me give you a quick example of the actuator attack. Let's assume we have a
1812290	1817970	robot, an actual robot, learning to navigate in an environment while avoiding obstacles.
1818130	1828590	If the robot commands or decides to, let's say, move the left wheel forward,
1829250	1833250	but there is some sort of obstacle in front of the left wheel, and the left wheel doesn't actually
1833250	1840390	move, then the actuator, the resulting data is going to be, the resulting observation is going
1840390	1845910	to be skewed, because the agent is going to assume that the actuation has happened and then
1845910	1852870	look at the changes in the observation and use that to retrain its policy, to optimize its policy
1852870	1857350	based on faulty data. What are the adversarial capabilities? Well, we first look at different
1857350	1864530	attack modes. The attacker can perform a passive attack where it's only observing the target,
1864530	1869830	it's not changing anything, or it can perform an active manipulation. In passive attacks,
1869830	1876770	the attacker can perform inverse reinforcement learning to learn about the reward function
1877430	1883250	of the agent, or later on we'll see it can perform imitation learning to steal the policy.
1883830	1890730	Active measures include attacks on the actuation, observation, or the reward channel. And attacks on
1890730	1896330	observation can be targeting their representation model, how the agent sees the environment,
1896330	1901150	or perturbing the transition dynamics, how the agent sees the changes in the environment.
1902050	1906290	Okay, going back to our initial proof of concept, remember our original goal was
1906290	1919590	to perform a black box attack. So, to achieve this objective, we introduce an approach based
1919590	1925730	on the transferability of adversarial examples. So, we create a second DQN, the adversary creates
1925830	1929630	a second DQN with similar architecture, but different initial parameters. When I say similar,
1929630	1932810	it doesn't necessarily have to be a match, it just needs to be a convolutional neural network
1932810	1938150	with some functional approximation techniques, but it doesn't need to have the same parameters
1938150	1947470	or the exact same architecture. And it trains that agent, that model, on the same environment.
1947470	1954110	The assumption is the adversary has access to that environment, and then uses the knowledge of that
1955750	1961790	architecture and the trained replica policy. It crafts adversarial examples the same way it did
1961790	1966250	for the white box case, and we know that many of those adversarial examples can transfer
1966250	1975930	to similar models, train on similar data, which also you can see applies to DQN policies as well.
1975930	1982370	And this is how we implemented a black box attack against deep RL policies. Now, what about training
1982370	1990690	time attacks? In the same paper we introduced the policy induction attack, where the attack
1990690	2000230	is of the adversarial example type against a DQN agent during training. Now, what are
2000230	2007550	the different steps in this attack? First, the adversary derives an adversarial policy
2007550	2014390	from the adversarial goal by training on the same environment where the target is going to
2014390	2022290	perform or be trained in. So, if the adversary wants to minimize the reward gained in the game
2022290	2030250	by the agent, then the goal, the optimization goal, is going to be the exact opposite of that of the
2030250	2035630	target policy. The target wants to maximize the reward and the adversary wants to minimize it.
2035630	2038730	Of course, there are different ways of formulating this adversarial goal.
2039790	2048090	Then, the adversary creates a replica of target's DQN and initializes it randomly. And then
2050810	2057510	comes the exploitation phase, where the attacker observes the current state and transitions in the
2057510	2064590	environment, then estimates best action according to the adversarial policy derived in step one.
2064590	2072210	Then, the attacker crafts perturbations to induce adversarial action based on the replica of
2072210	2077950	target's DQN. This is exactly the same as our black box test time attack. The attacker applies
2077950	2082690	the perturbation as a man in the middle in the observation channel. The perturbed input is
2082690	2089410	revealed to the target and the attacker waits for target's actions. And this is a loop. This loop can
2089410	2096870	go on until the training process of the target either converges to a suboptimal policy or up to
2097310	2104090	a certain number of iterations. This is a very rough plot. This is not smooth yet, but you can
2104090	2111670	see that the unperturbed agent moves towards... this is in the game of Pong still... moves towards
2111670	2121730	convergence to an optimal total sum of rewards, while the attack agent moves towards a convergence
2121730	2129450	to the minimum possible return of zero. It's getting closer and closer to zero, which indicates that
2129450	2135330	the training process of DQN and DeepRL in general can also be targeted through adversarial attacks.
2135330	2141450	Now, I'm going to introduce another type of training time attack. Again, this aims to
2141450	2148350	induce some form of misbehavior. We call this misbehavior addiction, and this is a follow-up to
2148450	2154070	a work I did with my colleague Roman Niempolski on psychopathological modeling of AI safety problems.
2154270	2161510	This is a proof of concept. We consider the game of Snake. Many of you probably remember Snake from
2162330	2170570	older Nokia phones. The DQN agent is a snake and is learning to play in this environment.
2170570	2176450	The attacker adds a drug seed with more instantaneous reward than the typical seed,
2176450	2182650	but it also results in more increase in the length of the tail. You can see that this can end up...
2184730	2196570	well, if the increase in the tail length is more than a certain amount, then a longer-tailed
2196570	2205170	snake is bound to eat its own tail sooner rather than later. So, we show that we actually derive
2205170	2213090	some theoretical closed-form solutions for what the additional reward and increase in the tail
2213090	2218610	length should be for addiction to emerge, meaning that the agent learns a more myopic policy instead
2218610	2224050	of the optimal policy, and you can see that it's actually possible to make the agent addicted
2225430	2231550	to the drug seed, as we call it, and this results in learning a sub-optimal policy.
2232390	2239510	And, of course, due to time limitations, I'm going to introduce only one more type of attack,
2239510	2246330	and it's that of targeting the confidentiality of a deep RL policy.
2247230	2251810	The problem here is, or the question is, is it possible to extract a deep RL policy from
2251810	2258830	observations of its actions? Why does this matter? Well, the security challenge posed by
2259570	2265510	this sort of action is, of course, model theft. A company like Google or, let's say, Uber or Waymo
2265510	2271870	may have spent billions or millions of dollars on coming up with a very accurate deep RL policy for
2271870	2277530	autonomous navigation, and if it can be stolen by an adversary, then the intellectual property
2277530	2284330	becomes worthless. And also, a stolen policy, an extracted policy, can be leveraged in integrity
2284330	2294230	attacks in the same way that we mounted black box attacks on DQN policies. So, let's see.
2294230	2299610	As it happens, a branch of reinforcement learning, or in general, one solution to the sequential
2299610	2304930	decision-making problem, is not in the RL domain, but in the supervised learning domain, and it's
2304930	2309670	called imitation learning. Imitation learning is the supervised learning of policies from observed
2309670	2316050	behavior of an expert, and by behavior, I mean state action behavior. What is the policy of an expert?
2317490	2327390	Based on this concept, in 2018, Hester et al. proposed DQFD, or DQ Learning from Demonstrations,
2327390	2338490	which is DQN, where the initial training is done based on observed data using deep learning. So,
2338490	2346150	they have data from human players playing a certain game, or human performance doing a
2346150	2351770	certain task that they want the agent to learn. The initial step of training for a DQFD agent is
2351770	2359810	supervised learning on observed data, and then it starts building on top of it through reinforcement
2359810	2364730	learning approaches, and it was shown that it can result in faster convergence, better sample
2364730	2372330	complexity, and sometimes more interesting and robust policies. As security researchers,
2372330	2378290	you can probably see where this is going. This wonderful algorithm, DQFD, can also be used to
2378290	2385390	replicate policies. Instead of applying it on observed data collected from human performance,
2385390	2391810	it can be applied on observed data from a target policy. So, here's a proof of concept attack the
2391810	2399430	procedure. The attacker observes and records n interactions of the SARSA type, state action,
2399430	2407010	the next state, and the reward based on this transition of the target agent in a particular
2407010	2413170	environment, and then the attacker applies DQFD to learn an imitation of the target policy and
2413170	2422650	Q function. Now, at this point, the attacker may either just go away and sell the extracted policy,
2422650	2428450	or it may decide to target it using different adversarial perturbation attacks, some of which
2428450	2435370	we've covered so far in this talk. So, as a proof of concept, we consider a slightly less complex
2435370	2441730	environment data cart poll, where the objective is to stabilize this poll on this cart by moving
2441730	2448970	the cart to right and left. The reason for choosing a simple environment is merely economical,
2449990	2457310	because we didn't want the experiments to take days or weeks. We start with a simple case,
2457310	2463990	and we consider different types of policies. DQN with prioritized replay, an enhanced version of
2463990	2470910	the classical DQN proximal, policy optimization, and asynchronous actor-critic. And, of course,
2470910	2478730	we also train an adversarial RL agent, a DQN agent, whose objective is to incur maximum loss
2478730	2484330	of reward, or in other terms, in more technical terms, maximize the regret of its target.
2485390	2489590	And here are the results. First, with regards to replication progress, based on only 5,000
2489590	2496650	demonstrations, 5,000 state action, next state reward observations, we see that all three
2497450	2506670	policies are almost exactly replicated. We can see convergence to the optimal performance
2507350	2514430	of those policies in the environment. And then we perform adversarial training. We
2514430	2521610	train an adversarial RL agent to attack and maximize the regret of those policies,
2521610	2529330	and you can see that this can also be easily achieved within very few iterations of training
2529330	2538550	in CardPol. I believe we can see that for PPO2, which is a somewhat robust deep RL
2539510	2545150	algorithm or approach, it's possible to incur maximum damage or find a policy that incurs
2545150	2553310	maximum regret on the target within 60,000 iterations, which is a relatively low amount.
2553310	2564210	Now, what about defenses? So, of course, similar to adversarial examples, one approach or one
2564210	2571550	technique for reducing the impact of adversarial example attacks is through regularization. And
2571550	2577130	one common type of regularization in supervised learning for mitigating adversarial example
2577130	2583410	attacks is adversarial training, training the model on adversarially perturbed samples to
2583410	2589870	make sure that it sees different perturbations of the same image and knows that all of those
2589870	2596110	results in the same label, in the correct label. This is called essentially data augmentation in
2596110	2602990	general as a regularization technique. So in 2017, when I was just starting to look into this
2603790	2610450	problem or this domain, I had gone through the adversarial training literature and thought that
2610450	2616490	the same may also hold true for deep RL. I made a hypothesis, actually two. One is with regards
2616490	2622190	to recovery. If training time attacks are not contiguous, if not all of the observations are
2622190	2629010	perturbed, then DRL adapts to the environment and adjusts the policy to overcome the attacks.
2629010	2635750	This is training time. And with regards to robustness, I made another hypothesis. Such policies,
2635750	2644650	policies trained under attack are more robust to test time attacks. And this particular
2644650	2648910	investigation is published in a paper titled whatever does not kill deep reinforcement
2648910	2656370	learning makes it stronger. So similar as before, we are looking at DQN in Atari games, Breakout,
2656370	2666230	Enduro, and Pong. Now, the way I designed the experiment was based on a probability of attack.
2666230	2673090	So as an attacker, I assign a certain probability for each state, for each observation during
2673090	2679630	training time to be perturbed. I perform experiments, different experiments with
2679630	2685670	different values of this P attack, 20%, 40%, 80%, and 1, which means contiguous attack.
2685670	2694420	And it's interesting to see that for values of P less than 50%, the agent actually recovers.
2695260	2703460	But for values greater than 50%, the training process plummets and either does not converge or
2703460	2716340	converges to a very, very low mean return value or mean total reward value. I later on publish a
2716340	2722040	theoretical analysis of why this happens. It's available in my PhD dissertation, which I'll
2722040	2728660	reference in the final slide. Also, it was interesting to see that the robustness is also,
2728660	2733300	the robustness hypothesis is also true. You can see here that after training,
2733300	2740780	if we attack the test time policy with probability of 1, the plane or vanilla policy,
2741000	2748560	a policy that was not trained adversarially, performs very poorly. However, policies trained
2748560	2755700	under adversarial attacks with probability 0.2 or 20% and 0.4 perform really well. For 80%,
2756320	2761360	it's even for policies trained at 80%, as you can see, the policy itself is already performing
2761360	2768880	very poorly. It's surprising to see that at P equals 1, the performance gets slightly better.
2768880	2774960	It's comparable with 40%. To this day, I'm not entirely sure why this happened. I've repeated
2774960	2778320	the experiments a number of times and still get the same result. I still don't know why this
2778320	2784240	happened. This is one of the interesting problems that we are looking at right now
2784800	2790160	as a research group. Another defense that we introduce is based on parameter space noise.
2790160	2796960	This is very much like dropout. The idea of parameter space noise was introduced in 2017,
2796960	2803080	I believe, independently by Plamper et al and Fortunato et al. And the idea here is, again,
2803080	2807780	similar to dropout, to introduce zero mean random noise to the learnable parameters of neural
2807780	2813120	network in deep RL to enhance exploration and convergence in deep RL benchmarks.
2813480	2822880	Now, in another paper in 2018, we investigate whether this approach can be used to mitigate
2822880	2830280	the impact or severity of policy manipulation attacks on DQN, and it was shown that it
2830280	2843380	actually performs very well compared to vanilla or classical DQN. And these are the training time
2843380	2849840	results. You can see that if noisy net is used, if parameter space noise is used, the performance
2849840	2855320	degradation for all environments is at a much lower slope, at a much lower rate than the
2855320	2866100	vanilla architecture. Finally, to have proposed a solution for the policy extraction problem,
2866980	2873940	I, along with William Hsu at K-State, came up with the idea of watermarking DRL policies.
2873940	2879860	Watermarking has already been introduced in deep learning in general. The idea is to come up with
2879960	2885980	a unique signature that is both difficult to remove and does not impact the performance of
2885980	2893180	the policy itself, or the model itself, but it still provides a unique signature, proof that
2893640	2902980	a model is the same as another model, or is a replica of the suspected model. So we introduce
2902980	2907560	an interesting watermarking procedure. I know it's arrogant of me to call my own work interesting,
2907560	2911420	but I still get excited when I think about the moment I came up with this idea. The idea is to
2911420	2917460	create a second environment whose state space is this drawing from the main environment. So,
2917460	2923140	create an environment. If you're training an agent to play a game, let's create another environment
2923140	2931500	which has no states. None of these states are the same as the original training environment,
2931500	2937020	or deployment environment of the agent. But the dimensionality of the states are the same. So,
2937020	2942640	if each state in the original environment is represented by, let's say, three values,
2943760	2948740	three features, then each state in the second environment is also represented by three
2948740	2954400	features. It doesn't really matter what the second environment looks like. It's just some
2954400	2961380	other environment that the agent may interact with. And then, we craft the transition dynamics
2961380	2966080	and reward procedure for the second environment such that the optimal policy follows a looping
2966080	2973240	trajectory. So, an optimal policy for an agent trained in the second environment is going to be
2973240	2977120	one that follows a loop, goes through, let's say, state one, then state two, state three,
2977120	2983160	and then goes back to state one. During training, what happens is, we periodically alternate
2983160	2990040	between the two environments. So, let's say, at every N iterations of the training process,
2990040	2995540	we take our RL agent from the original environment, we drop it in the second environment,
2995540	3000220	we train it for a few iterations, and then bring it back to the original environment.
3000460	3006920	Now, once trained, if we want to examine the authenticity or whether a policy is copied or not,
3006920	3013260	we apply the policy in the second environment and measure the total reward. Here's the experimental
3013260	3024340	setup. Again, we are working with CardPool. So, the watermarking environment is defined with
3024340	3032880	five states, states one to four, plus a terminal state, which should never be reached if the
3032880	3039980	policy is optimal in this environment. And none of these states, as represented here,
3039980	3044640	can be found in the original, can occur in the original CardPool environment. These are
3044640	3049780	all highly impossible, not highly, definitely and absolutely impossible to occur in the original
3049780	3058060	environment. As for the transition dynamics, this is how we've defined it. Let a particular state be
3058060	3070400	A0 and A1. If we are, I'm sorry, actions. Actions A0 and A1. If the agent is in state
3071360	3082220	I and performs action I modulo two, if I is even, this is going to be action zero. If I is odd,
3082220	3090520	this is going to be action one. Then the next state is going to be state I star or I modulo
3090520	3097460	four plus one. So, and it receives a reward of one. If the agent performs any other actions,
3097460	3103080	instead of going from one to two, it performs any other action, it will immediately go to the
3103080	3107560	terminal state and receives a reward of zero. So, the optimal trajectory is state one to two to three
3107560	3113700	to four back to state one and so on. All right. Let's see how it works. Let's look at the test
3113700	3118600	time performance comparison of watermarked and nominal, non-watermarked policies. The
3118600	3126620	watermarked policy performs exactly as well as unwatermarked policies. It reaches the
3127500	3137840	optimal or best performance of 500. And when it's applied to the watermark environment,
3137840	3143800	the second environment, it also performs optimally. It gets the maximum reward possible.
3144180	3152620	However, you can see that if we try to apply the non-watermarked policies to the watermark
3152620	3160560	environment, we'll see very, very small values of score, the total reward. So, you can see that
3160560	3166520	it's possible to determine whether a policy is authentic or not or whether a policy is an exact
3166520	3175000	copy of another policy by just applying it to a second environment and see whether it performs
3175000	3180260	optimally or not. Now, there's there are many other things that I really wanted to touch upon
3180260	3186360	in this talk, but unfortunately we are a little short in time. For practitioners, it may be
3186360	3191100	of interest to have some way of benchmarking or evaluating the resilience and robustness
3191100	3196260	of policies and compare different policies, different approaches with regards to the
3196260	3202780	resilience and robustness. Some of my work already introduces or proposes an RL-based
3202780	3210280	approach to perform this evaluation and benchmarking. I've also done some work on
3211180	3215480	investigating the impact of hyperparameter choices on resilience and robustness of
3215480	3220720	DQNs in particular, but also other model-free and actor-critic approaches.
3221920	3226860	This can be very helpful to those who want to engineer and design the new RL
3226860	3233680	agents to be deployed in critical environments. Also, something that I wanted to mention, but
3233680	3242020	unfortunately I don't have time to do so, is that adversarial training is not a silver bullet. It's
3242020	3247160	not an answer to all of the problems in DQN agents. There are certain limitations
3249200	3254340	for robustness and resilience obtained from adversarial training of DQN agents.
3254340	3261720	And also adversarial training is very costly in general, especially when it comes to real-world
3261720	3269480	scenarios, real-world environments and actions. And some of my recent work is focused on improving
3269480	3275920	the sample efficiency and computational cost of adversarial training via a new exploration
3275920	3281240	mechanism called adversarially-guided exploration, or AGE. All of this work can be found
3281940	3288360	in my Ph.D. dissertation, which bears the same name as this talk, False Narcotic Star Security
3288360	3293480	and Deep Reinforcement Learning. You can find it if you search my name on Google Scholar.
3293940	3298860	If you're interested, of course, all of these are published in separate papers in slightly
3299560	3308680	more details under the same titles. And finally, some of the open areas of research in this domain.
3309400	3315400	With regards to training time resilience and robustness, not much has been done with regards
3315400	3320320	to policy search and actor-critic methods, as well as model-based and hybrid methods. Of course,
3320320	3323880	when we talk about model-based, there are some approaches from optimal control theory and
3325620	3333140	approximate dynamic programming that may be applied here, but very few have looked at this
3333140	3337760	problem from a security point of view. So if you're interested, this is one of the areas that
3337760	3344260	is in dire need of a security-oriented investigation. As for mitigation of policy
3344260	3347500	replication, one of the ideas that my research group is currently working on is constrained
3347500	3354280	randomization of policy. So randomize the policy such that the replication through techniques like
3354280	3358660	imitation learning becomes more costly, more samples, more observations will be required,
3358660	3366100	while preserving the performance of the policy. There's almost no work done in multi-agent
3366100	3370600	settings. Of course, adversarial reinforcement learning has been investigated in settings where
3370600	3375600	there are zero-sum agents, but not really where there is an external adversary or adversaries
3375600	3381260	trying to exploit the inner workings of the agents, the RL components of the agent.
3381960	3387420	One more thing that is of note is the importance of discounting. The addiction problem that I
3387420	3394840	demonstrated earlier in the snake agent is mostly due to the constrained discounting solution. For
3394840	3397460	those of us who come from a reinforcement learning background, you're familiar with
3397460	3403060	the basics of reinforcement learning. You probably know that the discount factor is typically chosen
3403060	3410040	to be 0.99 or something in the same ballpark and is left the same. It's treated as a constant
3410040	3416460	throughout the training process. But this is very far from how our brain works and very far from
3417860	3423060	the optimal approach or accurate approach to discounting. Our research group has recently
3423060	3428140	started looking into this problem and is working on developing adaptive discounting
3429320	3436940	solutions to enhance the resilience and robustness of RL agents, particularly
3436940	3441590	deep RL agents in complex environments for AI safety and security purposes.
3441960	3446200	Now, of course, there are naturally inspired approaches that can be looked at. For example,
3446200	3453400	approaches coming from, let's say, TD lambda models or dopamine models of psychopathological
3453400	3458200	problems or neurological problems and the solutions prescribed to those, as well as
3458200	3463580	approaches in social sciences, which may help with the security problems arising in multi-agent
3463580	3471600	RL settings. Very well, thank you very much. And I believe at this time I should be available for
3471600	3473140	your questions.