Combating Deep Reinforcement Learning's Sisyphean Curse with Intrinsic Fear

By Zachary Chase Lipton, Jianfeng Gao, Lihong Li, Jianshu Chen, Li Deng. 2016

33 citations recorded by the source corpus.

To use deep reinforcement learning in the wild, we might hope for an agent that can avoid catastrophic mistakes. Unfortunately, even in simple environments, the popular deep Q-network (DQN) algorithm is doomed by a Sisyphean curse. Owing to the use of function approximation, these agents eventually forget experiences as they become exceedingly unlikely under a new policy. Consequently, for as long as they continue to train, DQNs may periodically relive catastrophic mistakes. Many real-world environments where people might be injured exhibit a special structure. We know a priori that catastrophes are not only bad, but that agents need not ever get near to a catastrophe state. In this paper, we exploit this structure to learn a reward-shaping that accelerates learning and guards oscillating policies against repeated catastrophes. First, we demonstrate unacceptable performance of DQNs on two toy problems. We then introduce intrinsic fear, a new method that mitigates these problems by avoiding dangerous states. Our approach incorporates a second model trained via supervised learning to predict the probability of catastrophe within a short number of steps. This score then acts to penalize the Q-learning objective, shaping the reward function away from catastrophic states.

Subfields: Artificial Intelligence, Reinforcement Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Safety, Robustness & Interpretability

Citations in the corpus

Troubling Trends in Machine Learning Scholarship (2018, background): “ without the need to argue for them. This often manifests in anthropomorphic characterizations of tasks (reading comprehension [31] and music composition [59]) and techniques (curiosity [66] and fear [48]). A number of papers name components of proposed models in a manner suggestive of human cogniti”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “ple, is the solution adopted in many works, whether a value based method [62, 63, 64, 65] or a policy gradient method [71]. In this regard, it is an open question to consider new DRL ideas for the ER [105, 106, 107] and how those would fare in a MDRL setting. Centralized learning with decentralized ”
Penalizing Side Effects using Stepwise Relative Reachability (2018, background): “, 2015], using intrinsic motivation [Lipton et al., 2016], preserving reversibility [Moldovan and Abbeel, 2012; Eysenbach et al.”
Avoiding Side Effects By Considering Future Tasks (2020, background): “Safe exploration methods that penalize risk [4] or use intrinsic motivation [14] help the agent avoid side effects that result in lower reward (such as getting trapped or damaged), but do not discourage the agent from damaging the environment in ways that are not penalized by the reward function (e.”
AGI Safety Literature Review (2018, background): “ an action to be too dangerous. Using this technique, they manage to avoid all catastrophes in simple settings, and a signicant fraction in more complex environments. A similar ideas was explored by Lipton et al. (2016). Instead of using human-generated labels, their catastrophe detector was trained”
Sample-Efficient Deep RL with Generative Adversarial Tree Search (2018, background): “In real-world applications, however, collecting samples takes considerable time and effort, e.g. robotics (Levine et al., 2016) and dialogue systems (Lipton et al., 2016).”
Measuring and avoiding side effects using relative reachability (2018, background): “, 2018), using intrinsic motivation (Lipton et al., 2016), penalizing risk (Chow et al.”