Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning

By Joshua Achiam, S. Sastry. 2017

251 citations recorded by the source corpus.

Exploration in complex domains is a key challenge in reinforcement learning, especially for tasks with very sparse rewards. Recent successes in deep reinforcement learning have been achieved mostly using simple heuristic exploration strategies such as $\epsilon$-greedy action selection or Gaussian control noise, but there are many tasks where these methods are insufficient to make any learning progress. Here, we consider more complex heuristics: efficient and scalable exploration strategies that maximize a notion of an agent's surprise about its experiences via intrinsic motivation. We propose to learn a model of the MDP transition probabilities concurrently with the policy, and to form intrinsic rewards that approximate the KL-divergence of the true transition probabilities from the learned model. One of our approximations results in using surprisal as intrinsic motivation, while the other gives the $k$-step learning progress. We show that our incentives enable agents to succeed in a wide range of environments with high-dimensional state spaces and very sparse rewards, including continuous control tasks and games in the Atari RAM domain, outperforming several other heuristic exploration techniques.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, positive): “Inspired by the success of intrinsic motivation methods—in particular, curiosity [14, 2, 75, 4]—we leverage the readily available action reconstruction errors from the generative decoder as intrinsic reward signals to boost exploration when training π g .”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “…generally achieved by defining a heuristic based on some measurement of surprise, information gain, compression, or empowerment (Schmidhuber, 1991; Achiam and Sastry, 2017; Schmidhuber, 1990; Klyubin, Polani, and Nehaniv, 2005; Kaplan and Oudeyer, 2006; Schmidhuber, 2008; Frank, Leitner,…”
Exploration by Random Network Distillation (2018, methodology): “Another class of exploration methods rely on errors in predicting dynamics (Schmidhuber, 1991b; Stadie et al., 2015; Achiam & Sastry, 2017; Pathak et al., 2017; Burda et al., 2018).”
Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “Another paradigm are transition-based methods, in which agents are rewarded for high prediction error in a learned forward or inverse dynamics model [42, 43, 44, 45, 46, 47, 48, 49].”
Parameter Space Noise for Exploration (2017, background): “In this section, we evaluate whether parameter noise enables existing RL algorithms to learn on environments with very sparse rewards, where uncorrelated action noise generally fails [7, 4].”
Learning to Play with Intrinsically-Motivated Self-Aware Agents (2018, background): “Achiam and Sastry [1] formulate surprise for intrinsic motivation as the KL-divergence of the true transition probabilities from learned model probabilities.”
Large-Scale Study of Curiosity-Driven Learning (2018, background): “Given a transition tuple { x t , x t +1 , a t } , the exploration reward is then deﬁned as r t = − log p ( φ ( x t +1 ) | x t , a t ) , also called the surprisal [2].”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, background): “To understand detachment, we must first consider that intrinsic reward is nearly always a consumable resource: a curious agent is curious about states to the extent that it has not often visited them (similar arguments apply for surprise, novelty, or prediction-error seeking agents [4, 14–16]).”
EX2: Exploration with Exemplar Models for Deep Reinforcement Learning (2017, methodology): “These methods usually use random, undirected exploration strategies, such as (cid:15) -greedy strategies (Mnih et al., 2015) or Gaussian noise (Lillicrap et al., 2015), which can struggle under sparse or delayed rewards.”
BeBold: Exploration Beyond the Boundary of Explored Regions (2020, methodology): “Later, to reduce computation, a deterministic approach has been adopted (Achiam and Sastry, 2017).”
World Discovery Models (2019, unknown): “…inputs from cognitive neuroscience, information-seeking algorithms have received signiﬁcant attention to improve the exploration capability of artiﬁcial learning agents (Schmid-huber, 1991b; Houthooft et al., 2016; Achiam & Sastry, 2017; Pathak et al., 2017; Burda et al., 2018; Shyam et al., 2018).”
Reinforcement learning for control: Performance, stability, and deep approximators (2018, background): “, 2016a), on the temporal difference error (Achiam and Sastry, 2017) or on the prediction accuracy of a simultaneously learned dynamics model (Schmidhuber, 1991).”
Unsupervised Real-Time Control Through Variational Empowerment (2017, unknown): “It has also been shown that a notion of surprise , quantifying the degree to which an agent is observing something unexpected, can improve reinforcement learning [Achiam and Sastry, 2017, Itti and Baldi, 2006].”
Improving Intrinsic Exploration with Language Abstractions (2022, methodology): “More recently, deep neural networks have been used to measure novelty with changes in state representations [6, 40, 54] or prediction errors in world models [1, 36, 46].”
Planning to Explore via Self-Supervised World Models (2020, background): “These approaches can also be understood as maximizing the agent’s surprise (Schmidhuber, 1991a; Achiam & Sastry, 2017).”
SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments (2021, background): “…et al., 2017), through state visitation counts (Bellemare et al., 2016), via surprise maximization (Achiam & Sastry, 2017b; Schmidhuber, 1991; Sun et al., 2011), and through other novelty-based reward bonuses (Lehman & Stanley, 2011; Achiam & Sastry, 2017a; Burda et al., 2018a; Kim et al., 2019).”
Continuous control for robot based on deep reinforcement learning (2019, background): “strategy by maximizing the agent’s surprise for its experiences [27], which is known as the intrinsic reward.”
Mega-Reward: Achieving Human-Level Play without Extrinsic Rewards (2019, methodology): “…novelty-driven (Bellemare et al. 2016; Martin et al. 2017; Ostrovski et al. 2017; Tang et al. 2017), prediction-errorbased novelty-driven (Achiam and Sastry 2017; Pathak et al. 2017; Burda et al. 2018; 2019), stochasticity-driven (Florensa, Duan, and Abbeel 2017), and diversity-driven…”