Unifying Count-Based Exploration and Intrinsic Motivation

By Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, R´emi Munos. 2016

1,688 citations recorded by the source corpus.

We consider an agent's uncertainty about its environment and the problem of generalizing this uncertainty across observations. Specifically, we focus on the problem of exploration in non-tabular reinforcement learning. Drawing inspiration from the intrinsic motivation literature, we use density models to measure uncertainty, and propose a novel algorithm for deriving a pseudo-count from an arbitrary density model. This technique enables us to generalize count-based exploration algorithms to the non-tabular case. We apply our ideas to Atari 2600 games, providing sensible pseudo-counts from raw pixels. We transform these pseudo-counts into intrinsic rewards and obtain significantly improved exploration in a number of hard games, including the infamously difficult Montezuma's Revenge.

Subfields: Artificial Intelligence, Economics and Econometrics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Reinforcement Learning

Citations in the corpus

Image Transformer (2018, methodology): “…the comparatively simple and stable training regime of directly maximizing log-likelihood, this enables the straightforward application of these models in problems such as image compression (van den Oord & Schrauwen, 2014) and probabilistic planning and exploration (Bellemare et al., 2016).”
Conditional Image Generation with PixelCNN Decoders (2016, methodology): “The basic idea of the architecture is to use autoregressive connections to model images pixel by pixel, decomposing the joint image distribution as a product of conditionals.”
Reward learning from human preferences and demonstrations in Atari (2018, background): “…actions and end states (Gregor et al., 2016), state prediction error or surprise (Pathak et al., 2017), state visit counts (Storck et al., 1995; Bellemare et al., 2016), distinguishability to a decoder (Eysenbach et al., 2018), or empowerment (Salge et al., 2014), which is also related to…”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, background): “, 2015), count based exploration (Bellemare et al., 2016), information theoretic notions of curiosity (Singh et al.”
Surprising Negative Results for Generative Adversarial Tree Search (2018, methodology): “For instance, for value learning (i): one can use Count-based methods [12].”
Rainbow: Combining Improvements in Deep Reinforcement Learning (2017, methodology): “2016), intrinsic motivation (Stadie, Levine, and Abbeel 2015) and count-based exploration (Bellemare et al. 2016).”
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, background): “Finally, intrinsically generated rewards (Bellemare et al., 2016b) may also provide a useful learning signal which the agent can use to build a representation.”
Neural Approaches to Conversational AI (2018, neutral): “, 2017), exploration when parametric models like neural networks are used is an active research topic (Bellemare et al., 2016; Osband et al., 2016; Houthooft et al., 2016; Jiang et al., 2017).”
Continual Lifelong Learning with Neural Networks: A Review (2018, background): “2017, Bellemare et al. 2016, Kulkarni et al. 2016). In a scenario with very sparse extrinsic rewards, curiosity-driven exploration provides intrinsic reward signals that enable the agent to autonomously and progressively learn tasks of increasing complexity. Pathak et al. (2017) proposed an approach”
Reinforcement Learning in Healthcare: A Survey (2019, methodology): “Apart from this kind of heuristic methods, there are several more advanced exploration strategies such as intrinsic motivation [17] and count-based methods [215].”
Automated Curriculum Learning for Neural Networks (2017, methodology): “…instantaneous change in loss for a sample x , before and after training on x : When p θ is a Bayesian mixture model, prediction gain upper bounds the model’s information gain (Bellemare et al., 2016), and is therefore closely related to the Bayesian pre-cept that learning is a change in posterior.”
Self-Imitation Learning (2018, methodology): “We observed that (Bellemare et al., 2016) and Reactor-PixelCNN respectively (Ostrovski et al., 2017).”
Information-theoretic Online Memory Selection for Continual Learning (2022, neutral): “Specifically, prediction gain (PG) (Bellemare et al., 2016; Graves et al., 2017) picks the curriculum according to L((x?, y?);θ)−L((x?, y?);θ), where θ′ is the parameter after the network updates θ for one step using (x?, y?).”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “This is generally achieved by defining a heuristic based on some measurement of surprise, information gain, compression, or empowerment (Schmidhuber, 1991; Klyubin et al., 2005; Kaplan and Oudeyer, 2006; Schmidhuber, 2008; Frank et al., 2014; Mohamed and Rezende, 2015; Bellemare et al., 2016; Houtho”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, positive): “In the direction of curiosity-driven exploration, recent work has used state novelty to reward an agent [5, 44, 33].”
RoboNet: Large-Scale Multi-Robot Learning (2019, methodology): “This includes demonstration data, data from modern exploration methods that scale to pixel observations [51, 52, 53], and task-driven data from running reinforcement learning on particular tasks.”
Safe Policy Improvement with Soft Baseline Bootstrapping (2019, methodology): “We recall that the computation of the policy improvement step relies on the estimates of an error function eP , which may, for instance, be indirectly inferred from pseudo-counts ÑD(x, a) (Bellemare et al., 2016; Fox et al., 2018; Burda et al., 2019).”
Safe Policy Improvement with Baseline Bootstrapping (2017, methodology): “Note that computing the SPIBB targets requires determining the bootstrapped set B , which relies on an estimate of the state-action counts (cid:101) N D ( x, a ) , also called pseudo-counts (Bellemare et al., 2016; Fox et al., 2018; Burda et al., 2019).”