Count-Based Exploration with Neural Density Models

By Georg Ostrovski, Marc G. Bellemare, Aäron van den Oord, R´emi Munos. 2017

695 citations recorded by the source corpus.

Bellemare et al. (2016) introduced the notion of a pseudo-count, derived from a density model, to generalize count-based exploration to non-tabular reinforcement learning. This pseudo-count was used to generate an exploration bonus for a DQN agent and combined with a mixed Monte Carlo update was sufficient to achieve state of the art on the Atari 2600 game Montezuma's Revenge. We consider two questions left open by their work: First, how important is the quality of the density model for exploration? Second, what role does the Monte Carlo update play in exploration? We answer the first question by demonstrating the use of PixelCNN, an advanced neural density model for images, to supply a pseudo-count. In particular, we examine the intrinsic difficulties in adapting Bellemare et al.'s approach when assumptions about the model are violated. The result is a more practical and general algorithm requiring no special apparatus. We combine PixelCNN pseudo-counts with different agent architectures to dramatically improve the state of the art on several hard Atari games. One surprising finding is that the mixed Monte Carlo update is a powerful facilitator of exploration in the sparsest of settings, including Montezuma's Revenge.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Generative Modeling by Estimating Gradients of the Data Distribution (2019, background): “To list a few, they have been used to generate high-fidelity images [26, 6], synthesize realistic speech and music fragments [58], improve the performance of semi-supervised learning [28, 10], detect adversarial examples and other anomalous data [54], imitation learning [22], and explore promising s”
Surprising Negative Results for Generative Adversarial Tree Search (2018, methodology): “[41], we deploy Wasserstein distance as a notion of count Ñ(x, a).”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, positive): “, 2018): each experiment is repeated with 3 seeds (Wu et al., 2017; Ostrovski et al., 2017), 10 additional validation seeds are used to pick the best model which is then tested on 100 testing seeds.”
Self-Imitation Learning (2018, methodology): “We observed that (Bellemare et al., 2016) and Reactor-PixelCNN respectively (Ostrovski et al., 2017).”
Deep Q-learning From Demonstrations (2017, methodology): “DQN with count-based exploration (Ostrovski et al. 2017) is designed for and achieves the best results on the most difficult exploration games.”
Safe Policy Improvement with Baseline Bootstrapping (2017, methodology): “Future work includes developing model-free versions of our algorithms in order to ease their use in continuous state MDPs and complex real-world applications, with state representation approximation, using density networks [36, 35] to compute pseudo-counts [3], in a similar way to that of optimism-m”
The Uncertainty Bellman Equation and Exploration (2017, background): “Recent work has further connected these approaches through the notion of ‘pseudo-count’ (Bellemare et al., 2016; Ostrovski et al., 2017), a generalization of the number of visits to a state and action.”
Exploration by Random Network Distillation (2018, background): “…are a natural and effective way to do exploration (Strehl & Littman, 2008) and a lot of work has studied how to tractably generalize count bonuses to large state spaces (Bellemare et al., 2016; Fu et al., 2017; Ostrovski et al., 2017; Tang et al., 2017; Machado et al., 2018; Fox et al., 2018).”
Variational Bayesian Reinforcement Learning with Regret Bounds (2018, methodology): “In the non-tabular case methods that estimate the pseudo-count have proven successful in some instances, indicating a possibility of combining K-learning with pseudo-counts in a deep RL setting [94, 11, 97, 80, 13, 79].”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, neutral): “Note that our penalization of exploration stands in direct contrast to online RL methods that specifically incentivize exploration [5, 44].”
Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “One example is count-based exploration, where agents are incentivized to reach infrequently visited states by maintaining state visitation counts [37, 38, 39] or density estimators [40, 41].”
Hindsight Experience Replay (2017, methodology): “Notice that using techniques for improving exploration (e.g. VIME (Houthooft et al., 2016), count-based exploration (Ostrovski et al., 2017) or bootstrapped DQN (Osband et al., 2016)) does not help here because the real problem is not in lack of diversity of states being visited, rather it is…”
Learning from Demonstrations for Real World Reinforcement Learning (2017, methodology): “DQN with count-based exploration Ostrovski et al. (2017) is designed for and achieves the best results on the most difficult exploration games.”
Randomized Prior Functions for Deep Reinforcement Learning (2018, background): “Crucially, this generalization is not learned from the task [53].”
Parameter Space Noise for Exploration (2017, background): “…plethora of methods have been proposed to tackle this challenge in high-dimensional and/or continuous-action MDPs, they often rely on complex additional structures such as counting tables [1], density modeling of the state space [2], learned dynamics models [3–5], or self-supervised curiosity [6].”
Large-Scale Study of Curiosity-Driven Learning (2018, background): “Smoothed versions of state visitation counts can be used for intrinsic rewards [3, 9, 24, 47].”
Efficient Exploration Through Bayesian Deep Q-Networks (2018, methodology): “We also desired to report the scores of count-based method (Ostrovski et al., 2017), but unfortunately there is no table of score in that paper in order to provide them here.”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, background): “Montezuma’s Revenge has become an important benchmark for exploration algorithms (including intrinsic motivation algorithms) [4, 16, 32–39] because precise sequences of hundreds of actions must be taken in between receiving rewards.”