Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models

By Bradly C. Stadie, Sergey Levine, Pieter Abbeel. 2015

542 citations recorded by the source corpus.

Achieving efficient and scalable exploration in complex domains poses a major challenge in reinforcement learning. While Bayesian and PAC-MDP approaches to the exploration problem offer strong formal guarantees, they are often impractical in higher dimensions due to their reliance on enumerating the state-action space. Hence, exploration in complex domains is often performed with simple epsilon-greedy methods. In this paper, we consider the challenging Atari games domain, which requires processing raw pixel inputs and delayed rewards. We evaluate several more sophisticated exploration strategies, including Thompson sampling and Boltzman exploration, and propose a new exploration method based on assigning exploration bonuses from a concurrently learned model of the system dynamics. By parameterizing our learned model with a neural network, we are able to develop a scalable and efficient approach to exploration bonuses that can be applied to tasks with complex, high-dimensional state spaces. In the Atari domain, our method provides the most consistent improvement across a range of games that pose a major challenge for prior methods. In addition to raw game-scores, we also develop an AUC-100 metric for the Atari Learning domain to evaluate the impact of exploration on this benchmark.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Dialog-based Language Learning (2016, methodology): “Finally, forward prediction models, which we make use of in this work, have been used for learning eye tracking [21], controlling robot arms [12] and vehicles [26], and action-conditional video prediction in atari games [16, 23].”
What can the brain teach us about building artificial intelligence? (2016, background): “More recent variants of the DQN have demonstrated superior performance (Hasselt, Guez, & Silver, 2016; Schaul et al., 2016; Stadie et al., 2016; Wang et al., 2016), reaching 83% of the professional gamer’s score by”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, background): “Previous work has extensively studied other exploration strategies including predictive error (Stadie et al., 2015), count based exploration (Bellemare et al.”
Rainbow: Combining Improvements in Deep Reinforcement Learning (2017, methodology): “Besides Noisy Nets, numerous other exploration methods could also be useful algorithmic ingredients: among these Bootstrapped DQN (Osband et al. 2016), intrinsic motivation (Stadie, Levine, and Abbeel 2015) and count-based exploration (Bellemare et al. 2016).”
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, background): “Recently, eschewing an explicit evaluation phase, Stadie, Levine, and Abbeel (2015) proposed the area under the learning curve as an evaluation metric.”
Prioritized Experience Replay (2015, methodology): “…proposed methods for playing Atari with deep reinforcement learning, including deep Q-networks (DQN) (Mnih et al., 2013; 2015; Guo et al., 2014; Stadie et al., 2015; Nair et al., 2015; Bellemare et al., 2016), and the Double DQN algorithm (van Hasselt et al., 2016), which is the current…”
Dueling Network Architectures for Deep Reinforcement Learning (2015, background): “The results of Schaul et al. (2016) are the current published state-of-the-art.”
Unifying Count-Based Exploration and Intrinsic Motivation (2016, methodology): “Recently, Stadie et al. (2015) used a squared prediction error bonus for exploring in Atari 2600 games.”
Curiosity-Driven Exploration by Self-Supervised Prediction (2017, methodology): “…the agent to perform actions that reduce the error/uncertainty in the agent’s ability to predict the consequence of its own actions (i.e. its knowledge about the environment) (Houthooft et al., 2016; Mohamed & Rezende, 2015; Schmidhuber, 1991; 2010; Singh et al., 2005; Stadie et al., 2015).”
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation (2016, methodology): “Existing methods like Boltzmann exploration and Thomson sampling [31, 19] offer significant improvements over -greedy, but are limited due to the underlying models functioning at the level of basic actions.”
Self-Imitation Learning (2018, background): “Exploration There has been a long history of work on improving exploration in RL, including recent work that can scale up to large state spaces (Stadie et al., 2015; Osband et al., 2016; Bellemare et al., 2016; Ostrovski et al., 2017).”
Deep Exploration via Bootstrapped DQN (2016, background): “This MDP is known to the agent a priori, with deterministic actions “left” and “right”.”
Exploration by Random Network Distillation (2018, methodology): “Another class of exploration methods rely on errors in predicting dynamics (Schmidhuber, 1991b; Stadie et al., 2015; Achiam & Sastry, 2017; Pathak et al., 2017; Burda et al., 2018).”
Variational Bayesian Reinforcement Learning with Regret Bounds (2018, methodology): “In the non-tabular case methods that estimate the pseudo-count have proven successful in some instances, indicating a possibility of combining K-learning with pseudo-counts in a deep RL setting [94, 11, 97, 80, 13, 79].”
Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “Another paradigm are transition-based methods, in which agents are rewarded for high prediction error in a learned forward or inverse dynamics model [42, 43, 44, 45, 46, 47, 48, 49].”
Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks (2016, background): “…al., 2015; Wat-ter et al., 2015), visual attention (Ba et al., 2015), and the Atari learning environment (ALE) (Guo et al., 2014; Mnih et al., 2015; Stadie et al., 2015; Wang et al., 2015; Schaul et al., 2016; van Hasselt et al., 2016; Oh et al., 2015; Belle-mare et al., 2016; Nair et al., 2015).”
Control of Memory, Active Perception, and Action in Minecraft (2016, background): “There have also been a few attempts to learn state-transition models using deep learning to improve exploration in RL (Oh et al., 2015; Stadie et al., 2015).”