Deep Exploration via Bootstrapped DQN

By Ian Osband, Charles Blundell, Alexander Pritzel, Benjamin Van Roy. 2016

1,541 citations recorded by the source corpus.

Efficient exploration in complex environments remains a major challenge for reinforcement learning. We propose bootstrapped DQN, a simple algorithm that explores in a computationally and statistically efficient manner through use of randomized value functions. Unlike dithering strategies such as epsilon-greedy exploration, bootstrapped DQN carries out temporally-extended (or deep) exploration; this can lead to exponentially faster learning. We demonstrate these benefits in complex stochastic MDPs and in the large-scale Arcade Learning Environment. Bootstrapped DQN substantially improves learning times and performance across most Atari games.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, background): “[31, 42]); for example, Bayesian uncertainty estimates have been proposed as an alternative to double DQN [3].”
Addressing Function Approximation Error in Actor-Critic Methods (2018, background): “Further, the variance of the value estimate has been considered directly (Osband et al., 2016; O’Donoghue et al., 2017) to estimate uncertainty of the agent as an exploration method.”
Surprising Negative Results for Generative Adversarial Tree Search (2018, background): “[40] suggests a bootstrapped-ensemble, Bellemare et al.”
Rainbow: Combining Improvements in Deep Reinforcement Learning (2017, methodology): “Besides Noisy Nets, numerous other exploration methods could also be useful algorithmic ingredients: among these Bootstrapped DQN (Osband et al. 2016), intrinsic motivation (Stadie, Levine, and Abbeel 2015) and count-based exploration (Bellemare et al. 2016).”
Neural Approaches to Conversational AI (2018, neutral): “, 2017), exploration when parametric models like neural networks are used is an active research topic (Bellemare et al., 2016; Osband et al., 2016; Houthooft et al., 2016; Jiang et al., 2017).”
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (2016, methodology): “It would be also interesting to investigate so-called implicit ensembles the where ensemble members share parameters, e.g. using multiple heads (Lee et al., 2015; Osband et al., 2016), snapshot ensembles (Huang et al., 2017) or swapout (Singh et al., 2016).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “When bootstrap ensembles3 are used to represent the Q-function, as is common in prior work (Osband et al., 2016; Eysenbach et al., 2017; Kumar et al., 2019a; Agarwal et al., 2019), typical choices of ‘Unc’ include variance across Q-value predictions of an ensemble of Q-functions (Kumar et al.,…”
Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “Following [47, 26, 45], we use concentration properties of B̂π to control this error.”
Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, positive): “QMIX is trained end-to-end to minimise the following loss: where b is the batch size of transitions sampled from the replay buﬀer, the DQN target is given by y tot = r + γ max u (cid:48) Q tot ( τ (cid:48) , u (cid:48) , s (cid:48) ; θ − ), and θ − are the parameters of a target network as in DQN. (”
Off-Policy Deep Reinforcement Learning without Exploration (2018, methodology): “Other methods have examined approximating the Bayesian posterior of the value function (Osband et al., 2016; 2018; Touati et al., 2018), again using the variance to encourage exploration to unseen regions of the state space.”
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, background): “…Bayesian NN models remains an open problem [MacKay, 1992, Neal, 1995, Osband, 2016, Guo et al., 2017], although recent promising work exists on incorporating dropout [Gal et al., 2017], ensembles [Osband et al., 2016, Lakshminarayanan et al., 2017], and α-divergence [Hernández-Lobato et al., 2016].”
Curiosity-Driven Exploration by Self-Supervised Prediction (2017, methodology): “Osband et al. (2016) train multiple value functions and makes use of bootstrapping and Thompson sampling for exploration.”
PathNet: Evolution Channels Gradient Descent in Super Neural Networks (2017, background): “Firstly, a possibility is that mutable paths provide a more useful form of diverse exploration in RL tasks [15].”
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation (2016, methodology): “This is a modified version of the MDP in [19], with the reward structure adding complexity to the task (see Figure 2).”
Self-Imitation Learning (2018, background): “Exploration There has been a long history of work on improving exploration in RL, including recent work that can scale up to large state spaces (Stadie et al., 2015; Osband et al., 2016; Bellemare et al., 2016; Ostrovski et al., 2017).”
Adaptive Online Planning for Continual Lifelong Learning (2019, methodology): “The value ensemble improves the exploration ability of the optimization procedure [25, 26].”
Multiplicative Normalizing Flows for Variational Bayesian Neural Networks (2017, methodology): “It should also be mentioned that uncertainty estimation in neural networks can also be performed without the Bayesian paradigm; frequentist methods such as Boot-strap (Osband et al., 2016) and ensembles (Lakshmi-narayanan et al., 2016) have shown that in certain scenarios they can provide reasonable”
Simple and Scalable Epistemic Uncertainty Estimation Using a Single Deep Deterministic Neural Network (2020, methodology): “Estimating uncertainty reliably and efﬁciently has remained an open problem with many important applications such as guiding exploration in Reinforcement Learning (Osband et al., 2016) or as a method for selecting data points for which to acquire labels in Active Learning (Houlsby et al., 2011).”