Efficient Exploration Through Bayesian Deep Q-Networks

By Kamyar Azizzadenesheli, Emma Brunskill, Anima Anandkumar. 2018

180 citations recorded by the source corpus.

We propose Bayesian Deep Q-Network (BDQN), a practical Thompson sampling based Reinforcement Learning (RL) Algorithm. Thompson sampling allows for targeted exploration in high dimensions through posterior sampling but is usually computationally expensive. We address this limitation by introducing uncertainty only at the output layer of the network through a Bayesian Linear Regression (BLR) model, which can be trained with fast closed-form updates and its samples can be drawn efficiently through the Gaussian distribution. We apply our method to a wide range of Atari games in Arcade Learning Environments. Since BDQN carries out more efficient exploration, it is able to reach higher rewards substantially faster than a key baseline, double deep Q network DDQN.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

Surprising Negative Results for Generative Adversarial Tree Search (2018, background): “To combat the sample complexity shortcoming, designing an efﬁcient exploration strategy in DRL has emerged as an active research topic, e.g. optimism (Ostrovski et al., 2017) and Thompson Sampling (Osband et al., 2016; Lipton et al., 2018; Azizzadenesheli et al., 2018).”
Off-Policy Deep Reinforcement Learning without Exploration (2018, neutral): “Uncertainty estimates in deep reinforcement learning have generally been used to encourage exploration (Dearden et al., 1998; Strehl & Littman, 2008; O’Donoghue et al., 2018; Azizzadenesheli et al., 2018).”
MOReL : Model-Based Offline Reinforcement Learning (2020, methodology): “Unknown state-action detector (USAD): In order to partition the state-action space into known and unknown regions, we use uncertainty quantification [69, 70, 71, 72].”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “By using an uncertainty estimation algorithm like bootstrap ensembles [43, 4, 37], we can estimate u ( s , a ) .”
Randomized Prior Functions for Deep Reinforcement Learning (2018, methodology): “To save computation we follow previous work and use a shared convnet for the ensemble uncertainty [47, 3].”
Deep Exploration via Randomized Value Functions (2017, background): “, 2016a), linear final layer approximation (Azizzadenesheli et al., 2018) or variational inference (Lipton et al.”
Bayesian Layers: A Module for Neural Network Uncertainty (2018, background): “Still relatively few works leverage deep Bayesian models for control (Gal et al., 2016; Azizzadenesheli et al., 2018).”
Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning (2018, unknown): “, continuously training an AEN with new data in order to learn good representation, but every few steps, learning a bandit model on top of it to gain accurate uncertainty estimates and better exploration (Levine et al., 2017; Azizzadenesheli et al., 2018; Riquelme et al., 2018).”
Action and Perception as Divergence Minimization (2020, methodology): “BNNs have been leveraged for RL for robustness (Okada et al., 2020; Tran et al., 2019) and exploration (Houthooft et al., 2016; Azizzadenesheli et al., 2018).”
Sample-Efficient Deep RL with Generative Adversarial Tree Search (2018, background): “To combat the sample complexity shortcoming, designing an efficient exploration strategy in DRL has emerged as an active research topic, e.g. optimism Ostrovski et al. (2017) and Thompson Sampling (Osband et al., 2016; Lipton et al., 2018; Azizzadenesheli et al., 2018).”
Offline Policy Selection under Uncertainty (2020, methodology): “Both BayesDICE and BDQN output a distribution of parameters, from which we conduct Monte Carlo sampling and use the resulting samples to compute a conﬁdence interval at a given conﬁdence level.”
Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Bound (2019, background): “Very recently, (Azizzadenesheli et al., 2018) studied the regret bound for linear function approximator.”
Bayesian Reinforcement Learning in Factored POMDPs (2018, background): “These methods often require millions of samples and combining them with effective exploration, although a topic of some studies [3, 5, 33], generally is difficult.”
Randomized Value Functions via Multiplicative Normalizing Flows (2018, methodology): “Azizzadenesheli et al. (2018) extend randomized least-square value iteration by Osband et al. (2016b) (which was restricted to linear approximator) to deep neural networks.”
Provably Efficient Reinforcement Learning with Linear Function Approximation (2019, unknown): “[9] K.”
Generating Adversarial Driving Scenarios in High-Fidelity Simulators (2019, methodology): “Methods in this category have modeled the uncertainty in the value function using the bootstrap method [14], Bayesian linear regression [15], and variational inference [16].”
Provably Efficient Q-learning with Function Approximation via Distribution Shift Error Checking Oracle (2019, background): “Later works further generalized sampling-based algorithms to Q -functions with neural network parameterization [6, 23, 13].”
On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation (2020, background): “…the idea of using the last layer of a pre-trained regression neural network to compute uncertainty estimates has been explored in several related ﬁelds: Bayesian optimization [SRS + 15], active learning [ZLA18], and as an uncertainty source for exploration in reinforcement learning [RTS18, ABA18].”