Randomized Prior Functions for Deep Reinforcement Learning

By Ian Osband, John Aslanides, Albin Cassirer. 2018

434 citations recorded by the source corpus.

Dealing with uncertainty is essential for efficient reinforcement learning. There is a growing literature on uncertainty estimation for deep learning from fixed datasets, but many of the most popular approaches are poorly-suited to sequential decision problems. Other methods, such as bootstrap sampling, have no mechanism for uncertainty that does not come from the observed data. We highlight why this can be a crucial shortcoming and propose a simple remedy through addition of a randomized untrainable `prior' network to each ensemble member. We prove that this approach is efficient with linear representations, provide simple illustrations of its efficacy with nonlinear representations and show that this approach scales to large-scale problems far better than previous attempts.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Off-Policy Deep Reinforcement Learning without Exploration (2018, methodology): “Other methods have examined approximating the Bayesian posterior of the value function (Osband et al., 2016; 2018; Touati et al., 2018), again using the variance to encourage exploration to unseen regions of the state space.”
Physics-Informed Neural Networks for Cardiac Activation Mapping (2020, methodology): “Instead, we borrow ideas from deep reinforcement learning and use randomized prior functions [ 16] to quantify the epistemic/model uncertainty associated with our neural network predictions.”
Adaptive Online Planning for Continual Lifelong Learning (2019, methodology): “1, as the mean and standard deviation of the ensemble represent the epistemic uncertainty of the value of the state [25].”
MOReL : Model-Based Offline Reinforcement Learning (2020, methodology): “Unknown state-action detector (USAD): In order to partition the state-action space into known and unknown regions, we use uncertainty quantification [69, 70, 71, 72].”
Generalized Off-Policy Actor-Critic (2019, background): “However, in many cases, we would prefer to learn off-policy to improve data efficiency (Lin, 1992) and exploration (Osband et al., 2018).”
Exploration by Random Network Distillation (2018, methodology): “The work Osband et al. (2018) provides motivation for random network distillation as discussed in Section 2.2.”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “By using an uncertainty estimation algorithm like bootstrap ensembles [43, 4, 37], we can estimate u(s,a).”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “Agents may incorporate ideas such as intrinsic rewards and auxiliary tasks (Jaderberg et al., 2017; Jaques et al., 2019), ensembling (Osband et al., 2018), prioritized experience replay (Schaul et al., 2015), distributional backups (Bellemare et al., 2017), specialised neural network architectures…”
Behaviour Suite for Reinforcement Learning (2019, methodology): “In this experiment, all agents correspond to different instantiations of a Bootstrapped DQN with prior networks [46, 45].”
Making Efficient Use of Demonstrations to Solve Hard Exploration Problems (2019, methodology): “For example, we tried adding randomized prior functions (Osband et al., 2018) to R2D2, but this approach was still unable to obtain reward on any of the Hard-Eight tasks.”
Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control (2018, methodology): “[17] demonstrate the benefits of uncertainty estimation for exploration.”
Deep Exploration via Randomized Value Functions (2017, methodology): “In place of explicit prior regularizationR we evaluate Q̃θk = f MLP θk +f θ0 k where f is a 2-layer MLP and the parameters θk, θ 0 k are sampled independently from Glorot initialization and henceforth θ k is held fixed (Glorot and Bengio, 2010; Osband et al., 2018).”
Making Sense of Reinforcement Learning and Probabilistic Inference (2020, methodology): “We define the value function VM,πh (s) = Eα∼πQ M,π h (s, α) and writeQM,?h (s, a) = maxπ∈ΠQ M,π h (s, a) for the optimal Q-values over policies, and the optimal value function is given by VM,?h (s) = maxaQ M,? h (s, a).”
Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning (2021, methodology): “Despite their limitations as noted by Osband et al. (2018), random prior based methods including dropout have been widely applied to capture uncertainty in RL (Gal & Ghahramani, 2016a; Osband et al., 2018; Fortunato et al., 2017; Lipton et al., 2018; Tang & Kucukelbir, 2017; Touati et al., 2020).”
VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning (2019, background): “Osband et al. (2018) note that non-Bayesian treatment for decision making can be arbitrarily suboptimal and propose a simple randomised prior based approach for structured exploration.”
Hypermodels for Exploration (2020, background): “o represent a sample from a prior distribution and for the differential to learn the difference between prior and posterior as training progresses. This additive decomposition was ﬁrst introduced in (Osband et al., 2018), which demonstrated its importance in training ensemble hypermodels with neural”
Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning (2021, background): “Another series of methods [52, 32, 33] aim to estimate uncertainty to determine the trustworthiness of a Q-value prediction.”
Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning (2019, methodology): “Osband et al. (2018) studied a method to improve ensemble-based approaches by adding a randomized network to each ensemble member to improve the uncertainty estimation and efﬁcient exploration in deep RL.”