Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling

By Carlos Riquelme, George Tucker, Jasper Snoek. 2018

393 citations recorded by the source corpus.

Recent advances in deep reinforcement learning have made significant strides in performance on applications such as Go and Atari games. However, developing practical methods to balance exploration and exploitation in complex domains remains largely unsolved. Thompson Sampling and its extension to reinforcement learning provide an elegant approach to exploration that only requires access to posterior samples of the model. At the same time, advances in approximate Bayesian methods have made posterior approximation for flexible neural network models practical. Thus, it is attractive to consider approximate Bayesian neural networks in a Thompson Sampling framework. To understand the impact of using an approximate posterior on Thompson Sampling, we benchmark well-established and recently developed methods for approximate posterior sampling combined with Thompson Sampling over a series of contextual bandit problems. We found that many approaches that have been successful in the supervised learning setting underperformed in the sequential decision-making scenario. In particular, we highlight the challenge of adapting slowly converging uncertainty estimates to the online setting.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, background): “, 2020), or computing a closed-form posterior for the last dense layer only (Riquelme et al., 2018; Snoek et al., 2015; Kristiadi et al., 2020).”
Functional Regularisation for Continual Learning using Gaussian Processes (2019, background): “Such GPs can be understood as Bayesian neural networks, where only the weights of the last layer are treated in a Bayesian fashion, while those in earlier layers are optimised (Riquelme et al., 2018).”
Neural Processes (2018, methodology): “We observe that Neural Processes are a highly competitive method, performing similar to MAML and the NeuralLinear baseline in Riquelme et al. (2018), which is consistently among the best out of 20 algorithms compared.”
Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks (2020, neutral): “, 2008) (especially in conjunction with ReLU features) and lastlayer Bayesian methods (Snoek et al., 2015; Wilson et al., 2016a;b; Riquelme et al., 2018); and complementing the empirical analyses of Ober & Rasmussen (2019) and Brosse et al.”
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift (2019, methodology): “Bayesian inference for the parameters of the last layer only (Riquelme et al., 2018) – ( LL SVI ) Mean ﬁeld stochastic variational inference on the last layer only – ( LL Dropout ) Dropout only on the activations before the last layer In addition to metrics (we use arrows to indicate which…”
Practical Deep Learning with Bayesian Principles (2019, background): “Slow convergence is found to be especially problematic for sequential problems [45].”
Continuous Meta-Learning without Tasks (2019, methodology): “We extend the wheel bandit problem [37], a common benchmark for meta-learning algorithms [15, 34].”
Uncertainty Sets for Image Classifiers using Conformal Prediction (2020, methodology): “This approach requires computational and algorithmic modiﬁcations; other approaches avoid these via ensembles (Lakshminarayanan et al., 2017; Jiang et al., 2018) or approximations of Bayesian inference (Riquelme et al., 2018; Sensoy et al., 2018).”
On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation (2020, background): “…the idea of using the last layer of a pre-trained regression neural network to compute uncertainty estimates has been explored in several related ﬁelds: Bayesian optimization [SRS + 15], active learning [ZLA18], and as an uncertainty source for exploration in reinforcement learning [RTS18, ABA18].”
Neural Contextual Bandits with UCB-based Exploration (2019, methodology): “Given the fact that DNNs enable the agent to make use of nonlinear models with less domain knowledge, existing work (Riquelme et al., 2018; Zahavy & Mannor, 2019) study neural-linear bandits .”
Benchmarking the Neural Linear Model for Regression (2019, methodology): “It has recently been used in active learning (Pinsler et al., 2019), Bayesian optimization (Snoek et al., 2015), reinforcement learning (Riquelme et al., 2018), and AutoML (Zhou and Precioso, 2019), among others; however, to the best of our knowledge, there has been no systematic attempt to…”
The Scientific Method in the Science of Machine Learning (2019, background): “, 2017), Bayesian neural networks (Riquelme et al., 2018), reinforcement learning (Henderson et al.”
Hybrid Models with Deep and Invertible Features (2019, background): “This could be done, but Riquelme et al. (2018) showed that Bayesian linear regression with deep features (i.e. computed by a deterministic neural network) is highly effective for contextual bandit problems, which suggests that capturing the uncertainty in prediction parameters β is more important…”
Functional Variational Bayesian Neural Networks (2019, methodology): “We ran the experiments for all algorithms and tasks using the default settings open sourced by Riquelme et al. (2018).”
Bao: Making Learned Query Optimization Practical (2020, background): “plex Bayesian neural networks to simple approaches [56].”
PHYRE: A New Benchmark for Physical Reasoning (2019, methodology): “Most CBs model the expected reward given the context and action using linear models [6, 8, 25], Gaussian processes [47], or deep neural networks [42].”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, methodology): “[181] Approximating the posterior sampling Balancing both exploration and exploitation in different complex domains Deep Bayesian Bandits Showdown using Thompson sampling √”