Better Optimism By Bayes: Adaptive Planning with Rich Models

By Arthur Guez, David Silver, Peter S. Dayan. 2014

8 citations recorded by the source corpus.

The computational costs of inference and planning have confined Bayesian model-based reinforcement learning to one of two dismal fates: powerful Bayes-adaptive planning but only for simplistic models, or powerful, Bayesian non-parametric models but using simple, myopic planning strategies such as Thompson sampling. We ask whether it is feasible and truly beneficial to combine rich probabilistic models with a closer approximation to fully Bayesian planning. First, we use a collection of counterexamples to show formal problems with the over-optimism inherent in Thompson sampling. Then we leverage state-of-the-art techniques in efficient Bayes-adaptive planning and non-parametric Bayesian methods to perform qualitatively better than both existing conventional algorithms and Thompson sampling on two contextual bandit-like problems.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Why is Posterior Sampling Better than Optimism for Reinforcement Learning? (2016, methodology): “In principle, dynamic programming can be applied to compute the Bayes-optimal solution to this problem (Guez et al., 2014).”
Bayesian Optimal Control of Smoothly Parameterized Systems (2015, background): “While in the case of episodic problems the issue of how long a policy should be kept does not arise, in a continuing problem with no episodic structure, if policies are changed too often, performance will suffer (see, e.g., Example 1 of Guez et al. (2014)).”