A Bayesian Framework for Reinforcement Learning

By Malcolm J. A. Strens. 2000

532 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

The Uncertainty Bellman Equation and Exploration (2017, methodology): “are a sample from the implicit posterior over Q-values, conditioned on the historyFt (Strens, 2000).”
Why is Posterior Sampling Better than Optimism for Reinforcement Learning? (2016, background): “This approach, originally proposed in [24], is called posterior sampling for reinforcement learning (PSRL).”
Deep Exploration via Bootstrapped DQN (2016, methodology): “The agent must also commit to this sample for several time steps in order to achieve deep exploration [21, 8].”
Variational Bayesian Reinforcement Learning with Regret Bounds (2018, background): “Approximations to the optimal Bayesian policy exist, one of the most successful being Thompson sampling [53, 55] wherein the agent samples from the posterior over value functions and acts greedily with respect to that sample [38, 42, 27, 41].”
Randomized Prior Functions for Deep Reinforcement Learning (2018, background): “1 Chain environments We begin our experiments with a family of chain-like environments that highlight the need for deep exploration [62].”
(More) Efficient Reinforcement Learning via Posterior Sampling (2013, methodology): “PSRL was originally introduced in the context of reinforcement learning by Strens [16] under the name “Bayesian Dynamic Programming”,2 where it appeared primarily as a heuristic method.”
Efficient Exploration Through Bayesian Deep Q-Networks (2018, methodology): “To approximate the posterior, they use BLR on one-hot encoding of state-action and improve the computation complexity of PSRL .”
BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems (2016, background): “Once a good estimate of θ is found so that Q ( · , · ; θ ) is a good approximation of Q ( · , · ) , the greedy policy, π ( s ; θ ) = arg max a Q ( s, a ; θ ) , is a near-optimal policy (Sutton and Barto 1998).”
A Tutorial on Thompson Sampling (2017, methodology): “, S} is specified by a positive vector of pseudo-observations α ∈ R+; updates to the Dirichlet posterior can be performed analytically simply by incrementing the appropriate column of α [4].”
Minimax Regret Bounds for Reinforcement Learning (2017, background): “…of concentration inequalities to the optimal value function as a whole, rather than to the transitions probabilities (to improve scaling in S), and we define Bernstein-based "exploration bonuses" that use the empirical variance of the estimated values at the next states (to improve scaling in H).”
Evolutionary Dynamics of Multi-Agent Learning: A Survey (2015, background): “Worth mentioning here as well is the related stream of work on Bayesian reinforcement learning (Dearden, Friedman, & Russell, 1998; Strens, 2000).”
Thompson Sampling for Contextual Bandits with Linear Payoffs (2012, methodology): “…, given context b i ( t ) and parameter µ , were given by the pdf of Gaussian distribution (cid:16)P (cid:17) Then, if the prior for µ at time t is given by N (ˆ µ ( t ) , v 2 B ( t ) − 1 ), it is easy to compute the posterior distribution at time t + 1, Pr(˜ µ r as N (ˆ µ ( t +1) , v 2 B ( t +…”
Near-optimal Reinforcement Learning in Factored MDPs (2014, methodology): “PSRL is motivated by the old heuristic of Thompson sampling [12] and has been previously shown to be e cient in non-factored MDPs [13, 6].”
Meta reinforcement learning as task inference (2019, background): “Reasoning about this belief state is at the heart of Bayesian reinforcement learning (Strens, 2000), and many algorithms with optimal regret guarantees such as Thompson sampling (Agrawal & Goyal, 2013) effectively separate the algorithm for estimating the belief state from that for acting based on…”
Near-Bayesian exploration in polynomial time (2009, methodology): “Simulated Domain In this section we present empirical results for BEB and other algorithms on a simple chain domain from the Bayesian exploration literature (Strens, 2000; Poupart et al., 2006), shown in Figure 1.”