Bayesian Q-Learning

By Richard Dearden, Nir Friedman, Stuart J. Russell. 1998

Subfields: Reinforcement Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

Taming the Noise in Reinforcement Learning via Soft Updates (2015, neutral): “This is in contrast to Bayesian Q-learning [38], which explicitly models the uncertainty about the Q function as a posterior distribution.”
Unifying Count-Based Exploration and Intrinsic Motivation (2016, background): “Most Bayesian exploration schemes use counts in their posterior estimates, for example when the uncertainty over transition probabilities is captured by an exponential family prior (Dearden et al., 1998; Duff, 2002; Poupart et al., 2006).”
Off-Policy Deep Reinforcement Learning without Exploration (2018, methodology): “voring seen actions, but is limited to discrete actions. Uncertainty in Reinforcement Learning. Uncertainty estimates in deep reinforcement learning have generally been used to encourage exploration (Dearden et al., 1998; Strehl & Littman, 2008; O’Donoghue et al., 2018; Azizzadenesheli et al., 2”
A Distributional Perspective on Reinforcement Learning (2017, background): “…1973; Sobel, 1982; White, 1988), in reinforcement learning it has thus far been subordinated to speciﬁc purposes: to model parametric uncertainty (Dearden et al., 1998), to design risk-sensitive algorithms (Morimura et al., 2010b;a), or for theoretical analysis (Azar et al., 2012; Lattimore &…”
Generalization and Exploration via Randomized Value Functions (2014, methodology): “Another related line of work [Dearden et al., 1998] proposes Q-value sampling, a Bayesian exploration scheme with randomized value functions.”
Efficient Exploration Through Bayesian Deep Q-Networks (2018, methodology): “Inevitably for PSRL, the act of posterior sampling for policy or value is computationally intractable in large systems, so PSRL can not be easily leveraged to high dimensional problems (Ghavamzadeh et al., 2015; Engel et al., 2003; Dearden et al., 1998; Tziortziotis et al., 2013).”
Deep Exploration via Randomized Value Functions (2017, background): “The only other work we know of involving exploration through random sampling of value functions is [33], which proposes a tabular algorithm.”
Model based Bayesian Exploration (1999, methodology): “In a recent paper, Dearden et al. (1998) examined modelfree Bayesian reinforcement learning.”
Algorithm-Directed Exploration for Model-Based Reinforcement Learning in Factored MDPs (2002, background): “Research on reinforcement learning has consistently focused on developing algorithms that effectively manage the exploration/exploitation tradeoff [1, 11 ] and attempting to scale up to large real-world problems [8, 7]. Most research in these directions has focused on the direct value-approximation”
Efficient exploration with Double Uncertain Value Networks (2017, background): “We believe this is due to a fundamental complication regarding uncertainties in RL, which has only been identified by Dearden et al. (1998) before.”
A Bayesian Framework for Reinforcement Learning (2000, background): “My approach is novel in that each hypothesis is retained over a period of time, ensuring goal-directed exploratory behavior without the need to use approximate measures such as myopic VPI (Dearden et al., 1998), or heuristic exploration bonuses combined with backpropagation of uncertainty (Meuleau &”
Bayes Meets Bellman: The Gaussian Process Approach to Temporal Difference Learning (2003, methodology): “For instance, the RL agent may use this information to actively direct its exploration (see Dearden et al., 1998); another possibility would be to use the confidence measure to provide stopping conditions for trials in episodic tasks, as suggested by Bertsekas and Tsitsildis (1996).”
Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control (2005, unknown): “Q(s, a) ≡  R(s) s is terminal (a=∅) ∑ s′ T (s, a, s ′) · maxa′ [Q(s′, a′)] otherwise (1)”
Using MDP Characteristics to Guide Exploration in Reinforcement Learning (2003, background): “In the value of information strategy [1, 2], the exploration-exploitation tradeoff is solved with a myopic approximation of the value of perfect information.”
Nonparametric Return Distribution Approximation for Reinforcement Learning (2010, background): “In the seminal paper of Dearden et al. (1998), a recursive formula for the second moment of the return is shown when the reward is defined by a deterministic function of the state-action”
Bayesian role discovery for multi-agent reinforcement learning (2010, unknown): “Previous work on Bayesian RL has considered priors on the domain dynamics in model-based RL [1, 7] and priors on expected returns, and action-value functions [2, 3].”
Temporal Difference Bayesian Model Averaging: A Bayesian Perspective on Adapting Lambda (2010, methodology): “We assume Q-values of each state-action pair are independently Gaussian distributed (Dearden et al., 1998), thus we write local models m over each q s,a as follows: 2 P (q s,a |m) = N (q s,a ; µ m s,a , (σ m s,a ) 2 ).”