Bayes Meets Bellman: The Gaussian Process Approach to Temporal Difference Learning

By Yaakov Engel, Shie Mannor, R. Meir. 2003

241 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, methodology): “Traditional approaches for solving the Bellman equation such as dynamic programming (Bellman 2003), parametric methods (Engel et al. 2003), Monte Carlo methods (Sutton and Barto 1998), and piecewise linear and convex (PWLC)-based methods (Smallwood and Sondik 1973) have several drawbacks.”
Variational Bayesian Reinforcement Learning with Regret Bounds (2018, methodology): “hue et al., 2017c). The beneﬁt of using a Bayesian approach include the ability to incorporate prior information, as well as not requiring a computationally difﬁcult search over transition functions (Engel et al., 2003, 2005). However, generally speaking, frequentist regret bounds are considered str”
Efficient Exploration Through Bayesian Deep Q-Networks (2018, methodology): “Inevitably for PSRL, the act of posterior sampling for policy or value is computationally intractable in large systems, so PSRL can not be easily leveraged to high dimensional problems (Ghavamzadeh et al., 2015; Engel et al., 2003; Dearden et al., 1998; Tziortziotis et al., 2013).”
Gaussian Processes for Data-Efficient Learning in Robotics and Control (2015, neutral): “Therefore, [17], [19], [46], [57] propose to learn GP value function models to address the issue of model errors in the value function.”
On-line policy optimisation of spoken dialogue systems via live interaction with human subjects (2011, methodology): “Gaussian process (GP) based RL [11] has been recently applied to POMDP dialogue policy optimisation in order to exploit the correlations between different belief states a nd thus speed up the learning process [12].”
Reinforcement learning with Gaussian processes (2005, methodology): “This bias was already observed and reported in Engel et al. (2003) as the " dampening " of the value estimates.”
A unifying framework for computational reinforcement learning theory (2009, background): “1997a], decision trees [McCallum 1995; Uther 2002] , and Gaussian processes [Engel et al. 2003; Rasmussen and Kuss 2004], just to name a few.”
Knowledge transfer between speakers for personalised dialogue management (2015, methodology): “Once the subset of points has been selected and assuming ∆Q(bi, ai) − γi∆Q(bi+1, ai+1) ∼ N (0, σ2) as in (Engel et al., 2003), the GP posterior can be approximated in O(t ·m2) with the DTC method as:”
Novelty or Surprise? (2013, unknown): “TD learning has also been developed in a Bayesian framework (Engle et al., 2003).”
A Clustering-Based Graph Laplacian Framework for Value Function Approximation in Reinforcement Learning (2014, background): “In recent years, there are some papers on applying Gaussian processes (GPs) or SVMs to RL problems, such as GPs in TD(0) learning [32], SVMs for RL [33], and policy iteration based on GPs [34].”
Policy evaluation with temporal differences: a survey and comparison (2015, methodology): “Besides KTD’s ability to deal with non-stationary environments, KTD and GPTD differ mostly in the way they handle value functions that are non-linear in the feature space.”
Convex Optimization: Algorithms and Complexity (2015, unknown): “Gaussian process temporal difference (GPTD) learning [Engel et al., 2003, 2005a, Engel, 2005] is a GP-based framework that uses linear statistical models (see §2.5.2) to relate, in a probabilistic way, the underlying hidden value function with the observed rewards, for a MDP controlled by some…”
Information Theory of Decisions and Actions (2011, background): “…are here not concerned with producing an algorithm which would compare performancewise with competitive state-of-the-art MDP learners (such as, e.g. Engel et al. 2003; Jung and Polani 2007); and neither are we concerned with proposing or investigating particular flavours of the perception-action…”
Non-parametric Policy Search with Limited Information Loss (2017, methodology): “Such sparsifications have been used in a number of RL algorithms (Engel et al., 2003; Xu et al., 2014; Jung and Polani, 2007; Xu et al., 2007; Lever and Stafford, 2015).”
Efficient exploration with Double Uncertain Value Networks (2017, background): “Most of these have focussed on Gaussian Process regression (Engel et al., 2003).”