Kalman Temporal Differences
By Matthieu Geist, Olivier Pietquin. 2010
102 citations recorded by the source corpus.
Because reinforcement learning suffers from a lack of scalability, online value (and Q-) function approximation has received increasing interest this last decade. This contribution introduces a novel approximation scheme, namely the Kalman Temporal Differences (KTD) framework, that exhibits the following features: sample-efficiency, non-linear approximation, non-stationarity handling and uncertainty management. A first KTD-based algorithm is provided for deterministic Markov Decision Processes (MDP) which produces biased estimates in the case of stochastic transitions. Than the eXtended KTD framework (XKTD), solving stochastic MDP, is described. Convergence is analyzed for special cases for both deterministic and stochastic transitions. Related algorithms are experimented on classical benchmarks. They compare favorably to the state of the art while exhibiting the announced features.
Subfields: Artificial Intelligence, Computer Networks and Communications, Management Science and Operations Research
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Yoshua Bengio
- Chelsea Finn
- Ruslan Salakhutdinov
- R´emi Munos
- Yuxi Li
- Marc G. Bellemare
In Computer Networks and Communications
- Olivier Pietquin
- Matthieu Geist
- Bilal Piot
- Bruno Scherrer
- Jan Peters
- Gerhard Neumann
- R´emi Munos
- Will Dabney
In Management Science and Operations Research
- Marc G. Bellemare
- Ian Osband
- R´emi Munos
- David Silver
- Sergey Levine
- Will Dabney
- Pieter Abbeel
- Brendan O'Donoghue
Citations in the corpus
- Implicit Quantile Networks for Distributional Reinforcement Learning (2018, methodology): “…to highlight the difference between intrinsic uncertainty , captured by the distribution over returns, and parametric uncertainty , the uncertainty over the value estimate typically associated with Bayesian approaches such as PSRL (Osband et al., 2013) and Kalman TD (Geist & Pietquin, 2010).”
- A Distributional Perspective on Reinforcement Learning (2017, background): “More recently, Geist & Pietquin (2010) proposed an alternative solution to the same problem based on unscented Kalman filters.”
- Noisy Networks for Exploration (2017, methodology): “…these methods are often limited to small state-action spaces or to linear function approximations and are not easily applied with more complicated function approximators such as neural networks (except from work by (Geist & Pietquin, 2010a;b) but it doesn’t come with convergence guarantees).”
- Off-policy learning with eligibility traces: a survey (2013, methodology): “We have explained how to derive recursive (memory and time-efficient) implementations of all these algorithms and discussed their known convergence properties (including an original analysis of BRM(λ) for sufficiently small λ, that implies the so far not known convergence of GPTD/KTD).”
- Policy evaluation with temporal differences: a survey and comparison (2015, methodology): “Kalman temporal-difference learning (KTD) by Geist and Pietquin (2010) is another probabilistic model very similar to GPTD.”