The Fixed Points of Off-Policy TD

By J. Kolter. 2011

65 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, background): “This has been generally aimed at resolving what is typically known as the “deadly triad” [31]: divergence caused by an interaction between function approximation, bootstrapped updates and off-policy sampling, resulting in the development of batch RL algorithms that choose sampling schemes [32, 33] f”
Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift (2019, background): “Further, under off-policy learning, the quality of the Bellman fixed point suffers as studied by Kolter (2011) and Munos (2003).”
An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning (2015, background): “Kolter (2011) also suggested adapting the distribution of states at which updates are made to improve convergence and solution quality, but did not provide a linear-complexity algorithm.”
Off-policy learning with eligibility traces: a survey (2013, background): “As far as we know, the only work to move in this direction is the off-policy approach of Kolter (2011): samples are weighted such that the projection operator composed with the Bellman operator is non-expansive (so, weaker than finding the projection of the stationary distribution, but offering…”
Finite-Sample Analysis of Proximal Gradient TD Algorithms (2015, background): “The proof of this lemma can be found in Kolter [2011].”
Online Off-policy Prediction (2018, background): “On the other hand, the fixed-point solution to the excursion objective can be arbitrarily poor compared with the best value function in the function approximation class if there is a significant mismatch between the behavior and target policy (Kolter, 2011).”
Provably Convergent Off-Policy Actor-Critic with Function Approximation (2019, methodology): “The proof from Kolter (2011), however, can not be applied to analyze themπ estimate until Lemma 2 is established.”
Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error (2022, background): “In a similar vein to our work, Kolter (2011) remarks that with off-policy sampling, the projected solution to linear TD can have arbitrarily large Bellman error, but focuses on the approximate fixed point, rather than any discrepancy between Bellman error and value error.”
Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation (2019, methodology): “The bound of the q π estimate follows directly from Kolter (2011).”
Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spaces (2014, background): “However, TD is only guaranteed to converge in the on-policy setting, although in many off-policy situations, it still has satisfactory performance [105].”
Regularized Off-Policy TD-Learning (2012, background): “However, TD is only guaranteed to converge in the on-policy setting, although in many off-policy situations, it still has satisfactory performance [21].”