True online TD(λ)

By Harm van Seijen, Richard S. Sutton. 2014

57 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

Q(λ) with Off-Policy Corrections (2016, methodology): “We refer to the direct eligibility trace extensions of these algorithms formed via Equations (11)-(13) by General Q(λ) and Expected SARSA(λ) (first mentioned by Sutton et al. [17]) Unfortunately, in an off-policy setting, General Q(λ) will not converge to the value function Qπ of the target policy, ”
Emphatic Temporal-Difference Learning (2015, background): “(6) The backward-view update with an offline equivalence (cf. van Seijen & Sutton 2014) with the above forward view can be written as: θt+1 .”
Unifying Task Specification in Reinforcement Learning (2016, methodology): “For TO-TD, the pseudo-code has been given for episodic tasks (van Seijen and Sutton, 2014), rather than more generally, and has treated v old carefully at the beginning of episodes, which is not necessary.”
Off-policy TD( l) with a true online equivalence (2014, methodology): “To construct a well-defined forward view, we can truncate the recursion at the current data horizon (van Seijen & Sutton, 2014; Sutton et al., 2014) to obtain interim -returns.”
Learning to Predict Independent of Span (2015, background): “2 The emergence of the dutch trace here is surprising and intriguing because, in contrast to previous work (van Seijen and Sutton 2014; van Hasselt, Mahmood, and Sutton 2014), the dutch trace has arisen in a setting without temporal-difference (TD) learning.”
Off-policy learning based on weighted importance sampling with linear computational complexity (2015, background): “Van Seijen and Sutton (2014) showed that such equivalences can also be achieved in the online case.”
DEVELOPING A PREDICTIVE APPROACH TO KNOWLEDGE (2015, background): “Recently, however, it has been shown that this conversion can be made equivalent in the interim (see van Seijen & Sutton, 2014).”
A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximation (2018, background): “As the name suggests, the components of z 0: t roughly capture the extent to which each feature is eligible for receiving credit or blame for an observed TD error [Sutton and Barto, 1998, Seijen and Sutton, 2014].”
Incremental Off-policy Reinforcement Learning Algorithms (2017, methodology): “Recently, a new TD(λ) algorithm has been developed that achieves equivalence with an online algorithm with superior properties than the original TD(λ) (van Seijen & Sutton 2014).”
A Deeper Look at Planning as Learning from Replay (2015, background): “It was recently introduced to derive an improved version of TD(λ), called true online TD(λ) (van Seijen & Sutton, 2014).”
True Online Temporal-Difference Learning (2015, methodology): “Van Seijen & Sutton (2014) did not provide a direct derivation of the backward view up-date equations; they simply proved that the forward view and the backward view equations result in the same weight vectors.”
RLLib: C++ Library to Predict, Control, and Represent Learnable Knowledge Using On/Off Policy Reinforcement Learning (2015, background): “The reader is referred to [23], [21], and [5] that describe the problems, feature extractions, and parameter settings respectively.”
Multi-step Off-policy Learning Without Importance Sampling Ratios (2017, methodology): “Equivalence for online updating could also be achieved by following the approach taken by van Seijen et al. (2014, 2016), but we leave that out for simplicity.”