Investigating Practical Linear Temporal Difference Learning

By Adam White, Martha White. 2016

44 citations recorded by the source corpus.

Off-policy reinforcement learning has many applications including: learning from demonstration, learning multiple goal seeking policies in parallel, and representing predictive knowledge. Recently there has been an proliferation of new policy-evaluation algorithms that fill a longstanding algorithmic void in reinforcement learning: combining robustness to off-policy sampling, function approximation, linear complexity, and temporal difference (TD) updates. This paper contains two main contributions. First, we derive two new hybrid TD policy-evaluation algorithms, which fill a gap in this collection of algorithms. Second, we perform an empirical comparison to elicit which of these new linear TD methods should be preferred in different situations, and make concrete suggestions about practical use.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

Multi-step Off-policy Learning Without Importance Sampling Ratios (2017, methodology): “Although importance sampling plays a key role in correcting the discrepancy between the policies, the highly varying nature of importance sampling ratios often results in large estimation variance (Liu 2001, Koller & Friedman 2009, Dann et al. 2014, Mahmood & Sutton 2015, White & White 2016a).”
Two-Timescale Networks for Nonlinear Value Function Approximation (2019, methodology): “Linear algorithms can also make use of eligibility traces, which can significantly speed learning (Sutton, 1988; Dann et al., 2014; White and White, 2016), but have not been able to be extended to nonlinear value function approximation.”