A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learning

By Martha White, Adam White. 2016

51 citations recorded by the source corpus.

One of the main obstacles to successful application of reinforcement learning methods is the parameter sensitivity of our core learning algorithms. In many larger scale applications, online computation and the use of function approximation represent key strategies in scaling up reinforcement learning algorithms. In this setting, we have effective and reasonably well understood algorithms for adapting the learning-rate parameter, online during learning. For temporal-difference learning algorithms which we study here, there is yet another parameter, λ, that similarly impacts learning speed and stability in practice. Unfortunately, unlike the learning-rate parameter, λ parametrizes the objective function that temporal-difference methods optimize. Different choices of λ produce different fixed-point solutions, and thus adapting λ online and characterizing the optimization is substantially more complex than adapting the learning-rate parameter. There are no methods available that can achieve (1) incremental updating, (2) compatibility with function approximation, and (3) maintain stability of learning under both on and off-policy sampling. In this paper we contribute a novel objective function for optimizing λ as a function of state rather than time. We derive a new incremental, linear complexity λ-adaption algorithm that does not require offline batch updating or access to a model of the world, and present a suite of experiments illustrating the practicality of our new algorithm in three different settings. Taken together, our contributions represent a concrete step towards black-box application of temporal-difference learning methods in real world problems.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

Citations in the corpus

Meta-Gradient Reinforcement Learning (2018, background): “Many researchers have sought to automate the selection of these parameters [Kearns and Singh, 2000, Downey and Sanner, 2010, Konidaris et al., 2011, White and White, 2016].”
Self-Tuning Deep Reinforcement Learning (2020, background): “Mann et al. (2016) and White & White (2016) proposed algorithms to adapt λ (the eligibility trace coefficient).”
Multi-step Off-policy Learning Without Importance Sampling Ratios (2017, methodology): “Although importance sampling plays a key role in correcting the discrepancy between the policies, the highly varying nature of importance sampling ratios often results in large estimation variance (Liu 2001, Koller & Friedman 2009, Dann et al. 2014, Mahmood & Sutton 2015, White & White 2016a).”