Emphatic Temporal-Difference Learning

By A. Mahmood, Huizhen Yu, Martha White, Richard S. Sutton. 2015

38 citations recorded by the source corpus.

Emphatic algorithms are temporal-difference learning algorithms that change their effective state distribution by selectively emphasizing and de-emphasizing their updates on different time steps. Recent works by Sutton, Mahmood and White (2015), and Yu (2015) show that by varying the emphasis in a particular way, these algorithms become stable and convergent under off-policy training with linear function approximation. This paper serves as a unified summary of the available results from both works. In addition, we demonstrate the empirical benefits from the flexibility of emphatic algorithms, including state-dependent discounting, state-dependent bootstrapping, and the user-specified allocation of function approximation resources.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning (2016, methodology): “…which includes the original linear-time algorithm (Sutton, 1988), least-squares formulations (Bradtke & Barto, 1996; Mahmood et al., 2014), methods for adapting λ (Downey & Sanner, 2010), true-online methods (van Hasselt et al., 2014), and the recent emphatic methods (Mahmood et al., 2015).”
Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction (2019, background): “Importance Sampling based distribution re-weighting [30, 16, 31, 27] has also been explored primarily in the context of off-policy policy evaluation.”
Generalized Emphatic Temporal Difference Learning: Bias-Variance Analysis (2015, methodology): “Such an analysis may proceed similarly to that in Munos (2003) for the on-policy case.”
Emphatic TD Bellman Operator is a Contraction (2015, positive): “Notice that in the case of generalλ, the Bellman operator is: T (λ)v = (I − γλPπ)−1rπ + Pλπ v. (3) Mahmood et al. (2015) show that ETD converges to someθ∗ that is a solution of the projected fixed-point equation: θ⊤Φ = ΠmT (λ)(θ⊤Φ).”
On Convergence of Emphatic Temporal-Difference Learning (2015, neutral): “For insights about ETD(λ), see (Sutton et al., 2015; Mahmood et al., 2015).”
An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning (2015, methodology): “Parametric temporal-difference learning was first studied as the key “learning by generalization” algorithm in Samuel’s (1959) checker player.”
Q(λ) with Off-Policy Corrections (2016, background): “There are several (recent) off-policy algorithms that reduce the variance of IS methods, at the cost of added bias [5, 6, 3].”
Unifying Task Specification in Reinforcement Learning (2016, background): “For example, ETD was introduced with state-based γ s ; however, (Hallak et al., 2015) analyzed approximation error bounds of ETD using a constant discount γ c .”
Safe and Efficient Off-Policy Reinforcement Learning (2016, methodology): “It is well known that IS estimates can suffer from large – even possibly infinite – variance (mainly due to the variance of the product π(a1|x1) μ(a1|x1) · · · π(at|xt) μ(at|xt) ), which has motivated further variance reduction techniques such as in (Mahmood and Sutton, 2015; Mahmood et al., 2015; H”
Weak Convergence Properties of Constrained Emphatic Temporal-difference Learning with Constant and Slowly Diminishing Stepsize (2015, background): “) Their behavior is demonstrated by experiments in [28, 53].”
Multi-step Off-policy Learning Without Importance Sampling Ratios (2017, background): “…environment and in that sense are model-free (Precup et al. 2000, 2001, Maei & Sutton 2010, Maei 2011, Yu 2012, Geist & Scherrer 2014, Dann et al. 2014, van Hasselt et al. 2014, Mahmood et al. 2014, Yu 2015, 2016, Hallak et al. 2015a, Mahmood et al. 2015, Sutton et al. 2016, White & White 2016a).”