An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning

By Richard S. Sutton, A. Mahmood, Martha White. 2015

297 citations recorded by the source corpus.

In this paper we introduce the idea of improving the performance of parametric temporal-difference (TD) learning algorithms by selectively emphasizing or de-emphasizing their updates on different time steps. In particular, we show that varying the emphasis of linear TD(γ)'s updates in a particular way causes its expected update to become stable under off-policy training. The only prior model-free TD methods to achieve this with per-step computation linear in the number of function approximation parameters are the gradient-TD family of methods including TDC, GTD(γ), and GQ(λ). Compared to these methods, our emphatic TD(λ) is simpler and easier to use; it has only one learned parameter vector and one step-size parameter. Our treatment includes general state-dependent discounting and bootstrapping functions, and a way of specifying varying degrees of interest in accurately valuing different states.

Subfields: Cancer Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cancer Research

In Reinforcement Learning

Citations in the corpus

When to use parametric models in reinforcement learning? (2019, methodology): “…and off-policy learning [Sutton and Barto, 2018, Precup et al., 2000] can be unstable [Williams and Baird III, 1993, Baird, 1995, Sutton, 1995, Tsitsiklis and Van Roy, 1997, Sutton et al., 2009, 2016] — this is sometimes called the deadly triad [Sutton and Barto, 2018, van Hasselt et al., 2018].”
Deep Reinforcement Learning with Double Q-Learning (2015, background): “If seen in isolation, one might perhaps be tempted to think the observed instability is related to inherent instability problems of off-policy learning with function approximation (Baird, 1995; Tsitsiklis and Van Roy, 1997; Sutton et al., 2008; Maei, 2011; Sutton et al., 2015).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Only recently have practical methods for estimating the marginalized importance ratio been introduced (Sutton et al., 2016; Yu, 2015; Hallak et al., 2015, 2016; Hallak and Mannor, 2017; Nachum et al., 2019a; Zhang et al., 2020; Nachum et al., 2019b).”
Meta-Gradient Reinforcement Learning (2018, background): “, 2015], emphasis on certain states [Sutton et al., 2016], reward clipping [Mnih et al.”
Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (2015, neutral): “It differs from, and is somewhat easier than, previous work, sometimes known as off-policy policy evaluation, that aims to estimate the whole value function (Precup et al., 2000; 2001; Sutton et al., 2015).”
Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning (2020, methodology): “This is different from value function or policy learning from off-policy data (e.g., Precup et al., 2001; Maei et al., 2010; Sutton et al., 2016; Munos et al., 2016; Metelli et al., 2018), where the major goal is to ensure stability and convergence.”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, background): “[33] Richard S.”
GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values (2020, background): “Zhang et al. (2019) show that some density ratios can be interpreted as special emphasis (Sutton et al., 2016).”
Reinforcement Learning via Fenchel-Rockafellar Duality (2020, background): “If d D has suﬃcient support or coverage (Sutton et al., 2016), we may re-write (35) as The optimal ζ ∗ of this problem satisﬁes ζ ∗ ( s, a ) = d π ( s,a ) d D ( s,a ) .”
DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections (2019, background): “Proper estimation of these ratios can improve the accuracy of policy evaluation [21] and the stability of policy learning [12, 14, 22, 40].”
Generalized Off-Policy Actor-Critic (2019, methodology): “This objective is referred to as the excursion objective (White, 2018; Ghiassian et al., 2018), as it corresponds to the excursion setting (Sutton et al., 2016).”
Diagnosing Bottlenecks in Deep Q-learning Algorithms (2019, background): “In the policy evaluation scenario, Tsit-siklis & Van Roy (1997) prove that on-policy TD-learning with linear function approximators converge, and methods such as GTD (Sutton et al., 2009a) and ETD (Sutton et al., 2016) have extended results to off-policy cases.”
Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift (2019, methodology): “The covariate shift method, studied by Hallak and Mannor (2017) and Sutton, Mahmood, and White (2016), reweights online updates according to the ratio of the target and behavior stationary distributions.”
Deep Reinforcement Learning and the Deadly Triad (2018, methodology): “This is not a concern when the agent updates the values used for bootstrapping as often as they are used (Sutton et al., 2016).”
An Off-policy Policy Gradient Theorem Using Emphatic Weightings (2018, methodology): “Secondly, derivation of the online algorithm assumes a fixed target policy [Sutton et al., 2016], while the actor is updated at every time step.”
Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation (2018, background): “Our off-policy setting is related to, but different from, off-policy value-function learning [30, 29, 37, 12, 25, 21].”
Emphatic TD Bellman Operator is a Contraction (2015, methodology): “Similarly to Sutton et al. (2015), we divide the analysis to the ‘pure bootstrapping’ caseλ = 0, and the more general case withλ ∈ [0, 1).”