A new Q(lambda) with interim forward view and Monte Carlo equivalence

By Richard S. Sutton, A. Mahmood, Doina Precup, Hado van Hasselt. 2014

29 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “Tree Backup (Precup et al., 2000), and Q( λ ) (Sutton et al., 2014) address this, but reduce the number of steps before bootstrapping even when this is undesirable (as in the on-policy case).”
Meta-Gradient Reinforcement Learning (2018, methodology): “The n -step or λ -return can be augmented with off-policy corrections [Precup et al., 2000, Sutton et al., 2014, Espeholt et al., 2018] if it is necessary to correct for the distribution used to generate the data.”
Reinforcement Learning via Fenchel-Rockafellar Duality (2020, methodology): “The original policy gradient theorem in Sutton et al. (2000) relies on on-policy settings, while in practice we would like to optimize the policy with only offline samples.”
An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning (2015, neutral): “…function λ : S → [0, 1] specifying a potentially different degree of bootstrapping, 1− λ(s), for each state s. General bootstrapping of this form has been partially developed in several previous works (Sutton 1995, Sutton & Barto 1998, Maei & Sutton 2010, Sutton et al. 2014, cf. Yu 2012).”
Multi-task Deep Reinforcement Learning with PopArt (2018, methodology): “In particular, we can write the n - step return as ) , and then apply appropriate importance sampling corrections to each error term (Sutton et al. 2014) This is unbiased, but has high variance.”
Off-policy learning based on weighted importance sampling with linear computational complexity (2015, methodology): “Based on the general off-policy forward view by Sutton et al. (2014), we combine truncated returns G and truncated corrected returns G k > t + ✓ scaled by corresponding weights due to discounting, bootstrapping and importance sampling to develop an overall return: where It can be readily veriﬁed…”
Revisiting Fundamentals of Experience Replay (2020, background): “These findings naturally open further questions as to the interaction between experience replay and more general classes of return estimators based on multi-step, off-policy data, such as variants of Q(λ) (Watkins, 1989; Peng & Williams, 1994; Sutton et al., 2014; Harutyunyan et al., 2016), TreeBack”
Weighted importance sampling for off-policy learning with linear function approximation (2014, methodology): “= E E E " 1 X t=0 R t+1 t Y k=1 (S k ) | S 0 = s, A t ⇠ ⇡(·|S t ), 8t # , where (S k ) 2 [0] [1] is a state-dependent degree of discounting on arrival in S k (as in Sutton et al. 2014).”
Off-policy TD( l) with a true online equivalence (2014, methodology): “To construct a well-defined forward view, we can truncate the recursion at the current data horizon (van Seijen & Sutton, 2014; Sutton et al., 2014) to obtain interim -returns.”
Learning to Predict Independent of Span (2015, background): “Therefore, as in Sutton et al. (2014), we allow the features, discounts and persistency parameters to be stationary functions of an underlying unobserved state, such that φt .”
Multi-step Off-policy Learning Without Importance Sampling Ratios (2017, methodology): “A closely related idea is state-dependent bootstrapping used by Sutton and Singh (1994) and Sutton et al. (2014) for state-value estimation, and by Maei and Sutton (2010) for action-value estimation.”
True Online Temporal-Difference Learning (2015, background): “Sutton et al. (2014) were the ﬁrst to attempt to come up with a general strategy for deriving a backward view (although for forward views based on Equation 19).”