A new Q ( � ) with interim forward view and Monte Carlo equivalence

By Richard S. Sutton, A. Rupam, M. Sutton. 2014

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “Tree Backup (Precup et al., 2000), and Q( λ ) (Sutton et al., 2014) address this, but reduce the number of steps before bootstrapping even when this is undesirable (as in the on-policy case).”
Meta-Gradient Reinforcement Learning (2018, methodology): “The n -step or λ -return can be augmented with off-policy corrections [Precup et al., 2000, Sutton et al., 2014, Espeholt et al., 2018] if it is necessary to correct for the distribution used to generate the data.”
Reinforcement Learning via Fenchel-Rockafellar Duality (2020, background): “…more typical to assume access to the distribution d π (for example, by simply performing Monte-Carlo rollouts enabled by online access), although instances exist in which d π is approximated by importance-weighting a diﬀerent ( i.e. , oﬄine) distribution (Precup et al., 2001; Sutton et al., 2014).”
Multi-task Deep Reinforcement Learning with PopArt (2018, methodology): “In particular, we can write the n - step return as ) , and then apply appropriate importance sampling corrections to each error term (Sutton et al. 2014) This is unbiased, but has high variance.”
Q(λ) with Off-Policy Corrections (2016, background): “[17]) Unfortunately, in an off-policy setting, General Q(λ) will not converge to the value function Q of the target policy, as stated by the following proposition.”
Multi-step Off-policy Learning Without Importance Sampling Ratios (2017, methodology): “A closely related idea is state-dependent bootstrapping used by Sutton and Singh (1994) and Sutton et al. (2014) for state-value estimation, and by Maei and Sutton (2010) for action-value estimation.”
Off-policy learning based on weighted importance sampling with linear computational complexity (2015, methodology): “Based on the general off-policy forward view by Sutton et al. (2014), we combine truncated returns G and truncated corrected returns G k > t + ✓ scaled by corresponding weights due to discounting, bootstrapping and importance sampling to develop an overall return: where It can be readily veriﬁed…”
Weighted importance sampling for off-policy learning with linear function approximation (2014, methodology): “In the following, we show that using such provisional parameters it is also possible to achieve an equivalent efficient update for (7).”
Q( λ ) with Oﬀ-Policy Corrections (unknown): “…trace extensions of these algorithms formed via Equations (11)-(13) by General Q( λ ) and Expected SARSA( λ ) (ﬁrst mentioned by Sutton et al. [17]) Unfortunately, in an oﬀ-policy setting, General Q( λ ) will not converge to the value function Q π of the target policy, as stated by the…”
True Online Temporal-Difference Learning (2015, background): “Sutton et al. (2014) were the ﬁrst to attempt to come up with a general strategy for deriving a backward view (although for forward views based on Equation 19).”
Off-policy TD( l) with a true online equivalence (2014, methodology): “To construct a well-defined forward view, we can truncate the recursion at the current data horizon (van Seijen & Sutton, 2014; Sutton et al., 2014) to obtain interim -returns.”
Revisiting Fundamentals of Experience Replay (2020, background): “These findings naturally open further questions as to the interaction between experience replay and more general classes of return estimators based on multi-step, off-policy data, such as variants of Q(λ) (Watkins, 1989; Peng & Williams, 1994; Sutton et al., 2014; Harutyunyan et al., 2016), TreeBack”