Q(λ) with Off-Policy Corrections

By Ani Harutyunyan, Marc G. Bellemare, T. Stepleton, R´emi Munos. 2016

105 citations recorded by the source corpus.

We propose and analyze an alternate approach to off-policy multi-step temporal difference learning, in which off-policy returns are corrected with the current Q-function in terms of rewards, rather than with the target policy in terms of transition probabilities. We prove that such approximate corrections are sufficient for off-policy convergence both in policy evaluation and control, provided certain conditions. These conditions relate the distance between the target and behavior policies, the eligibility trace parameter and the discount factor, and formalize an underlying tradeoff in off-policy TD($\lambda$). We illustrate this theoretical relationship empirically on a continuous-state control task.

Subfields: Artificial Intelligence, Software, Statistics and Probability, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Software

In Statistics and Probability

In Reinforcement Learning

Citations in the corpus

Sample Efficient Actor-Critic with Experience Replay (2016, background): “Updating the average policy network: θa ← αθa + (1− α)θ Because of the lack of the truncated importance ratio, the operator defined by Qopc is only a contraction if the target and behavior policies are close to each other (Harutyunyan et al., 2016).”
When to use parametric models in reinforcement learning? (2019, background): “Note that several algorithms exist that correct the return towards a desired policy (Harutyunyan et al., 2016; Munos et al., 2016), which is a separate issue from off-policy sampling of states.”
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018, methodology): “Related previous work on off-policy RL include (Precup et al., 2000; 2001; Wawrzynski, 2009; Geist & Scherrer, 2014) and (Harutyunyan et al., 2016).”
Safe and Efficient Off-Policy Reinforcement Learning (2016, methodology): “At one extreme ( λ = 0 ) we have the Bellman operator T πλ =0 Q = T π Q , while at the other ( λ = 1 ) we have the policy evaluation operator T πλ =1 Q = Q π which can be estimated using Monte Carlo methods (Sutton and Barto, 1998).”
Meta-Gradient Reinforcement Learning (2018, background): “…and Niranjan, 1994, van Seijen et al., 2009, Sutton and Barto, 2018], including several state-of-the-art deep RL algorithms [Mnih et al., 2015, van Hasselt et al., 2016, Harutyunyan et al., 2016, Hessel et al., 2018, Espeholt et al., 2018], are characterised by different choices of the return.”
A Distributional Perspective on Reinforcement Learning (2017, background): “We believe the convergence issues we outline here are a symptom of the inherent instability of greedy updates, as highlighted by e.g. Tsitsik-lis (2002) and most recently Harutyunyan et al. (2016).”
Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning (2019, methodology): “Q(λ) (Harutyunyan et al., 2016) can be viewed as a generalization of FQE that looks to the horizon limit to incorporate the long-term value into the backup step. Retrace(λ) (Munos et al., 2016) and Tree-Backup(λ) (Precup et al., 2000) also use full trajectories, but additionally incorporate varying ”
Evolution-Guided Policy Gradient in Reinforcement Learning (2018, background): “Offpolicy multi-step learning [34, 48] have been demonstrated to be stable in recent works but require complementary correction mechanisms like importance sampling, Retrace [37, 59] and V-trace [14] which can be computationally expensive and limiting.”
Flexible Option Learning (2021, background): “It would be possible to consider different approaches to correct for difference between intra-option policies [Mahmood et al., 2014, Li et al., 2015, Harutyunyan et al., 2016, Jiang and Li, 2016, Jain and Precup, 2018] which we leave for future work.”
Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update (2018, background): “Q(λ) [23], Q∗(λ) [6] and Retrace(λ) [15] modify the target values to allow the on-policy samples to be used interchangeably for on-policy and off-policy learning.”
Revisiting Fundamentals of Experience Replay (2020, methodology): “These findings naturally open further questions as to the interaction between experience replay and more general classes of return estimators based on multi-step, off-policy data, such as variants of Q(λ) (Watkins, 1989; Peng & Williams, 1994; Sutton et al., 2014; Harutyunyan et al., 2016), TreeBack”
Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning (2020, unknown): “Another approach is to use importance sampling to correct the targets for the value function V π to improve the approximation of the discounted sum of rewards under the target policy (Harutyunyan et al., 2016).”
Multi-step Off-policy Learning Without Importance Sampling Ratios (2017, methodology): “Only a few model-free algorithms have been proposed to learn oﬀ-policy without using importance sampling ratios (Precup et al. 2000, van Hasselt 2011, Harutyunyan et al. 2016).”
Convergent Tree-Backup and Retrace with Function Approximation (2017, methodology): “…et al., 2016) provided a uni-ﬁed perspective on several off-policy learning algorithms, namely: those using explicit importance sampling corrections (Precup, 2000) as well as T REE B ACKUP (TB( λ )) (Pre-cup, 2000) and Q ( λ ) π (Harutyunyan et al., 2016) which do not involve importance ratios.”
Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning (2021, methodology): “, Q(λ) [27] or Tree Backup), the corresponding algorithms (ICQ-MA (Q(λ)) or ICQ-MA (Tree Backup)) have poor performances and incorrect estimated values.”