Safe and Efficient Off-Policy Reinforcement Learning

By R´emi Munos, T. Stepleton, Ani Harutyunyan, Marc G. Bellemare. 2016

696 citations recorded by the source corpus.

In this work, we take a fresh look at some old and new algorithms for off-policy, return-based reinforcement learning. Expressing these in a common form, we derive a novel algorithm, Retrace($\lambda$), with three desired properties: (1) it has low variance; (2) it safely uses samples collected from any behaviour policy, whatever its degree of "off-policyness"; and (3) it is efficient as it makes the best use of samples collected from near on-policy behaviour policies. We analyze the contractive nature of the related operator under both off-policy policy evaluation and control settings and derive online sample-based algorithms. We believe this is the first return-based off-policy control algorithm converging a.s. to $Q^*$ without the GLIE assumption (Greedy in the Limit with Infinite Exploration). As a corollary, we prove the convergence of Watkins' Q($\lambda$), which was an open problem since 1989. We illustrate the benefits of Retrace($\lambda$) on a standard suite of Atari 2600 games.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Sample Efficient Actor-Critic with Experience Replay (2016, methodology): “Retrace is an off-policy, return-based algorithm which has low variance and is proven to converge (in the tabular case) to the value function of the target policy for any behavior policy, see Munos et al. (2016).”
Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, neutral): “[38]), or performing off-policy policy evaluation (e.”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “…is able to explore and collect more data (Degris et al., 2012; Riedmiller, 2005) such as learning online from an outdated replay buffer (e.g. (Munos et al., 2016)), or have performed off-policy policy evaluation (Farajtabar et al., 2018; Jiang and Li, 2016; Precup, 2000; Thomas and…”
Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, methodology): “But these methods can be notoriously unstable, and in practice, require a variety of stabilization techniques to ensure more consistent performance (Hasselt et al., 2016; Wang et al., 2016; Munos et al., 2016; Hessel et al., 2017; Fujimoto et al., 2018; Nachum et al., 2018; Fu et al., 2019).”
Addressing Function Approximation Error in Actor-Critic Methods (2018, methodology): “These methods have been shown to be an effective approach, through importance sampling (Precup et al., 2001; Munos et al., 2016), distributed methods (Mnih et al.”
The Reactor: A Sample-Efficient Actor-Critic Architecture (2017, methodology): “For the critic we use the Retrace( λ ) algorithm introduced in (Munos et al., 2016).”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, methodology): “Broadly, these methods can be understood as trading off bias (Sutton et al., 1999; Silver et al., 2014; Lillicrap et al., 2016; Gu et al., 2016) and variance (Precup, 2000; Munos et al., 2016).”
Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic (2016, methodology): “…example, Q-Prop can also be used with other on-policy policy gradient methods such as A3C (Mnih et al., 2016) and off-policy advantage estimation meth-ods such as Retrace( λ ) (Munos et al., 2016), GTD2 (Sutton et al., 2009), emphatic TD (Sutton et al., 2015), and WIS-LSTD (Mahmood et al., 2014).”
Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “RETRACE (Munos et al., 2016) makes use of full returns in the on-policy case, but it introduces a zero-mean random variable at each step, adding variance to empirical estimates in both on- and offpolicy cases.”
When to use parametric models in reinforcement learning? (2019, background): “Note that several algorithms exist that correct the return towards a desired policy (Harutyunyan et al., 2016; Munos et al., 2016), which is a separate issue from off-policy sampling of states.”
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018, methodology): “This property (which Retrace (Munos et al., 2016) does not have) allows one to use the same algorithm for off- and on-policy data.”
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, background): “…of value functions (van Hasselt et al., 2016), new operators that increase the action-gap of value function estimates (Bellemare, Ostrovski, Guez, Thomas, & Munos, 2016a), and more robust off-policy multi-step algorithms (Harutyunyan, Bellemare, Stepleton, & Munos, 2016; Munos et al., 2016).”
Experience Replay for Continual Learning (2018, methodology): “Moreover, while the V-Trace algorithm we use is effective at off-policy correction for small shifts between the present and past policy distributions, it is possible that off-policy approaches leveraging Q-functions, such as Retrace [21], may prove more powerful still.”
RUDDER: Return Decomposition for Delayed Rewards (2018, background): “Methods like Retrace [78] are able to correct for being too far from the actual policy but introduce an exponential decay.”
Meta-Gradient Reinforcement Learning (2018, background): “There are potentially many other design choices that may be represented in the return, including off-policy corrections [Espeholt et al., 2018, Munos et al., 2016], target networks [Mnih et al., 2015], emphasis on certain states [Sutton et al., 2016], reward clipping [Mnih et al., 2013], or even…”
Causally Correct Partial Models for Reinforcement Learning (2020, background): “…Importance-sampling has been extensively explored for off-policy policy evaluation (Sutton & Barto, 2018; Precup, 2000; Precup et al., 2001; Munos et al., 2016; Espeholt et al., 2018; Nachum et al., 2018) as well as for counter-factual policy evaluation (Buesing et al., 2019) and for…”
Structured agents for physical construction (2019, methodology): “RS0 learning algorithm : For continuous action outputs, we use an actor-critic learning algorithm that combines re-trace with stochastic value gradients (denoted RS0) (Munos et al., 2016; Heess et al., 2015; Riedmiller et al., 2018).”
Off-Policy Deep Reinforcement Learning without Exploration (2018, background): “Additionally, off-policy algorithms which rely on importance sampling (Precup et al., 2001; Jiang & Li, 2016; Munos et al., 2016) may not be applicable in a batch setting, requiring access to the action probabilities under the behavioral policy, and scale poorly to multi-dimensional action spaces.”