Sample Efficient Actor-Critic with Experience Replay

By Ziyun Wang, Victor Bapst, Nicolas Manfred Otto Heess, Volodymyr Mnih, R´emi Munos, Koray Kavukcuoglu, Nando de Freitas. 2016

821 citations recorded by the source corpus.

This paper presents an actor-critic deep reinforcement learning agent with experience replay that is stable, sample efficient, and performs remarkably well on challenging environments, including the discrete 57-game Atari domain and several continuous control problems. To achieve this, the paper introduces several innovations, including truncated importance sampling with bias correction, stochastic dueling network architectures, and a new trust region policy optimization method.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, methodology): “…of the policy parameters [36, 37], by enforcing the new policy to stay Importance Sampling for Reinforcement Learning In RL, IS is generally used for sample ef-ﬁciency purposes: in off-policy policy gradient methods, IS allows to re-use previously sampled sequences more than once [48, 44, 12].”
Emergence of Locomotion Behaviours in Rich Environments (2017, background): “One effective measure is to employ a trust region constraint that restricts the amount by which any update is allowed to change the policy [19, 7, 14].”
Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, background): “But these methods can be notoriously unstable, and in practice, require a variety of stabilization techniques to ensure more consistent performance (Hasselt et al., 2016; Wang et al., 2016; Munos et al., 2016; Hessel et al., 2017; Fujimoto et al., 2018; Nachum et al., 2018; Fu et al., 2019).”
Trust-PCL: An Off-Policy Trust Region Method for Continuous Control (2017, methodology): “These techniques have recently become popular in RL (Wang et al., 2016; Heess et al., 2017; Schulman et al., 2017b), although in terms of final reward performance on continuous control benchmarks, TRPO is still considered to be the state-of-the-art.”
The Reactor: A Sample-Efficient Actor-Critic Architecture (2017, methodology): “…using the β -LOO estimate with β deﬁned as: This truncated 1 /µ coefﬁcient shares similarities with the truncated IS gradient estimate introduced in (Wang et al., 2017) (which we call TISLR for truncated-ISLR): The differences are: which provides an additional variance reduction due to the…”
Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “Hence, most established actor-critic algorithms with replay such as (Wang et al., 2017; Gruslys et al., 2018; Haarnoja et al., 2018) employ and maintain Q-functions to learn from the replayed off-policy experience.”
Distributed Prioritized Experience Replay (2018, methodology): “For methods that use temporally extended sequences (e.g., Mnih et al., 2016; Wang et al., 2017), the Ape-X framework may be adapted to prioritize sequences of past experiences instead of individual transitions.”
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018, methodology): “, 2016) which introduced an off-policy correction for multi-step RL, and has been used in several agent architectures (Wang et al., 2017; Gruslys et al., 2018).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “This approximate gradient is used as a starting point in many widely used deep reinforcement learning algorithms (Silver et al., 2014; Lillicrap et al., 2015; Wang et al., 2016; Gu et al., 2017b).”
Counterfactual Multi-Agent Policy Gradients (2017, background): “…policy gradient is REINFORCE (Williams, 1992), in which the gradient is: (1) In actor-critic approaches (Kimura et al., 2000; Schulman et al., 2015; Wang et al., 2016; Hafner & Riedmiller, 2011), the actor , i.e., the policy, is trained by following a gradient that depends on a critic , which…”
The Mirage of Action-Dependent Baselines in Reinforcement Learning (2018, methodology): “In this direction, several recent methods have successfully mixed high-variance, unbiased on-policy gradient estimates directly with low-variance, biased off-policy or model-based gradient estimates to improve performance (O’Donoghue et al., 2016; Wang et al., 2016b; Gu et al., 2017a).”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, methodology): “Actor Critic with Experience Replay (ACER) [129], is a sample-efﬁcient policy gradient algorithm that makes use of a replay buffer, enabling it to perform more than one gradient update using each piece of sampled experience, as well as a trust region policy optimization method.”
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning (2017, methodology): “Finally, the value functions can be updated using the following gradient: To increase the episode efﬁciency, after running an episode, we conduct n mini-batch updates where n is sampled from a Poisson distribution with λ = 4 , similar to Wang et al. (2017).”
Self-Imitation Learning (2018, methodology): “• Can other off-policy actor-critic methods also exploit good experiences (e.g., ACER (Wang et al., 2017))?”
Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning (2017, methodology): “The ﬁrst approach is to mix some ratio of on-and off-policy gradients or update steps in order to update a policy, as in the ACER and PGQ algorithms (Wang et al., 2017; O’Donoghue et al., 2017).”
AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, background): “However, this approach is prone to high variance and instability without appropriate damping (Munos et al., 2016; Wang et al., 2016; Gruslys et al., 2017; Schulman et al., 2017).”
Generalized Off-Policy Actor-Critic (2019, methodology): “There have been numerous applications of OPPG including DDPG, ACER, IPG, EPG and IMPALA.”