The Reactor: A Sample-Efficient Actor-Critic Architecture

By Audrunas Gruslys, Mohammad Gheshlaghi Azar, Marc G. Bellemare, R´emi Munos. 2017

59 citations recorded by the source corpus.

In this work we present a new reinforcement learning agent, called Reactor (for Retrace-actor), based on an off-policy multi-step return actor-critic architecture. The agent uses a deep recurrent neural network for function approximation. The network outputs a target policy {\pi} (the actor), an action-value Q-function (the critic) evaluating the current policy {\pi}, and an estimated behavioral policy {\hat \mu} which we use for off-policy correction. The agent maintains a memory buffer filled with past experiences. The critic is trained by the multi-step off-policy Retrace algorithm and the actor is trained by a novel {\beta}-leave-one-out policy gradient estimate (which uses both the off-policy corrected return and the estimated Q-function). The Reactor is sample-efficient thanks to the use of memory replay, and numerical efficient since it uses multi-step returns. Also both acting and learning can be parallelized. We evaluated our algorithm on 57 Atari 2600 games and demonstrate that it achieves state-of-the-art performance.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, background): “…the standard, on-policy policy gradient formulation to update the actor (Peters & Schaal, 2008), and many of them also consider the entropy of the policy, but instead of maximizing the entropy, they use it as an regularizer (Schul-man et al., 2017b; 2015; Mnih et al., 2016; Gruslys et al., 2017).”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, methodology): “Hence, current methods remain potentially unstable and require specialized algorithmic and theoretical development as well as delicate tuning to be effective in practice [10, 41, 8].”
The Mirage of Action-Dependent Baselines in Reinforcement Learning (2018, background): “The state-action-dependent baselines (Gu et al., 2017b; Wu et al., 2018; Liu et al., 2018; Grathwohl et al., 2018; Gruslys et al., 2017) can be seen as the natural extension of the doubly robust estimator to the policy gradient setting.”
Self-Imitation Learning (2018, methodology): “For example, ACER (Wang et al., 2017) and Reactor (Gruslys et al., 2018) use Retrace (Munos et al., 2016) to evaluate the learner from the behavior policy.”
Deep Q-learning From Demonstrations (2017, neutral): “Note that we do not compare with A3C (Mnih et al. 2016) or Reactor (Gruslys et al. 2017) as the only published results are for human starts, and we do not compare with UNREAL (Jaderberg et al. 2016) as they select the best hyper-parameters per game.”
AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, background): “However, this approach is prone to high variance and instability without appropriate damping (Munos et al., 2016; Wang et al., 2016; Gruslys et al., 2017; Schulman et al., 2017).”
More Robust Doubly Robust Off-policy Evaluation (2018, background): “More generally, in off-policy actorcritic algorithms (such as the Reactor algorithm proposed in Gruslys et al. 2017), where one can view the gradient estimation part as an off-policy value evaluation problem, the DM state-action value function model is learned by minimizing the Bellman residual in…”
Soft Actor-Critic Algorithms and Applications (2018, background): “…on the standard, on-policy policy gradient formulation to update the actor (Peters & Schaal, 2008), and many of them also consider the entropy of the policy, but instead of maximizing the entropy, they use it as an regularizer (Schulman et al., 2017b, 2015; Mnih et al., 2016; Gruslys et al., 2017).”
Learning from Demonstrations for Real World Reinforcement Learning (2017, methodology): “We compared DQfD’s scores over 200 million steps with that of DQN, Double DQN, Prioritized DQN, Dueling DQN, Reactor, DQN+CTS, and DQN+PixelCNN Mnih et al. (2015); van Hasselt et al. (2016); Schaul et al. (2016); Wang et al. (2016); Gruslys et al. (2017); Ostrovski et al. (2017).”
Count-Based Exploration with Neural Density Models (2017, methodology): “To demonstrate the wide applicability of the PixelCNN exploration bonus, we also evaluate it with the more recent Reactor agent 3 (Gruslys et al., 2017).”
Multi-task Deep Reinforcement Learning with PopArt (2018, background): “…et al. 2017), continuous control (Lillicrap et al. 2016; Duan et al. 2016), classic video-games such as Atari (Mnih et al. 2015; Hessel et al. 2018; Gruslys et al. 2018; Schulman et al. 2015; Schulman et al. 2017; Bacon, Harb, and Precup 2017), and 3D ﬁrst person environments (Mnih et al. 2016;…”
Playing hard exploration games by watching YouTube (2018, background): “Despite the recent advancements in deep reinforcement learning algorithms [7, 9, 17, 19] and architectures [22, 35], there are many “hard exploration” challenges, characterized by particularly sparse environment rewards, that continue to pose a difficult challenge for existing RL agents.”
A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning (2017, methodology): “The main network architecture is based on the default Reactor network, except without a head for estimated behavior distributions (using purely memorized behavior probabilities), as illustrated in Figure 7.”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, methodology): “[3], and they were adopted as a primary source of stochasticity in most subsequent papers working in the Atari domain [3, 26, 27, 34, 38, 41, 42, 45, 67–73].”
Importance Sampling Policy Evaluation with an Estimated Behavior Policy (2018, background): “[5] who estimate the behavior policy for multi-step Q-learning with experience replay.”
Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents (2018, background): “For example, Gruslys et al. (2017); Wang et al. (2016); Gu et al. (2017) have proposed sample-efficient actor-critic algorithms based on experience replay (Mnih et al., 2015) and off-policy learning (Munos et al., 2016).”
An Introduction to Deep Reinforcement Learning (2018, unknown): “That approach was used in actor-critic architectures described by Wang et al. (2016b) and Gruslys et al. (2017).”