The Reactor: A fast and sample-efficient Actor-Critic agent for Reinforcement Learning

By Audrunas Gruslys, Will Dabney, Mohammad Gheshlaghi Azar, Bilal Piot, Marc G. Bellemare, R´emi Munos. 2017

103 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, background): “…the standard, on-policy policy gradient formulation to update the actor (Peters & Schaal, 2008), and many of them also consider the entropy of the policy, but instead of maximizing the entropy, they use it as an regularizer (Schul-man et al., 2017b; 2015; Mnih et al., 2016; Gruslys et al., 2017).”
Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “Prior art results are presented as horizontal lines (with scores cited from Gruslys et al. (2018), Hessel et al. (2017) and Mnih et al. (2013)).”
Observe and Look Further: Achieving Consistent Performance on Atari (2018, background): “[5] Audrunas Gruslys, Will Dabney, Mohammad Gheshlaghi Azar, Bilal Piot, Marc Bellemare, and Remi Munos.”
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018, methodology): “, 2016) which introduced an off-policy correction for multi-step RL, and has been used in several agent architectures (Wang et al., 2017; Gruslys et al., 2018).”
The Mirage of Action-Dependent Baselines in Reinforcement Learning (2018, background): “The state-action-dependent baselines (Gu et al., 2017b; Wu et al., 2018; Liu et al., 2018; Grathwohl et al., 2018; Gruslys et al., 2017) can be seen as the natural extension of the doubly robust estimator to the policy gradient setting.”
Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift (2019, methodology): “2016), which have also yielded good results on the Atari 2600 (Gruslys et al. 2018).”
More Robust Doubly Robust Off-policy Evaluation (2018, background): “More generally, in off-policy actorcritic algorithms (such as the Reactor algorithm proposed in Gruslys et al. 2017), where one can view the gradient estimation part as an off-policy value evaluation problem, the DM state-action value function model is learned by minimizing the Bellman residual in…”
Implicit Quantile Networks for Distributional Reinforcement Learning (2018, methodology): “…, on the pairwise TD-errors At the time of this writing, QR-DQN achieves the best performance on Atari-57, human-normalized mean and median, of all agents that do not combine distributional RL, prioritized replay, and n -step updates (Dabney et al., 2018; Hessel et al., 2018; Gruslys et al., 2018).”
Soft Actor-Critic Algorithms and Applications (2018, background): “…on the standard, on-policy policy gradient formulation to update the actor (Peters & Schaal, 2008), and many of them also consider the entropy of the policy, but instead of maximizing the entropy, they use it as an regularizer (Schulman et al., 2017b, 2015; Mnih et al., 2016; Gruslys et al., 2017).”
Count-Based Exploration with Neural Density Models (2017, methodology): “To demonstrate the wide applicability of the PixelCNN exploration bonus, we also evaluate it with the more recent Reactor agent 3 (Gruslys et al., 2017).”
Dopamine: A Research Framework for Deep Reinforcement Learning (2018, background): “Since then, it has become commonplace, if not expected, that an agent is composed of multiple interacting components; consider A3C (Mnih et al., 2016), ACER (Wang et al., 2017), Reactor (Gruslys et al., 2018), and IMPALA (Espeholt et al., 2018).”
Deep Reinforcement Learning Robot for Search and Rescue Applications: Exploration in Unknown Cluttered Environments (2019, methodology): “This procedure was repeated until the exploration task was finished or the maximum step size, , of 30 had been reached.”
Recurrent Experience Replay in Distributed Reinforcement Learning (2018, methodology): “…performance by leveraging distributed training architectures which separate learning from acting, collecting data from many actors running in parallel on separate environment instances (Horgan et al., 2018; Espeholt et al., 2018; Gruslys et al., 2018; OpenAI, 2018b;a; Jaderberg et al., 2018).”
Ranking Policy Gradient (2019, methodology): “One often overlooked but decisive factor is the usage of value functions , which have been widely adopted in state-of-the-art (Van Hasselt et al., 2016; Hessel et al., 2017; Mnih et al., 2016; Schulman et al., 2017; Gruslys et al., 2018).”
Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents (2018, methodology): “For example, Gruslys et al. (2017); Wang et al. (2016); Gu et al. (2017) have proposed sample-e ﬃ cient actor-critic algorithms based on experience replay (Mnih et al., 2015) and o ﬀ -policy learning (Munos et al., 2016).”
Mastering Diverse Domains through World Models (2023, methodology): “Voyager obtains items at a similar depth in the technology tree as Dreamer using API calls to a language model but operates on top of the MineFlayer bot scripting layer that was specifically engineered to the game and exposes high-level actions 54 .”