Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation

By Yuhuai Wu, Elman Mansimov, Roger Baker Grosse, Shun Liao, Jimmy Ba. 2017

674 citations recorded by the source corpus.

In this work, we propose to apply trust region optimization to deep reinforcement learning using a recently proposed Kronecker-factored approximation to the curvature. We extend the framework of natural policy gradient and propose to optimize both the actor and the critic using Kronecker-factored approximate curvature (K-FAC) with trust region; hence we call our method Actor Critic using Kronecker-Factored Trust Region (ACKTR). To the best of our knowledge, this is the first scalable trust region natural gradient method for actor-critic methods. It is also a method that learns non-trivial tasks in continuous control as well as discrete control policies directly from raw pixel inputs. We tested our approach across discrete domains in Atari games as well as continuous domains in the MuJoCo environment. With the proposed methods, we are able to achieve higher rewards and a 2- to 3-fold improvement in sample efficiency on average, compared to previous state-of-the-art on-policy actor-critic methods. Code is available at this https URL

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Electrical and Electronic Engineering, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Electrical and Electronic Engineering

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Addressing Function Approximation Error in Actor-Critic Methods (2018, methodology): “…our algorithm against DDPG (Lillicrap et al., 2015) as well as the state of art policy gradient algorithms: PPO (Schulman et al., 2017), ACKTR (Wu et al., 2017) and TRPO (Schulman et al., 2015), as implemented by OpenAI’s baselines repository (Dhariwal et al., 2017), and SAC (Haarnoja et al.,…”
Deep Reinforcement Learning that Matters (2017, methodology): “We also include all our experimental results, with baseline algorithms DDPG (Lillicrap et al. 2015b), TRPO (Schulman et al. 2015a), PPO (Schulman et al. 2017) and ACKTR (Wu et al. 2017)) as discussed in the paper.”
Model-Based Reinforcement Learning for Atari (2019, methodology): “, 2017) and actor-critic algorithms (Mnih et al., 2016; Schulman et al., 2017; Babaeizadeh et al., 2016; Wu et al., 2017; Espeholt et al., 2018).”
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018, neutral): “For a level-by-level comparison to ACKTR (Wu et al., 2017) and Reactor see Appendix C.”
Knowledge Flow: Improve Upon Your Teachers (2019, background): “The results of the state-of-the-art methods: A3C (Mnih et al., 2016), PPO (Schulman et al., 2017), and ACKTR (Wu et al., 2017) features are very likely irrelevant to a student’s top layer features.”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, positive): “, 2018): each experiment is repeated with 3 seeds (Wu et al., 2017; Ostrovski et al., 2017), 10 additional validation seeds are used to pick the best model which is then tested on 100 testing seeds.”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “In terms of manipulation, the tasks being considered for evaluating DRL algorithms are more standardized in the recent literature (Lillicrap et al., 2015; Schulman et al., 2015a; Mnih et al., 2016; Heess et al., 2017; Wu et al., 2017).”
Action Semantics Network: Considering the Effects of Actions in Multiagent Systems (2019, methodology): “Figure 9(a), (b) and (c) present the performance of multi-action ASN on Neural MMO compared with vanilla, attention and entity-attention networks under different IL methods (PPO, ACKTR (Wu et al., 2017) and A2C (Mnih et al., 2016)).”
Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning (2017, methodology): “We did not have source code for A3C, but were able to obtain source code for A2C, which has similar performance (Wu et al., 2017): the only difference is that it is synchronous instead of asynchronous.”
Model-Based Reinforcement Learning via Meta-Policy Optimization (2018, methodology): “We compare our method in sample complexity and performance to four state-of-the-art model free RL algorithms: Deep Deterministic Policy Gradient (DDPG) [2], Trust Region Policy Optimization [1], Proximal Policy Optimization (PPO) [45], and Actor Critic using Kronecker-Factored Trust Region (ACKTR) [”
Backpropagation through the Void: Optimizing control variates for black-box gradient estimation (2017, methodology): “, 2015b), or other optimization methods, such as KFAC (Wu et al., 2017).”
Simple random search provides a competitive approach to reinforcement learning (2018, methodology): “As a more scalable trust region method, Wu et al. [37] proposed an actor critic method which uses Kronecker-factor trust regions (ACKTR).”
Three Mechanisms of Weight Decay Regularization (2018, methodology): “Following on that work, K-FAC has been adopted in many tasks (Wu et al., 2017; Zhang et al., 2017) to gain optimization beneﬁts, and was shown to be amendable to distributed computation (Ba et al., 2016).”
Noisy Natural Gradient as Variational Inference (2017, methodology): “Following on that work, K-FAC has been adopted in many tasks (Grosse & Martens, 2016; Wu et al., 2017) to gain optimization beneﬁts, and was shown to be amenable to distributed computation (Ba et al., 2017).”
Deep Variational Reinforcement Learning for POMDPs (2018, methodology): “Despite looking complicated, it is just a very detailed implementation of n -step A2C with the additional changes: Inclusion of L ELBO and inclusing of the option to not delete the computation graph to allow longer backprop in n -step A2C.”
A Kernel Loss for Solving the Bellman Equation (2019, background): “, 2016) and their many variants (e.g., Gu et al., 2016; Schulman et al., 2016; Wang et al., 2016; Wu et al., 2017), whose stability largely depends on the contraction property.”
GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms (2017, positive): “For the exploitation phase, DDPG could be replaced by the more recent ACKTR (Wu et al., 2017) which can in principle be trained off-policy.”