Continuous Deep Q-Learning with Model-based Acceleration

By Shixiang Shane Gu, Timothy P. Lillicrap, Ilya Sutskever, Sergey Levine. 2016

1,075 citations recorded by the source corpus.

Model-free reinforcement learning has been successfully applied to a range of challenging problems, and has recently been extended to handle large neural network policies and value functions. However, the sample complexity of modelfree algorithms, particularly when using high-dimensional function approximators, tends to limit their applicability to physical systems. In this paper, we explore algorithms and representations to reduce the sample complexity of deep reinforcement learning for continuous control tasks. We propose two complementary techniques for improving the efficiency of such algorithms. First, we derive a continuous variant of the Q-learning algorithm, which we call normalized advantage functions (NAF), as an alternative to the more commonly used policy gradient and actor-critic methods. NAF representation allows us to apply Q-learning with experience replay to continuous tasks, and substantially improves performance on a set of simulated robotic control tasks. To further improve the efficiency of our approach, we explore the use of learned models for accelerating model-free reinforcement learning. We show that iteratively refitted local linear models are especially effective for this, and demonstrate substantially faster learning on domains where such models are applicable.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

Density estimation using Real NVP (2016, background): “For example, in reinforcement learning, these invertible functions can help extend the set of functions for which an argmax operation is tractable for continuous Qlearning [23] or find representation where local linear Gaussian approximations are more appropriate [67].”
Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, methodology): “DDPG, TD3, and SAC are not easily applicable to discrete action spaces due to their need to back-propagate from the Q-function to the policy.”
Addressing Function Approximation Error in Actor-Critic Methods (2018, methodology): “In contrast with our method, work has been done on adapting deep Q-learning methods for continuous actions, which replace selection of the action with the highest estimated value with various approximations (Gu et al., 2016; Metz et al., 2017; Tavakoli et al., 2017).”
Data-efficient Deep Reinforcement Learning for Dexterous Manipulation (2017, methodology): “And indeed, in simulation, recent work has used neural networks to learn solutions to a variety of control problems from scratch (e.g. [7, 20, 32, 31, 11, 17]).”
Reinforcement Learning with Deep Energy-Based Policies (2017, background): “Other beneﬁts include robustness in the face of uncertain dynamics (Ziebart, 2010), imitation learning (Ziebart et al., 2008), and improved convergence and computational properties (Gu et al., 2016a).”
Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic (2016, methodology): “…in RL but for approximate inference in stochastic models (Paisley et al., 2012), and the closest related work in that domain is the MuProp algorithm (Gu et al., 2016a) which uses a mean-field network as a surrogate for backpropagating a deterministic gradient through stochastic discrete variables.”
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates (2016, methodology): “algorithm (NAF) [11] can achieve training times that are suitable for real robotic systems.”
TreeQN and ATreeC: Differentiable Tree Planning for Deep Reinforcement Learning (2017, methodology): “Gu et al. (2016) use local linear models to generate additional samples for their model-free algorithm.”
Value Prediction Network (2017, background): “[7] extended these ideas to continuous control problems.”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “, 2019), but our method emphasizes the connections among neighboring pivotal states rather than transitions over any arbitrary pair, which is usually deemed as a much harder problem (Gu et al., 2016).”
Towards Deep Symbolic Reinforcement Learning (2016, background): “Each of these shortcomings is an active area of research in the DRL community, where data efﬁcient learning [8, 9], transfer learning [10, 11, 12, 13], planning [14, 15], and transparency [16] are all hot topics.”
Data Augmentation Generative Adversarial Networks (2017, background): “, 2017), learning from human play (Clark & Storkey, 2015) and reinforcement learning (Mnih et al., 2015; Silver et al., 2016; Foerster et al., 2016; Van Hasselt et al., 2016; Gu et al., 2016) among others.”
The Mirage of Action-Dependent Baselines in Reinforcement Learning (2018, methodology): “Prior work has explored model-based (Sutton, 1990; Heess et al., 2015; Gu et al., 2016) and off-policy critic-based gradient estimators (Lillicrap et al., 2015).”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “Gu et al. (2017) propose an asynchronous version of NAF.”
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, methodology): “…produce model-based RL algorithms that are competitive with their model-free counterparts in terms of asymptotic performance [Nagabandi et al., 2017], and often even found that simpler time-varying linear models can outperform expressive neural network models [Levine et al., 2016, Gu et al., 2016].”
Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning (2017, methodology): “Although a number of prior works have attempted to mitigate these shortcomings by using large, expressive neural networks to model the complex dynamical systems typically used in deep reinforcement learning benchmarks [7], [8], such models often do not perform well [9] and have been limited to relat”
Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning (2017, methodology): “If the policy is locally Gaussian, i.e. π θ ( a t | s t ) = N ( µ θ ( s t ) , Σ θ ( s t )) , then the quadratic Q w from Normalized Advantage Function (NAF) (Gu et al., 2016) can be directly used, Q w ( s t , a t ) = A w ( s t , a t ) + V w ( s t ) A w ( s t , a t ) = − 1 2( a t − µ w ( s t )) T P w”
The limits and potentials of deep learning for robotics (2018, background): “Deep reinforcement learning is a new learning paradigm that is capable of learning end-to-end robotic control tasks, but the accomplishments have been demonstrated primarily in simulation, rather than on actual robot platforms [37], [43], [62], [64], [77], [106], [107].”