Continuous control with deep reinforcement learning
By Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra. International Conference on Learning Representations, 2015
15,687 citations recorded by the source corpus.
We adapt the ideas underlying the success of Deep Q-Learning to the continuous action domain. We present an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces. Using the same learning algorithm, network architecture and hyper-parameters, our algorithm robustly solves more than 20 simulated physics tasks, including classic problems such as cartpole swing-up, dexterous manipulation, legged locomotion and car driving. Our algorithm is able to find policies whose performance is competitive with those found by a planning algorithm with full access to the dynamics of the domain and its derivatives. We further demonstrate that for many of the tasks the algorithm can learn policies end-to-end: directly from raw pixel inputs.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering, Reinforcement Learning
View Paper: https://arxiv.org/abs/1509.02971
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Yoshua Bengio
- Mohit Bansal
- Chelsea Finn
- Ruslan Salakhutdinov
- Quoc V. Le
- Jianfeng Gao
In Computational Theory and Mathematics
- Sergey Levine
- Pieter Abbeel
- Ofir Nachum
- Shimon Whiteson
- Chelsea Finn
- Shixiang Shane Gu
- Nicolas Manfred Otto Heess
- Jan Peters
In Control and Systems Engineering
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Abhishek Gupta
- Jan Peters
- Vikash Kumar
- Shixiang Shane Gu
- Abhinav Kumar Gupta
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Shimon Whiteson
- Ofir Nachum
- Shixiang Shane Gu
- Yuxi Li
- Abhishek Gupta
Citations in the corpus
- Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “While most RL methods use fixed target networks, BYOL uses a weighted moving average of previous networks (as in [54]) in order to provide smoother changes in the target representation.”
- Emergence of Locomotion Behaviours in Rich Environments (2017, background): “[10, 7, 11, 12, 6, 13]), although impressive, have focused on comparatively small problems, and the use of large-scale, distributed optimization is less widespread and the corresponding algorithms are less well developed (but see e.”
- Sample Efficient Actor-Critic with Experience Replay (2016, neutral): “Policy gradient methods have been at the heart of significant advances in AI and robotics (Silver et al., 2014; Lillicrap et al., 2015; Silver et al., 2016; Levine et al., 2015; Mnih et al., 2016; Schulman et al., 2015a; Heess et al., 2015).”
- Benchmarking Deep Reinforcement Learning for Continuous Control (2016, methodology): “Notable examples are training agents to play Atari games based on raw pixels (Guo et al., 2014; Mnih et al., 2015; Schulman et al., 2015a) and to acquire advanced manipulation skills using raw sensory inputs (Levine et al., 2015; Lillicrap et al., 2015; Watter et al., 2015).”
- Asynchronous Methods for Deep Reinforcement Learning (2016, methodology): “the physics models and task objectives) are near identical to the tasks examined in [Lillicrap et al., 2015].”
- High-Dimensional Continuous Control Using Generalized Advantage Estimation (2015, background): “In concurrent work, researchers have been developing policy gradient methods that involve differentiation with respect to the continuous-valued action (Lillicrap et al., 2015; Heess et al., 2015).”
- Memory-based control with recurrent neural networks (2015, methodology): “We primarily consider the Deterministic Policy Gradient algorithm (DPG) [24], which is an off-policy policy gradient algorithm that has recently produced promising results on a broad range of difficult, highdimensional continuous control problems, including direct control from pixels [14].”
- Learning Continuous Control Policies by Stochastic Value Gradients (2015, methodology): “Unlike DPG, instead of assuming a deterministic policy, SVG(0) estimates the derivative around the policy noise E p ( η ) Q a π θ (cid:12)(cid:12) η .”
- Programmable Agents (2017, methodology): “All of our agents are trained using Deterministic Policy Gradient [31, 21], where both the actor and the critic are Programmable Networks.”
- Scalable agent alignment via reward modeling: a research direction (2018, background): “In recent years the machine learning community has made great strides in designing more and more capable deep reinforcement learning algorithms, both value-based methods derived from Q-learning (Mnih et al., 2015) and policy-gradient methods (Schulman et al., 2015; Lillicrap et al., 2015).”
- Constrained Policy Optimization (2017, background): “…performance on many high-dimensional control tasks, including Atari games (using pixels as inputs) (Mnih et al., 2015; 2016), robot locomotion and manipulation (Schulman et al., 2015; Levine et al., 2016; Lillicrap et al., 2016), and even Go at the human grandmaster level (Silver et al., 2016).”
- Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, methodology): “DDPG, TD3, and SAC are not easily applicable to discrete action spaces due to their need to back-propagate from the Q-function to the policy.”
- Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control (2016, methodology): “Additional techniques such as a replay memory (Mnih et al., 2013) are used to stabilize and improve learning.”
- Pre-Trained Language Models for Interactive Decision-Making (2022, methodology): “Policy learning and LM. Traditional policy learning methods, such as PPO [37], DQN [27], DDPG [21], A3C [26], perform well on playing tasks on Atari, OpenAI gym [5], and MuJoCo [41].”
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, methodology): “A commonly used algorithm in such settings, deep deterministic policy gradient (DDPG) (Lillicrap et al., 2015), provides for sample-efficient learning but is notoriously challenging to use due to its extreme brittleness and hyperparameter sensitivity (Duan et al., 2016; Henderson et al., 2017).”
- A Convergence Theory for Deep Learning via Over-Parameterization (2018, background): “Neural networks have demonstrated a great success in numerous machine-learning tasks (Amodei et al., 2016; Graves et al., 2013; He et al., 2016; Krizhevsky et al., 2012; Lillicrap et al., 2015; Silver et al., 2016; 2017).”