End-to-End Training of Deep Visuomotor Policies
By Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel. 2015
3,853 citations recorded by the source corpus.
Policy search methods can allow robots to learn control policies for a wide range of tasks, but practical applications of policy search often require hand-engineered components for perception, state estimation, and low-level control. In this paper, we aim to answer the following question: does training the perception and control systems jointly end-to-end provide better performance than training each component separately? To this end, we develop a method that can be used to learn policies that map raw image observations directly to torques at the robot's motors. The policies are represented by deep convolutional neural networks (CNNs) with 92,000 parameters, and are trained using a partially observed guided policy search method, which transforms policy search into supervised learning, with supervision provided by a simple trajectory-centric reinforcement learning method. We evaluate our method on a range of real-world manipulation tasks that require close coordination between vision and control, such as screwing a cap onto a bottle, and present simulated comparisons to a range of prior policy search methods.
Subfields: Aerospace Engineering, Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Aerospace Engineering
- Sergey Levine
- Jitendra Malik
- Abhinav Kumar Gupta
- Silvio Savarese
- Kristen Grauman
- Pieter Abbeel
- Dieter Fox
- Vladlen Koltun
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Yoshua Bengio
- Chelsea Finn
- Mohit Bansal
- Ruslan Salakhutdinov
- Trevor Darrell
- Quoc V. Le
In Control and Systems Engineering
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Jan Peters
- Vikash Kumar
- Abhishek Gupta
- Dieter Fox
- Abhinav Kumar Gupta
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Ofir Nachum
- Shimon Whiteson
- Shixiang Shane Gu
- Yuxi Li
- Abhishek Gupta
Citations in the corpus
- Continuous control with deep reinforcement learning (2015, neutral): “, [17]) decomposes the problem into three phases that are relatively easy to solve: first, it uses full-state observations to create locally-linear approximations of the dynamics around one or more nominal trajectories, and then uses optimal control to find the locally-linear optimal policy along th”
- Domain randomization for transferring deep neural networks from simulation to the real world (2017, neutral): “Learning in simulation is especially promising for building on recent results using deep reinforcement learning to achieve human-level performance on tasks like Atari [29] and robotic control [26], [41].”
- Learning to Optimize Neural Nets (2017, methodology): “The reinforcement learning method we use is guided policy search (GPS) (Levine et al., 2015), which is a policy search method designed for searching over large classes of expressive non-linear policies in continuous state and action spaces.”
- RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, background): “We episodes We only need to sample from this distribution. use n to denote the total number of allowed to spend with a specific MDP.”
- Emergence of Locomotion Behaviours in Rich Environments (2017, background): “[10, 7, 11, 12, 6, 13]), although impressive, have focused on comparatively small problems, and the use of large-scale, distributed optimization is less widespread and the corresponding algorithms are less well developed (but see e.”
- Sample Efficient Actor-Critic with Experience Replay (2016, positive): “Policy gradient methods have been at the heart of significant advances in AI and robotics (Silver et al., 2014; Lillicrap et al., 2015; Silver et al., 2016; Levine et al., 2015; Mnih et al., 2016; Schulman et al., 2015a; Heess et al., 2015).”
- Benchmarking Deep Reinforcement Learning for Continuous Control (2016, background): “Notable examples are training agents to play Atari games based on raw pixels (Guo et al., 2014; Mnih et al., 2015; Schulman et al., 2015a) and to acquire advanced manipulation skills using raw sensory inputs (Levine et al., 2015; Lillicrap et al., 2015; Watter et al., 2015).”
- Asynchronous Methods for Deep Reinforcement Learning (2016, background): “The spatial softmax proposed by (Levine et al., 2015) could improve both value-based and policy-based methods by making it easier for the network to represent feature coordinates.”
- Memory-based control with recurrent neural networks (2015, methodology): “Several recent papers successfully apply model-free, direct policy search methods to the problem of learning neural network control policies for challenging continuous domains with many degrees of freedoms [2, 6, 14, 21, 22, 12].”
- Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies (2015, methodology): “Keywords: policy gradient, reinforcement learning, deep learning, gradient estimation, temporal difference learning”
- From Pixels to Torques: Policy Learning with Deep Dynamical Models (2015, methodology): “Along these lines, (Deisenroth et al., 2009; Abramova et al., 2012; Boedecker et al., 2014; Pan & Theodorou, 2014; Levine et al., 2015) suggested to first learn a transition model and then use optimal control methods to solve RL problems.”
- One-Shot Imitation Learning (2017, background): “…including Backgammon (Tesauro, 1995), helicopter control (Ng et al., 2003), Atari (Mnih et al., 2015), Go (Silver et al., 2016), continuous control in simulation (Schulman et al., 2015; Heess et al., 2015; Lillicrap et al., 2015) and on real robots (Peters & Schaal, 2008; Levine et al., 2016).”
- Constrained Policy Optimization (2017, background): “…performance on many high-dimensional control tasks, including Atari games (using pixels as inputs) (Mnih et al., 2015; 2016), robot locomotion and manipulation (Schulman et al., 2015; Levine et al., 2016; Lillicrap et al., 2016), and even Go at the human grandmaster level (Silver et al., 2016).”
- Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “…Nachum et al., 2017a,b), model-based algorithms based on the guided policy search framework (Levine and Abbeel, 2014; Levine and Koltun, 2014; Levine et al., 2016), and a variety of methods based on soft Q-learning (Haarnoja et al., 2017; Schulman et al., 2017) and soft actor-critic…”
- Adversarial Training for High-Stakes Reliability (2022, background): “Advances in deep learning have led to increasingly powerful AI systems, for example in sequential decision making [1, 2, 3, 4], robotics [5, 6], and language modeling and text-based reasoning [7, 8, 9, 10, 11].”
- Generative Language Modeling for Automated Theorem Proving (2020, background): “Artificial neural networks have enjoyed a spectacularly successful decade, having made considerable advances in computer vision [1, 2], translation [3, 4, 5], speech recognition [6, 7], image generation [8, 9, 10, 11, 12], game playing [13, 14, 15], and robotics [16, 17].”
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, background): “In guided policy search (Levine & Koltun, 2013; Levine et al., 2016), the maximum entropy distribution is used to guide policy learn-Soft ing towards high-reward regions.”
- Data-efficient Deep Reinforcement Learning for Dexterous Manipulation (2017, methodology): “Recent works have had some success learning flexible neural network policies directly on real robots (e.g. [18, 5, 39]), but tasks as complex as grasping-and-stacking remain daunting.”
- Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, background): “There has been a surge of recent interest in using neural networks for both value and policy based reinforcement learning (e.g., Mnih et al. (2013); Schulman et al. (2015); Levine et al. (2016); Silver et al. (2016)).”
- Reinforcement Learning with Deep Energy-Based Policies (2017, background): “…direction for autonomous acquisition of complex behaviors (Mnih et al., 2015; Silver et al., 2016), due to its ability to process complex sensory input (Jaderberg et al., 2016) and to acquire elaborate behavior skills using general-purpose neural network representations (Levine et al., 2016).”