Reset-free guided policy search: Efficient deep reinforcement learning with stochastic initial states

By William H. Montgomery, Anurag Ajay, Chelsea Finn, Pieter Abbeel, Sergey Levine. 2016

40 citations recorded by the source corpus.

Autonomous learning of robotic skills can allow general-purpose robots to learn wide behavioral repertoires without extensive manual engineering. However, robotic skill learning must typically make trade-offs to enable practical real-world learning, such as requiring manually designed policy or value function representations, initialization from human demonstrations, instrumentation of the training environment, or extremely long training times. We propose a new reinforcement learning algorithm that can train general-purpose neural network policies with minimal human engineering, while still allowing for fast, efficient learning in stochastic environments. We build on the guided policy search (GPS) algorithm, which transforms the reinforcement learning problem into supervised learning from a computational teacher (without human demonstrations). In contrast to prior GPS methods, which require a consistent set of initial states to which the system must be reset after each episode, our approach can handle random initial states, allowing it to be used even when deterministic resets are impossible. We compare our method to existing policy search algorithms in simulation, showing that it can train high-dimensional neural network policies with the same sample efficiency as prior GPS methods, and can learn policies directly from image pixels. We also present real-world robot results that show that our method can learn manipulation policies with visual features and random initial states.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Dual Policy Iteration (2018, methodology): “The instantiation shares similar spirit from some previous works from robotics and control literature, including works from [3, 5] and Guided Policy Search (GPS) [18] (and its variants (e.g., [19, 22, 21])), i.e., using local MBOC to speed up learning global policies.”
Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task (2017, background): “End-to-end approaches have had success both in the real world [1, 2, 3] and in simulated worlds [4, 5, 6, 7].”
SOLAR: Deep Structured Latent Representations for Model-Based Reinforcement Learning (2018, methodology): “These methods do not use models for long-term forward prediction, allowing for the use of simple models that enable policy improvement [29, 26].”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, background): “Although guided policy search methods that can handle random initial states have been developed (Montgomery and Levine 2016; Montgomery et al. 2017), they still rely on clustering the initial states into a discrete set of “similar” states, which may be impractical in some cases, such as the diverse…”
Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning (2017, methodology): “Recent work proposes a clustering method for lifting this restriction by sampling trajectories from random initial states and assembling them into task instances after the fact (Montgomery et al., 2017).”
Reinforcement learning for non-prehensile manipulation: Transfer from simulation to physical system (2018, background): “Another common issue with learning in the real world is how to reset the state of the system, with some work being done [31].”
Generalizing Skills with Semi-Supervised Reinforcement Learning (2016, methodology): “While we could in principle use any policy optimization method in this step, our prototype uses mirror descent guided policy search (MDGPS), a sample-efficient policy optimization method suitable for training complex neural network policies that has been validated on real-world physical robots (Mont”