Path integral guided policy search

By Yevgen Chebotar, Mrinal Kalakrishnan, Ali Yahya, Adrian Shuai Li, Stefan Schaal, Sergey Levine. 2016

157 citations recorded by the source corpus.

3Sergey Levine is with Google Brain, Mountain View, CA 94043, USA. We present a policy search method for learning complex feedback control policies that map from high-dimensional sensory inputs to motor torques, for manipulation tasks with discontinuous contact dynamics. We build on a prior technique called guided policy search (GPS), which iteratively optimizes a set of local policies for specific instances of a task, and uses these to train a complex, high-dimensional global policy that generalizes across task instances. We extend GPS in the following ways: (1) we propose the use of a model-free local optimizer based on path integral stochastic optimal control (PI2), which enables us to learn local policies for tasks with highly discontinuous contact dynamics; and (2) we enable GPS to train on a new set of task instances in every iteration by using on-policy sampling: this increases the diversity of the instances that the policy is trained on, and is crucial for achieving good generalization. We show that these contributions enable us to learn deep neural network policies that can directly perform torque control from visual input. We validate the method on a challenging door opening task and a pick-and-place task, and we demonstrate that our approach substantially outperforms the prior LQR-based local policy optimizer on these tasks. Furthermore, we show that on-policy sampling significantly increases the generalization ability of these policies.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates (2016, background): “As shown in recent work, this can be difficult in domains with severe discontinuities in the dynamics and reward function [24].”
Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning (2017, background): “Even in cases where failures are not catastrophic, some degree of human intervention is often required to reset the environment between attempts (e.g. Chebotar et al. (2017)).”
Self-Supervised Correspondence in Visuomotor Policy Learning (2019, background): “The model is close to [1]–[3] with the key architectural traits of having a few convolutional layers followed by a channel-wise spatial expectation operation, which has been widely used [10], [11], [25], [29], [36]–[38].”
Benchmarking Model-Based Reinforcement Learning (2019, background): “In this paper, we use the MD-GPS variant [35].”
End-to-End Robotic Reinforcement Learning without Reward Engineering (2019, background): “ids [35], door opening [44, 3], and cloth folding [28].”
Maximum a Posteriori Policy Optimisation (2018, methodology): “This is in contrast to other RL algorithms that use M-projection without KL constraint to ﬁt a parametric policy (Peters et al., 2010a; Wirth et al., 2016; Chebotar et al., 2016; Montgomery & Levine, 2016).”
Hindsight Experience Replay (2017, background): “…al., 2015), and defeating the best human player at the game of Go (Silver et al., 2016), as well as robotic tasks such as helicopter control (Ng et al., 2006), hitting a baseball (Peters and Schaal, 2008), screwing a cap onto a bottle (Levine et al., 2015), or door opening (Chebotar et al., 2016).”
Reinforcement and Imitation Learning for Diverse Visuomotor Skills (2018, methodology): “This result suggests that our model can learn end-to-end visuomotor policies without a pretraining phase or the need of auxiliary tasks, as opposed to previous work on visuomotor learning [3, 23, 52].”
Collective robot reinforcement learning with distributed asynchronous guided policy search (2016, methodology): “We can employ PI(2) to learn feedforward commands kt of time-varying linearGaussian controllers as described in [25].”
Deep predictive policy training using reinforcement learning (2017, methodology): “The encoding inherently preserves spatial distances in the input image and is therefore suitable for robotic manipulation tasks [6], [12], [16], [18].”
Unsupervised Perceptual Rewards for Imitation Learning (2016, methodology): “Following the experimental protocol in prior work [9], we adapt an imperfect kinesthetic demonstration which we ensure succeeds occasionally (about 10% of the time).”
Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methods (2018, unknown): “For example, guided policy search methods have been applied for solving a range of manipulation tasks, including contact-rich, visionbased skills [23], non-prehensile manipulation [10], and tasks involving significant discontinuities [5], [4].”
Robots in machining (2019, methodology): “Deep neural network policies have been combined with model-based learning in the context of guided policy search algorithms [221,222].”
Information asymmetry in KL-regularized RL (2019, methodology): “Related but often seen as distinct is the familiy of expectation maximization policy search algorithms (e.g. Peters et al., 2010; Rawlik et al., 2012; Levine & Koltun, 2013; Montgomery & Levine, 2016; Chebotar et al., 2016; Abdolmaleki et al., 2018).”
Policy transfer via modularity (2017, background): “In practice, a number of approximations are usually made to the dynamics model to make it easier to learn and to optimize over (such as assuming linear-Gaussian dynamics as a function of the state [18]), but these approximations might lead to a reduced task accuracy [24].”
Learning manipulation skills from a single demonstration (2018, methodology): “Chebotar et al. (2017) propose an extension of guided policy search with the reinforcement learning method PI 2 .”
The Ingredients of Real-World Robotic Reinforcement Learning (2020, background): “…of environmental instrumentation in order to deﬁne reward functions, reset between trials, and obtain ground truth state (Levine et al., 2016; Haarnoja et al., 2018a; Kumar et al., 2016; Andrychowicz et al., 2018; Zhu et al., 2019; Chebotar et al., 2016; Nagabandi et al., 2019; Gupta et al., 2016).”
A Game Theoretic Framework for Model Based Reinforcement Learning (2020, background): “GPS [7, 60], DPI [61], and related approaches [62, 63] learn a time varying linear model and perform a KL-constrained policy improvement step.”