Guided Policy Search as Approximate Mirror Descent

By William H. Montgomery, Sergey Levine. 2016

129 citations recorded by the source corpus.

Guided policy search algorithms can be used to optimize complex nonlinear policies, such as deep neural networks, without directly computing policy gradients in the high-dimensional parameter space. Instead, these methods use supervised learning to train the policy to mimic a "teacher" algorithm, such as a trajectory optimizer or a trajectory-centric reinforcement learning method. Guided policy search methods provide asymptotic local convergence guarantees by construction, but it is not clear how much the policy improves within a small, finite number of iterations. We show that guided policy search algorithms can be interpreted as an approximate variant of mirror descent, where the projection onto the constraint manifold is not exact. We derive a new guided policy search algorithm that is simpler and provides appealing improvement and convergence guarantees in simplified convex and linear settings, and show that in the more general nonlinear setting, the error in the projection step can be bounded. We provide empirical results on several simulated robotic navigation and manipulation tasks that show that our method is stable and achieves similar or better performance when compared to prior guided policy search methods, with a simpler formulation and fewer hyperparameters.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Benchmarking Model-Based Reinforcement Learning (2019, methodology): “In this paper, we use the MD-GPS variant [35].”
Maximum a Posteriori Policy Optimisation (2018, methodology): “This is in contrast to other RL algorithms that use M-projection without KL constraint to ﬁt a parametric policy (Peters et al., 2010a; Wirth et al., 2016; Chebotar et al., 2016; Montgomery & Levine, 2016).”
Collective robot reinforcement learning with distributed asynchronous guided policy search (2016, methodology): “Next, we run 16 iterations of asynchronous distributed MDGPS with PI2, where we randomly perturb each door pose at the start of every iteration.”
Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task (2017, background): “These end-to-end approaches have had success in both real world [10, 14, 13] and in simulated worlds [3].”
Batch Policy Learning under Constraints (2019, background): “Many policy search algorithms perform small policy update steps, requiring the new policy π to stay within a neighborhood of the most recent policy iterate πk to ensure learning stability (Levine & Abbeel, 2014; Schulman et al., 2015; Montgomery & Levine, 2016; Achiam et al., 2017).”
Learning by Playing - Solving Sparse Reward Tasks from Scratch (2018, background): “…et al. (2017) for recent examples), learning from demonstrations (Ross et al., 2011; Vecerik et al., 2017; Kober & Peters, 2011; Sermanet et al., 2017; Nair et al., 2017), learning with model guidance, see e.g. Montgomery & Levine (2016), or inverse RL (Ng & Russell, 2000; Ziebart et al., 2008).”
Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines (2018, methodology): “We use the blind peg-insertion task which is widely studied in the robot learning literature (Montgomery & Levine, 2016).”
A unified view of entropy-regularized Markov decision processes (2017, background): “This connection is not new: it has been ﬁrst shown by Zimin and Neu (2013) 2 , and has been recently rediscovered by Montgomery and Levine (2016).”
Reset-free guided policy search: Efficient deep reinforcement learning with stochastic initial states (2016, methodology): “Both of these quantities can be computed analytically under the current and previous linear-Gaussian dynamics and quadratic cost, via the formula for the expectation of a quadratic form under a Gaussian [16].”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, background): “Although guided policy search methods that can handle random initial states have been developed (Montgomery and Levine 2016; Montgomery et al. 2017), they still rely on clustering the initial states into a discrete set of “similar” states, which may be impractical in some cases, such as the diverse…”
Deep reinforcement learning for tensegrity robot locomotion (2016, methodology): “rithm that frames the guided policy search (GPS) alternating optimization framework as approximate mirror descent [10].”
Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning (2017, methodology): “MDGPS with LQR-FLM and MDGPS with PILQR perform competitively in terms of the ﬁnal distance from the end effector to the target, which is unsur-prising given the simplicity of the task, whereas MDGPS with PI 2 is again not able to make much progress.”
Information asymmetry in KL-regularized RL (2019, background): “Related but often seen as distinct is the familiy of expectation maximization policy search algorithms (e.g. Peters et al., 2010; Rawlik et al., 2012; Levine & Koltun, 2013; Montgomery & Levine, 2016; Chebotar et al., 2016; Abdolmaleki et al., 2018).”
Generalizing Skills with Semi-Supervised Reinforcement Learning (2016, unknown): “…method in this step, our prototype uses mirror descent guided policy search (MDGPS), a sample-efﬁcient policy optimization method suitable for training complex neural network policies that has been validated on real-world physical robots (Montgomery & Levine, 2016; Montgomery et al., 2016).”
Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Space (2018, methodology): “These methods include natural gradient descent [Kakade, 2002], trust region optimization [Schulman et al., 2015], proximal gradient descent [Schulman et al., 2017], mirror descent [Montgomery and Levine, 2016], and entropy regularization [O’Donoghue et al., 2017].”