Path Integral Policy Improvement with Covariance Matrix Adaptation

By Freek Stulp, Olivier Sigaud. 2012

222 citations recorded by the source corpus.

There has been a recent focus in reinforcement learning on addressing continuous state and action problems by optimizing parameterized policies. PI2 is a recent example of this approach. It combines a derivation from first principles of stochastic optimal control with tools from statistical estimation theory. In this paper, we consider PI2 as a member of the wider family of methods which share the concept of probability-weighted averaging to iteratively update parameters to optimize a cost function. We compare PI2 to other members of the same family - Cross-Entropy Methods and CMAES - at the conceptual level and in terms of performance. The comparison suggests the derivation of a novel algorithm which we call PI2-CMA for "Path Integral Policy Improvement with Covariance Matrix Adaptation". PI2-CMA's main advantage is that it determines the magnitude of the exploration noise automatically.

Subfields: Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

End-to-End Training of Deep Visuomotor Policies (2015, methodology): “CEM and RWR only dier in the weights. These methods represent a class of RL algorithms that t the policy to weighted samples, including PoWER and PI2 (Kober and Peters, 2009; Theodorou et al., 2010; Stulp and Sigaud, 2012). PILCO is a model-based method that uses a Gaussian process to learn a global”
Learning Parameterized Skills (2012, methodology): “In some cases, EM and cross entropy methods can be shown to be equivalent and correspond to special cases of a broader class of learning techniques based on probability-weighted averages of policy parameters [70].”
Reinforcement learning in robotics: A survey (2013, background): “Some recent work including (Tesch et al., 2011; Stulp and Sigaud, 2012) suggest that much of the benefit of policy search is achieved by black-box methods.”
Active learning of inverse models with intrinsically motivated goal exploration in robots (2013, neutral): “Such an architecture also leverages both techniques for optimizing action policy parameters for a single predefined tasks (e.g. [83,84]), as well as regression techniques allowing to infer the motor policy parameters corresponding to a given novel parameterized task, and based on the previously…”
Path integral guided policy search (2016, methodology): “Here, we adapt the approach described in [32] and update the covariance matrices based on the sample probabilities.”
Policy Improvement : Between Black-Box Optimization and Episodic Reinforcement Learning (2013, methodology): “Alternatively , ti , k can be set to have a constant value during a roll - out ( Stulp & Sigaud , 2012a ) .”
A Survey on Policy Search for Robotics (2013, methodology): “T−1 Policy Update: Compute new policy parameters θk+1 using D. Algorithms: REINFORCE, G(PO)MDP, NAC, eNAC, PoWER, PI2 until Policy converges θk+1 ≈ θk”
Information-Theoretic Model Predictive Control: Theory and Applications to Autonomous Driving (2017, background): “This is desirable in the case of open-loop trajectory optimization or parameterized policy learning [37], but in the model predictive control case it is problematic since the environment changes at every step, which means that a good policy can turn catastrophic in a few time-steps.”
Towards Deep Developmental Learning (2016, unknown): “to approximate the value function [141], [131].”
Value and reward based learning in neurorobots (2013, methodology): “The path integral approach has been successfully applied to learning of stochastic policies for robots with large degrees of freedom (Theodorou et al., 2010; Sugimoto and Morimoto, 2011; Stulp and Sigaud, 2012).”
Learning Neural Network Policies with Guided Policy Search under Unknown Dynamics (2014, background): “to weighted samples, including PoWER and PI2 [11, 24, 22].”