A Generalized Path Integral Control Approach to Reinforcement Learning

By Evangelos A. Theodorou, Jonas Buchli, Stefan Schaal. 2010

700 citations recorded by the source corpus.

Subfields: Control and Systems Engineering, Reinforcement Learning, Game & Decision Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Control and Systems Engineering

In Reinforcement Learning

In Game & Decision Theory

Citations in the corpus

Learning Parameterized Skills (2012, methodology): “We use the PI 2 algorithm [79] to learn individual policies.”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, methodology): “sidered the special case of learning from demonstration data. One class of algorithms initializes the policy via behavioral cloning from demonstrations, and then ﬁne-tunes with reinforcement learning [33, 15, 40, 18, 35, 10, 50]. Most such methods use on-policy ﬁnetuning, which is less sample-efﬁcie”
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations (2017, methodology): “b) Model-free reinforcement learning: Model-free RL methods [47], [49] and versions with deep function approximators [43], [29] do not require a model of the dynamics, and instead optimize the policy directly.”
Maximum a Posteriori Policy Optimisation (2018, methodology): “Among these are methods that perform trajectory optimisation using either linearised dynamics (Todorov, 2008; Toussaint, 2009; Levine & Koltun, 2013) or general dynamics as in path integral control (Kappen, 2005; Theodorou et al., 2010).”
Collective robot reinforcement learning with distributed asynchronous guided policy search (2016, methodology): “Policy search techniques show promising ability to learn feedback control policies for robotic tasks with highdimensional sensory inputs through trial and error [1, 2, 3, 4].”
Path integral guided policy search (2016, methodology): “In this section, we provide background on guided policy search (GPS) [17] and describe the general framework of model-free stochastic trajectory optimization using policy improvement with path integrals (PI(2)) [7], which serve as the foundations for our algorithm.”
Learning dexterous manipulation for a soft robotic hand from human demonstrations (2016, methodology): “ipulation problems can be found using reinforcement learning. A widely used approach is to learn the parameters of a dynamic motor primitive [13] (DMP) with relative entropy policy search [14] or PI2 [15]. This has been used to learn striking movements [16] and bi-manual transportation tasks [17]. A”
Unsupervised Perceptual Rewards for Imitation Learning (2016, methodology): “A.3 PI2 REINFORCEMENT LEARNING ALGORITHM We chose the PI2 reinforcement learning algorithm (Theodorou et al., 2010) for our experiments, with the particular implementation of the method based on a recently proposed deep reinforcement learning variant (Chebotar et al., 2016).”
Path Integral Policy Improvement with Covariance Matrix Adaptation (2012, methodology): “…Heidrich-Meisner and Igel, 2008; R¨uckstiess et al., 2010) (e.g. it is shown to outperform a particular gradient-based method), the PI 2 derivation (Theodorou et al., 2010) demonstrates that there is a theoretically sound motivation for using methods based on probability-weighted averaging, as…”
Value and reward based learning in neurorobots (2013, methodology): “, Theodorou et al. (2010) and Braun et al. (2011). These constraints are not necessary in active inference.”
Differentiable MPC for End-to-end Planning and Control (2018, methodology): “…synthetic training data can be generated by model-based control algorithms to guide the policy search or prime a model-free policy [Sutton, 1990, Theodorou et al., 2010, Levine and Abbeel, 2014, Gu et al., 2016b, Venkatraman et al., 2016, Levine et al., 2016, Chebotar et al., 2017, Nagabandi et…”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “The most common exploration strategies involve adding some form of noise to action selection, whether it be in the form of direct perturbation of policy parameters [384], Gaussian noise added to continuous actions [246], or epsilon-greedy or softmax action selection in discrete action spaces [264].”
Reinforcement learning in robotic applications: a comprehensive survey (2021, background): “This algorithm doesn’t have an open tuning parameter, except exploration noise.”