Reinforcement learning of motor skills in high dimensions: A path integral approach

By Evangelos A. Theodorou, Jonas Buchli, Stefan Schaal. 2010

303 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

End-to-End Training of Deep Visuomotor Policies (2015, methodology): “The number of samples required to learn each controller is around 20-25, substantially lower than many prior policy search methods in the literature (Peters and Schaal, 2008; Kober et al., 2010b; Theodorou et al., 2010; Deisenroth et al., 2013).”
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates (2016, methodology): “Reinforcement learning (RL) methods have been applied to range of robotic control tasks, from locomotion [1], [2] to manipulation [3], [4], [5], [6] and autonomous vehicle control [7].”
Reinforcement learning in robotics: A survey (2013, background): “Algorithms with closely related update rules can also be derived from different perspectives including policy improvements with path integrals (Theodorou et al., 2010) and relative entropy policy search (Peters et al.”
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations (2017, methodology): “d) Combining RL with demonstrations: Methods based on dynamic movement primitives (DMPs) [36], [50], [21], [18] have been used to effectively combine demonstrations and RL to enable faster learning.”
Optimal control as a graphical model inference problem (2009, neutral): “The interpretation of the Bellman equation as message passing for the KL control problems was first established in Todorov (2008). The difference between the KL control computation and the standard computation using the Bellman equation is schematically illustrated in Fig.”
Learning contact-rich manipulation skills with guided policy search (2015, methodology): “Previous policy search methods in robotics generally use one or more of the following techniques to achieve good performance and reasonable training times: the use of example demonstrations [15], [9], [16], the use of a prior controller that allows the policy to only set high level targets, such as ”
Learning Dexterous Manipulation Policies from Experience and Imitation (2016, methodology): “…in domains ranging from ﬂight (Abbeel, Coates, Quigley and Ng 2006) to locomotion (Tedrake, Zhang and Seung 2004) to manipulation (Peters, M¨ulling and Altun 2010a; Theodorou, Buchli and Schaal 2010; Peters and Schaal 2008), comparatively few methods have been applied to control dexterous hands.”
Exploring Deep and Recurrent Architectures for Optimal Control (2013, methodology): “The former challenge could be addressed with a stochastic trajectory optimization algorithm, perhaps modeled on recent work on path integrals for optimal control [20], while the latter could be addressed by including the recurrent hidden unit state in the trajectory optimization.”
Information theoretic MPC for model-based reinforcement learning (2017, methodology): “Model-free approaches to RL, such as policy gradient methods, have been successfully applied to many challenging tasks [2]–[5] These approaches typically require an expert demonstration to initialize the learning process, followed by many interactions with the actual robotic system.”
Learning concurrent motor skills in versatile solution spaces (2012, methodology): “The most prominent motor skill learning algorithms are Policy Search (PS) methods [7], [8], [4], [9], including Policy Gradient methods [2], [10], Expectation-Maximization (EM)-like approaches such as [1], [11] and policy improvements based on path integrals [4].”
Composable Deep Reinforcement Learning for Robotic Manipulation (2018, background): “to make reinforcement learning algorithms more sample efficient is to learn from demonstrations [20], [21], [22], [3], but this comes at the cost of additional instrumentation and human supervision.”
Learning compound multi-step controllers under unknown dynamics (2015, methodology): “…based on standard reinforcement learning assumptions, including likelihood ratio methods and methods based on stochastic optimal control (SOC) [10], [11], usually assume that the task is stationary, which means that aspects of the environment such as the dynamics and the initial state distribution…”
Data-Efficient Generalization of Robot Skills with Contextual Policy Search (2013, unknown): “As the lower-level policy needs to scale to anthropomorphic robotics, we implement them using the Dynamic Movement Primitive (Ijspeert and Schaal 2003) approach, which we will now briefly review.”
Reset-free guided policy search: Efficient deep reinforcement learning with stochastic initial states (2016, methodology): “Robotic reinforcement learning (RL) sets out to tackle the very general question of how new behaviors can be learned automatically from only high-level cost or reward specifications [10], [2], [1], [11], [12].”
Learning Complex Neural Network Policies with Trajectory Optimization (2014, methodology): “The related area of stochastic optimal control has developed model-free reinforcement learning algorithms under a similar objective (Theodorou et al., 2010; Rawlik et al., 2012).”
Model-based contextual policy search for data-efficient generalization of robot skills (2017, methodology): “3The PI2 is claimed to be unbiased [38].”
Socially guided intrinsic motivation for robot learning of motor skills (2013, methodology): “…reward function is deﬁned, techniques allowing efﬁcient and fast use of training data have been elaborated, such as natural actor-critic architectures (Peters and Schaal 2008), path integral approaches (Theodorou et al. 2010) or advanced Black Box optimisation techniques (Stulp and Sigaud 2012).”
Skill learning and task outcome prediction for manipulation (2011, methodology): “A more detailed derivation of the PI 2 algorithm as well as comparison with related work in the areas of stochastic optimal control and reinforcement learning can be found in [1], [7].”
Optimal control theory and the linear bellman equation (2011, methodology): “In other words, the agent faces the problem of finding the right compromise between learning and control, a problem which is known in control theory as dual control and was originally introduced by Feldbaum (1960) (see Filatov and Unbehauen (2004) for a recent review).”