Reinforcement learning of motor skills with policy gradients

By Jan Peters, Stefan Schaal. 2008

1,028 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Biomedical Engineering, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Trust Region Policy Optimization (2015, methodology): “In contrast, the single path algorithm requires no state resets and can be directly implemented on a physical system (Peters & Schaal, 2008b).”
Benchmarking Deep Reinforcement Learning for Continuous Control (2016, methodology): “We use the step size suggested by Peters & Schaal (2008): α = √ δKL (∇θη(πθ)T I(θ)−1∇θη(πθ))−1.”
One-Shot Imitation Learning (2017, background): “…including Backgammon (Tesauro, 1995), helicopter control (Ng et al., 2003), Atari (Mnih et al., 2015), Go (Silver et al., 2016), continuous control in simulation (Schulman et al., 2015; Heess et al., 2015; Lillicrap et al., 2015) and on real robots (Peters & Schaal, 2008; Levine et al., 2016).”
Constrained Policy Optimization (2017, background): “In local policy search (Peters & Schaal, 2008), the policy is iteratively updated by maximizing J ( π ) over a local neighborhood of the most recent iterate π k : where D is some distance measure, and δ > 0 is a step size.”
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, background): “Many actor-critic algorithms build on the standard, on-policy policy gradient formulation to update the actor (Peters & Schaal, 2008), and many of them also consider the entropy of the policy, but instead of maximizing the entropy, they use it as an regularizer (Schul-man et al., 2017b; 2015; Mnih…”
Trust-PCL: An Off-Policy Trust Region Method for Continuous Control (2017, background): “Previous work (Kakade, 2002; Bagnell & Schneider, 2003; Peters & Schaal, 2008; Schulman et al., 2015) has applied natural gradient to policy optimization, locally improving expected reward subject to variants of d θ T F ( θ )d θ ≤ (cid:15) .”
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates (2016, methodology): “Of course, many other linear representations are possible, including DMPs [39], splines [3], and task-specific representations [40].”
The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables (2016, background): “Though the basic version of the estimator can suffer from high variance, various variance reduction techniques can be used to make the estimator much more effective (Greensmith et al., 2004).”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, background): “One class of algorithms initializes the policy via behavioral cloning from demonstrations, and then fine-tunes with reinforcement learning [33, 15, 40, 18, 35, 10, 50].”
Deep learning in neural networks: An overview (2014, neutral): “…& Mahadevan, 2003; Grondman, Busoniu, Lopes, & Babuska, 2012; Grüttner et al., 2010;Heess, Silver, & Teh, 2012; Kohl & Stone, 2004; Peters, 2010; Peters & Schaal, 2008a, 2008b; Rückstieß, Felder, & Schmidhuber, 2008; Sehnke et al., 2010; Sutton, McAllester, Singh, &Mansour, 1999;Wierstra,…”
Reinforcement learning in robotics: A survey (2013, methodology): “…of additional structure within the reinforcement learning domain, including, for instance, the (approximate) Markov structure of problems, developing approximate models, value-function estimates when available (Peters and Schaal, 2008c), or even simply the causal ordering of actions and rewards.”
Active learning of inverse models with intrinsically motivated goal exploration in robots (2013, neutral): “[79, 112]), as well as regression techniques allowing to infer the motor policy parameters corresponding to a given novel parameterized task, and based on the previously learnt correspondences between policy and task parameters (e.”
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation (2018, methodology): “Reinforcement learning offers a promising avenue for tackling this problem, but current work on reinforcement learning tackles the problem of mastering individual skills, such as hitting a ball [1], opening a door [2, 3], or throwing [4].”
The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation (2018, background): “Peters et al. [25] use RL to learn motor primitives.”
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations (2017, methodology): “d) Combining RL with demonstrations: Methods based on dynamic movement primitives (DMPs) [36], [50], [21], [18] have been used to effectively combine demonstrations and RL to enable faster learning.”
Regularized Behavior Value Estimation (2021, methodology): “Our construction is based on direct policy differentiation (Peters and Schaal, 2008; Williams, 1992) where the objective”
Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation (2017, background): “While more traditional RL success stories in robotics (e.g., [45], [46], [47]) work in state space, more recent work has been able to learn deep neural net policies from pixels to actions (e.g., [17], [18], [20]).”