Policy Gradient Methods for Robotics

By Jan Peters, Stefan Schaal. 2006

623 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Biomedical Engineering, Control and Systems Engineering, Reinforcement Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Control and Systems Engineering

In Reinforcement Learning

In Model Architectures

Citations in the corpus

Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies (2015, methodology): “, 1983; Konda and Tsitsiklis, 2000), where the critic estimates the value of different actions and the actor maximizes rewards by following the policy gradient (Sutton et al., 1999; Peters and Schaal, 2006; Silver et al., 2014).”
Data-efficient Deep Reinforcement Learning for Dexterous Manipulation (2017, background): “…have been routinely applied in robotics, in part because they straightforwardly handle continuous and high-dimensional action spaces [3] and applications include manipulation [26, 13, 25, 37, 18, 5, 39, 8], locomotion e.g. [16, 21], and a range of other challenges such as helicopter ﬂight [1].”
Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic (2016, methodology): “Monte Carlo policy gradient methods (Peters & Schaal, 2006; Schulman et al., 2015) are popular on-policy methods that ar X iv :1 61 1.”
Continuous Deep Q-Learning with Model-based Acceleration (2016, background): “…this continuous Q-learning algorithm with learned models so as to accelerate learning while preserving the beneﬁts of model-free RL. Model-free reinforcement learning in domains with contin-a uous actions is typically handled with policy search meth-ods (Peters & Schaal, 2006; Peters et al., 2010).”
Reinforcement Learning Neural Turing Machines (2015, background): “In this setting, it is possible to reduce the variance of gradient estimates of action distributions near the end of an episode [12].”
The Mirage of Action-Dependent Baselines in Reinforcement Learning (2018, methodology): “Policy gradient methods (Williams, 1992; Sutton et al., 2000; Kakade, 2002; Peters & Schaal, 2006; Silver et al., 2014; Schulman See https://sites.google.com/view/mirage-rl for additional resources. et al., 2015a; 2017) are a class of model-free RL algorithms that have found widespread adoption due…”
MuProp: Unbiased Backpropagation for Stochastic Neural Networks (2015, methodology): “The likelihood-ratio (LR) estimator, of which the popular REINFORCE algorithm is a special case (Williams, 1992; Peters & Schaal, 2006), provides a convenient method for estimating this gradient, and serves as the basis for all of the unbiased estimators discussed in this paper, including MuProp.”
Uncertainty-Aware Reinforcement Learning for Collision Avoidance (2017, methodology): “Model-free methods have been particularly popular due to their simplicity and favorable computational properties [24].”
Reward-estimation variance elimination in sequential decision processes (2018, methodology): “Policy gradient (PG) methods [1, 5, 10, 12, 16], a class of RL algorithms, do offer convergence guaranties in principle, as they directly optimize the expected reward via gradient ascent.”
Safe Model-based Reinforcement Learning with Stability Guarantees (2017, methodology): “These update policies without a system model by repeatedly executing the same task [13].”
Watch, Try, Learn: Meta-Learning from Demonstrations and Reward (2019, methodology): “This ranges from simple techniques such as demonstration-based pre-training and initialization [28, 19, 21, 20, 35] to more complex methods that incorporate both demonstration data and reward information in the loop of training [40, 3, 37, 14, 38, 29, 26, 22].”
Model-Ensemble Trust-Region Policy Optimization (2018, methodology): “In Figure 3 we demonstrate the effect of using policy gradient methods that make use of a score function estimator, such as VPG and TRPO, while using a single learned model.”
Continuously Learning Neural Dialogue Management (2016, background): “Based on this, Peters and Schaal (2006) developed the Natural Actor-Critic (NAC).”
Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networks (2016, methodology): “The main focus nowadays is an on-line model-free context [14], whereas traditionally, deterministic model-based techniques have been used in discrete state space [23].”
Learning to Control a Low-Cost Manipulator using Data-Efficient Reinforcement Learning (2011, background): “Sampling for estimating the policy gradients [15] is unnecessary.”
Model-Based Reinforcement Learning via Meta-Policy Optimization (2018, methodology): “In our implementation, we use Trust-Region Policy Optimization (TPRO) [1] for maximizing the meta-objective, and employ vanilla policy gradient (VPG) [42] for the adaptation step.”
Gaussian Processes for Data-Efficient Learning in Robotics and Control (2015, neutral): “require many function evaluations, which limits the effective number of policy parameters [14], [42].”
Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management (2017, methodology): “s the objective (Martens, 2014). Empirically, the natural gradient has been found to signiﬁcantly speed up convergence. Based on these ideas, the Natural Actor-Critic (NAC) algorithm was developed by Peters and Schaal (2006). In its episodic version (eNAC), the Fisher matrix does not need to be expl”
Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing (2018, background): “There has been a recent surge of interest in applying policy gradient methods to various application domains including program synthesis [26, 17, 68, 10], dialogue generation [25, 11], architecture search [69, 71], game [53, 31] and continuous control [44, 50].”