Policy search via the signed derivative

By J. Kolter, Andrew Y. Ng. 2009

74 citations recorded by the source corpus.

We consider policy search for reinforcement learning: learning policy parameters, for some fixed policy class, that optimize performance of a system. In this paper, we propose a novel policy gradient method based on an approximation we call the Signed Derivative; the approximation is based on the intuition that it is often very easy to guess the direction in which control inputs affect future state variables, even if we do not have an accurate model of the system. The resulting algorithm is very simple, requires no model of the environment, and we show that it can outperform standard stochastic estimators of the gradient; indeed we show that Signed Derivative algorithm can in fact perform as well as the true (model-based) policy gradient, but without knowledge of the model. We evaluate the algorithm’s performance on both a simulated task and two realworld tasks — driving an RC car along a specified trajectory, and jumping onto obstacles with an quadruped robot — and in all cases achieve good performance after very little training.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Control and Systems Engineering

Citations in the corpus

Reinforcement learning in robotics: A survey (2013, background): “…flight (Bagnell and Schneider, 2001), learning biped walk patterns (Tedrake, 2004; Tedrake et al., 2005), driving a radiocontrolled (RC) car, learning a jumping behavior for a robot dog (Kolter and Ng, 2009a) (illustrated in Figure 5), and balancing a two-wheeled robot (Vlassis et al., 2009).”
Reinforcement learning with multi-fidelity simulators (2014, background): “For instance, several reinforcement learning (RL) algorithms use simulators to augment real robot data [1]–[3], but none of those agents actively decide when to sample simulated or real-world data.”
SOLAR: Deep Structured Latent Representations for Model-Based Reinforcement Learning (2018, methodology): “Another class of model-based methods rely only on local system models to compute the gradient for a policy update [1, 21, 14, 24, 5].”
Superhuman performance of surgical tasks by robots using iterative learning from human-guided demonstrations (2010, background): “The work of [20] suggests that only global knowledge of the robot dynamics is necessary to be able to iteratively improve control.”
Variational Bayesian Optimization for Runtime Risk-Sensitive Control (2012, methodology): “For this reason, model-free policy search methods have become one of the standard tools for constructing controllers for robot systems [27, 23, 12, 29, 17, 11].”
Autonomous navigation of stratospheric balloons using reinforcement learning (2020, background): “Unlike direct policy optimization schemes 42,43 , it is also trivially combined with experience replay 44 to reuse simulator interactions and achieves greater data efficiency.”
A simple learning strategy for high-speed quadrocopter multi-flips (2010, methodology): “The main idea behind this approach is similar to the algorithm described in [15], although we retain the full corrective matrix and not just the signs.”
Reducing Conservativeness in Safety Guarantees by Learning Disturbances Online: Iterated Guaranteed Safe Online Learning (2012, background): “…proven itself to be a powerful technique in control applications for robotics, with successes ranging from flying complex maneuvers on an autonomous helicopter [2] to teaching a quadruped robot to jump up steps [8], to helping a robot car navigate at high speeds in an unstructured environment [9].”
SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning (2018, unknown): “e still limited to short horizon prediction in more complex domains [27, 30, 13, 42]. Another class of model-based methods rely only on local system models to compute the gradient for a policy update [1, 21, 14, 24, 5]. These methods do not use models for long-term forward prediction, allowing for t”
Efficient reinforcement learning for robots using informative simulated priors (2015, methodology): “In this paper we introduce a method for using samples from a simulated robot to decrease the number of real-world samples that are needed to learn a good policy (see Figure 1).”
A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems (2017, methodology): “1), using a policy gradient algorithm [34].”
Reachability-based safe learning with Gaussian processes (2014, methodology): “We present the algorithm here briefly and refer the reader to [12] for a more detailed description.”
Learning to pour with a robot arm combining goal and shape learning for dynamic movement primitives (2011, result): “Our observations on direct policy search being nonsensitive to details are also supported by those of Kolter and Ng [19], who found that quite a coarse algorithm can perform very well in direct policy learning.”