Learning Continuous Control Policies by Stochastic Value Gradients

By Nicolas Manfred Otto Heess, Greg Wayne, David Silver, Timothy P. Lillicrap, Tom Erez, Yuval Tassa. 2015

600 citations recorded by the source corpus.

We present a unified framework for learning continuous control policies using backpropagation. It supports stochastic control by treating stochasticity in the Bellman equation as a deterministic function of exogenous noise. The product is a spectrum of general policy gradient algorithms that range from model-free methods with value functions to model-based methods without value functions. We use learned models but only require observations from the environment in- stead of observations from model-predicted trajectories, minimizing the impact of compounded model errors. We apply these algorithms first to a toy stochastic control problem and then to several physics-based control problems in simulation. One of these variants, SVG(1), shows the effectiveness of learning models, value functions, and policies simultaneously in continuous domains.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Emergence of Locomotion Behaviours in Rich Environments (2017, background): “[10, 7, 11, 12, 6, 13]), although impressive, have focused on comparatively small problems, and the use of large-scale, distributed optimization is less widespread and the corresponding algorithms are less well developed (but see e.”
Sample Efficient Actor-Critic with Experience Replay (2016, background): “Instead of a trust region update, one could alternatively add an appropriately scaled KL cost to the objective function as proposed by Heess et al. (2015).”
Benchmarking Deep Reinforcement Learning for Continuous Control (2016, methodology): “Similar tasks have been explored in Gomez & Miikkulainen (1998); Sch¨afer & Udluft (2005); Heess et al. (2015a); Wierstra et al. (2007).”
Memory-based control with recurrent neural networks (2015, methodology): “We also consider DPG’s stochastic counterpart, SVG(0) ([6]; SVG stands for “Stochastic Value Gradients”) which similarly updates the policy via backpropagation of action-value gradients from an action-value critic but learns a stochastic policy.”
One-Shot Imitation Learning (2017, background): “…including Backgammon (Tesauro, 1995), helicopter control (Ng et al., 2003), Atari (Mnih et al., 2015), Go (Silver et al., 2016), continuous control in simulation (Schulman et al., 2015; Heess et al., 2015; Lillicrap et al., 2015) and on real robots (Peters & Schaal, 2008; Levine et al., 2016).”
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, methodology): “A commonly used algorithm in such settings, deep deterministic policy gradient (DDPG) (Lillicrap et al., 2015), provides for sample-efﬁcient learning but is notoriously challenging to use due to its extreme brittleness and hyperparameter sensitivity (Duan et al., 2016; Henderson et al., 2017).”
Mastering Atari, Go, chess and shogi by planning with a learned model (2019, background): “[16] Nicolas Heess, Greg Wayne, David Silver, Timothy Lillicrap, Yuval Tassa, and Tom Erez.”
Data-efficient Deep Reinforcement Learning for Dexterous Manipulation (2017, background): “Finally, DPG can be seen as the deterministic limit of a particular instance of the stochastic value gradients (SVG) family [11], which also computes policy gradient via back-propagation of value gradients, but optimizes stochastic policies.”
Model-Based Reinforcement Learning for Atari (2019, methodology): “Variants of this basic algorithm have been proposed in a number of prior works, starting from Dyna (Sutton, 1991) to more recent methods that incorporate deep networks (Heess et al., 2015; Feinberg et al., 2018; Kalweit & Boedecker, 2017; Kurutach et al., 2018).”
TreeQN and ATreeC: Differentiable Tree Planning for Deep Reinforcement Learning (2017, background): “A class of value gradient methods (Deisenroth & Rasmussen, 2011; Fairbank & Alonso, 2012; Heess et al., 2015) also differentiates through models to train a policy.”
Value Prediction Network (2017, methodology): “In cases where the observation-prediction model is differentiable with respect to continuous actions, backpropagation can be used to find the optimal action [19] or to compute value gradients [11].”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “…works have studied applying sequence models of various types to represent components in standard RL algorithms, such as policies, value functions, and models (Bakker, 2002; Heess et al., 2015a; Chiappa et al., 2017; Parisotto et al., 2020; Parisotto & Salakhutdinov, 2021; Kumar et al., 2020b).”
Continuous Deep Q-Learning with Model-based Acceleration (2016, methodology): “While prior methods propose a variety of model classes, including neural networks (Heess et al., 2015), Gaussian processes (Deisenroth & Rasmussen, 2011), and locally-weighted regression (Atkeson et al.”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “There are a number of different variants of actor-critic methods, including on-policy variants that directly estimate V (s) (Konda and Tsitsiklis, 2000), and off-policy variants that estimate Q(s,a) via a parameterized state-action value function Qφ(s,a) (Haarnoja et al., 2018, 2017; Heess et al., 2”
Decoupled Neural Interfaces using Synthetic Gradients (2016, methodology): “These approaches have previously been used in control (Werbos, 1992; Fairbank, 2014) and model-based reinforcement learning (Heess et al., 2015).”
The Mirage of Action-Dependent Baselines in Reinforcement Learning (2018, methodology): “Prior work has explored model-based (Sutton, 1990; Heess et al., 2015; Gu et al., 2016) and off-policy critic-based gradient estimators (Lillicrap et al., 2015).”
Causally Correct Partial Models for Reinforcement Learning (2020, background): “…for planning (Oh et al., 2017; Schrittwieser et al., 2019), generation of synthetic training data (Weber et al., 2017), better credit assignment (Heess et al., 2015), learning useful internal representations and belief states (Gregor et al., 2019; Guo et al., 2018), or exploration via…”