On the model-based stochastic value gradient for continuous reinforcement learning

By Brandon Amos, Samuel Stanton, Denis Yarats, Andrew Gordon Wilson. 2020

81 citations recorded by the source corpus.

Model-based reinforcement learning approaches add explicit domain knowledge to agents in hopes of improving the sample-efficiency in comparison to model-free agents. However, in practice model-based methods are unable to achieve the same asymptotic performance on challenging continuous control tasks due to the complexity of learning and controlling an explicit world model. In this paper we investigate the stochastic value gradient (SVG), which is a well-known family of methods for controlling continuous systems which includes model-based approaches that distill a model-based value expansion into a model-free policy. We consider a variant of the model-based SVG that scales to larger systems and uses 1) an entropy regularization to help with exploration, 2) a learned deterministic world model to improve the short-horizon value estimate, and 3) a learned model-free value estimate after the model's rollout. This SVG variation captures the model-free soft actor-critic method as an instance when the model rollout horizon is zero, and otherwise uses short-horizon model rollouts to improve the value estimate for the policy update. We surpass the asymptotic performance of other model-based methods on the proprioceptive MuJoCo locomotion tasks from the OpenAI gym, including a humanoid. We notably achieve these results with a simple deterministic world model without requiring an ensemble.

Subfields: Artificial Intelligence, Molecular Biology, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “…most obvious in model-based RL, where it is used to train predictive models that can then be used for planning or policy learning (Sutton, 1990; Silver et al., 2008; Fairbank, 2008; Deisenroth & Rasmussen, 2011; Lampe & Riedmiller, 2014; Heess et al., 2015b; Janner et al., 2020; Amos et al., 2021).”
Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “Though model-based methods have been applied to the humanoid task, prior works tend to keep the horizon intentionally short to prevent the accumulation of model errors [25, 1].”
PlaTe: Visually-Grounded Planning With Transformers in Procedural Tasks (2021, background): “Learning to Plan from Pixels: Another related work is learning dynamics models for model-based RL [6]–[8].”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, background): “[315] found an effective combination of model-free soft value estimation for policy evaluation and model-based stochastic value gradient for policy improvement for model-based high-dimensional”
Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning (2021, methodology): “Similar to Amos et al. [2020], we instantiate this idea by using linear decay σ ( t ) for the variance σ 2 of the exploration noise deﬁned as: where σ init and σ ﬁnal are the initial and ﬁnal values for standard deviation, and T is the decay horizon.”