Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning

By Anusha Nagabandi, Gregory Kahn, Ronald S. Fearing, Sergey Levine. 2017

1,091 citations recorded by the source corpus.

Model-free deep reinforcement learning algorithms have been shown to be capable of learning a wide range of robotic skills, but typically require a very large number of samples to achieve good performance. Model-based algorithms, in principle, can provide for much more efficient learning, but have proven difficult to extend to expressive, high-capacity models such as deep neural networks. In this work, we demonstrate that neural network dynamics models can in fact be combined with model predictive control (MPC) to achieve excellent sample complexity in a model-based reinforcement learning algorithm, producing stable and plausible gaits that accomplish various complex locomotion tasks. We further propose using deep neural network dynamics models to initialize a model-free learner, in order to combine the sample efficiency of model-based approaches with the high task-specific performance of model-free methods. We empirically demonstrate on MuJoCo locomotion tasks that our pure model-based approach trained on just random action data can follow arbitrary trajectories with excellent sample efficiency, and that our hybrid algorithm can accelerate model-free learning on high-speed benchmark tasks, achieving sample efficiency gains of $3-5\times$ on swimmer, cheetah, hopper, and ant agents. Videos can be found at https://sites.google.com/view/mbmf

Subfields: Artificial Intelligence, Biomedical Engineering, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Learning Latent Dynamics for Planning from Pixels (2018, methodology): “…et al., 2018; Henaff et al., 2018; Chua et al., 2018), combining the beneﬁts of model-based and model-free approaches (Kalweit and Boedecker, 2017; Nagabandi et al., 2017; Weber et al., 2017; Kurutach et al., 2018; Buckman et al., 2018; Ha and Schmidhuber, 2018; Wayne et al., 2018; Igl et al.,…”
Recurrent World Models Facilitate Policy Evolution (2018, methodology): “Recent work [57] combines the model-based approach with traditional model-free RL training by first initializing the policy network with the learned policy, but must subsequently rely on model-free methods to fine-tune this policy in the actual environment.”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, background): “The alternative, of treating action sequences as unconstrained optimization variables that do not depend on state (Nagabandi et al., 2018), can more readily lead to model exploitation, as the problem of maximizing reward under a learned model closely resembles that of ﬁnding adversarial examples…”
E(n) Equivariant Graph Neural Networks (2021, background): “, 2018), model based dynamics in reinforcement learning (Nagabandi et al., 2018), and physical systems simulations (Grzeszczuk et al.”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, positive): “…θk,g+1 ← θk,g + απ∇θk,gJ(πθk,g ) 25: end for 26: θk + 1← θk,Gπ . update policy parameters 27: end for utilize it for planning at test time, often by means of model-predictive control (MPC) (Tassa et al., 2012) with various trajectory optimization methods (Nagabandi et al., 2018; Chua et al., 2018).”
World Models (2018, methodology): “Recent work (Nagabandi et al., 2017) combines the model-based approach with traditional model-free RL training by ﬁrst initializing the policy network with the learned policy, but must subsequently rely on model-free methods to ﬁne-tune this policy in the actual environment.”
Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data (2020, background): “Accurate transition models for macroscopic physical systems are critical components in control systems (Lenz et al., 2015; Kamthe and Deisenroth, 2017; Chua et al., 2018) and data-efﬁcient reinforcement learning algorithms (Nagabandi et al., 2018; Janner et al., 2019).”
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, methodology): “• Model-based model-free hybrid (MBMF): [33] is a”
Adaptive Online Planning for Continual Lifelong Learning (2019, methodology): “[2, 24] learn dynamics models and perform MPC using them.”
MOReL : Model-Based Offline Reinforcement Learning (2020, background): “These advantages have enabled MBRL to be highly sample efficient in the online RL setting, often substantially outperforming model-free counterparts [28, 29, 30, 31].”
When to Trust Your Model: Model-Based Policy Optimization (2019, methodology): “Neural network predictive models (Draeger et al., 1995; Gal et al., 2016; Depeweg et al., 2016; Nagabandi et al., 2018), are appealing because they allow for algorithms that combine the sample efﬁciency of a model-based approach with the asymptotic performance of high-capacity function…”
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, methodology): “All environments that we test have a maximum horizon of 500, which is longer than most of the existing model-based RL work (Nagabandi et al., 2017; Kurutach et al., 2018).”
Model-Ensemble Trust-Region Policy Optimization (2018, background): “…their applications on more challenging domains have either required a combination with model-free techniques (Oh et al., 2015; Heess et al., 2015; Nagabandi et al., 2017), or domain-speciﬁc policy learning or planning algorithms (Lenz et al., 2015; Agrawal et al., 2016; Pinto & Gupta, 2016;…”
Graph networks as learnable physics engines for inference and control (2018, background): “We also explore jointly learning a model and policy (Heess et al., 2015; Gu et al., 2016; Nagabandi et al., 2017).”
Model-Based Reinforcement Learning via Meta-Policy Optimization (2018, methodology): “[22], which uses MPC for planning (MB-MPC).”
Dynamics-Aware Unsupervised Discovery of Skills (2019, background): “More recently, due to the challenges of applying these methods to highdimensional state spaces, MBRL approaches employs Bayesian deep neural networks (Nagabandi et al., 2018; Chua et al., 2018b; Gal et al., 2016; Fu et al., 2016; Lenz et al., 2015) to learn dynamics models.”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “that can be used in order to improve their data efficiency [90].”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “We believe model-based RL is better suited to this setting due to its higher demonstrated sample efﬁciency than model-free RL (Kurutach et al., 2018; Nagabandi et al., 2018).”