When to Trust Your Model: Model-Based Policy Optimization

By Michael Janner, Justin Fu, Marvin Zhang, Sergey Levine. 2019

1,258 citations recorded by the source corpus.

Designing effective model-based reinforcement learning algorithms is difficult because the ease of data generation must be weighed against the bias of model-generated data. In this paper, we study the role of model usage in policy optimization both theoretically and empirically. We first formulate and analyze a model-based reinforcement learning algorithm with a guarantee of monotonic improvement at each step. In practice, this analysis is overly pessimistic and suggests that real off-policy data is always preferable to model-generated on-policy data, but we show that an empirical estimate of model generalization can be incorporated into such analysis to justify model usage. Motivated by this analysis, we then demonstrate that a simple procedure of using short model-generated rollouts branched from real data has the benefits of more complicated model-based algorithms without the usual pitfalls. In particular, this approach surpasses the sample efficiency of prior model-based methods, matches the asymptotic performance of the best model-free algorithms, and scales to horizons that cause other model-based methods to fail entirely.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Software, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Software

In Reinforcement Learning

Citations in the corpus

Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, methodology): “Furthermore, TD-learning is the dominant paradigm in RL for sample efficiency, and also features prominently as a sub-routine in many model-based RL algorithms (Sutton, 1990; Janner et al., 2019).”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, background): “intentionally short to prevent the accumulation of model errors (Janner et al., 2019; Amos et al., 2021).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Following Janner et al. (2019), if we assume that the total variation distance (TVD) between the learned model Tψ and true model T is bounded by m = maxt Edπt DTV (Tψ(st+1|st, at)‖T (st+1|st, at)), and the TVD between π and πβ is likewise bounded on sampled states by π, then the true policy value…”
Causally Correct Partial Models for Reinforcement Learning (2020, neutral): “…next-step models (Oh et al., 2015; Ha & Schmidhuber, 2018; Chiappa et al., 2017; Schmidhuber, 2010; Xie et al., 2016; Deisenroth & Rasmussen, 2011; Lin & Mitchell, 1992; Li et al., 2015; Diuk et al., 2008; Igl et al., 2018; Ebert et al., 2018; Kaiser et al., 2019; Janner et al., 2019).”
Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data (2020, background): “Accurate transition models for macroscopic physical systems are critical components in control systems (Lenz et al., 2015; Kamthe and Deisenroth, 2017; Chua et al., 2018) and data-efﬁcient reinforcement learning algorithms (Nagabandi et al., 2018; Janner et al., 2019).”
Adaptive Online Planning for Continual Lifelong Learning (2019, methodology): “[7, 15, 18] utilize model ensembles to reduce model overfitting for policy learning.”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “While prior approaches have used these models to select actions using planning [66, 16, 53, 50, 58], we choose to build upon Dyna-style approaches that optimize for a policy [63, 65, 71, 31, 25, 27, 43], speciﬁcally MBPO [28].”
MOReL : Model-Based Offline Reinforcement Learning (2020, background): “As a result, MBRL algorithms have been highly sample efficient for online RL [28, 29].”
Provable Representation Learning for Imitation with Contrastive Fourier Features (2021, background): “We emphasize that the performance difference below measures the difference in returns in the true MDPM, rather than, as is commonly seen in model-based RL [23], the difference in the latent MDP defined byRZ ,PZ .”
Benchmarks for Deep Off-Policy Evaluation (2021, methodology): “Our models are deep neural networks trained to maximize the log likelihood of the next state and reward given the current state and action, similar to models from successful model-based RL algorithms (Chua et al., 2018; Janner et al., 2019).”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “While any RL or planning algorithm can be used to learn the optimal policy for (cid:99) M , we focus speciﬁcally on MBPO [20, 57] which was used in MOPO.”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, methodology): “…proposed to relieve the issue of model bias, such as the use of multiple dynamics models as an ensemble (Chua et al., 2018; Kurutach et al., 2018; Janner et al., 2019), meta-learning (Clavera et al., 2018), energy-based regularizer (Boney et al., 2019), game-theoretic framework (Rajeswaran et…”
Deep Dynamics Models for Learning Dexterous Manipulation (2019, methodology): “…propagation; NPG [33] is a model-free natural policy gradient method, and has been used in prior work on learning manipulation skills [4]; SAC [34] is an off-policy model-free RL algorithm; MBPO [35] is a recent hybrid approach that uses data from its model to accelerate policy learning.”
Randomized Ensembled Double Q-Learning: Learning Fast Without a Model (2021, methodology): “For MBPO, we use the default network architectures for the Q networks, policy network, and model ensembles (Janner et al., 2019).”
Few-shot Language Coordination by Modeling Theory of Mind (2021, methodology): “Recently, neural networks have been used widely for model-based RL (Gal et al., 2016; Depeweg et al., 2016; Nagabandi et al., 2018; Chua et al., 2018; Janner et al., 2019).”