MOPO: Model-based Offline Policy Optimization

By Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, Tengyu Ma. 2020

951 citations recorded by the source corpus.

Offline reinforcement learning (RL) refers to the problem of learning policies entirely from a batch of previously collected data. This problem setting is compelling, because it offers the promise of utilizing large, diverse, previously collected datasets to acquire policies without any costly or dangerous active exploration, but it is also exceptionally difficult, due to the distributional shift between the offline training data and the learned policy. While there has been significant progress in model-free offline RL, the most successful prior methods constrain the policy to the support of the data, precluding generalization to new states. In this paper, we observe that an existing model-based RL algorithm on its own already produces significant gains in the offline setting, as compared to model-free approaches, despite not being designed for this setting. However, although many standard model-based RL methods already estimate the uncertainty of their model, they do not by themselves provide a mechanism to avoid the issues associated with distributional shift in the offline setting. We therefore propose to modify existing model-based RL methods to address these issues by casting offline model-based RL into a penalized MDP framework. We theoretically show that, by using this penalized MDP, we are maximizing a lower bound of the return in the true MDP. Based on our theoretical results, we propose a new model-based offline RL algorithm that applies the variance of a Lipschitz-regularized model as a penalty to the reward function. We find that this algorithm outperforms both standard model-based RL methods and existing state-of-the-art model-free offline RL approaches on existing offline RL benchmarks, as well as two challenging continuous control tasks that require generalizing from data collected for a different task.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, background): “, 2020), or (c) incorporate pessimism into learned dynamics models (Kidambi et al., 2020; Yu et al., 2020).”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “…well, such as ensembles in the online setting (Kurutach et al., 2018; Buckman et al., 2018; Malik et al., 2019) or conservatism mechanisms in the ofﬂine setting (Yu et al., 2020; Kidambi et al., 2020; Argenson & Dulac-Arnold, 2021), our method does not require explicit handling of these components.”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “Then, conservative behavior can be induced either by modifying the reward function to obtain a conservative reward of the form r̃(s,a) = r(s,a) − λu(s,a), as in MOPO (Yu et al., 2020), or by modifying the MDP under the learned model so that the agent enters an absorbing state with a low reward value”
MOReL : Model-Based Offline Reinforcement Learning (2020, neutral): “[54] also study a model-based approach to offline RL.”
Regularized Behavior Value Estimation (2021, background): “3) model-based approaches similarly lower reward estimates for unseen state-action pairs (Yu et al., 2020; Kidambi et al., 2020).”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, background): “[26, 64] consider model-based approaches for batch RL, and [1] find ensembles partly address some of the issues that arise in batch settings.”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “…this intuition, we perform empirical evaluations to study whether uncertainty quantiﬁcation with deep neural networks, especially in the setting of dynamics model learning, is challenging and could cause problems with uncertainty-based model-based ofﬂine RL methods such as MOReL [26] and MOPO [67].”
Representation Matters: Offline Pretraining for Sequential Decision Making (2021, background): “Accordingly, much of the recent oﬄine RL research focuses on proposing new policy optimization algorithms amenable to learning from oﬄine datasets (e.g., Fujimoto et al. (2019); Wu et al. (2019); Agarwal et al. (2020); Kumar et al. (2020); Yu et al. (2020); Matsushima et al. (2020)).”
TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations (2021, neutral): “Although much work has been proposed, such as model-based methods (Kidambi et al., 2020; Yu et al., 2020), policy constraint methods (Fujimoto, Meger, and Precup, 2019; Wu, Tucker, and Nachum, 2019), value regularization methods (Kumar et al., 2020) and uncertainty-based methods (Agarwal,…”
QPLEX: Duplex Dueling Multi-Agent Q-Learning (2020, background): “This ability not only provides QPLEX with high stability and sample efficiency but also with opportunities to efficiently utilize multi-source offline data without additional online exploration (Fujimoto et al., 2019; Fu et al., 2020; Levine et al., 2020; Yu et al., 2020).”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, methodology): “We therefore augmented BREMEN with either a hard (MOReL-like, green) or soft (MOPO-like, orange) reward penalty according to model uncertainty.”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, methodology): “Works such as MOPO (Yu et al. 2020) and MoREL (Kidambi et al. 2020) leverage the learnt model to learn a model-free policy, and approaches such as MBOP (Argenson and Dulac-Arnold 2020) leverage the model directly using an MPC-based planner.”
Offline Reinforcement Learning from Images with Latent Space Models (2020, methodology): “For ofﬂine RL, model-based algorithms (Kidambi et al., 2020; Yu et al., 2020b) have been shown to be mimimax optimal, obtain state of the art results in a variety of benchmark tasks, as well as generalize to new out-of-distribution tasks.”
Locality Sensitive Sparse Encoding for Learning World Models Online (2024, background): “| This technique enables more stable predictions and is widely adopted (Chua et al., 2018; Janner et al., 2019; Yu et al., 2020).”
Instabilities of Offline RL with Pre-Trained Neural Representation (2021, background): “…2, there are a number of algorithms for oﬄine RL [Fujimoto et al., 2019, Kumar et al., 2019, Wu et al., 2020, Jaques et al., 2020, Nachum et al., 2019b, Peng et al., 2019, Siegel et al., 2020, Kumar et al., 2020, Agarwal et al., 2019, Yu et al., 2020, Kidambi et al., 2020, Rafailov et al., 2020].”
Overcoming Model Bias for Robust Offline Deep Reinforcement Learning (2020, methodology): “It has been shown however, that this issue can, even in the ofﬂine RL setting, usually be circumvented by penalizing model uncertainty in the policy training process, e.g., by employing Bayesian models (Depeweg et al. 2016, 2017; Kaiser et al. 2020), or by ensembling, as in MOPO and MOReL (Yu et al.”
COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning (2020, background): “While offline (or “batch”) RL is a well-studied area [35, 36, 37, 1], there has been a significant amount of recent interest in offline deep RL, where deep RL agents are trained on a static, previously collected dataset without any environment interaction [1, 3, 2, 4, 5, 38, 39, 40].”
Planning with Diffusion for Flexible Behavior Synthesis (2022, positive): “…compare to a variety of prior algorithms spanning other approaches to data-driven control, including the model-free reinforcement learning algorithms CQL (Kumar et al., 2020) and IQL (Kostrikov et al., 2022) (Yu et al., 2020), MOReL (Kidambi et al., 2020), and MBOP (Argenson & Dulac-Arnold, 2021).”