MOReL : Model-Based Offline Reinforcement Learning

By Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, Thorsten Joachims. 2020

834 citations recorded by the source corpus.

In offline reinforcement learning (RL), the goal is to learn a highly rewarding policy based solely on a dataset of historical interactions with the environment. The ability to train RL policies offline can greatly expand the applicability of RL, its data efficiency, and its experimental velocity. Prior work in offline RL has been confined almost exclusively to model-free RL approaches. In this work, we present MOReL, an algorithmic framework for model-based offline RL. This framework consists of two steps: (a) learning a pessimistic MDP (P-MDP) using the offline dataset; and (b) learning a near-optimal policy in this P-MDP. The learned P-MDP has the property that for any policy, the performance in the real environment is approximately lower-bounded by the performance in the P-MDP. This enables it to serve as a good surrogate for purposes of policy evaluation and learning, and overcome common pitfalls of model-based RL like model exploitation. Theoretically, we show that MOReL is minimax optimal (up to log factors) for offline RL. Through experiments, we show that MOReL matches or exceeds state-of-the-art results in widely studied offline RL benchmarks. Moreover, the modular design of MOReL enables future advances in its components (e.g. generative modeling, uncertainty estimation, planning etc.) to directly translate into advances for offline RL.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “…well, such as ensembles in the online setting (Kurutach et al., 2018; Buckman et al., 2018; Malik et al., 2019) or conservatism mechanisms in the ofﬂine setting (Yu et al., 2020; Kidambi et al., 2020; Argenson & Dulac-Arnold, 2021), our method does not require explicit handling of these components.”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “[32] compares to model-free approaches, we make the further observation that even a vanilla model-based RL method outperforms model-free ones in the offline setting, opening interesting questions for future investigation.”
Regularized Behavior Value Estimation (2021, background): “3) model-based approaches similarly lower reward estimates for unseen state-action pairs (Yu et al., 2020; Kidambi et al., 2020).”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, background): “[26, 64] consider model-based approaches for batch RL, and [1] find ensembles partly address some of the issues that arise in batch settings.”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “In this section, we perform empirical evaluations to show that uncertainty quantification with deep neural networks, especially in the setting of dynamics model learning, is challenging and could cause problems with uncertainty-based model-based offline RL methods such as MOReL (Kidambi et al., 2020”
TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations (2021, neutral): “Although much work has been proposed, such as model-based methods (Kidambi et al., 2020; Yu et al., 2020), policy constraint methods (Fujimoto, Meger, and Precup, 2019; Wu, Tucker, and Nachum, 2019), value regularization methods (Kumar et al., 2020) and uncertainty-based methods (Agarwal,…”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “…as an ensemble (Chua et al., 2018; Kurutach et al., 2018; Janner et al., 2019), meta-learning (Clavera et al., 2018), energy-based regularizer (Boney et al., 2019), game-theoretic framework (Rajeswaran et al., 2020), and explicit penalty for unknown states (Kidambi et al., 2020; Yu et al., 2020).”
What are the Statistical Limits of Offline RL with Linear Function Approximation? (2020, methodology): “…2019, Fujimoto et al., 2019, Jaques et al., 2020, Kumar et al., 2019, Agarwal et al., 2020] and model-based [Ross and Bagnell, 2012, Kidambi et al., 2020] offline RL methods, with steady improvements in performance [Fujimoto et al., 2019, Kumar et al., 2019, Wu et al., 2020, Kidambi et al., 2020].”
Reset-Free Lifelong Learning with Skill-Space Planning (2020, methodology): “Note that we do not compare against MOReL here since the asymptotic performance will be similar to SAC, as they both rely on Q-learning policy updates.”
PLAS: Latent Action Space for Offline Reinforcement Learning (2020, methodology): “Besides model-free approaches, recent work such as [15, 16] learns a dynamics model with the dataset and then trains the policy using either off-policy RL or on-policy RL with the learned model.”
Is Pessimism Provably Efficient for Offline RL? (2020, background): “…et al., 2019; Laroche et al., 2019; Jaques et al., 2019; Wu et al., 2019; Kumar et al., 2019; 2020; Agarwal et al., 2020b; Yu et al., 2020; Kidambi et al., 2020; Wang et al., 2020c; Siegel et al., 2020; Nair et al., 2020; Liu et al., 2020) demonstrate the empirical success of various…”
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems (2022, methodology): “[20] propose a method named model-based offline RL (MOReL), which measures their model’s epistemic uncertainty through an ensemble of dynamics models.”
Representation Balancing Offline Model-based Reinforcement Learning (2021, methodology): “MOReL (Kidambi et al., 2020) trains a state-action novelty detector and use it to pe-nalize rewards in the data-sparse region.”
Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization (2021, methodology): “Our contributions to ofﬂine model-based policy optimization are orthogonal to prior work that uses ensembles to lower the values when ensemble components disagree (Kidambi et al., 2020).”
RvS: What is Essential for Offline RL via Supervised Learning? (2021, methodology): “Recent offline RL methods use many algorithmic techniques, including value functions [15, 27], dynamics estimation alongisde value functions [24, 37, 42], dynamics estimation alone [3, 7, 39], and uncertainty quantification [1, 41, 42].”