COMBO: Conservative Offline Model-Based Policy Optimization

By Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, Chelsea Finn. 2021

535 citations recorded by the source corpus.

Model-based algorithms, which learn a dynamics model from logged experience and perform some sort of pessimistic planning under the learned model, have emerged as a promising paradigm for offline reinforcement learning (offline RL). However, practical variants of such model-based algorithms rely on explicit uncertainty quantification for incorporating pessimism. Uncertainty estimation with complex models, such as deep neural networks, can be difficult and unreliable. We overcome this limitation by developing a new model-based offline RL algorithm, COMBO, that regularizes the value function on out-of-support state-action tuples generated via rollouts under the learned model. This results in a conservative estimate of the value function for out-of-support state-action tuples, without requiring explicit uncertainty estimation. We theoretically show that our method optimizes a lower bound on the true policy value, that this bound is tighter than that of prior methods, and our approach satisfies a policy improvement guarantee in the offline setting. Through experiments, we find that COMBO consistently performs as well or better as compared to prior offline model-free and model-based methods on widely studied offline RL benchmarks, including image-based tasks.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021, methodology): “BCQ outperforms the other methods; however, it is possible that recent batch RL methods [38, 43] that have been shown to work on pixel observations might be able to perform even better.”
Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism (2021, methodology): “Based on this intuition, a variety of model-based [55, 19, 56] and model-free [22, 33, 11, 34, 25, 37, 44, 14, 28, 23, 3] ofﬂine RL algorithms are proposed that achieve promising empirical results.”
Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models (2023, methodology): “For instance, [21, 37, 63, 60, 53, 22] train actionconditioned dynamics models and run RL in the model.”
Skill-based Meta-Reinforcement Learning (2022, methodology): “In particular, the field of offline reinforcement learning (Levine et al., 2020; Siegel et al., 2020; Kumar et al., 2020; Yu et al., 2021) aims to devise methods that can perform RL fully offline from pre-collected data, without the need for environment interactions.”
Instabilities of Offline RL with Pre-Trained Neural Representation (2021, background): “To mitigate the issue of error amplification, prior empirical work usually constrains the learned policy to be closer to the behavioral policy (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2020; Jaques et al., 2020; Nachum et al., 2019b; Peng et al., 2019; Siegel et al., 2020; Kumar et al.,”
DayDreamer: World Models for Physical Robot Learning (2022, background): “World models also learn representations that fuse multiple sensor modalities and integrate them into latent states, removing the need for manual state estimation.”
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems (2022, methodology): “strictly a lower bound to the true value function [17], [18].”