RvS: What is Essential for Offline RL via Supervised Learning?

By Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, Sergey Levine. 2021

241 citations recorded by the source corpus.

Recent work has shown that supervised learning alone, without temporal difference (TD) learning, can be remarkably effective for offline RL. When does this hold true, and which algorithmic components are necessary? Through extensive experiments, we boil supervised learning for offline RL down to its essential elements. In every environment suite we consider, simply maximizing likelihood with a two-layer feedforward MLP is competitive with state-of-the-art results of substantially more complex methods based on TD learning or sequence modeling with Transformers. Carefully choosing model capacity (e.g., via regularization or architecture) and choosing which information to condition on (e.g., goals or rewards) are critical for performance. These insights serve as a field guide for practitioners doing Reinforcement Learning via Supervised Learning (which we coin"RvS learning"). They also probe the limits of existing RvS methods, which are comparatively weak on random data, and suggest a number of open problems.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Pretraining Language Models with Human Preferences (2023, background): “Most work in offline RL has focused on pretraining policies for robotic control environments (Nair et al., 2020; Kumar et al., 2020; Emmons et al., 2022).”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, methodology): “We also compare to prior offline RL methods that eschew TD learning: (unconditional) behavioral cloning (BC), the implementation of GCBC from [25] (which refers to GCBC as RvS-G), and a recent method based on the transformer architecture (DT [16]).”
Is Conditional Generative Modeling all you need for Decision-Making? (2022, methodology): “Emmons et al. (2021) obtained similar performance as Chen et al. (2021) without using a transformer policy but relied on careful capacity tuning of MLP policy.”
OpenChat: Advancing Open-source Language Models with Mixed-Quality Data (2023, methodology): “…we introduce C-RLFT, which is inspired by the insight from the goal-conditioned supervised learning in offline RL, that by conditioning on proper information in a supervised goal/outcome-conditioned policy, it is possible to recover optimized performance (Chen et al., 2021; Emmons et al., 2021).”
You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments (2022, unknown): “1 to zero? Clearly the approach works empirically on deterministic environments [1, 2] since”
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems (2022, methodology): “[47] show that this simple formulation combined with an increased network capacity and regularization is capable of learning policies that are as good or better than more complex approaches (e.”