Representation Matters: Offline Pretraining for Sequential Decision Making

By Mengjiao Yang, Ofir Nachum. 2021

126 citations recorded by the source corpus.

The recent success of supervised learning methods on ever larger offline datasets has spurred interest in the reinforcement learning (RL) field to investigate whether the same paradigms can be translated to RL algorithms. This research area, known as offline RL, has largely focused on offline policy optimization, aiming to find a return-maximizing policy exclusively from offline data. In this paper, we consider a slightly different approach to incorporating offline data into sequential decision-making. We aim to answer the question, what unsupervised objectives applied to offline datasets are able to learn state representations which elevate performance on downstream tasks, whether those downstream tasks be online RL, imitation learning from expert demonstrations, or even offline policy optimization based on the same offline dataset? Through a variety of experiments utilizing standard offline RL datasets, we find that the use of pretraining with unsupervised learning objectives can dramatically improve the performance of policy learning algorithms that otherwise yield mediocre performance on their own. Extensive ablations further provide insights into what components of these unsupervised objectives -- e.g., reward prediction, continuous or discrete representations, pretraining or finetuning -- are most important and in which settings.

Subfields: Renewable Energy, Sustainability and the Environment, Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Renewable Energy, Sustainability and the Environment

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021, background): “The choice of observation space for imitation merits careful consideration – other work has also confirmed the importance of feature representations used for offline policy learning [52, 53].”
Provable Representation Learning for Imitation with Contrastive Fourier Features (2021, background): “Similar contrastive learning objectives have appeared in related works [10, 40], and so our theoretical bounds can be used to explain these previous empirical successes.”
Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error (2022, background): “This highlights the importance of feature learning (Jaderberg et al., 2016; Yang & Nachum, 2021), as neural network features are unlikely to be relevant early in training.”
Representation Learning for Online and Offline RL in Low-rank MDPs (2021, background): “…several empirical works have shown that leveraging representation learning techniques developed in supervised or unsupervised learning settings can accelerate the search for good decision-making strategies (Silver et al., 2018; Stooke et al., 2021; Srinivas et al., 2020; Yang and Nachum, 2021).”
Transfer Learning in Deep Reinforcement Learning: A Survey (2020, methodology): “Specifically, [71] illustrated that using demonstrations for representation learning can remarkably improve various downstream tasks, including both offline and online RL.”