Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism

By Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, Stuart J. Russell. 2021

360 citations recorded by the source corpus.

Offline reinforcement learning (RL) algorithms seek to learn an optimal policy from a fixed dataset without active data collection. Based on the composition of the offline dataset, two main methods are used: imitation learning which is suitable for expert datasets, and vanilla offline RL which often requires uniform coverage datasets. From a practical standpoint, datasets often deviate from these two extremes and the exact data composition is usually unknown. To bridge this gap, we present a new offline RL framework, called single-policy concentrability, that smoothly interpolates between the two extremes of data composition, hence unifying imitation learning and vanilla offline RL. Under this new framework, we ask: can one develop an algorithm that achieves a minimax optimal rate adaptive to unknown data composition? To address this question, we consider a lower confidence bound (LCB) algorithm developed based on pessimism in the face of uncertainty in offline RL. We study finite-sample properties of LCB as well as information-theoretic limits in multi-armed bandits, contextual bandits, and Markov decision processes (MDPs). Our analysis reveals surprising facts about optimality rates. In particular, in both contextual bandits and RL, LCB achieves a fast convergence rate for nearly-expert datasets, analogous to the one achieved by imitation learning, contrary to the slow rate achieved in offline RL. In contextual bandits, we prove that LCB is adaptively optimal for the entire data composition range, achieving a smooth transition from imitation learning to offline RL. We further show that LCB is almost adaptively optimal in MDPs.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

COMBO: Conservative Offline Model-Based Policy Optimization (2021, background): “Note that in general, the required value of β may be quite large similar to prior works, which typically utilize a large constant β , which may be in the form of a penalty on a regularizer [36, 29] or as constants in theoretically optimal algorithms [23, 49].”
Representation Learning for Online and Offline RL in Low-rank MDPs (2021, background): “On the theoretical side, pessimism allows us to obtain the PAC guarantee on various models when a comparator policy is covered by offline data in some forms (Jin et al., 2020b; Rashidinejad et al., 2021; Yin et al., 2021; Zanette et al., 2021b; Zhang et al., 2021b; Chang et al., 2021).”
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (2023, background): “As suggested in the offline learning literature (Jin et al., 2021b; Rashidinejad et al., 2021), the first term can typically controlled by the property of D off , while the second term is far more challenging to control because both the ˆ π and ˆ r are estimated from D off , and hence spuriously…”
Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons (2023, background): “Compared to the traditional pessimism principle (Rashidinejad et al., 2021; Li et al., 2022; Xie et al., 2021b; Zanette, 2022; Zanette et al., 2021), we subtract an extra reference vector v in all the feature vectors ϕ .”