Bellman-consistent Pessimism for Offline Reinforcement Learning

By Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, Alekh Agarwal. Neural Information Processing Systems, 2021

335 citations recorded by the source corpus.

The use of pessimism, when reasoning about datasets lacking exhaustive exploration has recently gained prominence in offline reinforcement learning. Despite the robustness it adds to the algorithm, overly pessimistic reasoning can be equally damaging in precluding the discovery of good policies, which is an issue for the popular bonus-based pessimism. In this paper, we introduce the notion of Bellman-consistent pessimism for general function approximation: instead of calculating a point-wise lower bound for the value function, we implement pessimism at the initial state over the set of functions consistent with the Bellman equations. Our theoretical guarantees only require Bellman closedness as standard in the exploratory setting, in which case bonus-based pessimism fails to provide guarantees. Even in the special case of linear function approximation where stronger expressivity assumptions hold, our result improves upon a recent bonus-based approach by $\mathcal{O}(d)$ in its sample complexity when the action space is finite. Remarkably, our algorithms automatically adapt to the best bias-variance tradeoff in the hindsight, whereas most prior approaches require tuning extra hyperparameters a priori.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/e2ad21dae85950ab3631f65a0f142924c99fb9c4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Masatoshi Uehara
2. Nan Jiang
3. Wen Sun
4. Quanquan Gu
5. Sergey Levine
6. Amy Zhang
7. Audrey Huang
8. Dongruo Zhou

Citations in the corpus

• Provably efficient representation selection in Low-rank Markov Decision Processes: from online to offline RL (2021, background): “Following that, Xie et al. [2021] introduces the notion of Bellman’s consistent pessimism for general function approximation.”
• Beyond the Return: Off-policy Function Estimation under User-specified Error-measuring Distributions (2022, methodology): “4 Moreover, many general methods—including MIS [UHJ20] and other minimax methods [ASM08; XCJMA21]—coincide with LSTDQ in the linear case, so the aforementioned results can be viewed as a specialized analysis leveraging the properties of linear classes.”
• Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons (2023, background): “Compared to the traditional pessimism principle (Rashidinejad et al., 2021; Li et al., 2022; Xie et al., 2021b; Zanette, 2022; Zanette et al., 2021), we subtract an extra reference vector v in all the feature vectors ϕ .”
• Representation Learning for Online and Offline RL in Low-rank MDPs (2021, methodology): “Though the pessimism in the initial state distribution was recently also used in Xie et al. (2021); Zanette et al. (2021b), the derivation is totally different.”
• Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (2023, methodology): “Here we do not maintain a confidence set but use a modified target that is biased toward pessimism, similar to Xie et al. (2021a); Zhang (2022), which may be easier to approximate in practice (Liu et al., 2023b).”