The Importance of Pessimism in Fixed-Dataset Policy Optimization

By Jacob Buckman, Carles Gelada, Marc G. Bellemare. 2020

147 citations recorded by the source corpus.

We study worst-case guarantees on the expected return of fixed-dataset policy optimization algorithms. Our core contribution is a unified conceptual and mathematical framework for the study of algorithms in this regime. This analysis reveals that for naive approaches, the possibility of erroneous value overestimation leads to a difficult-to-satisfy requirement: in order to guarantee that we select a policy which is near-optimal, we may need the dataset to be informative of the value of every policy. To avoid this, algorithms can follow the pessimism principle, which states that we should choose the policy which acts optimally in the worst possible world. We show why pessimistic algorithms can achieve good performance even when the dataset is not informative of every policy, and derive families of algorithms which follow this principle. These theoretical findings are validated by experiments on a tabular gridworld, and deep learning experiments on four MinAtar environments.

Subfields: Artificial Intelligence, Computer Networks and Communications, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Management Science and Operations Research

Citations in the corpus

Offline RL Without Off-Policy Evaluation (2021, background): “However, being able to properly quantify uncertainty about unknown states is notoriously difﬁcult when dealing with neural network value functions [Buckman et al., 2020].”
A Minimalist Approach to Offline Reinforcement Learning (2021, background): “Another direction is to implement divergence regularization as a form of pessimism into the value estimate [Nachum et al., 2019, Kumar et al., 2020, Buckman et al., 2020].”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, methodology): “We use these three aspects to design a penalty for Q-value updates to be more pessimistic (Buckman et al., 2021; Jin et al., 2020).”
Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism (2021, methodology): “However, it is observed empirically that existing model-free methods perform better when the dataset is nearly expert-driven whereas existing model-based methods perform better when the dataset is randomly-collected [55, 5, 56].”
Is Pessimism Provably Efficient for Offline RL? (2020, background): “In other words, our work complements (Buckman et al., 2020) by proving that pessimism is not only “important” but also optimal in the sense of information theory.”
Offline Reinforcement Learning with Pseudometric Learning (2021, background): “…policy (Jaques et al., 2019; Wu et al., 2019; Kumar et al., 2019; Siegel et al., 2020; Peng et al., 2019; Fujimoto et al., 2018), or a pessimistic (Buckman et al., 2021) estimate of either the Q-value or the bootstrapped Q-value (Kumar et al., 2020; Laroche et al., 2019; Simão et al., 2020;…”