You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments

By Keiran Paster, Sheila A. McIlraith, Jimmy Ba. 2022

82 citations recorded by the source corpus.

Recently, methods such as Decision Transformer that reduce reinforcement learning to a prediction task and solve it via supervised learning (RvS) have become popular due to their simplicity, robustness to hyperparameters, and strong overall performance on offline RL tasks. However, simply conditioning a probabilistic model on a desired return and taking the predicted action can fail dramatically in stochastic environments since trajectories that result in a return may have only achieved that return due to luck. In this work, we describe the limitations of RvS approaches in stochastic environments and propose a solution. Rather than simply conditioning on the return of a single trajectory as is standard practice, our proposed method, ESPER, learns to cluster trajectories and conditions on average cluster returns, which are independent from environment stochasticity. Doing so allows ESPER to achieve strong alignment between target return and expected performance in real environments. We demonstrate this in several challenging stochastic offline-RL tasks including the challenging puzzle game 2048, and Connect Four playing against a stochastic opponent. In all tested domains, ESPER achieves significantly better alignment between the target return and achieved return than simply conditioning on returns. ESPER also achieves higher maximum performance than even the value-based baselines.

Subfields: Artificial Intelligence, General Decision Sciences, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yilun Du
2. Josh Tenenbaum
3. Sergey Levine
4. Tommi S. Jaakkola
5. Pulkit Agrawal
6. Anurag Ajay
7. Ruslan Salakhutdinov
8. Pieter Abbeel

In General Decision Sciences

1. Josh Tenenbaum

In Management Science and Operations Research

1. Anurag Ajay
2. Yilun Du
3. Josh Tenenbaum
4. Pulkit Agrawal
5. Tommi S. Jaakkola

In Reinforcement Learning

1. Josh Tenenbaum
2. Yilun Du
3. Anurag Ajay
4. Tommi S. Jaakkola
5. Pulkit Agrawal
6. Sergey Levine
7. Michael L. Littman
8. Csaba Szepesvari

Citations in the corpus

• Is Conditional Generative Modeling all you need for Decision-Making? (2022, background): “Several works (Paster et al., 2022; Yang et al., 2022) have shown that the performance of returnconditioned policies suffers as the stochasticity in environment dynamics increases.”