Decision Transformer: Reinforcement Learning via Sequence Modeling

By Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, A. Srinivas, Igor Mordatch. Neural Information Processing Systems, 2021

2,391 citations recorded by the source corpus.

We introduce a framework that abstracts Reinforcement Learning (RL) as a sequence modeling problem. This allows us to draw upon the simplicity and scalability of the Transformer architecture, and associated advances in language modeling such as GPT-x and BERT. In particular, we present Decision Transformer, an architecture that casts the problem of RL as conditional sequence modeling. Unlike prior approaches to RL that fit value functions or compute policy gradients, Decision Transformer simply outputs the optimal actions by leveraging a causally masked Transformer. By conditioning an autoregressive model on the desired return (reward), past states, and actions, our Decision Transformer model can generate future actions that achieve the desired return. Despite its simplicity, Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL baselines on Atari, OpenAI Gym, and Key-to-Door tasks.

Subfields: Artificial Intelligence, Reinforcement Learning

View Paper: https://arxiv.org/abs/2106.01345

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, background): “(2022) adapt reward-conditioned transformers (Chen et al., 2021) for several language generation tasks.”
Teaching Autoregressive Language Models Complex Tasks By Demonstration (2021, methodology): “…‘environment forcing’ (Section 4.2) is inspired by techniques for minimizing exposure bias [33]–[35] and corresponds to the conditioning on sequences of prior actions and states undertaken by [17], [18] when using ‘decision transformers’ and ‘trajectory transformers’ in imitation learning contexts.”
Pre-Trained Language Models for Interactive Decision-Making (2022, methodology): “We train Decision Transformer (DT) [7] using the data collected by LID-ADG.”
TALM: Tool Augmented Language Models (2022, methodology): “While our tasks are currently single-hop, this formulation can be extended further into RL: modelling multi-hop tool-use tasks as markov decision processes (MDPs), or integrating algorithms like Decision Transformer [Chen et al., 2021].”
Language Model Cascades (2022, background): “…may be used to reject incorrect examples in ancestral sampling, and the thought generator may itself be conditioned on the veriﬁers being correct by ﬁnetuning or prompting, reminiscent of RL as inference (Levine, 2018) and goal-conditioned policies such as decision-transformer (Chen et al., 2021).”
Shaking the foundations: delusions in sequence models for interaction and control (2021, methodology): “If we are interested in purposeful adaptive behavior (rather than mere prediction), it is natural to ask whether the above auto-regressive prediction scheme extends to the interactive setting , where the model is used both for generating and predicting data (see e.g. [6, 11]).”
Pix2seq: A Language Modeling Framework for Object Detection (2021, background): “…of modeling various sequential data, such as machine translation (Sutskever et al., 2014; Bahdanau et al., 2014), image captioning (Vinyals et al., 2015b; Karpathy & Fei-Fei, 2015; Xu et al., 2015), and many others (Vinyals et al., 2015a; Huang et al., 2018; Ramesh et al., 2021; Chen et al., 2021).”
Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “…2020), language modeling (Radford et al., 2018; 2019; Brown et al., 2020), coding (Chen et al., 2021b; Li et al., 2022), , reinforcement learning (Chen et al., 2021a; Janner et al., 2021), image synthesis (Esser et al., 2020; Ramesh et al., 2021; Yu et al., 2022) and protein folding (Jumper et…”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “We compare against ﬁve other methods spanning other approaches to data-driven control: (1) behavior-regularized actor-critic (BRAC; Wu et al. 2019) 2021)) is a concurrently-developed sequence-modeling approach that uses return-conditioning instead of planning; and behavior-cloning (BC) provides the…”
Offline Reinforcement Learning with Implicit Q-Learning (2021, methodology): “We did not reimplement Decision Transformers due to their complexity and report runtime of the original implementation.”
A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “The Decision Transformer (DT) is concurrent work which examines the use of the transformer architecture in offline RL, by framing the problem as sequence modeling [Chen et al., 2021].”
Quark: Controllable Text Generation with Reinforced Unlearning (2022, methodology): “(2) Inspired by the Decision Transformer [9] which frames reinforcement learning as next-token prediction, we train a model capable of conditioning on the desired reward of the trajectory, prior to observing it, i.”
Boosting Search Engines with Interactive Agents (2021, positive): “We plan to investigate approaches like Decision Transformers (Chen et al., 2021) next, as a natural framework for expressing the search task within LMs, while at the same time bringing back in key RL concepts such as expected returns and learning also from negative experiences, possibly produced by ”
Can Wikipedia Help Offline Reinforcement Learning? (2022, methodology): “RL Baselines In addition to benchmarking our pre-trained language models, we compare to popular state-of-the-art offline RL algorithms as follows: Decision Transformer (DT) (Chen et al., 2021), CQL (Kumar et al.”