Offline Reinforcement Learning as One Big Sequence Modeling Problem

By Michael Janner, Qiyang Li, Sergey Levine. 2021

901 citations recorded by the source corpus.

Reinforcement learning (RL) is typically concerned with estimating stationary policies or single-step models, leveraging the Markov property to factorize problems in time. However, we can also view RL as a generic sequence modeling problem, with the goal being to produce a sequence of actions that leads to a sequence of high rewards. Viewed in this way, it is tempting to consider whether high-capacity sequence prediction models that work well in other domains, such as natural-language processing, can also provide effective solutions to the RL problem. To this end, we explore how RL can be tackled with the tools of sequence modeling, using a Transformer architecture to model distributions over trajectories and repurposing beam search as a planning algorithm. Framing RL as sequence modeling problem simplifies a range of design decisions, allowing us to dispense with many of the components common in offline RL algorithms. We demonstrate the flexibility of this approach across long-horizon dynamics prediction, imitation learning, goal-conditioned RL, and offline RL. Further, we show that this approach can be combined with existing model-free algorithms to yield a state-of-the-art planner in sparse-reward, long-horizon tasks.

Subfields: Artificial Intelligence, Information Systems, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Reinforcement Learning

Citations in the corpus

Pretraining Language Models with Human Preferences (2023, background): “, 2019) and demonstrated strong results when paired with transformers (Chen et al., 2021a; Janner et al., 2021).”
Scalable Diffusion Models with Transformers (2022, background): “Transformers [57] have replaced domainspecific architectures across language, vision [10], reinforcement learning [5, 23] and meta-learning [36].”
Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “, 2022), , reinforcement learning (Chen et al., 2021a; Janner et al., 2021), image synthesis (Esser et al.”
Quark: Controllable Text Generation with Reinforced Unlearning (2022, methodology): “The Trajectory Transformer [27] and Decision Transformer [9] are both ofﬂine RL methods that use transformers to produce a sequence of actions with high rewards given observed states.”
Boosting Search Engines with Interactive Agents (2021, neutral): “We have started to investigate approaches in the direction of decision/trajectory transformers (Chen et al., 2021; Janner et al., 2021).”
Reinforcement Learning for Language Models (2023, background): “Several prior papers have applied offline RL to NLP and more broadly to sequence generation problems [23] [24] [25], suggesting potential benefits of an off-policy approach.”
IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies (2023, neutral): “The Decision Transformer and Trajectory Transformer [8, 21] clone the entire trajectory space using transformers and apply reward conditioning or beam search, respectively, to bias sampled trajectories towards higher rewards.”
A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks (2023, methodology): “In summary, approaches like Decision Transformer and Trajectory Transformer have addressed RL problems as sequence modeling tasks, harnessing the power of transformer architectures to model sequential trajectories (Chen et al., 2021, Janner et al., 2021).”
Multi-Agent Reinforcement Learning is a Sequence Modeling Problem (2022, background): “Coming with effective and expressive network architectures such as Transformer [40], sequence modeling techniques have also attracted tremendous attention from the RL community, which results in a series of successful ofﬂine RL developments based on the Transformer architecture [5, 14, 30, 23].”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, background): “1) Novel agent architecture: Decision Transformer [19, 58, 144] applies the powerful self-attention models to sequential decision making.”
RvS: What is Essential for Offline RL via Supervised Learning? (2021, background): “Prior work has also looked at conditioning on different information, such as many previous timesteps [5, 23], tasks inferred by inverse RL [10], and other task information [6, 21].”
Consistent Dropout for Policy Gradient Reinforcement Learning (2022, background): “Examples such as the Decision Transformer (Chen et al., 2021) and Trajectory Transformer (Janner et al., 2021) show that transformer models are capable of learning high quality policies from offline expert data.”
AnyMorph: Learning Transferable Polices By Inferring Agent Morphology (2022, methodology): “We implement our policy as a variant of the Transformer (Vaswani et al., 2017) due to its success in multiple reinforcement learning settings (Kurin et al., 2021; Janner et al., 2021; Chen et al., 2021).”
Offline RL for Natural Language Generation with Implicit Language Q Learning (2022, background): “Several prior works have applied offline RL to NLP and more broadly sequence generation problems [12, 11, 13, 14, 7, 8].”
Transformers are Sample Efficient World Models (2022, methodology): “The Trajectory Transformer [25] and the Decision Transformer [26] represent offline trajectories as a static dataset of sequences.”
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems (2022, methodology): “’s [28] in (51), but has a modified training procedure and opts for a different planning strategy.”
History Compression via Language Models in Reinforcement Learning (2022, methodology): “Further, it has been successfully applied for sequence modeling in the domain of RL (Janner et al., 2021; Chen et al., 2021).”