Is Conditional Generative Modeling all you need for Decision-Making?

By Anurag Ajay, Yilun Du, Josh Tenenbaum, Tommi S. Jaakkola, Pulkit Agrawal. International Conference on Learning Representations, 2022

657 citations recorded by the source corpus.

Recent improvements in conditional generative modeling have made it possible to generate high-quality images from language descriptions alone. We investigate whether these methods can directly address the problem of sequential decision-making. We view decision-making not through the lens of reinforcement learning (RL), but rather through conditional generative modeling. To our surprise, we find that our formulation leads to policies that can outperform existing offline RL approaches across standard benchmarks. By modeling a policy as a return-conditional diffusion model, we illustrate how we may circumvent the need for dynamic programming and subsequently eliminate many of the complexities that come with traditional offline RL. We further demonstrate the advantages of modeling policies as conditional diffusion models by considering two other conditioning variables: constraints and skills. Conditioning on a single constraint or skill during training leads to behaviors at test-time that can satisfy several constraints together or demonstrate a composition of skills. Our results illustrate that conditional generative modeling is a powerful tool for decision-making.

Subfields: Artificial Intelligence, Cultural Studies

View Paper: https://www.semanticscholar.org/paper/f19dfc360088922cf1d423c538662aae8d542c28

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cultural Studies

Citations in the corpus

Diffusion policy: Visuomotor policy learning via action diffusion (2023, positive): “Diffusion models (Ho et al., 2020; Sohl-Dickstein et al., 2015) have recently been applied to solve various different control tasks (Ajay et al., 2022; Janner et al., 2022a; Urain et al., 2022).”
Training Diffusion Models with Reinforcement Learning (2023, neutral): “Denoising diffusion models [47, 19] have emerged as an effective class of generative models for modalities including images [38, 41], videos [20, 46], 3D shapes [60, 57], and robotic trajectories [21, 1, 6].”
Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models (2023, methodology): “While diffusion-based variants of these methods [32, 1] use diffusion models like our approach, they still require training data from the target task to learn a policy, unlike our zero-shot planning approach.”
Learning Universal Policies via Text-Guided Video Generation (2023, methodology): “Ajay et al. (2022) on the other hand, trained a conditional diffusion model to guide behavior synthesis from desired re-wards, constraints or agent skills.”
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, background): “Video diffusion models now play important roles in robotics, showing a new era where robots can generate and interpret complex video sequences for enhanced perception [163, 164] and decision-making [165, 166, 167].”
IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies (2023, neutral): “[1] use diffusion to directly model and sample the trajectory space; samples are guided with gradient guidance or reward conditioning.”
Goal-Conditioned Imitation Learning using Score-based Diffusion Policies (2023, methodology): “Right: the internal denoising score model, which uses a transformer architecture with causal masking to iteratively predict the denoised action given the sequence of prior observations, actions, and the goal sequence. action or state-only trajectories using large U-net architectures [16, 1].”
DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving (2024, positive): “Subsequently, numerous works have applied diffusion models to various robotic tasks, including stationary manipulation [1, 53], mobile manipulation [47], autonomous nav-12039 igation [37, 51], quadruped locomotion [38], and dexterous manipulation [46].”
Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion (2024, neutral): “Commonly used in video generation and long-horizon planning, one directly models the joint distribution of a fixed number of tokens by diffusing their concatenation [31, 1], where the noise level is identical across all tokens.”
Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning (2022, background): “Goo & Niekum (2022) emphasize the necessity of conducting explicit behavioral cloning in offline RL, while Ajay et al. (2022) admit the power of conditional generative models for decision making.”
Imitating Human Behaviour with Diffusion Models (2023, neutral): “Ajay et al. (2023) provide a further concurrent work.”