Planning with Diffusion for Flexible Behavior Synthesis

By Michael Janner, Yilun Du, Josh Tenenbaum, Sergey Levine. International Conference on Machine Learning, 2022

1,286 citations recorded by the source corpus.

Model-based reinforcement learning methods often use learning only for the purpose of estimating an approximate dynamics model, offloading the rest of the decision-making work to classical trajectory optimizers. While conceptually simple, this combination has a number of empirical shortcomings, suggesting that learned models may not be well-suited to standard trajectory optimization. In this paper, we consider what it would look like to fold as much of the trajectory optimization pipeline as possible into the modeling problem, such that sampling from the model and planning with it become nearly identical. The core of our technical approach lies in a diffusion probabilistic model that plans by iteratively denoising trajectories. We show how classifier-guided sampling and image inpainting can be reinterpreted as coherent planning strategies, explore the unusual and useful properties of diffusion-based planning methods, and demonstrate the effectiveness of our framework in control settings that emphasize long-horizon decision-making and test-time flexibility.

Subfields: Artificial Intelligence, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/3ebdd3db0dd91069fa0cd31cbf8308b60b1b565e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

Citations in the corpus

WorldSimBench: Towards Video Generation Models as World Simulators (2024, background): “Predictive Video Model (Du et al., 2024; 2023), based on video generation models (Janner et al., 2022), have made some progress in embodied control.”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “Furthermore, diffusion models have close connections with other research area, such as robust learning [101, 156, 205], representative learning [1, 131, 232, 249] and reinforcement learning [92].”
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, background): “Video diffusion models now play important roles in robotics, showing a new era where robots can generate and interpret complex video sequences for enhanced perception [163, 164] and decision-making [165, 166, 167].”
Goal-Conditioned Imitation Learning using Score-based Diffusion Policies (2023, methodology): “Right: the internal denoising score model, which uses a transformer architecture with causal masking to iteratively predict the denoised action given the sequence of prior observations, actions, and the goal sequence. action or state-only trajectories using large U-net architectures [16, 1].”
EDGE: Editable Dance Generation From Music (2022, background): “In the past few years, diffusion models have been shown to be a promising avenue for generative modeling, exceeding the state-of-the-art in generative tasks [18, 24, 50].”
Imitating Human Behaviour with Diffusion Models (2023, methodology): “Up to our knowledge, the only previous work that used diffusion models for RL is by Janner et al. (2022) who proposed Diffuser: a model that predicts all states and actions of a complete trajectory; and uses classiﬁer-like guidance for trajectory optimization and similar U-Net architectures to…”
NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration (2023, methodology): “The noise prediction network, ϵ θ , consists of a 1D conditional U-Net [29, 31] with 15 convolutional layers.”
Compositional Visual Generation with Composable Diffusion Models (2022, background): “While diffusion models have been developed for image generation [39], they have further proven successful in the generation of waveforms [3], 3D shapes [47], decision making [16] and text [1], suggesting that our proposed composition operators may further be applied in such domains.”
Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models (2023, methodology): “While diffusion-based variants of these methods [32, 1] use diffusion models like our approach, they still require training data from the target task to learn a policy, unlike our zero-shot planning approach.”
IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies (2023, neutral): “Janner et al. [22] and Ajay et al. [1] use diffusion to directly model and sample the trajectory space; samples are guided with gradient guidance or reward conditioning.”
Is Conditional Generative Modeling all you need for Decision-Making? (2022, methodology): “• We represent the noise model εθ with a temporal U-Net (Janner et al., 2022), consisting of a U-Net structure with 6 repeated residual blocks.”
DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving (2024, neutral): “(c) Vanilla diffusion policy [7, 19].”
Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion (2024, positive): “[36] uses full-sequence diffusion models as planners in offline reinforcement learning.”
Training Diffusion Models with Reinforcement Learning (2023, positive): “Their flexibility in representing complex, high-dimensional distributions has led to the adoption of diffusion models in applications including image and video synthesis [38, 41, 20], drug and material design [56, 55, 42], and continuous control [21, 52, 15].”
Diffusion-based Generation, Optimization, and Planning in 3D Scenes (2023, methodology): “With the learned p θ p τ 0 | S q , we sample p p τ 0 | S , G q by taking the advantage of the diffusion model’s flexible conditioning [10,23].”
Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning (2022, methodology): “For sequence modelling approaches, we compare with Decision Transformer (DT) (Chen et al., 2021) and Diffuser (Janner et al., 2022).”