Photorealistic Video Generation with Diffusion Models

By Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Fei-Fei Li, Irfan Essa, Lu Jiang, José Lezama. European Conference on Computer Vision, 2023

341 citations recorded by the source corpus.

We present W.A.L.T, a transformer-based approach for photorealistic video generation via diffusion modeling. Our approach has two key design decisions. First, we use a causal encoder to jointly compress images and videos within a unified latent space, enabling training and generation across modalities. Second, for memory and training efficiency, we use a window attention architecture tailored for joint spatial and spatiotemporal generative modeling. Taken together these design decisions enable us to achieve state-of-the-art performance on established video (UCF-101 and Kinetics-600) and image (ImageNet) generation benchmarks without using classifier free guidance. Finally, we also train a cascade of three models for the task of text-to-video generation consisting of a base latent video diffusion model, and two video super-resolution diffusion models to generate videos of $512 \times 896$ resolution at $8$ frames per second.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/905ba940236b00bebb2fd348d4d932e7887b0c0a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (2024, neutral): “Based on these improvements, Pixart-α (Chen et al., 2024b), Pixart-Σ (Chen et al., 2024a), Stable diffusion 3 (Esser et al., 2024) show pure transformers can be scaled up for challenging text-to-image generation, and CMD (Yu et al., 2024), WALT (Gupta et al., 2024), and Sora (Brooks et al., 2024) de”
Lumiere: A Space-Time Diffusion Model for Video Generation (2024, neutral): “To reconcile the per-segment SSR predictions, { Φ( J i ) } Ni =1 , we define the result of the denoising step to be the solution of the optimization problem arg min The solution to this problem is given by linearly combining the predictions over overlapping windows.”
CameraCtrl: Enabling Camera Control for Text-to-Video Generation (2024, neutral): “When selecting the dataset for training our camera control model, we first choose three datasets as candidates, they are Objaverse (Deitke et al., 2023), MVImageNet (Yu et al., 2023), and RealEstate10K (Zhou et al., 2018).”
Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (2025, neutral): “Video diffusion models typically adopt bidirectional attention mechanisms to simultaneously denoise all video frames [3, 4, 6, 13, 23–26, 39, 64, 80, 83, 97].”
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, background): “T) [65] proposes to unify the latent space of both videos and images with 3D causal CNN and a special coding strategy for the first frame.”
BLOCK DIFFUSION: INTERPOLATING BETWEEN AU-TOREGRESSIVE AND DIFFUSION LANGUAGE MODELS (2025, neutral): “Diffusion models are widely used to generate images (Ho et al., 2020; Dhariwal & Nichol, 2021; Sahoo et al., 2024b) and videos (Ho et al., 2022; Gupta et al., 2023), and are becoming increasingly effective at generating discrete data such as text (Lou et al., 2024; Sahoo et al., 2024a) or…”
VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, methodology): “For each video task, we randomly select 20% from a training subset of 50 million videos.”
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, neutral): “Diffusion models [23, 59, 16, 53] are considered the forefront of visual synthesis, given their superior generation quality and diversity.”
LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, neutral): “Several latest works (Brooks et al., 2024; Ma et al., 2024; Gupta et al., 2023; Lu et al., 2023; Yang et al., 2024) extend such idea on video generation and show promising results on large-scale text-to-video generation.”
SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers (2024, background): “Many concurrent works [23, 31, 40, 45] explore similar approaches in a wide variety of downstream tasks, and we leave the application of SiT to these tasks for future works.”
Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (2025, neutral): “The second strategy is to de-liberately limit the tokenizer’s reconstruction capacity, e.g. Sana [4, 42], W.A.L.T [15], for faster convergence of diffusion model training.”