Lumiere: A Space-Time Diffusion Model for Video Generation

By Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli and 4 more. ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia, 2024

526 citations recorded by the source corpus.

We introduce Lumiere – a text-to-video diffusion model designed for synthesizing videos that portray realistic, diverse and coherent motion – a pivotal challenge in video synthesis. To this end, we introduce a Space-Time U-Net architecture that generates the entire temporal duration of the video at once, through a single pass in the model. This is in contrast to existing video models which synthesize distant keyframes followed by temporal super-resolution – an approach that inherently makes global temporal consistency difficult to achieve. By deploying both spatial and (importantly) temporal down- and up-sampling and leveraging a pre-trained text-to-image diffusion model, our model learns to directly generate a full-frame-rate, low-resolution video by processing it in multiple space-time scales. We demonstrate state-of-the-art text-to-video generation results , and show that our design easily facilitates a wide range of content creation tasks and video editing applications, including image-to-video, video inpainting, and stylized generation.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/94f7d8bce3bb848d127c8f113afc5bb0243579df

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Simplified and Generalized Masked Diffusion for Discrete Data (2024, unknown): “Since their inception [1, 2, 3], diffusion models have emerged as the workhorse for generative media, achieving state-of-the-art in tasks such as image synthesis [4, 5, 6], audio [7, 8] and video generation [9, 10, 11, 12, 13].”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, unknown): “The trade-off between perceptual quality and motion intensity has been widely acknowledged in video generation [3, 32, 73, 144], and our case is no exception.”
CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, background): “Video diffusion models have demonstrated an astonishing capability of generating realistic videos [48, 49, 10, 12, 15, 13, 50], and are thought to implicitly reason about 3D.”
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, unknown): “Diffusion models [23, 59, 16, 53] are considered the forefront of visual synthesis, given their superior generation quality and diversity.”