Imagen Video: High Definition Video Generation with Diffusion Models

By Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey A. Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet and 1 more. arXiv.org, 2022

2,108 citations recorded by the source corpus.

We present Imagen Video, a text-conditional video generation system based on a cascade of video diffusion models. Given a text prompt, Imagen Video generates high definition videos using a base video generation model and a sequence of interleaved spatial and temporal video super-resolution models. We describe how we scale up the system as a high definition text-to-video model including design decisions such as the choice of fully-convolutional temporal and spatial super-resolution models at certain resolutions, and the choice of the v-parameterization of diffusion models. In addition, we confirm and transfer findings from previous work on diffusion-based image generation to the video generation setting. Finally, we apply progressive distillation to our video models with classifier-free guidance for fast, high quality sampling. We find Imagen Video not only capable of generating videos of high fidelity, but also having a high degree of controllability and world knowledge, including the ability to generate diverse videos and text animations in various artistic styles and with 3D object understanding. See https://imagen.research.google/video/ for samples.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/498ac9b2e494601d20a3d0211c16acf2b7954a54

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, methodology): “For text-to-video/4D synthesis, the expansion of pretrained spatial to a spatial-temporal factorized U-Net with joint image and video data training has yielded significant success (Ho et al., 2022a;b; Singer et al., 2023a;b).”
Training Diffusion Models with Reinforcement Learning (2023, positive): “Their flexibility in representing complex, high-dimensional distributions has led to the adoption of diffusion models in applications including image and video synthesis [38, 41, 20], drug and material design [56, 55, 42], and continuous control [21, 52, 15].”
EDGE: Editable Dance Generation From Music (2022, methodology): “Future work may explore the use of non-uniform sampling patterns such as those implemented in Harvey et al. [16], or a variation of the frame in-betweening scheme used in Ho et al. [18].”
I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models (2023, methodology): “Imagen Video [13] and Lavie [47] synthesize high-definition videos to achieve performance improvement in a progressive manner.”
RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion (2022, background): “While diffusion models have been successfully applied to different modalities [11, 23, 26, 32], its generative capability is much less explored in 3D generation, with only a few attempts on modeling 3D primitives [36, 74, 76].”
Structure and Content-Guided Video Synthesis with Diffusion Models (2023, methodology): “Diffusion models for video synthesis Recently, diffusion models, masked generative models and autoregressive models have been applied to text-conditioned video synthesis [17, 13, 58, 67, 18, 49].”
Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization (2023, background): “To meet the specific needs of different users, Ruiz et al .”
Imitating Human Behaviour with Diffusion Models (2023, background): “CFG can be interpreted as guiding the denoising sampling procedure towards higher values of p ( o | a ) (Ho et al., 2022).”
Synthetic Data from Diffusion Models Improves ImageNet Classification (2023, methodology): “Diffusion models have been successfully applied to image generation [25–27], speech generation [8,35], and video generation [24,58,68], and have found applications in various image processing areas, including image colorization, super-resolution, inpainting, and semantic editing [49,51,61,69].”
VideoCrafter1: Open Diffusion Models for High-Quality Video Generation (2023, methodology): “Regarding text-to-video (T2V) models, Make-A-Video [45] and Imagen Video [27] are cascaded models, while most other works, such as LVDM [24], Magic Video [63], Mod-elScope [50], and Align your Latents [10], are SD-based models.”
Latent Video Diffusion Models for High-Fidelity Long Video Generation (2022, background): “Besides, most of the text-to-video generation models are not available to the public [8, 29, 48, 12], which hinders the research progress of this ﬁeld.”
Continuous diffusion for categorical data (2022, methodology): “Generative models have seen a rapid increase in scale and capabilities over the past few years, across many modalities, including images, audio signals, video and text (Borsos et al., 2022; Brown et al., 2020; Dhariwal et al., 2020; Ho et al., 2022a; Ramesh et al., 2022; Sa-haria et al., 2022b).”
ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (2024, methodology): “PixelDance [68] and SVD [4] leverage pixel-based and latent-based techniques to produce high-resolution videos, while Make-A-Video [54] and Imagen Video [19] extend text-to-image models and techniques to the text-to-video domain.”
MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation (2022, background): “Previous studies [13, 19] drew similar conclusions as well.”