Flexible Diffusion Modeling of Long Videos

By William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, Frank D. Wood. Neural Information Processing Systems, 2022

371 citations recorded by the source corpus.

We present a framework for video modeling based on denoising diffusion probabilistic models that produces long-duration video completions in a variety of realistic environments. We introduce a generative model that can at test-time sample any arbitrary subset of video frames conditioned on any other subset and present an architecture adapted for this purpose. Doing so allows us to efficiently compare and optimize a variety of schedules for the order in which frames in a long video are sampled and use selective sparse and long-range conditioning on previously sampled frames. We demonstrate improved video modeling over prior work on a number of datasets and sample temporally coherent videos over 25 minutes in length. We additionally release a new video modeling dataset and semantically meaningful metrics based on videos generated in the CARLA autonomous driving simulator.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8299db3c4c58f0962e2cd6ebb5319cbed621e9dd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers (2022, background): “Recently, several text-to-video diffusion models were proposed and achieved high-quality video generation results [20, 25, 29, 67, 85].”
A Survey on Generative Diffusion Models (2022, methodology): “Repre-sentative works include Video Diffusion Models [12], Make-A-Video [154], AnimatedDiff [159], RVD [157], FDM [156], MCVD [155].”
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, methodology): “Our training approach for prediction and interpolation models is inspired by recent works [24,33,93] that use similar masking techniques (also see Appendix C).”
DiffusionDet: Diffusion Model for Object Detection (2022, background): “Diffusion models have recently demonstrated remarkable results in fields including computer vision [4,21,33,35,39,68,71,75,76,82,104,108], nature language processing [3,30,52], audio processing [41,48,50,70, 91,100,103], graph-related topics [42], interdisciplinary applications…”
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, positive): “Inspired by the notable success of the diffusion model in image synthesis, several recent studies have ventured into applying diffusion models for both conditional and unconditional video synthesis (Voleti et al., 2022; Harvey et al., 2022; Zhou et al., 2022; Wu et al., 2022b; Blattmann et al., 2023”
GAIA-1: A Generative World Model for Autonomous Driving (2023, methodology): “We choose to train a single video diffusion model for all tasks as it has been shown that multi-task training improves performance on individual tasks [17].”
Imagen Video: High Definition Video Generation with Diffusion Models (2022, background): “Harvey et al. (2022) generates videos up to 25 minutes in length with video diffusion models, however the domain is restricted.”
ModelScope Text-to-Video Technical Report (2023, positive): “Recent works have utilized diffusion models [53, 37] to generate authentic videos [66, 14, 21, 59].”
Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models (2023, background): “Recently, driven by the tremendous success of applying the diffusion model to image synthesis, multiple works have proposed to explore diffusion models for conditional and unconditional video synthesis [57, 12, 69, 61, 4, 22, 19, 57, 64, 33, 28, 1, 59].”
Imitating Human Behaviour with Diffusion Models (2023, background): “In particular, this paper focuses on diffusion models, which currently lead in image, video and audio generation (Saharia et al., 2022; Harvey et al., 2022; Kong et al., 2020), and avoid issues of training instability in generative adversarial networks (Srivastava et al.”
Photorealistic Video Generation with Diffusion Models (2023, background): “Diffusion models have shown impressive results in image [33, 38, 52, 61, 67, 68] and video generation [4, 24, 29, 34, 36, 66].”
Latte: Latent Diffusion Transformer for Video Generation (2024, positive): “Finally, recent advances in video generation have focused on building systems based on diffusion models (Ho et al, 2020; Harvey et al, 2022; Ho et al, 2022; Singer et al, 2022; Mei and Patel, 2023; Blattmann et al, 2023b; Wang et al, 2023b; Chen et al, 2023c; Wang et al, 2023c), resulting in…”
VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation (2023, background): “Inspired by the success of DPMs on image generation, many researchers are trying to apply a similar idea to video prediction/interpolation [13, 44, 48].”
Phenaki: Variable Length Video Generation From Open Domain Textual Description (2022, background): “…42, 13, 50], variational based stochastic meth-ods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion models [47, 14, 56, 19] ﬂow based models [24], and ﬁnally adversarial based methods [48, 39, 43, 9, 40, 27].”
Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow (2022, neutral): “It has been quickly popularized in other domains, such as video [e.g., 24, 92, 21], music [51], audio [e.g., 33, 40, 60], and text [41, 88], and more tasks such as image editing [97, 50].”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, background): “Recent research has demonstrated the potential of employing diffusion models for video synthesis [6, 18, 23, 28, 32, 68].”
VBench: Comprehensive Benchmark Suite for Video Generative Models (2023, methodology): “We then elaborate on the prompt suites we use in Section 3.2.”