Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models

By Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David W. Jacobs, Jia-Bin Huang, Ming-Yu Liu, Yogesh Kumar Balaji. IEEE International Conference on Computer Vision, 2023

326 citations recorded by the source corpus.

Despite tremendous progress in generating high-quality images using diffusion models, synthesizing a sequence of animated frames that are both photorealistic and temporally coherent is still in its infancy. While off-the-shelf billion-scale datasets for image generation are available, collecting similar video data of the same scale is still challenging. Also, training a video diffusion model is computationally much more expensive than its image counterpart. In this work, we explore finetuning a pretrained image diffusion model with video data as a practical solution for the video synthesis task. We find that naively extending the image noise prior to video noise prior in video diffusion leads to sub-optimal performance. Our carefully designed video noise prior leads to substantially better performance. Extensive experimental validation shows that our model, Preserve Your Own COrrelation (PYoCo), attains SOTA zero-shot text-to-video results on the UCF-101 and MSR-VTT benchmarks. It also achieves SOTA video generation quality on the small-scale UCF-101 benchmark with a 10× smaller model using significantly less computation than the prior art. The project page is available at https://research.nvidia.com/labs/dir/pyoco/.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/02bc11de9d2f75bad48166098aa6b30fffee4d70

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, unknown): “The Bund, Shanghai, with the ship moving on the river, oil painting.”
Photorealistic Video Generation with Diffusion Models (2023, methodology): “Nevertheless, to compare with prior work on text-to-video, we also report results on the UCF-101 dataset in the zero-shot evaluation protocol in Table 5 [24, 37, 66].”
VBench: Comprehensive Benchmark Suite for Video Generative Models (2023, background): “Recently, diffusion models [19, 38, 89, 91] have achieved significant progress in image synthesis [31, 46, 47, 78, 81, 84, 86], and enabled a line of works towards video generation [8, 13, 28, 33– 35, 39, 40, 50, 58, 72, 88, 98, 104, 118, 124, 132, 133].”
OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (2024, methodology): “Current text-to-video generation methods can be divided into two categories from architecture design: UNet based methods [19, 20, 7, 21, 22, 23, 24], and DiT based methods [6, 8, 5].”
Lumiere: A Space-Time Diffusion Model for Video Generation (2024, unknown): “The conditioning signal C contains this first frame followed by blank frames for the rest of the video.”
Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation (2023, methodology): “Text-to-image and text-to-video generation has garnered significant rapid advancements using both language models (Yu et al., 2023b; Chang et al., 2023) and diffusion models (Ho et al., 2022a; Blattmann et al., 2023; Singer et al., 2022; Ge et al., 2023; Ramesh et al., 2022).”
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, unknown): “, 2022a), PYoCo (Ge et al., 2023), and (2) Latent-based VDMs that manipulate the compacted latent space within a variational autoencoder (VAE), like Video LDM (Blattmann et al.”
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (2023, methodology): “CogVideo (ZH) [43] 751.34 CogVideo (EN) [43] 701.59 Make-A-Video [82] 367.23 Video LDM [9] 550.61 MagicVideo [115] 655.00 PYOCO [29] 355.20”
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, background): “To address this, one can either sample a specific number of frames (padding or temporal interpolation [34] may be needed for much shorter videos) or define a universally extended (super long) input length for subsequent processing (more details are described in Section 3.2.4); Utilization of…”
VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, methodology): “Most video generation works use diffusion-based methods for text-to-video [4, 5, 25, 30, 33, 56, 67, 69, 70, 80, 81, 81, 85] and video-to-video editing [15, 23, 24, 39].”
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (2024, methodology): “For UCF101 [66], we produce a text prompt for each class [23] and generate 10,000 videos which share the same class distribution as the original dataset [8, 72].”