MagicVideo: Efficient Video Generation With Latent Diffusion Models

By Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, Jiashi Feng. arXiv.org, 2022

510 citations recorded by the source corpus.

We present an efficient text-to-video generation framework based on latent diffusion models, termed MagicVideo. MagicVideo can generate smooth video clips that are concordant with the given text descriptions. Due to a novel and efficient 3D U-Net design and modeling video distributions in a low-dimensional space, MagicVideo can synthesize video clips with 256x256 spatial resolution on a single GPU card, which takes around 64x fewer computations than the Video Diffusion Models (VDM) in terms of FLOPs. In specific, unlike existing works that directly train video models in the RGB space, we use a pre-trained VAE to map video clips into a low-dimensional latent space and learn the distribution of videos' latent codes via a diffusion model. Besides, we introduce two new designs to adapt the U-Net denoiser trained on image tasks to video data: a frame-wise lightweight adaptor for the image-to-video distribution adjustment and a directed temporal attention module to capture temporal dependencies across frames. Thus, we can exploit the informative weights of convolution operators from a text-to-image model for accelerating video training. To ameliorate the pixel dithering in the generated videos, we also propose a novel VideoVAE auto-encoder for better RGB reconstruction. We conduct extensive experiments and demonstrate that MagicVideo can generate high-quality video clips with either realistic or imaginary content. Refer to \url{https://magicvideo.github.io/#} for more examples.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/94b690162ead76af6a487d6e10998ea585c035d1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (2024, background): “For instance, MagicVideo [73] introduces a 3D U-Net-based architecture and directs temporal attention to ensure efficient, high-fidelity video generation while maintaining temporal consistency and authenticity.”
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (2024, background): “The high-level view of our approach is shown in Figure 2.”
MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model (2023, unknown): “Specifically, we inflate the original 2D UNet to 3D temporal UNet by inserting temporal attention layers [10, 40, 53].”
ModelScope Text-to-Video Technical Report (2023, methodology): “This encompasses textual content [40, 5, 56], visual content [39, 24, 48, 44, 27, 71, 16, 38, 51, 59] and auditory content [4, 32, 31].”
VBench: Comprehensive Benchmark Suite for Video Generative Models (2023, background): “Recently, diffusion models [19, 38, 89, 91] have achieved significant progress in image synthesis [31, 46, 47, 78, 81, 84, 86], and enabled a line of works towards video generation [8, 13, 28, 33– 35, 39, 40, 50, 58, 72, 88, 98, 104, 118, 124, 132, 133].”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, methodology): “In contrast, MagicVideo [75] models the video distribution in the latent space, following the paradigm of LDMs [44], signiﬁcantly reducing computational overhead.”
MotionCtrl: A Unified and Flexible Motion Controller for Video Generation (2023, unknown): “With advancements in diffusion models and the availability of robust computational resources, a plethora of diffusion-based video generation models have emerged[4, 5, 8–10, 23, 25, 32].”
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, unknown): “Several studies have investigated the hierarchical structure, encompassing separate keyframes, interpolation, and super-resolution modules for high-fidelity video generation.”
Photorealistic Video Generation with Diffusion Models (2023, background): “CogVideo (Chinese) [37] 23.6 751.3 CogVideo (English) [37] 25.3 701.6 MagicVideo [88] - 699.0 Make-A-Video [66] 33.0 367.2 Video LDM [4] 33.5 550.6 PYoCo [24] 47.8 Scaling our base model size leads to significant improvements on both the metrics.”
From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (2024, unknown): “The denoiser network in video diffusion models can be instantiated by different neural architectures, such as U-Net [10, 24, 67, 111] or Transformers [5, 25, 80, 96].”
VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, methodology): “Most video generation works use diffusion-based methods for text-to-video [4, 5, 25, 30, 33, 56, 67, 69, 70, 80, 81, 81, 85] and video-to-video editing [15, 23, 24, 39].”
Learning Interactive Real-World Simulators (2023, unknown): “In generative modeling of videos, previous works have leveraged text prompts [64, 65], driving motions [66, 67], 3D geometries [68, 69], physical simulations [70], frequency information [71], and user annotations [72] to introduce movements into videos.”
VideoCrafter1: Open Diffusion Models for High-Quality Video Generation (2023, methodology): “Regarding text-to-video (T2V) models, Make-A-Video [45] and Imagen Video [27] are cascaded models, while most other works, such as LVDM [24], Magic Video [63], Mod-elScope [50], and Align your Latents [10], are SD-based models.”
EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (2023, methodology): “Powered by the image generation priors in Stable Diffusion, LVDM [20] and Mag-icVideo [69] have been proposed to train the temporal layers to efficiently generate videos.”
I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models (2023, methodology): “Instead of modeling the video distribution in the pixels space, MagicVideo [61] does not use temporal convolution by designing an extra adapter.”
DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors (2023, methodology): “Following [7, 97], we evaluate the zero-shot generation performance of all the methods on UCF-101 [70] and MSR-VTT [85].”