Make-A-Video: Text-to-Video Generation without Text-Video Data

By Uriel Singer, Adam Polyak, Thomas Hayes, Xiaoyue Yin, Jie An, Songyang Zhang, Qiyuan Hu, Qi-Yu Hu, Harry Yang, Oron Ashual and 4 more. 2022

2,111 citations recorded by the source corpus.

We propose Make-A-Video -- an approach for directly translating the tremendous recent progress in Text-to-Image (T2I) generation to Text-to-Video (T2V). Our intuition is simple: learn what the world looks like and how it is described from paired text-image data, and learn how the world moves from unsupervised video footage. Make-A-Video has three advantages: (1) it accelerates training of the T2V model (it does not need to learn visual and multimodal representations from scratch), (2) it does not require paired text-video data, and (3) the generated videos inherit the vastness (diversity in aesthetic, fantastical depictions, etc.) of today's image generation models. We design a simple yet effective way to build on T2I models with novel and effective spatial-temporal modules. First, we decompose the full temporal U-Net and attention tensors and approximate them in space and time. Second, we design a spatial temporal pipeline to generate high resolution and frame rate videos with a video decoder, interpolation model and two super resolution models that can enable various applications besides T2V. In all aspects, spatial and temporal resolution, faithfulness to text, and quality, Make-A-Video sets the new state-of-the-art in text-to-video generation, as determined by both qualitative and quantitative measures.

Subfields: Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

In Multimodal Learning

Citations in the corpus

The Gradient of Generative AI Release: Methods and Considerations (2023, background): “These releases can be cherry-picked, for example showing only non-human animals or human silhouettes [89], which does not give robust insight to broad capabilities or social impacts such as biases.”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, methodology): “For text-to-video/4D synthesis, the expansion of pretrained spatial to a spatial-temporal factorized U-Net with joint image and video data training has yielded significant success (Ho et al., 2022a;b; Singer et al., 2023a;b).”
Imagen Video: High Definition Video Generation with Diffusion Models (2022, methodology): “The concurrent work of Singer et al. (2022) also applied text-to-video modelling with diffusion models, but built on a pretrained text-to-image model.”
TokenFlow: Consistent Diffusion Features for Consistent Video Editing (2023, methodology): “Such models have been extended for text-to-video generation, by extending 2D architectures to the temporal dimen-sion (e.g., using temporal attention [15]) and performing large-scale training on video datasets [13, 46].”
GAIA-1: A Generative World Model for Autonomous Driving (2023, background): “They can operate either at the pixel level [38, 17, 66, 67, 68, 69, 16] or in the latent space of a pre-trained image tokenizer [70, 18, 71].”
EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (2023, methodology): “For example, they only evaluate the performance using FVD [56] (LVDM [20], MagicVideo [69], Align Your Latent [8]), IS [46] (Align Your Latent [8]), CLIP similarity [43] (Gen1 [16], Imagen Video [22], Make-A-Video [49]), or user studies to show the performance level.”
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation (2022, positive): “To replicate this success in Text-to-Video (T2V) generation, recent works [43, 16, 19, 54, 48] have extended spatial-only T2I generation models to the spatio-temporal domain.”
One-Step Diffusion with Distribution Matching Distillation (2023, background): “Diffusion Model Diffusion models [2, 21, 71, 74] have emerged as a powerful generative modeling framework, achieving unparalleled success in diverse domains such as image generation [61, 63, 64], audio synthesis [6, 35], and video generation [11, 22, 70].”
Training Diffusion Models with Reinforcement Learning (2023, positive): “Denoising diffusion models [47, 19] have emerged as an effective class of generative models for modalities including images [38, 41], videos [20, 46], 3D shapes [60, 57], and robotic trajectories [21, 1, 6].”
Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models (2023, methodology): “Similar to prior works [17, 46], we adapt the image-based U-Net model for the video synthesis task by making the following changes: (1) Transforming 2D convolutions to 3D by adding a dimension 1 to temporal axis and (2) Adding temporal attention layers.”
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, positive): “Although hierarchical structures require more inference time compared to single-stage models, their outcomes are significantly superior, as evidenced by advanced generation methods like (Ho et al., 2022a; Singer et al., 2022; Blattmann et al., 2023a).”
NExT-GPT: Any-to-Any Multimodal LLM (2023, background): “…tasks, such as Image/Video Captioning [99, 16, 56, 56, 27, 49], Image/Video Question Answering [94, 90, 48, 98, 3], Text-to-Image/Video/Speech Synthesis [74, 30, 84, 23, 17, 51, 33], Image-to-Video Synthesis [18, 37] and more, all of which have experienced rapid advancements in past decades.”
Driving Into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (2023, methodology): “Text-conditional video generation has garnered more attention due to its controllable generation, and a plethora of methods have emerged [2, 25, 48, 66, 73].”
Pix2Video: Video Editing using Image Diffusion (2023, positive): “With the success of text-to-image generation models, there has been recent attempts in text-to-video generation models using architectures such as transformers [19, 53, 60] or diffusion models [14, 17, 46, 61].”
VideoCrafter1: Open Diffusion Models for High-Quality Video Generation (2023, methodology): “Regarding text-to-video (T2V) models, Make-A-Video [45] and Imagen Video [27] are cascaded models, while most other works, such as LVDM [24], Magic Video [63], Mod-elScope [50], and Align your Latents [10], are SD-based models.”
Genie: Generative Interactive Environments (2024, neutral): “…video (Blattmann et al., 2023b; Brooks et al., 2024; Clark et al., 2019; Finn et al., 2016; Ho et al., 2022a,b; Höppe et al., 2022; Kalchbrenner et al., 2017; Le Moing et al., 2021; Lotter et al., 2017; Luc et al., 2020; Singer et al., 2023; Walker et al., 2021; Yan et al., 2021; Yu et al., 2023).”