GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions

By Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, Nan Duan. arXiv.org, 2021

317 citations recorded by the source corpus.

Generating videos from text is a challenging task due to its high computational requirements for training and infinite possible answers for evaluation. Existing works typically experiment on simple or small datasets, where the generalization ability is quite limited. In this work, we propose GODIVA, an open-domain text-to-video pretrained model that can generate videos from text in an auto-regressive manner using a three-dimensional sparse attention mechanism. We pretrain our model on Howto100M, a large-scale text-video dataset that contains more than 136 million text-video pairs. Experiments show that GODIVA not only can be fine-tuned on downstream video generation tasks, but also has a good zero-shot capability on unseen texts. We also propose a new metric called Relative Matching (RM) to automatically evaluate the video generation quality. Several challenges are listed and discussed as future work.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c64025f83864ec9c40e2970a24314b6b84d4c753

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer (2022, methodology): “We hence consider several kinds of conditional information that provide additional temporal information such as audio [10], text [76], and so on.”
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, methodology): “Zero-Shot CLIPSIM ( ↑ ) GODIVA [98] No 0.2402 N ¨UWA [99] No 0.2439 CogVideo (Chinese) [32] Yes 0.2614 CogVideo (English) [32] Yes 0.2631 Make-A-Video [76] Yes 0.3049 Video LDM (SD 1.4) (Ours) Yes 0.2848 Video LDM (SD 2.1) (Ours) Yes 0.2929 low-resolution conditioning following our…”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, unknown): “Method Zero-shot FVD ↓ CLIPSIM ↑ GODIVA [64] No - 0.2402 Nüwa [65] No - 0.2439 CogVideo (Chinese) [24] Yes - 0.2614 CogVideo (English) [24] Yes 1294 0.2631 MagicVideo [75] Yes 1290 - Make-A-Video [49] Yes - 0.3049 Video LDM [”
Diffusion Probabilistic Modeling for Video Generation (2022, background): “Videos can be generated based on side information of various types, such as images [30,31], text [32–37] or other videos [38].”
Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos (2023, background): “Thanks to the ability of large language models [24] and trans-former [34], more current works generate video from the text description. e.g ., GODIVA [37] extends VQ-VAE [33] to text-to-video generation by mapping text tokens to video tokens.”
ModelScope Text-to-Video Technical Report (2023, methodology): “We compare ModelScopeT2V with several state-of-the-art models using FID-vid [15], FVD [57], and CLIPSIM [61] metrics.”
ControlVideo: Training-free Controllable Text-to-Video Generation (2023, methodology): “Earlier works [12,36,38,39] adopt an autoregressive framework to synthesize videos according to given descriptions.”
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, unknown): “Table 2: Comparisons with SOTA models on MSR-VTT dataset (Xu et al., 2016).”
LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, unknown): “In terms of memory consumption, our base, interpolation and VSR models require 13GB, 40GB and 42GB respectively.”
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation (2022, unknown): “GODIVA [43] is the first work to extend VQ-VAE [39] to T2V generation by mapping text tokens to video tokens.”
VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, methodology): “We measure CLIP similarity scores [71] following an implementation given by Villegas et al .”
Language-Driven Image Style Transfer (2021, methodology): “Following GODIVA [12], we adopt CLIP [13] that provides visual-text matching: • Vision-and-Language Similarity (VLS) : VLS calculates the cosine similarity between the joint embedding of the instruction ( X ) and the transferred result ( ˆ O ) from CLIP; • Relative Similarity (RS) : Since VLS only…”
NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion (2021, methodology): “Recently, with the arise of VQ-VAE [40] as a discrete visual tokenization approach, efﬁcient and large-scale pre-1 training can be applied to visual synthesis tasks for images (e.g., DALL-E [33] and CogView [9]) and videos (e.g., GO-DIVA [45]).”
Make-A-Video: Text-to-Video Generation without Text-Video Data (2022, methodology): “Method Zero-Shot Samples/Input FID ( ↓ ) CLIPSIM ( ↑ ) GODIVA (Wu et al., 2021a) No 30 − 0 .”
Transformers are Sample Efficient World Models (2022, background): “GODIVA (Wu et al., 2021) models sequences of frames instead of a single frame for text conditional video generation.”
Phenaki: Variable Length Video Generation From Open Domain Textual Description (2022, methodology): “We compare C-ViViT against per-frame image based encoder-decoders that have been used as video quantizers for conditional video generation [57, 54, 18, 54, 18, 52]: a ViT [58] and a convolutional VQ-GAN[12].”
Photorealistic Video Generation with Diffusion Models (2023, methodology): “Subsequently, this approach was applied to multiple applications including class-conditional image generation [20, 79], text-to-image [17, 59, 76, 80] or image-to-image translation [21, 77].”
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (2024, methodology): “We report FVD [68] on UCF101 [66] and CLIP similarity (CLIPSim) [76] on MSR-VTT [79].”
Learning Interactive Real-World Simulators (2023, background): “Internet-scale generative models can synthesize realistic images and videos [44, 3, 45, 46, 47], but have mostly been applied to generative media [48] as opposed to empowering sophisticated agents capable of multi-turn interactions.”