CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers
By Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, Jie Tang. 2022
1,140 citations recorded by the source corpus.
Large-scale pretrained transformers have created milestones in text (GPT-3) and text-to-image (DALL-E and CogView) generation. Its application to video generation is still facing many challenges: The potential huge computation cost makes the training from scratch unaffordable; The scarcity and weak relevance of text-video datasets hinder the model understanding complex movement semantics. In this work, we present 9B-parameter transformer CogVideo, trained by inheriting a pretrained text-to-image model, CogView2. We also propose multi-frame-rate hierarchical training strategy to better align text and video clips. As (probably) the first open-source large-scale pretrained text-to-video model, CogVideo outperforms all publicly available models at a large margin in machine and human evaluations.
Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Chen Change Loy
- Andrew Zisserman
In Generative Models
- Ying Shan
- Jun-Yan Zhu
- Xintao Wang
- Robin Rombach
- Karsten Kreis
- Ming-Hsuan Yang
- Eli Shechtman
- Andreas Blattmann
In Multimodal Learning
In Natural Language Generation
Citations in the corpus
- GLM-130B: An Open Bilingual Pre-trained Model (2022, background): “Large language models, together with other machine learning models in different modalities (e.g., Image (Ramesh et al., 2021; Ding et al., 2021; Saharia et al.) and Video (Hong et al., 2022)), could be used to generate synthetic text for harmful applications, such as telemarketing fraud, political p”
- MusicLM: Generating Music From Text (2023, neutral): “(Wu et al., 2022a; Hong et al., 2022; Vil-legas et al., 2022; Ho et al., 2022).”
- Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, neutral): “CogVideo (Hong et al., 2022) is built on top of a CogView2 (Ding et al.”
- Genie: Generative Interactive Environments (2024, methodology): “Unlike prior works that focus on spatial-only compression in the tokenization phase (Gupta et al., 2023; Hong et al., 2022; Wu et al., 2022), we utilize the ST-transformer in both the encoder and decoder to incorporate temporal dynamics in the encodings, which improves the video generation quality.”
- Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation (2022, neutral): “We compare our method against three base-lines: 1) CogVideo [20]: a T2V model trained on a dataset of 5.4 million captioned videos, and is capable of generating videos directly from text prompts in a zero-shot manner.”
- VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, methodology): “…weights Prompt : A large blob of exploding splashing rainbow paint, with an apple emerging, 8k Parti [74] and [19]) and text-to-video ( e.g ., [36]), language models have not reached a level of quality on par with video diffusion models in tasks like text-to-video generation as shown in…”
- Point-E: A System for Generating 3D Point Clouds from Complex Prompts (2022, methodology): “Inspired by these results, recent works have explored text-conditional generation in other modalities, such as video (Hong et al., 2022; Singer et al., 2022; Ho et al., 2022b;a) and 3D objects (Jain et al., 2021; Poole et al., 2022; Lin et al., 2022a; Sanghi et al., 2021; 2022).”
- LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, neutral): “In terms of memory consumption, our base, interpolation and VSR models require 13GB, 40GB and 42GB respectively.”
- Make-A-Video: Text-to-Video Generation without Text-Video Data (2022, methodology): “Second, we fine-tune the T2I model for video generation, gaining the advantage of adapting the model weights effectively, compared to freezing the weights as in CogVideo (Hong et al., 2022).”
- MAGVIT: Masked Generative Video Transformer (2022, methodology): “Inspired by the recent success of generative image transformers such as DALL·E [46] and other approaches [12,18,20,72], we propose an efficient and effective video generation model by leveraging masked token modeling and multi-task learning.”
- Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos (2023, background): “CogVideo [14] improves video generation quality by extending CogView-2 [5] to T2V generation through the incorporation of temporal attention modules and pre-trained text-to-image models.”
- Latent Video Diffusion Models for High-Fidelity Long Video Generation (2022, methodology): “We thus utilize a common strategy, hierarchical generation [6, 13, 3], to alleviate this problem.”
- Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer (2022, methodology): “2% over previous method with synthetic initial frames [37] and is competitive against concurrent works [61,24,22].”
- Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (2023, background): “Recently, benefiting from advancements in text-to-image research[2, 19, 29, 32, 34, 36], video generation (e.g., text-to-video, video editing)[4, 10, 12, 15–17, 23, 30, 40, 48, 52] has also achieved notable progress in terms of visual quality and diversity.”
- ControlVideo: Training-free Controllable Text-to-Video Generation (2023, methodology): “Several works [5,9,11,12,34] attempt to replicate this success in the video counterpart, i.e ., modeling higher-dimensional complex video distributions in the wild world.”
- Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (2025, neutral): “In contrast, autoregressive (AR) models [17, 27, 38, 94, 104] generate videos sequentially, a paradigm that naturally aligns with the causal structure of temporal media.”
- Structure and Content-Guided Video Synthesis with Diffusion Models (2023, methodology): “Diffusion models for video synthesis Recently, diffusion models, masked generative models and autoregressive models have been applied to text-conditioned video synthesis [17, 13, 58, 67, 18, 49].”
- Pix2Video: Video Editing using Image Diffusion (2023, methodology): “With the success of text-to-image generation models, there has been recent attempts in text-to-video generation models using architectures such as transformers [20, 52, 59] or diffusion models [15, 18, 45, 60].”
- Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, neutral): “…2017; Tulyakov et al., 2018; Tian et al., 2021; Shen et al., 2023), autoregressive models (Srivastava et al., 2015; Yan et al., 2021; Le Moing et al., 2021; Ge et al., 2022; Hong et al., 2022; Kondratyuk et al., 2023), and implicit neural representations (Skorokhodov et al., 2021; Yu et al., 2021).”