CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers

By Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, Jie Tang. 2022

1,140 citations recorded by the source corpus.

Large-scale pretrained transformers have created milestones in text (GPT-3) and text-to-image (DALL-E and CogView) generation. Its application to video generation is still facing many challenges: The potential huge computation cost makes the training from scratch unaffordable; The scarcity and weak relevance of text-video datasets hinder the model understanding complex movement semantics. In this work, we present 9B-parameter transformer CogVideo, trained by inheriting a pretrained text-to-image model, CogView2. We also propose multi-frame-rate hierarchical training strategy to better align text and video clips. As (probably) the first open-source large-scale pretrained text-to-video model, CogVideo outperforms all publicly available models at a large margin in machine and human evaluations.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Ming-Hsuan Yang
  3. Dacheng Tao
  4. Trevor Darrell
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Chen Change Loy
  8. Andrew Zisserman

In Generative Models

  1. Ying Shan
  2. Jun-Yan Zhu
  3. Xintao Wang
  4. Robin Rombach
  5. Karsten Kreis
  6. Ming-Hsuan Yang
  7. Eli Shechtman
  8. Andreas Blattmann

In Multimodal Learning

  1. Ying Shan
  2. Devi Parikh
  3. Sanja Fidler
  4. Xintao Wang
  5. Yu Qiao
  6. Jie Tang
  7. Ming Ding
  8. Jingren Zhou

In Natural Language Generation

  1. Devi Parikh
  2. Ming-Hsuan Yang
  3. Dumitru Erhan
  4. Yaniv Taigman
  5. Adam Polyak

Citations in the corpus