CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer

By Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng and 9 more. International Conference on Learning Representations, 2024

2,147 citations recorded by the source corpus.

We present CogVideoX, a large-scale text-to-video generation model based on diffusion transformer, which can generate 10-second continuous videos aligned with text prompt, with a frame rate of 16 fps and resolution of 768 * 1360 pixels. Previous video generation models often had limited movement and short durations, and is difficult to generate videos with coherent narratives based on text. We propose several designs to address these issues. First, we propose a 3D Variational Autoencoder (VAE) to compress videos along both spatial and temporal dimensions, to improve both compression rate and video fidelity. Second, to improve the text-video alignment, we propose an expert transformer with the expert adaptive LayerNorm to facilitate the deep fusion between the two modalities. Third, by employing a progressive training and multi-resolution frame pack technique, CogVideoX is adept at producing coherent, long-duration, different shape videos characterized by significant motions. In addition, we develop an effective text-video data processing pipeline that includes various data preprocessing strategies and a video captioning method, greatly contributing to the generation quality and semantic alignment. Results show that CogVideoX demonstrates state-of-the-art performance across both multiple machine metrics and human evaluations. The model weight of both 3D Causal VAE, Video caption model and CogVideoX are publicly available at https://github.com/THUDM/CogVideo.

Subfields: Artificial Intelligence, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/7b248d78573ccf0dca6aa2cec2743d3eccaa9d1a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

Citations in the corpus

GR00T N1: An Open Foundation Model for Generalist Humanoid Robots (2025, neutral): “4 shows 𝑥 𝑡 and 𝑥 𝑡 + 𝐻 pairs from 8 distinct embodiments including both robot and human embodiment, all retrieved from similar latent actions; the first latent action shows all embodiments moving right arm to the left and the second latent action shows moving right arm to the right .”
HunyuanVideo: A Systematic Framework For Large Video Generative Models (2024, positive): “Most previous work focus on providing either brief captions [14, 50] or dense captions [93, 9, 10].”
Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (2025, positive): “Video diffusion models typically adopt bidirectional attention mechanisms to simultaneously denoise all video frames [3, 4, 6, 13, 23–26, 39, 64, 80, 83, 97].”
LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, neutral): “Several latest works (Brooks et al., 2024; Ma et al., 2024; Gupta et al., 2023; Lu et al., 2023; Yang et al., 2024) extend such idea on video generation and show promising results on large-scale text-to-video generation.”
CameraCtrl: Enabling Camera Control for Text-to-Video Generation (2024, neutral): “Besides, some methods jointly train the video generators using images and videos (Yang et al., 2024b).”
From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (2024, positive): “While most video diffusion models have bidirectional dependencies [5, 61, 96, 109], autoregressive video generation using diffusion models has also been explored.”
VACE: All-in-One Video Creation and Editing (2025, positive): “…[9, 22, 64] generation in the field, there has been a proliferation of down-stream tasks and applications, such as repainting [3, 82], editing [4, 42, 68, 70, 75], controllable generation [30, 76], frame reference generation [20, 73], and ID-referenced video synthesis [11, 35, 47, 74].”