Movie Gen: A Cast of Media Foundation Models

By Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang and 78 more. 2024

592 citations recorded by the source corpus.

We present Movie Gen, a cast of foundation models that generates high-quality, 1080p HD videos with different aspect ratios and synchronized audio. We also show additional capabilities such as precise instruction-based video editing and generation of personalized videos based on a user's image. Our models set a new state-of-the-art on multiple tasks: text-to-video synthesis, video personalization, video editing, video-to-audio generation, and text-to-audio generation. Our largest video generation model is a 30B parameter transformer trained with a maximum context length of 73K video tokens, corresponding to a generated video of 16 seconds at 16 frames-per-second. We show multiple technical innovations and simplifications on the architecture, latent spaces, training objectives and recipes, data curation, evaluation protocols, parallelization techniques, and inference optimizations that allow us to reap the benefits of scaling pre-training data, model size, and training compute for training large scale media generation models. We hope this paper helps the research community to accelerate progress and innovation in media generation models. All videos from this paper are available at https://go.fb.me/MovieGenResearchVideos.

Subfields: Economics and Econometrics

View Paper: https://www.semanticscholar.org/paper/ed92a7be6b2acc36baf0ed43454a97dbade0c893

Citations in the corpus

• Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (2025, unknown): “We use all 1003 prompts from MovieGenBench [64] and each prompt is evaluated by a single user.”
• Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (2024, unknown): “They achieve remarkably successful results in challenging tasks such as zero-shot text-to-image (Podell et al., 2023; Saharia et al., 2022; Esser et al., 2024) or text-to-video (Polyak et al., 2024; Brooks et al., 2024) generation.”
• Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (2025, unknown): “Diffusion Transformers (DiT) [30] has gained considerable success as a foundational model for text-to-image [3, 8] and text-to-video generation tasks [28, 32].”
• From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (2024, unknown): “While most video diffusion models have bidirectional dependencies [5, 61, 96, 109], autoregressive video generation using diffusion models has also been explored.”
• π0: A Vision-Language-Action Flow Model for General Robot Control (2024, unknown): “Recent work in high-resolution image [14] and video [38] synthesis has shown that flow matching can achieve strong empirical performance when combined with a simple linear-Gaussian (or optimal transport) probability path [28], given by q In practice, the network is trained by sampling random noise…”
• HunyuanVideo: A Systematic Framework For Large Video Generative Models (2024, unknown): “In contrast to most previous work [67, 11, 104], we do not rely on a pre-trained image VAE for parameter initialization; instead, we train our model from scratch.”
• Mean Flows for One-step Generative Modeling (2025, unknown): “Since its introduction, Flow Matching has seen widespread adoption in modern generative modeling [10, 33, 35].”