VideoPoet: A Large Language Model for Zero-Shot Video Generation

By D. Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Y. Alon, Vighnesh Birodkar and 21 more. International Conference on Machine Learning, 2023

521 citations recorded by the source corpus.

We present VideoPoet, a language model capable of synthesizing high-quality video, with matching audio, from a large variety of conditioning signals. VideoPoet employs a decoder-only transformer architecture that processes multimodal inputs -- including images, videos, text, and audio. The training protocol follows that of Large Language Models (LLMs), consisting of two stages: pretraining and task-specific adaptation. During pretraining, VideoPoet incorporates a mixture of multimodal generative objectives within an autoregressive Transformer framework. The pretrained LLM serves as a foundation that can be adapted for a range of video generation tasks. We present empirical results demonstrating the model's state-of-the-art capabilities in zero-shot video generation, specifically highlighting VideoPoet's ability to generate high-fidelity motions. Project page: http://sites.research.google/videopoet/

View Paper: https://www.semanticscholar.org/paper/0c4f46e4dcae5527018e6432fb60cfe8c3354e97

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Xinlong Wang
2. Xiaosong Zhang
3. Jinheng Xie
4. Sicong Leng
5. Zhengxiong Luo
6. Weijia Mao
7. Zechen Bai
8. Quan Sun

Citations in the corpus

• Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (2024, methodology): “For instance, VideoPoet (Kondratyuk et al., 2023) also employs the decoder-only transformer architecture for synthesizing high-quality videos from multimodal inputs.”
• Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, unknown): “The trade-off between perceptual quality and motion intensity has been widely acknowledged in video generation [3, 32, 73, 144], and our case is no exception.”
• CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (2025, unknown): “Looking forward, we believe recent advances in video/image generation and world models [13, 20, 27, 62, 66] present promising opportunities to enhance generalization capabilities through improved visual reasoning and predictive modeling. dreamer: Learning to follow instructions via…”
• VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, background): “Despite these successes, the domain of video understanding and generation (Kondratyuk et al., 2023; Menapace et al., 2024; Kondratyuk et al., 2023; Brooks et al., 2024) remains relatively nascent.”
• Pyramidal Flow Matching for Efficient Video Generative Modeling (2024, unknown): “…diffusion models (Ho et al., 2022c; Blattmann et al., 2023a; OpenAI, 2024) and autoregressive models (Yan et al., 2021; Hong et al., 2023; Kondratyuk et al., 2024) have made remarkable breakthroughs in generating realistic and long-duration video through scaling of data and computation.”
• Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (2025, unknown): “In contrast, autoregressive (AR) models [17, 27, 38, 94, 104] generate videos sequentially, a paradigm that naturally aligns with the causal structure of temporal media.”
• Lumiere: A Space-Time Diffusion Model for Video Generation (2024, unknown): “To reconcile the per-segment SSR predictions, { Φ( J i ) } Ni =1 , we define the result of the denoising step to be the solution of the optimization problem arg min The solution to this problem is given by linearly combining the predictions over overlapping windows.”
• From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (2024, unknown): “Inspired by the success of LLMs [6], some works choose to tokenize video frames into discrete tokens and apply autoregressive transformers to generate tokens one by one [17, 36, 42, 86, 90, 95].”
• Latte: Latent Diffusion Transformer for Video Generation (2024, unknown): “While these approaches generally offer good video quality and exhibit more stable convergence, they come with the drawback of requiring significant computational resources.”
• SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (2024, unknown): “The VideoLLaVA (Lin et al., 2023) and VideoPoet (Kondratyuk et al., 2023) models allocate thousands of tokens to afford the representation of multiple frames.”
• CameraCtrl: Enabling Camera Control for Text-to-Video Generation (2024, unknown): “…attempts include leveraging scalable transformer backbone (Ma et al., 2024), operating in space-temporal compressed latent space, e.g. , W.A.L.T. (Gupta et al., 2023) and Sora (Brooks et al., 2024), and using discrete token with language model for video generation (Kondratyuk et al., 2023).”
• Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, unknown): “…2017; Tulyakov et al., 2018; Tian et al., 2021; Shen et al., 2023), autoregressive models (Srivastava et al., 2015; Yan et al., 2021; Le Moing et al., 2021; Ge et al., 2022; Hong et al., 2022; Kondratyuk et al., 2023), and implicit neural representations (Skorokhodov et al., 2021; Yu et al., 2021).”
• Infinity∞: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (2024, unknown): “Parti [79], Emu3 [70], chameleon[62] , Liquid[73], and VideoPoet[32] scaled up autoregressive models in text-to-image or video synthesis.”
• Emu3: Next-Token Prediction is All You Need (2024, methodology): “VideoGPT [93] and VideoPoet [38] Unified Understanding and Generation.”