VideoGPT: Video Generation using VQ-VAE and Transformers

By Wilson Yan, Yunzhi Zhang, Pieter Abbeel, A. Srinivas. 2021

771 citations recorded by the source corpus.

We present VideoGPT: a conceptually simple architecture for scaling likelihood based generative modeling to natural videos. VideoGPT uses VQ-VAE that learns downsampled discrete latent representations of a raw video by employing 3D convolutions and axial self-attention. A simple GPT-like architecture is then used to autoregressively model the discrete latents using spatio-temporal position encodings. Despite the simplicity in formulation and ease of training, our architecture is able to generate samples competitive with state-of-the-art GAN models for video generation on the BAIR Robot dataset, and generate high fidelity natural videos from UCF-101 and Tumbler GIF Dataset (TGIF). We hope our proposed architecture serves as a reproducible reference for a minimalistic implementation of transformer based video generation models. Samples and code are available at https://wilson1yan.github.io/videogpt/index.html

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “To scale to higher resolutions, several two-stage approaches [23, 65, 97, 99] use ARMs to model a compressed latent image space instead of raw pixels.”
Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, methodology): “VQ-VAE’s latent variables are used as a means for video generation tasks [59].”
NaturalSpeech: End-to-End Text-to-Speech Synthesis With Human-Level Quality (2022, methodology): “Inspired by image/video generation [21], [36], [37], [38], [39] that uses VQ-VAE [20], [40], [41] to compress high-dimensional image into low-dimensional representations to ease the generation [42], we leverage VAE [22] to compress high-dimensional speech x into frame-level representations z (i.e.,…”
Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory (2022, background): “The two-stage approaches, which first encode data and afterwards learn a probabilistic model to generate the encoding, have been applied in multiple generative areas [7, 42, 8].”
Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (2023, methodology): “Our method is different in that it adapts video prediction tasks (Yan et al., 2021; 2023) for large-scale pre-training and focuses on language-conditioned multi-tasking.”
A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks (2023, background): “VideoGPT (Yan et al., 2021) Video Generation 2021 Decoder No NA BAIR RobotNet, Moving MNIST, ViZDoom, UCF-101, Tumblr GIF”
Emu3: Next-Token Prediction is All You Need (2024, methodology): “VideoGPT [93] and VideoPoet [38] Unified Understanding and Generation.”
Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer (2022, methodology): “Our paper is a small step but we hope it can encourage future works on more interesting forms of video synthesis with a realistic number of frames, perhaps even movies. compared with VQGAN [15] and VideoGPT [80].”
CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers (2022, methodology): “However, most previous works use GPT [34, 36, 35], which is unidirectional.”
Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (2025, positive): “In contrast, autoregressive (AR) models [17, 27, 38, 94, 104] generate videos sequentially, a paradigm that naturally aligns with the causal structure of temporal media.”
Transformers are Sample Efficient World Models (2022, background): “VideoGPT (Yan et al., 2021) introduces video-level discrete autoencoders, and Transformers with spatial and temporal attention patterns, for unconditional and action conditional video generation.”
Latent Video Diffusion Models for High-Fidelity Long Video Generation (2022, methodology): “Existing works capitalize on different types of generative models including GANs [40, 26, 27, 36, 35, 30, 2], VAEs [7, 44], autoregressive models [6, 42, 21], and normalizing ﬂows [17].”
StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2 (2021, positive): “Training was done on × 4 32 GB NVidia V100 GPUs for all the methods except VideoGPT, which was trained on × 4 NVidia A6000 GPUs (with 48.5 GB of memory each) due to its high memory consumption.”
CelebV-HQ: A Large-Scale Video Facial Attributes Dataset (2022, methodology): “We construct a benchmark of unconditional video generation task, for four currently prevalent models (VideoGPT [90], MoCoGAN-HD [75], DIGAN [94], and StyleGAN-V [73]) on 4 face video datasets (FaceForensics [65], Vox [59], MEAD [82] and CelebV-HQ).”
Pyramidal Flow Matching for Efficient Video Generative Modeling (2024, neutral): “Nowadays, the cutting-edge diffusion models (Ho et al., 2022c; Blattmann et al., 2023a; OpenAI, 2024) and autoregressive models (Yan et al., 2021; Hong et al., 2023; Kondratyuk et al., 2024) have made remarkable breakthroughs in generating realistic and long-duration video through scaling of data…”