Scaling Autoregressive Video Models

By Dirk Weissenborn, Oscar Täckström, Jakob Uszkoreit. 2019

256 citations recorded by the source corpus.

Due to the statistical complexity of video, the high degree of inherent stochasticity, and the sheer amount of data, generating natural video remains a challenging task. State-of-the-art video generation models often attempt to address these issues by combining sometimes complex, usually video-specific neural network architectures, latent variable models, adversarial training and a range of other methods. Despite their often high complexity, these approaches still fall short of generating high quality video continuations outside of narrow domains and often struggle with fidelity. In contrast, we show that conceptually simple autoregressive video generation models based on a three-dimensional self-attention mechanism achieve competitive results across multiple metrics on popular benchmark datasets, for which they produce continuations of high fidelity and realism. We also present results from training our models on Kinetics, a large scale action recognition dataset comprised of YouTube videos exhibiting phenomena such as camera movement, complex object interactions and diverse human movement. While modeling these phenomena consistently remains elusive, we hope that our results, which include occasional realistic continuations encourage further research on comparatively complex, large scale datasets such as Kinetics.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, background): “An alternative way to scale attention is to apply it in blocks of varying sizes (Weissenborn et al., 2019), in the extreme case only along individual axes (Ho et al.”
Axial Attention in Multidimensional Transformers (2019, methodology): “To our knowledge, sampling speedups of this type are not possible with contemporary work on scaling Transformers to images and videos (Child et al., 2019; Weissenborn et al., 2019).”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “In addition, action-conditioned samples are visually consistent with the input 3 SV2P (Babaeizadeh et al., 2017), SAVP (Lee et al., 2018), DVD-GAN-FP (Clark et al., 2019), Video Transformer (Weis-senborn et al., 2019), Latent Video Transformer (LVT) (Rakhimov et al., 2020), and TrIVD-GAN (Luc et al.”
Taming Transformers for High-Resolution Image Synthesis (2020, background): “[48] and [66] restrict the receptive fields of the attention modules, which reduces the expressivity and, especially for high-resolution images, introduces unjustified assumptions on the independence of pixels.”
Scaling Laws for Autoregressive Generative Modeling (2020, methodology): “…across a wide variety of data modalities, including generative language [KMH + 20, BMR + 20], image [TSF + 15, CRC + 20], and video modeling [WTU19], multimodal modeling [TBL + 19] of text-image correlations, and even mathematical problem solving [SGHK19], a task requiring a degree of…”
RoboNet: Large-Scale Multi-Robot Learning (2019, background): “We expect larger, state-of-the-art models [50, 49] to produce significantly better predictions, which would hopefully translate to better control performance.”
Is Space-Time Attention All You Need for Video Understanding? (2021, methodology): “While Transformers have been recently used for video generation (Weissenborn et al., 2020), we are not aware of prior video recognition architectures using self-attention as the exclusive building block.”
ViViT: A Video Vision Transformer (2021, neutral): “We note that factorising attention over input dimensions has also been explored in [29, 78], and concurrently in the context of video by [4] in their “Di-vided Space-Time” model.”
Video Diffusion Models (2022, neutral): “Method FVD ↓ DVD-GAN [14] 109.8 VideoGPT [62] 103.3 TrIVD-GAN-FP [33] 103.3 Transframer [35] 100 CCVS [31] 99 VideoTransformer [59] 94 FitVid [4] 93.6 NUWA [61] 86.9 Video Diffusion (ours) ancestral sampler, 512 steps 68.19 Langevin sampler, 256 steps 66.92 Table 3: Video prediction on Kinetics-600.”
GAIA-1: A Generative World Model for Autonomous Driving (2023, background): “Likewise, they can operate at the pixel level [72, 73], or in a discrete learned token space [74, 75, 76, 77].”
NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion (2021, methodology): “Earlier visual auto-regressive models [5,28,39,41,44] performed visual synthesis in a “pixel-by-pixel” manner.”
TokenLearner: What Can 8 Learned Tokens Do for Images and Videos? (2021, methodology): “Attention-based architectures have also been used in the context of video generation [63].”
Latte: Latent Diffusion Transformer for Video Generation (2024, unknown): “Secondly, some methods propose learning the data distribution using autoregressive models (Ge et al, 2022; Rakhimov et al, 2021; Weissenborn et al, 2020; Yan et al, 2021).”
Which transformer architecture fits my data? A vocabulary bottleneck in self-attention (2021, background): “, 2020), video (Weissenborn et al., 2020), mathematical problem solving (Saxton et al.”
Flexible Diffusion Modeling of Long Videos (2022, background): “30 frames [35], 48 frames [6] or 64 frames [15]), generating longer videos that are both coherent and photo-realistic remains an open challenge.”
Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (2025, unknown): “Autoregressive models, in contrast, are trained with next-token prediction objectives and generate spatiotemporal tokens sequentially at inference time [7, 38, 66, 86, 88, 94].”
iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis (2021, background): “However, also more complex models based on transformers [75], pixel-level autoregression [51, 40, 45, 16, 70, 10], factorization of dynamics and content [53, 24] and image warping using optical flow [71, 44, 25] have been proposed.”