Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer

By Songwei Ge, Thomas Hayes, Harry Yang, Xiaoyue Yin, Guan Pang, David W. Jacobs, Jia-Bin Huang, Devi Parikh. 2022

309 citations recorded by the source corpus.

Videos are created to express emotion, exchange information, and share experiences. Video synthesis has intrigued researchers for a long time. Despite the rapid progress driven by advances in visual synthesis, most existing studies focus on improving the frames' quality and the transitions between them, while little progress has been made in generating longer videos. In this paper, we present a method that builds on 3D-VQGAN and transformers to generate videos with thousands of frames. Our evaluation shows that our model trained on 16-frame video clips from standard benchmarks such as UCF-101, Sky Time-lapse, and Taichi-HD datasets can generate diverse, coherent, and high-quality long videos. We also showcase conditional extensions of our approach for generating meaningful long videos by incorporating temporal information with text and audio. Videos and code can be found at https://songweige.github.io/projects/tats/index.html.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

AudioLM: A Language Modeling Approach to Audio Generation (2022, methodology): “This approach has been used to generate high-resolution images [11], [12], [42] and long videos [43].”
CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers (2022, methodology): “The framework of VQVAE [28] and autoregressive transformers [29, 1] quickly becomes the mainstream method [34, 35, 9].”
Make-A-Video: Text-to-Video Generation without Text-Video Data (2022, result): “This is a signiﬁcant advantage compared to prior work, that has to be restricted to narrow domains (Mittal et al., 2017; Gupta et al., 2018; Ge et al., 2022; Hayes et al., 2022), or require large-scale paired text-video data (Hong et al., 2022; Ho et al., 2022).”
GAIA-1: A Generative World Model for Autonomous Driving (2023, background): “Likewise, they can operate at the pixel level [72, 73], or in a discrete learned token space [74, 75, 76, 77].”
From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (2024, unknown): “Inspired by the success of LLMs [6], some works choose to tokenize video frames into discrete tokens and apply autoregressive transformers to generate tokens one by one [17, 36, 42, 86, 90, 95].”
Latte: Latent Diffusion Transformer for Video Generation (2024, unknown): “Secondly, some methods propose learning the data distribution using autoregressive models (Ge et al, 2022; Rakhimov et al, 2021; Weissenborn et al, 2020; Yan et al, 2021).”
LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, unknown): “The Bund, Shanghai, with the ship moving on the river, oil painting.”
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (2023, methodology): “For FVD, we use the pretrained InceptionI3d model and followed the TATS method [80].”
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation (2022, unknown): “To reduce the computational complexity, we propose a novel variant of causal attention [8, 19], termed as Sparse-Causal Attention (SC-Attn).”
VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models (2024, methodology): “Then, since transformers have been successfully applied in various fields, they are also introduced for video synthesis, e.g. , CogVideo [28], VideoGPT [57], NUVA-infinity [53], TATS [19], MAGVIT [58], Phenaki [45].”
Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation (2023, background): “Later research (Yan et al., 2021; Ge et al., 2022; Yu et al., 2023a) integrates 3D CNNs to tokenize spatial-temporal volumes.”
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, unknown): “…2017; Tulyakov et al., 2018; Tian et al., 2021; Shen et al., 2023), autoregressive models (Srivastava et al., 2015; Yan et al., 2021; Le Moing et al., 2021; Ge et al., 2022; Hong et al., 2022; Kondratyuk et al., 2023), and implicit neural representations (Skorokhodov et al., 2021; Yu et al., 2021).”
Driving Into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (2023, methodology): “Various generation methods have been proposed in the past, including VAE-based (Variational Autoen-coder) [16, 33, 58, 61], GAN-based (generative adversarial networks) [3, 15, 31, 49, 56, 70], flow-based [10, 34] and auto-regressive models [18, 64, 68].”
MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation (2022, methodology): “(2) Our model with only video generation (Ours-v) even outperforms SOTA methods DIGAN and TATS-base on most metrics in both tables (comparing #5 to #2 and #3).”