MAGVIT: Masked Generative Video Transformer

By Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G. Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa and 1 more. Computer Vision and Pattern Recognition, 2022

394 citations recorded by the source corpus.

We introduce the MAsked Generative VIdeo Transformer, MAGVIT, to tackle various video synthesis tasks with a single model. We introduce a 3D tokenizer to quantize a video into spatial-temporal visual tokens and propose an embedding method for masked video token modeling to facilitate multi-task learning. We conduct extensive experiments to demonstrate the quality, efficiency, and flexibility of MAGVIT. Our experiments show that (i) MAGVIT performs favorably against state-of-the-art approaches and establishes the best-published FVD on three video generation benchmarks, including the challenging Kinetics-600. (ii) MAGVIT outperforms existing methods in inference time by two orders of magnitude against diffusion models and by 60x against autoregressive models. (iii) A single MAGVIT model supports ten diverse generation tasks and generalizes across videos from different visual domains. The source code and trained models will be released to the public at https://magvit.cs.cmu.edu.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/fe34137e5cc07235eae65ce53a54cd226b9f8b23

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Autoregressive Image Generation without Vector Quantization (2024, background): “But discrete tokenizers are difficult to train, which has recently drawn special focus [27, 54, 32].”
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, neutral): “MagViT [73] adapts this approach to video data, and MagViT-2 [74] enhances MaskGIT by introducing an improved VQVAE.”
DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimization (2023, background): “The intuition for the cosine schedule is that diffusion models can achieve better generation quality when iterating more steps in the low-noise regime [85, 121, 13].”
Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation (2023, methodology): “The Masked LM (MLM) is another type of language model for visual generation, such as: MaskGIT (Chang et al., 2022), MAGVIT (Yu et al., 2023a), Phenaki (Villegas et al., 2022), and MUSE (Chang et al., 2023), among others.”
MoMask: Generative Masked Modeling of 3D Human Motions (2023, methodology): “Magvit [47] suggests a versatile masking strategy for multi-task video generation.”
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, neutral): “Additionally, masked prediction models [7, 8] draw upon BERT-style [19] masking meth-ods, predicting masked sections of visual inputs to improve synthesis efficiency, and have been adapted for video generation [86].”
VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, methodology): “During inference, we use non-autoregressive sampling [12, 75] with classifier-free guidance [7] independently on both the LR condition and the text embeddings.”
DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors (2023, neutral): “Moreover, recent proprietary T2V models [47, 66, 73, 90] have been demonstrated to be extensible to image-to-video synthesis.”
GAIA-1: A Generative World Model for Autonomous Driving (2023, background): “A known limitation is the slow generation speed, but this issue could be alleviated by future research on parallel sampling [78, 79, 80], reducing the number of latent variables [81], and improvements in hardware accelerators.”
An Image is Worth 32 Tokens for Reconstruction and Generation (2024, neutral): “In recent years, image generation has experienced remarkable progress, driven by the significant advancements in both transformers [19, 62, 66, 10, 67, 68] and diffusion models [16, 55, 29, 49, 21].”
Pix2Video: Video Editing using Image Diffusion (2023, neutral): “With the success of text-to-image generation models, there has been recent attempts in text-to-video generation models using architectures such as transformers [19, 53, 60] or diffusion models [14, 17, 46, 61].”
Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models (2023, background): “Video generation models: Generating realistic and novel videos have long been an attractive and essential research direction [58, 39, 65].”
VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models (2024, methodology): “Then, since transformers have been successfully applied in various fields, they are also introduced for video synthesis, e.g. , CogVideo [28], VideoGPT [57], NUVA-infinity [53], TATS [19], MAGVIT [58], Phenaki [45].”
Photorealistic Video Generation with Diffusion Models (2023, methodology): “To alleviate this issue, non-autoregressive sampling [10, 11], i.e . parallel token prediction, has been adopted as a more efficient solution for transformer-based video generation [27, 74, 81].”