MaskGIT: Masked Generative Image Transformer

By Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, William T. Freeman. 2022

1,238 citations recorded by the source corpus.

Generative transformers have experienced rapid popularity growth in the computer vision community in synthesizing high-fidelity and high-resolution images. The best generative transformer models so far, however, still treat an image naively as a sequence of tokens, and decode an image sequentially following the raster scan ordering (i.e. line-by-line). We find this strategy neither optimal nor efficient. This paper proposes a novel image synthesis paradigm using a bidirectional transformer decoder, which we term MaskGIT. During training, MaskGIT learns to predict randomly masked tokens by attending to tokens in all directions. At inference time, the model begins with generating all tokens of an image simultaneously, and then refines the image iteratively conditioned on the previous generation. Our experiments demonstrate that MaskGIT significantly outperforms the state-of-the-art transformer model on the ImageNet dataset, and accelerates autoregressive decoding by up to 48x. Besides, we illustrate that MaskGIT can be easily extended to various image editing tasks, such as inpainting, extrapolation, and image manipulation. Project page: masked-generative-image-transformer.github.io.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Model Architectures

Citations in the corpus

Scalable Diffusion Models with Transformers (2022, background): “They have also been trained on discrete codebooks [56] as both autoregressive models [11,44] and masked generative models [4, 14]; the former has shown excellent scaling behavior up to 20B parameters [59].”
VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, methodology): “For example, Soundstorm [Borsos et al., 2023] leveraged the confidence-based parallel decoding scheme [Chang et al., 2022] to generate the acoustic codec codes with a non-autoregressive model.”
SoundStorm: Efficient Parallel Audio Generation (2023, positive): “Our proposed decoding scheme can be seen as the extension of MaskGIT’s decoding to token sequences produced by residual quantization.”
Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation (2023, methodology): “The Masked LM (MLM) is another type of language model for visual generation, such as: MaskGIT (Chang et al., 2022), MAGVIT (Yu et al., 2023a), Phenaki (Villegas et al., 2022), and MUSE (Chang et al., 2023), among others.”
Improved Masked Image Generation with Token-Critic (2022, methodology): “The ﬁeld has seen impressive progress in the hand of mainly three techniques: large Generative Adversarial Networks (GANs) [3], diﬀusion models [9,19], and transformer-based models over a vector-quantized (VQ) latent space [12,5].”
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, positive): “Starting from the vanilla AR transformer baseline implemented by [11], we replace its methodology with our VAR and keep other settings unchanged to get row 2.”
An Image is Worth 32 Tokens for Reconstruction and Generation (2024, positive): “We further confirm the model’s efficacy in image generation through the MaskGIT [9] framework.”
Self-conditioned Embedding Diffusion for Text Generation (2022, positive): “MaskGIT (Chang et al., 2022) demonstrated excellent results in modeling VQ-discretized images.”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “Recent advances in deep learning have facilitated significant progress for various data modalities [25, 162, 207], such as images [34, 237], audio [112, 236], and text [12, 18, 80, 160, 163].”
Simplified and Generalized Masked Diffusion for Discrete Data (2024, neutral): “The cosine schedule was originally proposed in MaskGIT [39], an iterative unmasking generative model inspired by diffusion.”
Discrete Flow Matching (2024, methodology): “For the MaskGIT we train and sample with the Cosine scheduler κ ( t ) = 1 − cos π 2 t (cid:1) which is reported to achieve best results by Chang et al. (2022).”
Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (2024, neutral): “MAGE (Li et al., 2023c) bridge MAE training and masked image modeling (Chang et al., 2022) by adjusting the masking ratio in training, which leads to a single model both capable of discrimination and generation tasks.”
BLOCK DIFFUSION: INTERPOLATING BETWEEN AU-TOREGRESSIVE AND DIFFUSION LANGUAGE MODELS (2025, neutral): “We also compare to SSD-LM (Han et al., 2022), an alternative block diffusion formulation.”
Large Language Diffusion Models (2025, positive): “However, inspired by the annealing tricks of sampling in LLMs [4, 30], we adopt a low-confidence remasking strategy, where of predicted tokens with the lowest confidence are remarked based on the predictions, same as the approach of Chang et al. [23].”
Generating Images with Multimodal Language Models (2023, background): “Several recent papers also formulate the text-to-image generation task as a sequence modeling problem [45, 17, 13], training large Transformer [57] models on discretized image tokens [46].”