Improved Masked Image Generation with Token-Critic

By José Lezama, Huiwen Chang, Lu Jiang, Irfan Essa. 2022

74 citations recorded by the source corpus.

Non-autoregressive generative transformers recently demonstrated impressive image generation performance, and orders of magnitude faster sampling than their autoregressive counterparts. However, optimal parallel sampling from the true joint distribution of visual tokens remains an open challenge. In this paper we introduce Token-Critic, an auxiliary model to guide the sampling of a non-autoregressive generative transformer. Given a masked-and-reconstructed real image, the Token-Critic model is trained to distinguish which visual tokens belong to the original image and which were sampled by the generative transformer. During non-autoregressive iterative sampling, Token-Critic is used to select which tokens to accept and which to reject and resample. Coupled with Token-Critic, a state-of-the-art generative transformer significantly improves its performance, and outperforms recent diffusion models and GANs in terms of the trade-off between generated image quality and diversity, in the challenging class-conditional ImageNet generation.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, neutral): “Inspire by [50, 51, 52], we first sample ˆX 0 from p θ ( X 0 | X t , X p , C ) , and then sample X t − ∆ t from q ( X t − ∆ t | ˆX 0 , X t ) , which involves remask ⌊ N · σ ( t − ∆ t ) ⌋ tokens in ˆX 0 with the lowest confidence score, where we define the confidence score of ˆ x i in ˆX 0 to p θ (ˆ…”
An Image is Worth 32 Tokens for Reconstruction and Generation (2024, unknown): “Apart from that, further studies looked into improved sampling strategies [38, 39, 37].”
Muse: Text-To-Image Generation via Masked Generative Transformers (2023, background): “…for multiple generative models due to the ability to move the bulk of the computation from input (pixel) space to latents (Rom-bach et al., 2022), or to enable more effective loss functions such as classiﬁcation instead of regression (Chang et al., 2022; Lezama et al., 2022; Li et al., 2022).”