Muse: Text-To-Image Generation via Masked Generative Transformers

By Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, José Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin P. Murphy, William T. Freeman, Michael Rubinstein and 2 more. 2023

783 citations recorded by the source corpus.

We present Muse, a text-to-image Transformer model that achieves state-of-the-art image generation performance while being significantly more efficient than diffusion or autoregressive models. Muse is trained on a masked modeling task in discrete token space: given the text embedding extracted from a pre-trained large language model (LLM), Muse is trained to predict randomly masked image tokens. Compared to pixel-space diffusion models, such as Imagen and DALL-E 2, Muse is significantly more efficient due to the use of discrete tokens and requiring fewer sampling iterations; compared to autoregressive models, such as Parti, Muse is more efficient due to the use of parallel decoding. The use of a pre-trained LLM enables fine-grained language understanding, translating to high-fidelity image generation and the understanding of visual concepts such as objects, their spatial relationships, pose, cardinality etc. Our 900M parameter model achieves a new SOTA on CC3M, with an FID score of 6.06. The Muse 3B parameter model achieves an FID of 7.88 on zero-shot COCO evaluation, along with a CLIP score of 0.32. Muse also directly enables a number of image editing applications without the need to fine-tune or invert the model: inpainting, outpainting, and mask-free editing. More results are available at https://muse-model.github.io

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

FastComposer: Tuning-Free Multi-subject Image Generation with Localized Attention (2023, neutral): “Recent advancements in text-to-image generation [4, 8, 17, 28], particularly diffusion models [13, 27, 31, 36, 37], have opened new frontiers in content creation.”
MACE: Mass Concept Erasure in Diffusion Models (2024, background): “In large-scale text-to-image (T2I) models [9, 14, 34, 40, 49, 54, 57, 73, 74, 76, 77], the task of concept erasure aims to remove concepts that may be harmful, copyrighted, or offensive.”
Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation (2024, background): “Recently, vision-and-language models [Liu et al. 2024; Zhu et al. 2023; Dai et al. 2024; Peng et al. 2023; Zhang et al. 2023; Ma et al. 2024] have achieved versatile visual understanding through visual instruction tuning [Liu et al. 2024; Zhu et al. 2023].”
One-Step Diffusion with Distribution Matching Distillation (2023, background): “Distribution Matching Recently, a few classes of generative models have shown success in scaling up to complex datasets by recovering samples that are corrupted by a pre-deﬁned mechanism, such as noise injection [21, 61, 64] or token masking [5, 60, 86].”
Adversarial Diffusion Distillation (2023, background): “5-XL 4 ), and OpenMUSE 5 [48], a reimplementation of MUSE [6], a transformer model speciﬁcally developed for fast inference.”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “Furthermore, developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1 [162], Muse [163], and PixArt-α [164], presents a viable solution.”
A Cookbook of Self-Supervised Learning (2023, background): “, 2022] • Muse reaches state-of-the-art text conditional image generation with a masked transformer approach [Chang et al., 2023].”
DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data (2023, background): “In recent years, text-driven generative models (e.g., Stable Diffusion [71], Imagen [72], DALLE-2 [68], MUSE [13]) have emerged as powerful tools for image synthesis.”
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, positive): “MUSE [10] scales MaskGIT’s architecture to 3 billion parameters and merges it with the T5 language model [50], setting new benchmarks in text-to-image synthesis.”
GAIA-1: A Generative World Model for Autonomous Driving (2023, methodology): “Because these text sources are imperfect, to improve the alignment between generated futures and the text prompt, we employ classifier-free guidance [45, 46] at inference time.”
Ablating Concepts in Text-to-Image Diffusion Models (2023, background): “Large-scale text-to-image models have demonstrated remarkable ability in synthesizing photorealistic images [51, 43, 56, 54, 76, 14].”
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, positive): “For image generation, we utilize classifier-free guidance (CFG) 2 , similar to prior works [8, 25, 70].”
Discrete Flow Matching (2024, methodology): “MaskGIT (Chang et al., 2022) followed a similar path, for the task of class-conditioned image synthesis, Chang et al. (2023) extended this approach to high-quality textually guided image generation over low-resolution images followed by a super-resolution module.”
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (2024, methodology): “Following the work (Chang et al., 2023), in inference, we compute both the conditional logit ℓ c and unconditional logit ℓ u for masked tokens.”
InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (2023, methodology): “We encode X into a global concept embedding c g via Concept Encoder E g and obtain”