Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors

By Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, Yaniv Taigman. 2022

639 citations recorded by the source corpus.

Recent text-to-image generation methods provide a simple yet exciting conversion capability between text and image domains. While these methods have incrementally improved the generated image fidelity and text relevancy, several pivotal gaps remain unanswered, limiting applicability and quality. We propose a novel text-to-image method that addresses these gaps by (i) enabling a simple control mechanism complementary to text in the form of a scene, (ii) introducing elements that substantially improve the tokenization process by employing domain-specific knowledge over key image regions (faces and salient objects), and (iii) adapting classifier-free guidance for the transformer use case. Our model achieves state-of-the-art FID and human evaluation results, unlocking the ability to generate high fidelity images in a resolution of 512x512 pixels, significantly improving visual quality. Through scene controllability, we introduce several new capabilities: (i) Scene editing, (ii) text editing with anchor scenes, (iii) overcoming out-of-distribution text prompts, and (iv) story illustration generation, as demonstrated in the story we wrote.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Generative Models

In Multimodal Learning

Citations in the corpus

Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, background): “[17] conditions a generative image model on segmentation masks, allowing for a generative process that first samples a semantic map of an image and then conditions the generated image on this information.”
Visual Instruction Tuning (2023, background): “To this end, the community has witnessed an emergent interest in developing language-augmented foundation vision models [24, 14], with strong capabilities in open-world visual understanding such as classification [36, 18, 53, 50, 35], detection [26, 58, 29], segmentation [23, 59, 54] and captioning ”
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, methodology): “Make-A-Scene [10] further expands on this two-stage modeling approach to support both text and scene-guided image generation.”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, methodology): “Autoregressive models [39], GANs [79, 84], VQ-VAE Transformer-based methods [55, 22], and diffusion models have seen remarkable progress in text-to-image [59, 43, 59], including the concurrent”
AudioGen: Textually Guided Audio Generation (2022, methodology): “Then, a Transformer-decoder model was trained to model these codes in an autoregressive fashion, while Gafni et al. (2022) suggested a similar approach and incorporated segmentation maps for increased controllability.”
Underspecification in Scene Description-to-Depiction Tasks (2022, background): “Recent text-to-image generation systems have demonstrated impressive capabilities (Zhang et al., 2021; Ramesh et al., 2021; Ding et al., 2021; Nichol et al., 2021; Gafni et al., 2022; Ramesh et al., 2022; Saharia et al., 2022; Ramesh et al., 2022; Yu et al., 2022).”
Emu Edit: Precise Image Editing via Recognition and Generation Tasks (2023, neutral): “The emergence of high-performing text-to-image diffusion models [7, 16, 17, 19] facilitated the development of effective text-based image editing methods.”
Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models (2023, methodology): “To provide users with more control over the synthesis process, several works employ a segmentation map or spatial conditioning [Avrahami et al. 2022b; Gafni et al. 2022; Zhao et al. 2019].”
Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks (2024, background): “…88, 79, 25, 79, 16, 28], referring object detection and segmentation [41, 37, 86], object tracking [67, 84], image generation [75, 54, 48, 45, 23, 14, 52, 22, 82, 44], image editing [42, 1, 2, 53, 21], human-centric perception and understanding [72, 71, 73, 70, 69, 4, 33, 74], and human-centric…”
Generating Images with Multimodal Language Models (2023, neutral): “Model FID ( ↓ ) GLIDE [38] 12.24 Make-A-Scene [21] 11.84 DALL-E 2 [44] 10.39 LAFITE2 [72] 8.42 Imagen [50] 7.27 Parti [65] 7.23 Re-Imagen [14] 6.88 SD [49] v1.”
ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation (2023, background): “Text-to-image generative models, including auto-regressive [39, 10, 13, 15, 11, 56] and diffusionbased [34, 41, 38, 42] approaches, have experienced rapid advancements in recent years.”
Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models (2023, background): “Recently, text-to-image models [10, 23, 53, 52, 57, 66, 54, 65] have shown impressive performance in synthesizing high-quality images according to text prompts.”
BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion (2023, background): “Layout-to-image literature [19, 32, 33, 36, 40, 12, 2, 1] has studied on the way to synthesize images adhering to the spatial conditioning input.”
All are Worth Words: A ViT Backbone for Diffusion Models (2022, background): “86 Discrete diffusion Conceptual Caption Subset (7M) 370M Make-A-Scene [18] 7.”
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, positive): “For image generation, we utilize classifier-free guidance (CFG) 2 , similar to prior works [8, 25, 70].”