Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis

By Seunghoon Hong, Dingdong Yang, Jongwook Choi, Honglak Lee. 2018

359 citations recorded by the source corpus.

We propose a novel hierarchical approach for text-to-image synthesis by inferring semantic layout. Instead of learning a direct mapping from text to image, our algorithm decomposes the generation process into multiple steps, in which it first constructs a semantic layout from the text by the layout generator and converts the layout to an image by the image generator. The proposed layout generator progressively constructs a semantic layout in a coarse-to-fine manner by generating object bounding boxes and refining each box by estimating object shapes inside the box. The image generator synthesizes an image conditioned on the inferred semantic layout, which provides a useful semantic structure of an image matching with the text description. Our model not only generates semantically more meaningful images, but also allows automatic annotation of generated images and user-controlled generation process by modifying the generated scene layout. We demonstrate the capability of the proposed model on challenging MS-COCO dataset and show that the model can substantially improve the image quality, interpretability of output and semantic alignment to input text over existing approaches.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

Citations in the corpus

Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, background): “An earlier work by [23] suggests coarse control in the form of bounding boxes resulting in low resolution images.”
Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020, methodology): “In [18], [23], [39], a two-step approach is used in image synthesis: generating the semantic layout (class label, bounding boxes and semantic mask) from a text description or a scene graph, and synthesizing images conditioned on the predicted semantic layout and text description (if present).”
Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, background): “Researchers have explored various models for generating images based on text [18,44,52,55].”
Vector Quantized Diffusion Model for Text-to-Image Synthesis (2021, methodology): “Other works [22,29] adopt a two-step process which first infer the semantic layout then generate different objects, but this kind of method requires fine-grained object labels, e.”
Improving Text-to-Image Synthesis Using Contrastive Learning (2021, methodology): “Recently, a great number of studies [1, 4, 6, 12, 13, 14, 17, 18, 19, 22, 23, 24, 25, 32, 33, 36, 42, 43, 44, 45, 46, 47, 48] present promising results on the text-to-image synthesis task, most of which make use of GANs as the backbone model.”
Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, methodology): “Others have proposed hierarchical models that explicitly generate different objects after inferring semantic layouts [18, 16, 22].”
Self-Attention Generative Adversarial Networks (2018, background): “, 2017) and text-to-image synthesis (Reed et al., 2016b;a; Zhang et al., 2017; Hong et al., 2018).”
Object-Driven Text-To-Image Synthesis via Adversarial Training (2019, methodology): “However, [9] encodes the text into a single global sentence vector, which loses word-level fine-grained information.”
MirrorGAN: Learning Text-To-Image Generation by Redescription (2019, background): “Although significant progress has been made in generating visually realistic images using generative adversarial networks (GANs) such as in [39, 42, 35, 13], guaranteeing semantic alignment of the generated image with the input text remains challenging.”
Image Synthesis From Reconfigurable Layout and Style (2019, methodology): “In [16, 12, 11, 21], layout and object information is utilized in text-to-image generation.”
Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instruction (2018, methodology): “ject annotations with the intermediate steps of generating bounding boxes and object masks before generating the final images can improve results [5].”
Specifying Object Attributes and Relations in Interactive Scene Generation (2019, background): “In another line of work, images are generated to match input sentences, without constructing the scene graph as an intermediate representation [18, 26, 6].”
X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers (2020, background): “There is another line of works predicting object layouts from text and generating image based on the layouts (Hong et al., 2018; Tan et al., 2019).”
ChatPainter: Improving Text to Image Generation using Dialogue (2018, neutral): “02 Hong et al. (2018) 11 .”
Semantic Bottleneck Scene Generation (2019, methodology): “The text-to-image synthesis model proposed in [17] de-composes the synthesis task into multiple steps.”
Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis (2022, background): “As text inputs are relatively weak conditions, previous work leverage layout or scene graph to enhance compositionality (Johnson et al., 2018; Hong et al., 2018; Yang et al., 2022; Gafni et al., 2022).”