Image Generation from Scene Graphs

By Justin Johnson, Agrim Gupta, Li Fei-Fei. 2018

933 citations recorded by the source corpus.

To truly understand the visual world our models should be able not only to recognize images but also generate them. To this end, there has been exciting recent progress on generating images from natural language descriptions. These methods give stunning results on limited domains such as descriptions of birds or flowers, but struggle to faithfully reproduce complex sentences with many objects and relationships. To overcome this limitation we propose a method for generating images from scene graphs, enabling explicitly reasoning about objects and their relationships. Our model uses graph convolution to process input graphs, computes a scene layout by predicting bounding boxes and segmentation masks for objects, and converts the layout to an image with a cascaded refinement network. The network is trained adversarially against a pair of discriminators to ensure realistic outputs. We validate our approach on Visual Genome and COCO-Stuff, where qualitative results, ablations, and user studies demonstrate our method's ability to generate complex images with multiple objects.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, background): “As a beneficial prior knowledge describing the detailed semantics of the image and caption for the visual scene, scene graphs have led to many state-of-the-art models in image captioning [22], image retrieval [23], VQA [24] and image generation [25].”
High-Resolution Complex Scene Synthesis with Transformers (2021, methodology): “We hypothesize that this is partly due to the fact that most concurrent approaches rely on many steps and auxiliary losses such as scene-matching or object-specific losses [13, 2, 28, 11], which turns out limiting the models’ overall performance.”
Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020, methodology): “In [18], [23], [39], a two-step approach is used in image synthesis: generating the semantic layout (class label, bounding boxes and semantic mask) from a text description or a scene graph, and synthesizing images conditioned on the predicted semantic layout and text description (if present).”
Object-Centric Image Generation from Layouts (2020, methodology): “These are constructed from the objects and spatial relations inferred from the bounding box positions following the setup in [23].”
Improving Text-to-Image Synthesis Using Contrastive Learning (2021, methodology): “Recently, a great number of studies [25, 45, 46, 42, 13, 47, 23, 48, 43, 18, 22, 17, 1, 12, 6, 19, 33, 4, 24, 32, 14, 36, 44] present promising results on the text-to-image synthesis task, most of which make use of GANs as the back-bone model.”
Object-Driven Text-To-Image Synthesis via Adversarial Training (2019, methodology): “[12] also proposes such a two-step process to generate images from scene graphs, and their process can be trained end-to-end.”
Auto-Encoding Scene Graphs for Image Captioning (2018, methodology): “In particular, we use scene graphs [13, 47] to bridge the gap between the two worlds.”
Paint by Word (2021, background): “As an alternative approach, generating images from scene graphs instead of sentences [21] can allow a generator to exploit logical structure explicitly.”
Semantics Disentangling for Text-To-Image Generation (2019, methodology): “Following prior arts on text-to-image generation [37, 40, 41, 14, 42, 18], we apply the numerical assessment approach “inception score” [31] for quantitative evaluation.”
Text-to-Image Generation Grounded by Fine-Grained User Attention (2020, background): “Several other approaches also use intermediate scene graph representations [18, 41] for improving image synthesis.”
Image Synthesis From Reconfigurable Layout and Style (2019, methodology): “In comparison, the state-of-the-art methods include the very recent Layout2Im method [38], the scene graph to image (sg2im) method [16] and the pix2pix method [15].”
Specifying Object Attributes and Relations in Interactive Scene Generation (2019, methodology): “Some of the statistics reported for [9] are computed by us, based on the published model.”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, background): “Progress in image-based scene graph prediction has cascaded to improvements across multiple Computer Vision tasks, including image captioning [2], image retrieval [36, 64], visual question answering [35], relationship modeling [41] and image generation [34].”
Unbiased Scene Graph Generation From Biased Training (2020, neutral): “— yet on which are our core efforts made [69, 53, 6], then pretend that there is a graph — nothing but a sparse object layout with binary links, and finally shroud it into graph neural networks [63] for merely more contextual object representations [65, 17, 54].”
Investigating object compositionality in Generative Adversarial Networks (2018, background): “The goal of this work is not to improve upon state-of-the-art approaches to multi-object image generation that rely on prior knowledge by conditioning on explicit scene graphs (Johnson et al., 2018; Xu et al., 2018), images of individual objects (Lin et al.”