Interactive Image Generation Using Scene Graphs

By Gaurav Mittal, Shubham Agrawal, Anuva Agarwal, Sushant Mehta, Tanya Marwah. 2019

59 citations recorded by the source corpus.

Recent years have witnessed some exciting developments in the domain of generating images from scene-based text descriptions. These approaches have primarily focused on generating images from a static text description and are limited to generating images in a single pass. They are unable to generate an image interactively based on an incrementally additive text description (something that is more intuitive and similar to the way we describe an image). We propose a method to generate an image incrementally based on a sequence of graphs of scene descriptions (scene-graphs). We propose a recurrent network architecture that preserves the image content generated in previous steps and modifies the cumulative image as per the newly provided scene information. Our model utilizes Graph Convolutional Networks (GCN) to cater to variable-sized scene graphs along with Generative Adversarial image translation networks to generate realistic multi-object images without needing any intermediate supervision during training. We experiment with Coco-Stuff dataset which has multi-object images along with annotations describing the visual scene and show that our model significantly outperforms other approaches on the same dataset in generating visually consistent images for incrementally growing scene graphs.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

Citations in the corpus

Object-Centric Image Generation from Layouts (2020, background): “2019b), scene graphs (Johnson, Gupta, and Fei-Fei 2018; Mittal et al. 2019; Ashual and Wolf 2019), layouts (Zhao et al.”
Adversarial Text-to-Image Synthesis: A Review (2021, background): “…+ dialogue [93], [95], [99] caption + layout [104], [97], [108], [103] caption + semantic masks [109], [110], [113], [114], [115], [38] scene graphs [116], [121], [122], [124], [125] multiple captions [91], [92] multiple captions + mouse traces [126] Table 2: Methods grouped by their supervision.”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, background): “These applications include visual-textual transformers [139], [140], [141], image text retrieval [3], [6], [142], visual question answering [143], [144], image understanding and reasoning [145], [146], [147], 3D scene graphs [7], [148], and the detection and recognition of human-object and human-hum”
Semantic Object Accuracy for Generative Text-to-Image Synthesis (2019, background): “[41] do interactive image generation but do not use text as direct input but instead update the scene graph from the text over the course of the interaction.”