Specifying Object Attributes and Relations in Interactive Scene Generation

By Oron Ashual, Lior Wolf. 2019

193 citations recorded by the source corpus.

We introduce a method for the generation of images from an input scene graph. The method separates between a layout embedding and an appearance embedding. The dual embedding leads to generated images that better match the scene graph, have higher visual quality, and support more complex scene graphs. In addition, the embedding scheme supports multiple and diverse output images per scene graph, which can be further controlled by the user. We demonstrate two modes of per-object control: (i) importing elements from other images, and (ii) navigation in the object space by selecting an appearance archetype. Our code is publicly available at https://www.github.com/ashual/scene_generation.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

High-Resolution Complex Scene Synthesis with Transformers (2021, methodology): “We hypothesize that this is partly due to the fact that most concurrent approaches rely on many steps and auxiliary losses such as scene-matching or object-speciﬁc losses [13, 2, 28, 11], which turns out limiting the models’ overall performance.”
Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020, methodology): “(a) Input Layout, and Images generated by (b) Layout2Im [20] 64× 64, (c) our LostGAN-V1 128× 128, (d) Grid2Im [19] 256× 256, (e) our LostGAN-V2 256× 256, and (f) Ground Truth.”
Object-Centric Image Generation from Layouts (2020, methodology): “Past methods focus on different input types, including scene graphs (Johnson, Gupta, and Fei-Fei 2018; Ashual and Wolf 2019), pixel-level semantic segmentation (Li et al.”
Towards causal generative scene models via competition of experts (2020, methodology): “Turkoglu et al. (2019) proposed a layer based model to add objects onto a background, Ashual & Wolf (2019) proposed a scene-generation method allowing for ﬁne grained user control, Karras et al. (2019a;b) have achieved impressive image generation results by exclusively training on a single class of…”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, background): “Recent studies have explored image generation from scene graphs [3, 34].”
LayoutDiffusion: Controllable Diffusion Model for Layout-to-Image Generation (2023, background): “Grid2Im [2], LostGANv2 [41] and PLGAN [45] generate images with distorted and unreal objects.”
GAN Steerability without optimization (2020, background): “These range from disentangled models for controlling e.g., the hair color or gender of facial images (Karras et al., 2019a;b; Choi et al., 2018), to models that even allow specifying object relations (Ashual & Wolf, 2019).”
Spatial-Temporal Transformer for Dynamic Scene Graph Generation (2021, background): “Recently, scene graphs have been successfully applied in different vision tasks, such as image retrieval [26, 46], object detection, semantic segmentation, human-object interaction [15], image synthesis [24, 3], and high-level vision-language tasks like image captioning [13, 62] or visual question a”
Differentiable Scene Graphs (2019, background): “SGs have been shown to be useful for semantic-level interpretation and reasoning about a visual scene [20, 1, 15, 47].”
House-GAN: Relational Generative Adversarial Networks for Graph-constrained House Layout Generation (2020, methodology): “such as the number of rooms with room types and approximate square footage. These bubble diagrams are later converted to oorplans using the Metropolis algorithm. Jonhson et al. [12] and Ashual et al. [3] aim to generate image layouts and synthesize realistic images from input scene-graphs via GCNs. ”
Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling (2019, background): “Furthermore, counterfactual thinking is also applied to augment training targets [37, 9, 6].”
Image Synthesis from Layout with Locality-Aware Mask Adaption (2021, methodology): “Layout-to-image generation also serves as a useful subpro-cess in text-to-image translation [11, 12, 13, 20] and image generation from scene graphs [1, 15].”
Instance-Conditioned GAN (2021, methodology): “Existing methods for complex scene generation, where natural looking scenes contain multiple objects, most often aim at controllability and rely on detailed conditionings such as a scene graphs [3, 25], bounding box layouts [48–50, 58], semantic segmentation masks [9, 43, 44, 52, 55] or more recentl”
AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning (2021, methodology): “The Computer Vision community has utilized the scene graph representation for a variety of tasks including visual question answering [22], relationship modeling [38], object localization [28], evaluation [1], generation [20, 3], retrieval [3, 23] and few-shot ActivityNet QA”
Semantic Bottleneck Scene Generation (2019, background): “In conditional image synthesis one aims to generate images by conditioning on an input which can be provided in the form of an image [18, 47, 3, 5, 25], a text phrase [37, 45, 35, 2, 17], a scene graph [20, 2], a class label or a semantic lay-out [31, 8, 39, 32].”
Adversarial Text-to-Image Synthesis: A Review (2021, methodology): “…+ dialogue [93], [95], [99] caption + layout [104], [97], [108], [103] caption + semantic masks [109], [110], [113], [114], [115], [38] scene graphs [116], [121], [122], [124], [125] multiple captions [91], [92] multiple captions + mouse traces [126] Table 2: Methods grouped by their supervision.”