Object-Centric Image Generation from Layouts

By Tristan Sylvain, Pengchuan Zhang, Yoshua Bengio, R. Devon Hjelm, Shikhar Sharma. 2020

109 citations recorded by the source corpus.

We begin with the hypothesis that a model must be able to understand individual objects and relationships between objects in order to generate complex scenes with multiple objects well. Our layout-to-image-generation method, which we call Object-Centric Generative Adversarial Network (or OC-GAN), relies on a novel Scene-Graph Similarity Module (SGSM). The SGSM learns representations of the spatial relationships between objects in the scene, which lead to our model's improved layout-fidelity. We also propose changes to the conditioning mechanism of the generator that enhance its object instance-awareness. Apart from improving image quality, our contributions mitigate two failure modes in previous approaches: (1) spurious objects being generated without corresponding bounding boxes in the layout, and (2) overlapping bounding boxes in the layout leading to merged objects in images. Extensive quantitative evaluation and ablation studies demonstrate the impact of our contributions, with our model outperforming previous state-of-the-art approaches on both the COCO-Stuff and Visual Genome datasets. Finally, we address an important limitation of evaluation metrics used in previous works by introducing SceneFID -- an object-centric adaptation of the popular Fréchet Inception Distance metric, that is better suited for multi-object images.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

High-Resolution Complex Scene Synthesis with Transformers (2021, methodology): “[28] further introduce a differentiable graph-scene matching loss to enhance the separation of objects.”
Training-Free Layout Control with Cross-Attention Guidance (2023, background): “Image generation with spatial conditioning is closely related to layout con-trol and typically done with bounding boxes or semantic maps [12, 32, 44, 45, 50, 58].”
ReCo: Region-Controlled Text-to-Image Generation (2022, methodology): “At the region level, ReCo achieves a SceneFID of 6.51, indicating strong capabilities in both generating highfidelity objects and precisely placing them in the queried position.”
Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis (2022, methodology): “In addition, we considered other task-speciﬁc metrics such as CLIP score (Hessel et al. 2021), Precision and Recall (Sajjadi et al. 2018), SceneFID (Sylvain et al. 2021), YOLO score (Li et al. 2021), PSNR (Hore and Ziou 2010), and SSIM (Wang et al. 2004) when applicable.”
BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion (2023, background): “Layout-to-image literature [19, 32, 33, 36, 40, 12, 2, 1] has studied on the way to synthesize images adhering to the spatial conditioning input.”
Instance-Conditioned GAN (2021, unknown): “Moreover, the diversity in the generations using two random seeds for LostGANv2 and OC-GAN is lower than for IC-GAN.”
Image Scene Graph Generation (SGG) Benchmark (2021, background): “…for building more visually-grounded and compositional models for other computer vision and vision-language tasks, like image/text retrieval [9, 17], image cap-tioning [4, 20, 6], visual question answering [3, 5], action recognition [7], image generation [8, 14], and 3D scene understanding [2].”
Image Synthesis from Layout with Locality-Aware Mask Adaption (2021, methodology): “We mainly compare our method with leading layout-to-images methods, including Layout2Im [37] as well as its extension [38], LostGAN-V1 [32], LostGAN-V2 [34] and OC-GAN [35].”
Adversarial Text-to-Image Synthesis: A Review (2021, methodology): “Besides the IS and FID, there have been multiple other proposals such the detection based score [140], SceneFID [103], the classiﬁcation accuracy score (CAS) [139], precision and recall metrics [166, 167], and the recently proposed density and coverage metrics [168], which have not yet been adopted…”
Modeling Image Composition for Complex Scene Generation (2022, methodology): “Similarly, SceneFID is proposed for complex scene generation tasks.”
Few-Shot Incremental Learning for Label-to-Image Translation (2022, methodology): “All cropped objects in real and generated images are resized and used to calculate SceneFID.”