Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis

By Tianfu Wu. 2020

96 citations recorded by the source corpus.

With the remarkable recent progress on learning deep generative models, it becomes increasingly interesting to develop models for controllable image synthesis from reconfigurable structured inputs. This paper focuses on a recently emerged task, layout-to-image, whose goal is to learn generative models for synthesizing photo-realistic images from a spatial layout (i.e., object bounding boxes configured in an image lattice) and its style codes (i.e., structural and appearance variations encoded by latent vectors). This paper first proposes an intuitive paradigm for the task, layout-to-mask-to-image, which learns to unfold object masks in a weakly-supervised way based on an input layout and object style codes. The layout-to-mask component deeply interacts with layers in the generator network to bridge the gap between an input layout and synthesized images. Then, this paper presents a method built on Generative Adversarial Networks (GANs) for the proposed layout-to-mask-to-image synthesis with layout and style control at both image and object levels. The controllability is realized by a proposed novel Instance-Sensitive and Layout-Aware Normalization (ISLA-Norm) scheme. A layout semi-supervised version of the proposed method is further developed without sacrificing performance. In experiments, the proposed method is tested in the COCO-Stuff dataset and the Visual Genome dataset with state-of-the-art performance obtained.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “9 on the COCO dataset, we follow common practice [37, 87, 89] and compute FID scores the 2048 unaugmented examples of the COCO Segmentation Challenge split.”
High-Resolution Complex Scene Synthesis with Transformers (2021, methodology): “256×256† LostGAN-V2∗ [27] OC-GAN∗ [28] SPADE ∗ [20, 27] ours∗ 42.”
Object-Centric Image Generation from Layouts (2020, neutral): “2019), LostGAN (Sun and Wu 2019), LostGAN-v2 (Sun and Wu 2020)).”
GLIGEN: Open-Set Grounded Text-to-Image Generation (2023, methodology): “Model FID (↓) YOLO score (AP/AP50/AP75) (↑) LostGAN-V2 [56] 42.”
Image Synthesis from Layout with Locality-Aware Mask Adaption (2021, methodology): “We mainly compare our method with leading layout-to-images methods, including Layout2Im [37] as well as its extension [38], LostGAN-V1 [32], LostGAN-V2 [34] and OC-GAN [35].”
Instance-Conditioned GAN (2021, unknown): “We also compare to state-of-the-art complex scene generation pipelines which rely on labeled bounding box annotations as conditioning – LostGANv2 [49] and OC-GAN [50].”
Modeling Image Composition for Complex Scene Generation (2022, methodology): “Although CNN-based layout-to-image generation methods have achieved promising performance on texture synthesis, they may still suffer from accurately modeling the dependencies among pixels (or object parts), which hinders the model generated more realistic scene images.”
Adversarial Text-to-Image Synthesis: A Review (2021, background): “There is an increasing interest in the layout-to-image generation task [100, 101, 102, 103] where each object is defined by a bounding box and class label.”