Image Synthesis From Reconfigurable Layout and Style

By Tianfu Wu. 2019

165 citations recorded by the source corpus.

Despite remarkable recent progress on both unconditional and conditional image synthesis, it remains a long- standing problem to learn generative models that are capable of synthesizing realistic and sharp images from re- configurable spatial layout (i.e., bounding boxes + class labels in an image lattice) and style (i.e., structural and appearance variations encoded by latent vectors), especially at high resolution. By reconfigurable, it means that a model can preserve the intrinsic one-to-many mapping from a given layout to multiple plausible images with different styles, and is adaptive with respect to perturbations of a layout and style latent code. In this paper, we present a layout- and style-based architecture for generative adversarial networks (termed LostGANs) that can be trained end-to-end to generate images from reconfigurable layout and style. Inspired by the vanilla StyleGAN, the proposed LostGAN consists of two new components: (i) learning fine-grained mask maps in a weakly-supervised manner to bridge the gap between layouts and images, and (ii) learning object instance-specific layout-aware feature normalization (ISLA-Norm) in the generator to realize multi-object style generation. In experiments, the proposed method is tested on the COCO-Stuff dataset and the Visual Genome dataset with state-of-the-art performance obtained. The code and pretrained models are available at https://github.com/iVMCL/LostGANs.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

High-Resolution Complex Scene Synthesis with Transformers (2021, background): “128×128† LostGAN∗ [26] LostGAN-V2∗ [27] OC-GAN∗ [28] ours∗ 29.”
Object-Centric Image Generation from Layouts (2020, methodology): “Generator As a means of disentangling our model’s performance from a speciﬁc choice of generator architecture, we used a classical residual [16] architecture consisting of 4 layers for 64 × 64 inputs, and 5 layers for 128 × 128 inputs, as used recently in [41,53,58].”
Semantic Object Accuracy for Generative Text-to-Image Synthesis (2019, methodology): “[35] introduce a new feature normalization method and fine-grained mask maps to generate visually different images from a given layout.”
Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis (2022, methodology): “One can find that Frido outperforms previous methods, including LostGAN-v2 (Sun and Wu 2019), OC-GAN (Sylvain et al.”
Modeling Image Composition for Complex Scene Generation (2022, methodology): “For better controllability, LostGANs [32, 33] first synthesize the semantic masks from layouts, and then ISLA-Norm is proposed for generating color images from specific masks and style codes.”
Composer: Creative and Controllable Image Synthesis with Composable Conditions (2023, background): “Reconﬁgurations: Image reconﬁguration (Sun & Wu, 2019) refers to manipulating an image through direct modiﬁcation of one or more of its representations.”
Adversarial Text-to-Image Synthesis: A Review (2021, methodology): “to [100, 101], we could adopt the LPIPS metric [169] as the Diversity Score (DS) on two sets of generated images from the same captions to specifically evaluate the diversity of generated images.”
Training-Free Layout Control with Cross-Attention Guidance (2023, background): “√ It is designed to take a noised sample z t = α t z + 1 − α t (cid:2) t , as input, where (cid:2) t is normally distributed noise and α t is a decreasing sequence, from α 0 ≈ 1 to α T ≈ 0 , representing the noise schedule.”
Image Synthesis from Layout with Locality-Aware Mask Adaption (2021, methodology): “It stacks ResNet blocks [9] with ISLA-Norm layers [32].”
Few-shot Semantic Image Synthesis Using StyleGAN Prior (2021, unknown): “The input semantic masks are the same as those used in Figure 7. ity [6, 22, 23, 32, 31], generate multi-modal outputs [23, 18, 44, 10], and simplify input annotations using bounding boxes [41, 30, 19].”
Instance-Conditioned GAN (2021, unknown): “Existing methods for complex scene generation, where natural looking scenes contain multiple objects, most often aim at controllability and rely on detailed condition-ings such as a scene graphs [3, 25], bounding box layouts [48–50, 58 ], semantic segmentation masks [ 9,”