Diverse Image Synthesis From Semantic Layouts via Conditional IMLE

By Ke Li, Tianhao Zhang, Jitendra Malik. 2018

107 citations recorded by the source corpus.

Most existing methods for conditional image synthesis are only able to generate a single plausible image for any given input, or at best a fixed number of plausible images. In this paper, we focus on the problem of generating images from semantic segmentation maps and present a simple new method that can generate an arbitrary number of images with diverse appearance for the same semantic layout. Unlike most existing approaches which adopt the GAN framework, our method is based on the recently introduced Implicit Maximum Likelihood Estimation (IMLE) framework. Compared to the leading approach, our method is able to generate more diverse images while producing fewer artifacts despite using the same architecture. The learned latent space also has sensible structure despite the lack of supervision that encourages such behaviour.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

CNN-Generated Images Are Surprisingly Easy to Spot… for Now (2019, methodology): “IMLE [26] 10 We take ofﬁcially released IMLE model pretrained on GTA, and generate synthesized images from pre-processed segmentation maps.”
Towards Universal Fake Image Detectors that Generalize Across Generative Models (2023, methodology): “First, we evaluate on the models used in [49]: Pro-GAN [28], StyleGAN [29], BigGAN [7], CycleGAN [53], StarGAN [13], GauGAN [37], CRN [12], IMLE [30], SAN [18], SITD [11], and DeepFakes [46].”
Inclusive GAN: Improving Data and Minority Coverage in Generative Models (2020, background): “, GLANN [20] and conditional IMLE [31], also validate the improved effectiveness of IMLE framework in a reconstructive generation context.”
A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentials (2022, methodology): “…[88] Style/object transfer 2.6k GauGAN [58] COCO 10.0k StarGAN [13] CelebA 16.8k Non-GAN Model Glow [37] CelebA 16.8k CRN [11] GTA 12.8k IMLE [43] GTA 12.8k SAN [15] Standard SR benchmark 440 FaceForensics++ [64] YouTube 5.4k Unknown Model WhichFaceReal [3] Internet 2.0k WildDeepfake [89]…”
Deep learning for deepfakes creation and detection: A survey (2019, methodology): “Images A dataset of images generated by ten GAN models, including CycleGAN [164], StarGAN [153], AttGAN [154], GDWCT [152], StyleGAN [51], StyleGAN2 [155], PGGAN [62], FaceForensics++ [105], IMLE [165], and SPADE [42].”
Semantically Multi-Modal Image Synthesis (2020, methodology): “Similar to most label-to-image methods [41, 36, 28], our network also supports semantic manipulation.”
You Only Need Adversarial Supervision for Semantic Image Synthesis (2020, methodology): “Further, (Li et al., 2019) employed implicit maximum likelihood estimation of Li & Malik (2018) to increase the variety of the CRN model.”
BlobGAN: Spatially Disentangled Scene Representations (2022, methodology): “Other setups include using perceptual losses [14], implicit likelihood estimation [43], and more recently, diffusion models [49, 72].”
Text as Neural Operator:Image Manipulation by Text Instruction (2020, background): “The input context can be images [18, 56, 24, 16, 37], audio sequences [25], human poses [34], or semantic segmentation masks [49, 41, 28].”
Few-shot Semantic Image Synthesis Using StyleGAN Prior (2021, unknown): “The input semantic masks are the same as those used in Figure 7. ity [6, 22, 23, 32, 31], generate multi-modal outputs [23, 18, 44, 10], and simplify input annotations using bounding boxes [41, 30, 19].”