Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts

By Levent Karacan, Zeynep Akata, Aykut Erdem, Erkut Erdem. 2016

192 citations recorded by the source corpus.

Automatic image synthesis research has been rapidly growing with deep networks getting more and more expressive. In the last couple of years, we have observed images of digits, indoor scenes, birds, chairs, etc. being automatically generated. The expressive power of image generators have also been enhanced by introducing several forms of conditioning variables such as object names, sentences, bounding box and key-point locations. In this work, we propose a novel deep conditional generative adversarial network architecture that takes its strength from the semantic layout and scene attributes integrated as conditioning variables. We show that our architecture is able to generate realistic outdoor scene images under different conditions, e.g. day-night, sunny-foggy, with clear object boundaries.

Subfields: Computer Vision and Pattern Recognition, Geology, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Geology

In Generative Models

Citations in the corpus

Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, background): “The GANs consist of a generator and a discriminator where the goal of the generator is to produce realistic images so that the discriminator cannot tell the synthesized images apart from real ones.”
Image-to-Image Translation with Conditional Adversarial Networks (2016, background): “The image-conditional models have tackled image prediction from a normal map [52], future frame prediction [37], product photo generation [56], and image generation from sparse annotations [28, 45] (c.”
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, methodology): “Existing image synthesis methods only utilize semantic label maps [5, 21, 25], an image where each pixel value represents the object class that the pixel belongs to.”
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks (2017, methodology): “Similar ideas have been applied to various tasks such as generating photographs from sketches [40] or from attribute and semantic layouts [22].”
Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis (2018, methodology): “The problem of generating images from pixel-wise semantic labels has been explored recently [3, 10, 12, 22].”
CyCADA: Cycle-Consistent Adversarial Domain Adaptation (2017, background): “Recent work (Isola et al., 2016; Sangkloy et al., 2016; Karacan et al., 2016) adopt conditional GANs (Mirza & Osindero, 2014) for these image-to-image translation problems (Isola et al., 2016), but they require input-output image pairs for training, which is in general not available in domain…”
Generating Multiple Objects at Spatially Distinct Locations (2019, background): “Karacan et al. (2016), Johnson et al. (2018), and Wang et al. (2018) use a scene layout to generate images in which given objects are drawn within their specified segments based on the generated scene layout.”
DA-GAN: Instance-Level Image Translation by Deep Attention Generative Adversarial Networks (2018, methodology): “Similar ideas have been applied to various tasks such as generating photographs from sketches [29] or from attribute and semantic layouts [15].”
How Generative Adversarial Networks and Their Variants Work (2017, methodology): “[58] synthesized outdoor images having specific scenery attributes using the CGAN framework, and they also concatenated an attribute latent vector to z for the generator.”
Adaptive Appearance Rendering (2021, methodology): “Zhu et al. [42], Karacan et al. [16], Johnson et al. [15] and Ma et al. [22] share a similar spirit of generating images conditioned on high-level semantic annotations.”
CartoonGAN: Generative Adversarial Networks for Photo Cartoonization (2018, background): “Several works [5, 14, 16] have provided GAN solutions to pixel-to-pixel image synthesis problems.”
InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs (2020, unknown): “It is widely used for real image editing considering its fast inference speed [5], [57], [58], [59], [60], [61].”
Diverse Image Synthesis From Semantic Layouts via Conditional IMLE (2018, unknown): “[17] developed a conditional GAN-based model for generating images from semantic layouts and labelled image attributes.”
Image De-Raining Using a Conditional Generative Adversarial Network (2017, background): “in [23] proposed a deep GAN conditioned on semantic layout and scene attributes to synthesize realistic outdoor scene images under different conditions.”
Image Generation From Layout (2018, background): “Of specific relevance are approaches for text-to-image [11, 15, 25, 33, 41, 47] generation.”
How Generative Adversarial Networks and Its Variants Work : An Overview of GAN (2017, background): “AL-CGAN [55] also takes this conditional approach that takes side information of the layout and content of the scene.”
MIXGAN: Learning Concepts from Different Domains for Mixture Generation (2018, methodology): “…[Gatys et al., 2016; Johnson et al., 2016; Ulyanov et al., 2016] and image-to-image translation [Hertzmann et al., 2001; Sangkloy et al., 2017; Karacan et al., 2016] models can translate an existing image to another style, they are also restricted in that they require off-the-shelf content…”
Semantic Object Accuracy for Generative Text-to-Image Synthesis (2019, background): “Many approaches also generate images directly from semantic layouts without any additional textual input [13], [14].”