Generative Image Modeling Using Style and Structure Adversarial Networks

By X. Wang, Abhinav Kumar Gupta. 2016

637 citations recorded by the source corpus.

Current generative frameworks use end-to-end learning and generate images by sampling from uniform noise distribution. However, these approaches ignore the most basic principle of image formation: images are product of: (a) Structure: the underlying 3D model; (b) Style: the texture mapped onto structure. In this paper, we factorize the image generation process and propose Style and Structure Generative Adversarial Network (\({\text {S}^2}\)-GAN). Our \({\text {S}^2}\)-GAN has two components: the Structure-GAN generates a surface normal map; the Style-GAN takes the surface normal map as input and generates the 2D image. Apart from a real vs. generated loss function, we use an additional loss with computed surface normals from generated images. The two GANs are first trained independently, and then merged together via joint learning. We show our \({\text {S}^2}\)-GAN model is interpretable, generates more realistic images and can be used to learn unsupervised RGBD representations.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, background): “Compared to L1 loss, which often leads to blurry images [21, 22], the adversarial loss [16] has become a popular choice for many image-to-image tasks [10, 24, 25, 31, 40, 45, 53, 58, 64].”
Generating Videos with Scene Dynamics (2016, background): “generating short videos with realistic temporal semantics, rather than detecting or retrieving them. Our technical approach builds on recent work in generative adversarial networks for image modeling [9, 31, 4, 48, 28], which we extend to video. To our knowledge, there has been relatively little wor”
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks (2016, methodology): “[36] factorized the indoor scene generation process into structure generation and style generation with the proposed S(2)-GAN.”
Parallel Multiscale Autoregressive Density Estimation (2017, methodology): “These networks have been used in a wide variety of conditional image generation schemes such as text and spatial structure to image (Mansimov et al., 2015; Reed et al., 2016b;a; Wang & Gupta, 2016).”
StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks (2017, methodology): “[46] utilized a structure GAN and a style GAN to synthesize images of the indoor scenes.”
Mode Regularized Generative Adversarial Networks (2016, methodology): “Motivated from this, several conditional variants of GAN has been applied to a wide range of tasks, including image prediction from a normal map Wang & Gupta (2016), image synthesis from text Reed et al. (2016) and edge map Isola et al. (2016), real-time image manipulation Zhu et al. (2016),…”
Stacked Generative Adversarial Networks (2016, methodology): “Our conditional loss Lcond Gi is similar to the “feature loss” used by [7] and the “FCN loss” in [62].”
Learning to Discover Cross-Domain Relations with Generative Adversarial Networks (2017, background): “The translated images clearly match the rotation feature of the input images while preserving visual features of car and face domain, respectively.”
Photographic Image Synthesis with Cascaded Refinement Networks (2017, background): “Wang and Gupta [47] synthesize 128×128 images of indoor scenes by factorizing the image generation process into synthesis of a normal map and subsequent synthesis of a corresponding color image.”
Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis (2018, background): “Due to adversarial training, images generated by the other methods, especially StackGAN [36], tend to be clear and exhibits high frequency details.”
RenderGAN: Generating Realistic Labeled Data (2016, background): “Wang & Gupta (2016) trained two separate GANs, one to model the object normals and another one for the texture conditioned on the normals.”
Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations (2017, methodology): “In the unsupervised setting, the InfoGAN model of Chen et al. [2016] adapts GANs to the learning of an interpretable representation with the use of mutual information theory, and Wang and Gupta [2016] use two sequentially connected GANs.”
DualGAN: Unsupervised Dual Learning for Image-to-Image Translation (2017, background): “Conventionally, these tasks have been tackled separately due to their intrinsic disparities [7, 8, 21, 12, 4, 18].”
LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation (2016, methodology): “…image generation including part-based models (Im et al., 2016; Kwak & Zhang, 2016), image generation conditioned on input text or attributes (Mansimov et al., 2015; Reed et al., 2016b;a), image generation based on 3D structure (Wang & Gupta, 2016), and even video generation (Vondrick et al., 2016).”