High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs
By Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, Bryan Catanzaro. 2017
4,428 citations recorded by the source corpus.
We present a new method for synthesizing high-resolution photo-realistic images from semantic label maps using conditional generative adversarial networks (conditional GANs). Conditional GANs have enabled a variety of applications, but the results are often limited to low-resolution and still far from realistic. In this work, we generate 2048 × 1024 visually appealing results with a novel adversarial loss, as well as new multi-scale generator and discriminator architectures. Furthermore, we extend our framework to interactive visual manipulation with two additional features. First, we incorporate object instance segmentation information, which enables object manipulations such as removing/adding objects and changing the object category. Second, we propose a method to generate diverse results given the same input, allowing users to edit the object appearance interactively. Human opinion studies demonstrate that our method significantly outperforms existing methods, advancing both the quality and the resolution of deep image synthesis and editing.
Subfields: Computer Vision and Pattern Recognition, Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Andrea Vedaldi
- Andrew Zisserman
In Generative Models
- Jun-Yan Zhu
- Eli Shechtman
- Daniel Cohen-Or
- Richard Zhang
- Alexei A. Efros
- Ming-Hsuan Yang
- Bolei Zhou
- Stefano Ermon
Citations in the corpus
- Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, methodology): “Human perception, which is also an important and reliable measure to evaluate synthesis quality, has been used in [71], [45], [72], [63].”
- A Style-Based Generator Architecture for Generative Adversarial Networks (2018, methodology): “, using multiple discriminators [17, 43, 10], multiresolution discrimination [55, 51], or self-attention [57].”
- Progressive Growing of GANs for Improved Quality, Stability, and Variation (2017, background): “, 2016a), image-to-image translation (Zhu et al., 2017; Liu et al., 2017; Wang et al., 2017), and image inpainting (Iizuka et al.”
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “Both methods do not require supervision between domains. pix2pix [24] utilized conditional GANs together with a supervised reconstruction loss. pix2pixHD [62] improved the latter by increasing output image resolution thanks to improved network architecture.”
- Generative Adversarial Networks (2021, neutral): “By providing a thorough overview of GANs, we hope to make this complex and challenging topic more accessible to researchers and practitioners in the field, and inspire new innovations and applications.”
- Projected GANs Converge Faster (2021, background): “Providing multi-scale feedback with one or multiple discriminators has been helpful for both image synthesis [23, 24] and image-to-image translation [43, 64].”
- Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, methodology): “In the final loss we also use R1 = Ex||∇Dξ(x)||(2) gradient penalty [31, 38, 7], and a discriminator-based perceptual loss or so-called feature matching loss—a perceptual loss on the features of discriminator network LDiscPL [45].”
- D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “Unlike many GAN-based methods [12, 72, 97, 45, 98], D2C does not need to optimize an inversion procedure at evaluation time, and thus the latent value is much faster to compute; D2C is also better at retaining fine-grained features of the original image due to the reconstruction loss.”
- High-Resolution Complex Scene Synthesis with Transformers (2021, methodology): “in class-conditional [4, 24], text-conditional [23, 34] or pixelwise conditional settings [20, 33, 30].”
- Taming Transformers for High-Resolution Image Synthesis (2020, background): “Dataset ours SPADE [46] Pix2PixHD (+aug) [65] CRN [9]”
- Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020, methodology): “The conditions may take many forms such as category labels [3], [9], paired or unpaired source images [10], [11], [12], [13], semantic maps [14], [15], text description [16], [17] and scene graphs [18], [19].”
- Object-Centric Image Generation from Layouts (2020, methodology): “Generator As a means of disentangling our model’s performance from a specific choice of generator architecture, we used a classical residual [16] architecture consisting of 4 layers for 64 × 64 inputs, and 5 layers for 128 × 128 inputs, as used recently in [41,53,58].”
- A U-Net Based Discriminator for Generative Adversarial Networks (2020, background): “Other lines of work focused mainly on improving the discriminator by exploiting multiple [36, 13, 11] and multi-resolution [45, 42] discriminators, using spatial feedback of the discriminator [17], an auto-encoder architecture with the reconstructionbased feedback to the generator [52] or self-super”
- Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, methodology): “In this paper, we show that the conven tional network architecture [22,45], which is built by stack ing convolutional, normalization, and nonlinearity layers, is at best sub-optimal, because their normalization layers tend to “wash away” information in input semantic masks.”
- EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning (2019, methodology): “The network is trained with an objective comprised of an adversarial loss and feature-matching loss [46]”
- Free-Form Image Inpainting With Gated Convolution (2018, background): “[42] propose to synthesize high-resolution photo-realistic images from semantic label maps using conditional generative adversarial networks.”
- GAN Inversion: A Survey (2021, methodology): “In recent years, GANs have been applied to computer vision tasks ranging from image translation [2], [3], [4], image manipulation [5], [6], [7], to image restoration [8], [9], [10].”
- GLIGEN: Open-Set Grounded Text-to-Image Generation (2023, methodology): “We compare with pix2pixHD [66], a classic image-to-image translation model.”