Image-to-Image Translation with Conditional Adversarial Networks
By Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros. 2016
22,722 citations recorded by the source corpus.
We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Moreover, since the release of the pix2pix software associated with this paper, hundreds of twitter users have posted their own artistic experiments using our system. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without handengineering our loss functions either.
Subfields: Computer Vision and Pattern Recognition, Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Yi Yang
- Andrew Zisserman
In Generative Models
- Jun-Yan Zhu
- Eli Shechtman
- Richard Zhang
- Stefano Ermon
- Daniel Cohen-Or
- Alexei A. Efros
- Yoshua Bengio
- Arash Vahdat
Citations in the corpus
- Group Normalization (2018, background): “1986; Hochreiter and Schmidhuber 1997) or generativemodels (GANs; Goodfellow et al. 2014; Isola et al. 2017).”
- Contrastive Multiview Coding (2019, background): “[27], adversarial loss functions [20] that learn the loss in addition to the representation, or contrastive losses e.”
- Making Convolutional Networks Shift-Invariant Again (2019, methodology): “We test on the Labels→Facades (Tyleček & Šára, 2013; Isola et al., 2017) dataset, where a network is tasked to generated a 256×256 photorealistic image from a label map.”
- Zero-Shot Text-to-Image Generation (2021, background): “This previously required custom approaches (Isola et al., 2017), rather emerging as a capability of a single, large generative model.”
- CNN-Generated Images Are Surprisingly Easy to Spot… for Now (2019, background): “[24] likewise showed that simple classifiers can detect images created by an image translation network [17], but did not consider cross-model transfer.”
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “Both methods do not require supervision between domains. pix2pix [24] utilized conditional GANs together with a supervised reconstruction loss. pix2pixHD [62] improved the latter by increasing output image resolution thanks to improved network architecture.”
- Generative Adversarial Networks (2021, neutral): “By providing a thorough overview of GANs, we hope to make this complex and challenging topic more accessible to researchers and practitioners in the field, and inspire new innovations and applications.”
- Vector-quantized Image Modeling with Improved VQGAN (2021, methodology): “, 2019) is more stable and yields better reconstruction quality than PatchGAN (Isola et al., 2017)”
- Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, positive): “We define a discriminator Dξ(·) that works on a local patch-level [19], discriminating between “real” and “fake” patches.”
- D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “Unlike many GAN-based methods [12, 72, 97, 45, 98], D2C does not need to optimize an inversion procedure at evaluation time, and thus the latent value is much faster to compute; D2C is also better at retaining fine-grained features of the original image due to the reconstruction loss.”
- Designing a Practical Degradation Model for Deep Blind Image Super-Resolution (2021, methodology): “Thirdly, we train the BSRGAN by minimizing a weighted combination of L1 loss, VGG perceptual loss and spectral norm-based least square PatchGAN loss [19] with weights 1 , 1 and 0 .”
- Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, methodology): “We follow the common setting (DeVries et al., 2019; Wang et al., 2019b) on the edges to photos datasets (Isola et al., 2017) at 256×256 resolution, where FID samples once per validation image (200 in total) and the training set is used as the reference distribution; LPIPS measures the…”
- Dual Contradistinctive Generative Autoencoder (2020, critical): “However, it relies on a pixel-wise reconstruction loss, which is known not ideal to be reflective of perceptual realism [32, 30], often resulting in blurry images.”
- Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020, methodology): “The conditions may take many forms such as category labels [3], [9], paired or unpaired source images [10], [11], [12], [13], semantic maps [14], [15], text description [16], [17] and scene graphs [18], [19].”
- Object-Centric Image Generation from Layouts (2020, background): “Models that perform well on fine-grained pixel-level semantic maps also can be easily applied to this setting (Park et al. 2019; Isola et al. 2017; Wang et al. 2018).”
- A U-Net Based Discriminator for Generative Adversarial Networks (2020, neutral): “Local discriminator feedback is also commonly applied through PatchGAN discriminators [18].”