A U-Net Based Discriminator for Generative Adversarial Networks

By Edgar Schönfeld, Bernt Schiele, Anna Khoreva. 2020

348 citations recorded by the source corpus.

Among the major remaining challenges for generative adversarial networks (GANs) is the capacity to synthesize globally and locally coherent images with object shapes and textures indistinguishable from real images. To target this issue we propose an alternative U-Net based discriminator architecture, borrowing the insights from the segmentation literature. The proposed U-Net based architecture allows to provide detailed per-pixel feedback to the generator while maintaining the global coherence of synthesized images, by providing the global image feedback as well. Empowered by the per-pixel response of the discriminator, we further propose a per-pixel consistency regularization technique based on the CutMix data augmentation, encouraging the U-Net discriminator to focus more on semantic and structural changes between real and fake images. This improves the U-Net discriminator training, further enhancing the quality of generated samples. The novel discriminator improves over the state of the art in terms of the standard distribution and image quality metrics, enabling the generator to synthesize images with varying structure, appearance and levels of detail, maintaining global and local realism. Compared to the BigGAN baseline, we achieve an average improvement of 2.7 FID points across FFHQ, CelebA, and the proposed COCO-Animals dataset.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Training Generative Adversarial Networks with Limited Data (2020, methodology): “Methods that do not employ augmentation (BigGAN, StyleGAN2, and our baseline) degrade noticeably as the size of the training set decreases, generally yielding poor image quality and diversity with fewer than 30k training images.”
Projected GANs Converge Faster (2021, background): “Notable exceptions are [55,70] which utilize an encoder-decoder discriminator architecture.”
Taming Transformers for High-Resolution Image Synthesis (2020, neutral): “7 U-Net GAN (+aug) [57] 10.”
You Only Need Adversarial Supervision for Semantic Image Synthesis (2020, background): “, 2017) and unconditional image synthesis (Schönfeld et al., 2020), but to the best of our knowledge have not been explored for semantic image synthesis and our work is the first to apply adversarial semantic segmentation loss for this task.”
Perception Prioritized Training of Diffusion Models (2022, neutral): “Type FID ↓ FFHQ BigGAN ICLR’19 [3] GAN 12.4 UNet GAN CVPR’20 [37] GAN 10.9 StyleGAN2 CVPR’20 [21] GAN 3.73 NVAE NeurIPS’20 [44] VAE 26.02 VDVAE ICLR’21 [5] VAE 33.5 VQGAN CVPR’21 [11] GAN+AR 9.6 D2C NeurIPS’21 [38] Diff [20] GAN 64.70 MSG-GAN CVPR’20 [16] [22] Flow 68.93 ALAE CVPR’20 [33] GAN 19.21…”
Dual Contrastive Loss and Attention for GANs (2021, methodology): “Besides StyleGAN2 [43], we also compare to a parallel state-of-the-art study, U-Net GAN [69], which was build upon and improved on BigGAN [5].”
TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up (2021, methodology): “After its origin, GANs quickly embraced fully convolutional backbones [26], and inherited most successful designs from CNNs such as batch normalization, pooling, (Leaky) ReLU and more [38, 39, 40, 18].”
Training GANs with Stronger Augmentations via Contrastive Discriminator (2021, background): “…the literature (Arjovsky et al., 2017; Gulrajani et al., 2017; Miyato et al., 2018; Wei et al., 2018; Jolicoeur-Martineau, 2019; Chen et al., 2019; Schonfeld et al., 2020), and the idea of incorporating data augmentation techniques has recently gained a particular attention on this line of…”
Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data (2021, methodology): “Inspired by [39, 50], we also improve the VGG-style discriminator in ESRGAN to an U-Net design with skip connections (Fig.”
Improving Augmentation and Evaluation Schemes for Semantic Image Synthesis (2020, methodology): “We first compare our proposed augmentation technique with the CutMix augmentation [68], which has recently been used to regularize unconditional GAN training [49].”
DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior (2023, methodology): “Then the adversarial loss [31; 17; 56; 41; 49] is incorporated for learning the reconstruction process in an end-to-end manner.”
E2Style: Improve the Efficiency and Effectiveness of StyleGAN Inversion (2021, background): “[14] in 2014, it has evolved considerably in terms of training strategies [2], [15], loss functions [16], [17], [18], regularizations [19], [20], [21], and network structures [22], [23].”