You Only Need Adversarial Supervision for Semantic Image Synthesis

By Vadim Sushko, Edgar Schönfeld, Dan Zhang, Juergen Gall, Bernt Schiele, Anna Khoreva. 2020

221 citations recorded by the source corpus.

Despite their recent successes, GAN models for semantic image synthesis still suffer from poor image quality when trained with only adversarial supervision. Historically, additionally employing the VGG-based perceptual loss has helped to overcome this issue, significantly improving the synthesis quality, but at the same time limiting the progress of GAN models for semantic image synthesis. In this work, we propose a novel, simplified GAN model, which needs only adversarial supervision to achieve high quality results. We re-design the discriminator as a semantic segmentation network, directly using the given semantic label maps as the ground truth for training. By providing stronger supervision to the discriminator as well as to the generator through spatially- and semantically-aware discriminator feedback, we are able to synthesize images of higher fidelity with better alignment to their input label maps, making the use of the perceptual loss superfluous. Moreover, we enable high-quality multi-modal image synthesis through global and local sampling of a 3D noise tensor injected into the generator, which allows complete or partial image change. We show that images synthesized by our model are more diverse and follow the color and texture distributions of real images more closely. We achieve an average improvement of $6$ FID and $5$ mIoU points over the state of the art across different datasets using only adversarial supervision.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

StyleGAN-NADA (2021, methodology): “In addition to StyleGAN, we investigated our model’s ability to convert existing classes in a more localized manner using OASIS [43], a SPADE-like [36] model that synthesizes images from segmentation masks.”
Pretraining is All You Need for Image-to-Image Translation (2022, methodology): “We compare with three strong baselines: Pix2PixHD [50], SPADE [36], and OASIS [44].”
Chunked Autoregressive GAN for Conditional Waveform Synthesis (2021, methodology): “Pitch discriminator Inspired by the OASIS model for semantic image segmentation (Sch¨onfeld et al., 2020), we design a discriminator that jointly performs pitch estimation and real/fake classiﬁ-cation.”
T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models (2023, background): “, some GAN-base [25, 35] and diffusion-based [43] methods.”
Few-Shot Incremental Learning for Label-to-Image Translation (2022, methodology): “We adopt the evaluation metics from previous works [25, 36,40,46] including mean Intersection-over-Union (mIoU), pixel accuracy (accu) and Fréchet Inception Distance (FID) [14].”
Collaging Class-specific GANs for Semantic Image Synthesis (2021, methodology): “There are also many great works exploring conditional image generation [39, 36, 19, 2, 30, 35, 34].”
BBDM: Image-to-Image Translation with Brownian Bridge Diffusion Models (2022, background): “We additionally compare BBDM with OASIS [34] and SPADE [24] on CelebAMask-HQ dataset.”
Enhancing Photorealism Enhancement (2021, background): “feature space [8], [15], [71], processed via a separate network stream [72], or guide an auxiliary classifier [73].”