Diversity-Sensitive Conditional Generative Adversarial Networks

By Dingdong Yang, Seunghoon Hong, Yunseok Jang, Tianchen Zhao, Honglak Lee. 2019

223 citations recorded by the source corpus.

We propose a simple yet highly effective method that addresses the mode-collapse problem in the Conditional Generative Adversarial Network (cGAN). Although conditional distributions are multi-modal (i.e., having many modes) in practice, most cGAN approaches tend to learn an overly simplified distribution where an input is always mapped to a single output regardless of variations in latent code. To address such issue, we propose to explicitly regularize the generator to produce diverse outputs depending on latent codes. The proposed regularization is simple, general, and can be easily integrated into most conditional GAN objectives. Additionally, explicit regularization on generator allows our method to control a balance between visual quality and diversity. We demonstrate the effectiveness of our method on three conditional generation tasks: image-to-image translation, image inpainting, and future video prediction. We show that simple addition of our regularization to existing models leads to surprisingly diverse generations, substantially outperforming the previous approaches for multi-modal conditional generation specifically designed in each individual task.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

StarGAN v2: Diverse Image Synthesis for Multiple Domains (2019, methodology): “To further enable the generator G to produce diverse images, we explicitly regularize G with the diversity sensitive loss [34, 48] where the target style codes (cid:101) s 1 and (cid:101) s 2 are produced by F conditioned on two random latent codes z 1 and z 2 ( i.e .”
Palette: Image-to-Image Diffusion Models (2021, methodology): “and has been a key limitation of many existing GAN-based methods [Yang et al. 2019b; Zhu et al. 2017b].”
Regularizing Generative Adversarial Networks under Limited Data (2021, background): “On the other hand, a variety of regularization approaches [5, 8, 41, 43, 59, 67] are proposed to alleviate the model-collapse issue, thus increasing the diversity of the generated images.”
Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning (2019, background): “However, Conditional GANs are severely prone to mode collapse, where the generator’s output is restricted to a limited subset of the distribution [7, 51].”
SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models (2021, unknown): “Although some techniques are proposed to alleviate mode collapse in GAN (e.g., bayesian inference [40] and explicit regularization loss [41]), they have not been veriﬁed in the super-resolution task to achieve state-of-the-art and diverse results.”
Inclusive GAN: Improving Data and Minority Coverage in Generative Models (2020, methodology): “0209 DSGAN [50] 14.”
The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks (2019, background): “Despite the difference, our approach to MI attacks leverages some training strategies from the venerable line of work on image inpainting [25, 13, 23] and significantly improves the realism of the reconstructed images over the existing attack methods.”
Manipulating Medical Image Translation with Manifold Disentanglement (2020, methodology): “This is done by imposing an additional diversiﬁcation regulariser (similar to [38]) as below to ensure the output is well-conditioned on the style codes.”
Semantically Multi-Modal Image Synthesis (2020, unknown): “We choose four very recent state-of-the-art meth-ods: BicycleGAN [58], DSCGAN [47], pix2pixHD [43] and SPADE [38], as the comparison methods.”
Speech gesture generation from the trimodal context of text, audio, and speaker identity (2020, methodology): “us, we use diversity regularization (Yang et al. 2019) to avoid ignoring style features.”
Improved Consistency Regularization for GANs (2020, methodology): “Though motivated differently, this can be seen as related to the Jacobian Clamping technique from Odena et al. (2018) and diversity increase technique in Yang et al. (2019).”
DLow: Diversifying Latent Flows for Diverse Human Motion Prediction (2020, background): “These methods [68, 46, 6, 61, 42, 71, 3] often build upon popular generative models such as conditional generative adversarial networks (CGANs; [21]) or conditional variational autoencoders (CVAEs; [38]).”