Mode Seeking Generative Adversarial Networks for Diverse Image Synthesis

By Qi Mao, Hsin-Ying Lee, Hung-Yu Tseng, Siwei Ma, Ming-Hsuan Yang. 2019

444 citations recorded by the source corpus.

Most conditional generation tasks expect diverse outputs given a single conditional context. However, conditional generative adversarial networks (cGANs) often focus on the prior conditional information and ignore the input noise vectors, which contribute to the output variations. Recent attempts to resolve the mode collapse issue for cGANs are usually task-specific and computationally expensive. In this work, we propose a simple yet effective regularization term to address the mode collapse issue for cGANs. The proposed method explicitly maximizes the ratio of the distance between generated images with respect to the corresponding latent codes, thus encouraging the generators to explore more minor modes during training. This mode seeking regularization term is readily applicable to various conditional generation tasks without imposing training overhead or modifying the original network structures. We validate the proposed algorithm on three conditional image synthesis tasks including categorical generation, image-to-image translation, and text-to-image synthesis with different baseline models. Both qualitative and quantitative results demonstrate the effectiveness of the proposed regularization method for improving diversity without loss of quality.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “A branch of works aims to enforce the intra-conditioning diversity using VAE-based latent sampling strategies (Zhu et al., 2017b) or imposing distance-based loss terms (Huang et al., 2018; Mao et al., 2019; Qin et al., 2018).”
GAN Inversion: A Survey (2021, background): “In recent years, GANs have been applied to numerous tasks ranging from image translation [3], [4], [5], image manipulation [6], [7], [8] to image restoration [9], [10], [11], [12], [13].”
InfoMax-GAN: Improved Adversarial Image Generation via Information Maximization and Contrastive Learning (2020, background): “Subsequent works include MSGAN [53], which introduces a regularization encouraging conditional GANs to seek out minor modes often”
StarGAN v2: Diverse Image Synthesis for Multiple Domains (2019, methodology): “Along with improving the visual quality of generated images, their diversity also has been considered as an important objective which has been tackled by either devoted loss functions [27, 28] or architectural design [3, 18].”
Regularizing Generative Adversarial Networks under Limited Data (2021, background): “On the other hand, a variety of regularization approaches [5, 8, 41, 43, 59, 67] are proposed to alleviate the model-collapse issue, thus increasing the diversity of the generated images.”
SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models (2021, unknown): “However, GAN-driven methods can easily suffer from the mode collapse problem and only produce samples from a single or few modes of the distribution but ignore other modes, according to the papers [37,38].”
PriorGAN: Real Data Prior for Generative Adversarial Nets (2020, background): “MS-GAN [18] proposes a loss term that maximizes the ratio of the distance between generated images with respect to the corresponding latent codes.”
Large-Scale Generative Data-Free Distillation (2020, background): “Mode collapse is a common problem plaguing various generative models like GANs [16, 40, 47], where instead of producing a variety of different images, the generator produces a distribution with single image or just a few variations and the generated samples are almost independent of the latent varia”
Describe What to Change: A Text-guided Unsupervised Image-to-image Translation Approach (2020, methodology): “s motivated by previous works [21] showing that L1 produces sharper results than L2. To encourage the generator to produce diverse images, we explicitly regularize G with the diversity sensitive loss [8, 32]: Lds = E x∼p Xk ,a 1 2∼N(µµℓ,ΣΣΣℓ) ∥G(c ,a 1)−G(c ,a2)∥ Domain losses. Similarly to StarGAN ”
Learning Object Placement by Inpainting for Compositional Data Augmentation (2020, methodology): “While there is generally a trade-oﬀ between plausibility and diversity in generative models [34][23][21], we emphasize that our model aims to produce diverse results without sacriﬁcing the plausibility in the meantime.”
Image-to-Image Translation: Methods and Applications (2021, methodology): “Paint-to-image \ \ [139], [13] [76] \ [155] \ \ \ \ \ Text-to-image \ \ [140] [153] \ \ \ [173] \ \ [80] Semantic manipulation [3], [4], [5] \ [186], [139], [187] \ \ \ \ \ \ \ \”
Unpaired Image-to-Image Translation via Latent Energy Transport (2020, unknown): “6 MSGAN [29] 33.”
On the Evaluation of Conditional GANs (2019, background): “A flurry of works have tackled the problem of image-to-image translation, either from a deterministic perspective [16, 57, 47], or from that of conditional generation [58, 1, 15, 22, 29], to account for the multimodal nature of the problem.”
Semantic Pyramid for Image Generation (2020, background): “Finally, Ldiv is a diversity loss, based on [25].”
Data-Free Knowledge Distillation for Heterogeneous Federated Learning (2021, methodology): “To further increase the diversity of the generator output, we also leverage the idea of diversity loss from prior work (Mao et al., 2019) to train the generator model.”
LFS-GAN: Lifelong Few-Shot Image Generation (2023, methodology): “The mode seeking loss introduced in [31] is as follows:”