Mode Regularized Generative Adversarial Networks

By Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, Wenjie Li. 2016

593 citations recorded by the source corpus.

Although Generative Adversarial Networks achieve state-of-the-art results on a variety of generative tasks, they are regarded as highly unstable and prone to miss modes. We argue that these bad behaviors of GANs are due to the very particular functional shape of the trained discriminators in high dimensional spaces, which can easily make training stuck or push probability mass in the wrong direction, towards that of higher concentration than that of the data generating distribution. We introduce several ways of regularizing the objective, which can dramatically stabilize the training of GAN models. We also show that our regularizers can help the fair distribution of probability mass across the modes of the data generating distribution, during the early phases of training and thus providing a unified solution to the missing modes problem.

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

In Signal Processing

Citations in the corpus

Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, background): “Previous work on GANs focused on improving stability and mode dropping issues (Metz et al., 2016; Che et al., 2016; Arjovsky et al., 2017; Gulrajani et al., 2017; Roth et al., 2017), which are believed to be the main difficulties in training GANs.”
Unrolled Generative Adversarial Networks (2016, methodology): “Contrary to (Che et al., 2016), we found that reasonably sized models (such as the one used in Section 3.”
Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “1, we show additional metrics on generated sentences, which show low diversity but high quality for RKL-DPG, compared to other f -DPGs, suggesting it captures a subset of the target distribution (“mode collapse”), as commonly observed in other generative models (Huszar, 2015; Che et al., 2017; Mesch”
Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields (2017, methodology): “To avoid mode collapsing without batch normalization, several methods have been proposed [5, 33, 38].”
Gradient descent GAN optimization is locally stable (2017, background): “A wide variety of optimization heuristics and architectures have also been proposed to address challenges such as mode collapse [Salimans et al., 2016, Metz et al., 2016, Che et al., 2016, Radford et al., 2015].”
Self-Attention Generative Adversarial Networks (2018, background): “…works have attempted to stabilize the GAN training dynamics and improve the sample diversity by designing new network architectures [22, 37, 10], modifying the learning objectives and dynamics [1, 27, 15, 3, 39], adding regularization methods [7, 16] and introducing heuristic tricks [26, 19].”
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks (2016, background): “Several techniques [23, 29, 18, 1, 3] have been proposed to stabilize the training process and generate compelling results.”
Least Squares Generative Adversarial Networks (2016, background): “Many works have been proposed to address this problem by analyzing the objective functions of GANs [2, 3, 19, 23, 24].”
A Note on the Inception Score (2018, methodology): “, 2016), Mode Score (Che et al., 2016) and Fréchet Inception Distance (FID) (Heusel et al.”
StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks (2017, methodology): “In order to stabilize the GANs’ training process and improve sample diversity, several methods tried to address the challenges by proposing new network architectures [32], introducing heuristic tricks [40] or modifying the learning objectives [1], [3], [5].”
GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium (2017, methodology): “Different methods have been proposed to avoid mode collapsing [10, 39].”
Geometric GAN (2017, methodology): “The synthetic data consists of 100K data points generated from a mixture of 25 Gaussians, akin to the data that have been used for describing mode collapsing behaviors of GANs [19, 20, 21].”
InfoMax-GAN: Improved Adversarial Image Generation via Information Maximization and Contrastive Learning (2020, background): “These works often focused on assessing the number of modes recovered by a GAN based on synthetic datasets [48], [51], [52].”
Task Agnostic Continual Learning via Meta Learning (2019, background): “…ﬁnd a single φ that maximizes both l Thanh-Tung & Tran (2019) and Liang et al. (2019) showed that a notorious problem for GAN training called mode collapse (Che et al., 2016) is interrelated with catastrophic forgetting and can cause the training process to never converge, since the generator is…”
Maximum-Likelihood Augmented Discrete Generative Adversarial Networks (2017, background): “When D is correctly trained but not optimal in the sense of Theorem 3.1, when m → ∞ , we almost surely have ∀ 0 ≤ N ≤ T , 13: end for Second, this normalization scheme makes our model robust to mode missing, which is a common failure pattern when training GANs (Che et al., 2016).”
Stabilizing Training of Generative Adversarial Networks through Regularization (2017, background): “This includes the work of [7] which incorporates a supervised training signal as a regularizer on top of the discriminator target.”
VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning (2017, methodology): “As the true locations of the modes in this data are unknown, the number of modes are estimated using a trained classifier as described originally in [1].”