How to Train Your DRAGAN

By Naveen Kodali, Jacob D. Abernethy, James Hays, Zsolt Kira. 2017

89 citations recorded by the source corpus.

Generative Adversarial Networks have emerged as an effective technique for estimating data distributions. The basic setup consists of two deep networks playing against each other in a zero-sum game setting. However, it is not understood if the networks reach an equilibrium eventually and what dynamics makes this possible. The current GAN training procedure, which involves simultaneous gradient descent, lacks a clear game-theoretic justification in the literature. In this paper, we introduce regret minimization as a technique to reach equilibrium in games and use this to justify the success of simultaneous GD in GANs. In addition, we present a hypothesis that mode collapse, which is a common occurrence in GAN training, happens due to the existence of spurious local equilibria in non-convex games. Motivated by these insights, we develop an algorithm called DRAGAN that is fast, simple to implement and achieves competitive performance in a stable fashion across different architectures (150 random setups), datasets (MNIST, CIFAR-10, and CelebA), and divergence measures with almost no hyperparameter tuning. We show significant improvements over the recently proposed Wasserstein GAN variants.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

Progressive Growing of GANs for Improved Quality, Stability, and Variation (2017, background): “GANs produce sharp images, albeit only in fairly small resolutions and with somewhat limited variation, and the training continues to be unstable despite recent progress (Salimans et al., 2016; Gulrajani et al., 2017; Berthelot et al., 2017; Kodali et al., 2017).”
Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence At Every Step (2017, methodology): “To assess the effectiveness of the gradient penalty on standard datasets for the non-saturating GAN formulation, we train a non-saturating GAN, a non-saturating GAN with the gradient penalty introduced by [9] (denoted by GAN-GP), a non-saturating GAN with the gradient penalty introduced by [12] (den”
Which Training Methods for GANs do actually Converge? (2018, methodology): “As a result, a lot of recent research has focused on ﬁnding better training algorithms (Arjovsky et al., 2017; Gulrajani et al., 2017; Kodali et al., 2017; Sønderby et al., 2016; Roth et al., 2017) for GANs as well as gaining better theoretically understanding of their training dynamics (Arjovsky…”
The relativistic discriminator: a key element missing from standard GAN (2018, background): “The constraint of WGAN-GP (i.e., discriminator with gradient norm equal to 1 around real and fake data) has been shown to be beneﬁcial in SGAN [Fedus et al., 2017] (along with a very similar gradient penalty by Kodali et al. [2017]).”
How Generative Adversarial Networks and Its Variants Work : An Overview of GAN (2017, background): “[35]: Standard GAN, [73]: LSGAN, [138]: EBGAN, [6]: WGAN, [97]: Improved WGAN, [80]: McGAN, [10]: CramerGAN, [65]: MMDGAN, [79]: FisherGAN, [76]: Unrolled GAN, [14]: MRGAN, [59]: DRAGAN, [77]: Conditional GAN, [16]: InfoGAN, [106]: ss-InfoGAN, [27]: ALI, [25]: BiGAN, [116]: AGE, [58]: ARAE, [61]: VA”
On the regularization of Wasserstein GANs (2017, unknown): “A qualitative way to evaluate the learned critic function for a two-dimensional data set is by displaying its level sets, as it was done by Gulrajani et al. (2017) and Kodali et al. (2017).”
Parametric Adversarial Divergences are Good Task Losses for Generative Modeling (2017, background): “Recently, it was shown that neural adversarial divergences other than the Wasserstein can also be made stable by regularizing the discriminator properly (Kodali et al., 2017; Roth et al., 2017).”
MedGAN: Medical Image Translation using GANs (2018, background): “Naveen Kodali [32] pointed out that the excellent performance of W-GAN is due to the regularization of discriminator through the gradient penalty.”
A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, methodology): “To curb this issue, Style-Based Generator Architecture for Generative Adversarial Network (StyleGAN) [31] redesigns the architecture of the generator network, makes it possible to control the image synthesis through scale-specific amendments to the styles with-out compromising the generated image…”
Towards the High-quality Anime Characters Generation with Generative Adversarial Networks (2017, methodology): “Here, we use DRAGAN [10] as GAN model which are fast and stable under several network architecture.”
Towards the Automatic Anime Characters Creation with Generative Adversarial Networks (2017, methodology): “Here, we use DRAGAN proposed by Kodali et al.[11] as the basic GAN model.”
Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow (2018, background): “These challenges have led to widespread interest in a variety of stabilization methods for adversarial learning algorithms (Arjovsky et al., 2017; Kodali et al., 2017; Berthelot et al., 2017).”
SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis (2018, methodology): “In practice, we use DRAGAN loss [31] in order to stabilize training and use focal loss [36] as classification loss.”
How Generative Adversarial Networks and Their Variants Work (2017, methodology): “Deep regret analytic GAN (DRAGAN) [61] suggests that a mode collapse occurs due to the existence of a spurious local Nash Equilibrium in the nonconvex problem.”
Online Adaptative Curriculum Learning for GANs (2018, background): “ion corresponds to the generator producing samples only from a small subset of the true distribution. Among numerous works on GANs, some studied the complex dynamics of min-max optimization problems (Kodali et al., 2017, Mescheder et al., 2017) to stabilize the training, while others proposed an alt”
BAGAN: Data Augmentation with Balancing GAN (2018, methodology): “r and generator in the GAN with an autoencoder. This enables us to start training the generative model from a more stable point and helps mitigating convergence problems arising with traditional GANs [4, 5, 6]. Additionally, we leverage the autoencoder to learn class-conditioning in the latent space”
An Improved Evaluation Framework for Generative Adversarial Networks (2018, methodology): “It is worth noting that the recently proposed DRAGAN [25] and WGAN-GP [26] outperform other models by a relatively large margin.”
Dropout-GAN: Learning from a Dynamic Ensemble of Discriminators (2018, background): “Training instability has been noticeably reported as one of GANs biggest problems [3, 4, 8, 21, 33].”