AdaGAN: Boosting Generative Models

By Ilya O. Tolstikhin, Sylvain Gelly, Olivier Bousquet, Carl-Johann Simon-Gabriel, Bernhard Schölkopf. 2017

232 citations recorded by the source corpus.

Generative Adversarial Networks (GAN) are an effective method for training generative models of complex data such as natural images. However, they are notoriously hard to train and can suffer from the problem of missing modes where the model is not able to produce examples in certain regions of the space. We propose an iterative procedure, called AdaGAN, where at every step we add a new component into a mixture model by running a GAN algorithm on a re-weighted sample. This is inspired by boosting algorithms, where many potentially weak individual predictors are greedily aggregated to form a strong composite predictor. We prove analytically that such an incremental procedure leads to convergence to the true distribution in a finite number of steps if each step is optimal, and convergence at an exponential rate otherwise. We also illustrate experimentally that this procedure addresses the problem of missing modes.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

Imperfect ImaGANation: Implications of GANs Exacerbating Biases on Facial Data Augmentation and Snapchat Selfie Lenses (2020, methodology): “We show experiments on two other GAN architectures designed to address mode collapse – Wasserstein GAN (Arjovsky et al., 2017) and AdaGAN (Tolstikhin et al., 2017) – in the appendix.”
Generalization and Equilibrium in Generative Adversarial Nets (GANs) (2017, methodology): “ADAGAN (Tolstikhin et al., 2017) uses ideas similar to boosting and maintains weights on training examples.”
Boundary-Seeking Generative Adversarial Networks (2017, background): “Tolstikhin et al. (2017) proposed to iteratively re-weight training examples in order to train a series of GAN’s, and showed that this procedure signiﬁcantly stabilizes learning.”
PacGAN: The Power of Two Samples in Generative Adversarial Networks (2017, background): “Mode collapse is a phenomenon commonly reported in the GAN literature [17], [37], [38], [39], [40], which can refer to two distinct concepts: (i) the generative model loses some modes that are present in the samples of the target distribution.”
An Online Learning Approach to Generative Adversarial Networks (2017, methodology): “The work of Tolstikhin et al. (2017) also uses a similar mixture approach based on boosting.”
Pros and Cons of GAN Evaluation Measures (2018, methodology): “[80] proposed to use the probability mass of the real data “covered” by the model distribution Pmodel as a metric.”
Handling Incomplete Heterogeneous Data using VAEs (2018, unknown): “L G et al. (2018), DVAE++ (Vahdat et al., 2018), Shape Variational Autoencoder (Nash and Williams, 2017) and GAN-style objective functions (f-GAN (Nowozin et al., 2016), DR-GAN (Tran et al., 2019), Wasserstein GANs (Gulrajani et al., 2017), MMD-GAN (Li et al., 2015), Gated-GAN (Chen et al., 2019), A”
A Survey of Unsupervised Deep Domain Adaptation (2018, methodology): “using a variety of tricks [77, 155, 181, 192, 201], network architecture choices [101, 166, 181], objective modifications [4, 13, 54, 74, 97, 105, 138, 140, 142, 152– 154, 244], mixtures or ensembles [5, 51, 66, 80, 102, 147, 157, 207, 232], maximum mean discrepancy (MMD) [14, 52, 117, 120, 200], ma”
PassGAN: A Deep Learning Approach for Password Guessing (2017, unknown): “[25], there have been several improvements on GANs [3, 4, 6–8, 10, 14, 16, 27, 35, 36, 42, 47, 53, 55, 56, 58–60, 63, 66, 69, 74, 76, 80, 87, 90, 94, 96], where each new paper provides novel improvements in the domain.”
Online Adaptative Curriculum Learning for GANs (2018, background): “2018), multiple discriminators (Neyshabur, Bhojanapalli, and Chakrabarti 2017; Juefei-Xu, Boddeti, and Savvides 2017), or multiple generators (Tolstikhin et al. 2017; Hoang et al. 2017; Kwak and Zhang 2016).”
MGAN: Training Generative Adversarial Nets with Multiple Generators (2018, methodology): “Tolstikhin et al. (2017), however, show that this heuristic fails to address the mode collapsing problem.”
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium (2017, background): “For special GAN variants, convergence c n be p oved under c rtain assumptions [38, 22, 55], as can local stability [45] (s e also Appe dix Section A2.”
Applications of Generative Adversarial Networks (GANs): An Updated Review (2019, background): “Diversity in the generator can be increased by practical hacks to balance the distribution of samples produced by the discriminator for real and fake batches, or by employing multiple GANs to cover the different modes of the probability distribution [107].”
Generative Adversarial Networks (GANs) (2020, background): “istribution which previous GANs could not captured efficiently. However, additive procedure of cGANs is not motivated by the theoretical analysis of optimality conditions. Further, Tolstikhin, et al. [42] have shown through the empirical analysis that cGANs heuristic fails to address the mode collap”
Evaluation of Mode Collapse in Generative Adversarial Networks (2018, methodology): “Assuming that the data has 10 modes (i.e., digits 1-10) [10], a pretrained MNIST classiﬁer is used to select samples with high conﬁdence classiﬁcations (i.e., score > 0 .”
Style Aggregated Network for Facial Landmark Detection (2018, background): “Following that, many researchers devoted great efforts to improve this research topic regarding theory [2, 8, 25, 35, 54] and applications [36, 41, 50, 71].”
A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, background): “ve also been proposed such as WGAN [23], Multi-Batch Discriminator (MBD) [196], Energy-Based GAN (EBGAN) [22]), Unrolled GAN (UGAN) [197], Deep Regret Analytic GAN (DRAGAN) [45], Adaptive GAN(AdaGAN) [231], Mode Regularized GAN (MRGAN) [232], MultiAgent Diverse GAN (MAD-GAN) [233] that show how thes”
Evolutionary Generative Adversarial Networks (2018, methodology): “[48] applied boosting techniques to train a mixture of generators by continually adding new generators to the mixture.”