The GAN Landscape: Losses, Architectures, Regularization, and Normalization

By Karol Kurach, Mario Lucic, Xiaohua Zhai, Marcin Michalski, Sylvain Gelly. 2018

159 citations recorded by the source corpus.

Generative adversarial networks (GANs) are a class of deep generative models which aim to learn a target distribution in an unsupervised fashion. While they were successfully applied to many problems, training a GAN is a notoriously challenging task and requires a significant amount of hyperparameter tuning, neural architecture engineering, and a non-trivial amount of "tricks". The success in many practical applications coupled with the lack of a measure to quantify the failure modes of GANs resulted in a plethora of proposed losses, regularization and normalization schemes, and neural architectures. In this work we take a sober view of the current state of GANs from a practical perspective. We reproduce the current state of the art and go beyond fairly exploring the GAN landscape. We discuss common pitfalls and reproducibility issues, open-source our code on Github, and provide pre-trained models on TensorFlow Hub.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

Are GANs Created Equal? A Large-Scale Study (2017, background): “Finally, instead of random search it might be beneficial to apply (carefully tuned) sequential Bayesian optimization which is computationally beyond the scope of this study, but nevertheless a great candidate for future work [16].”
Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, positive): “In the quest to draw sound and robust conclusions, we join other recent papers in highlighting the under-acknowledged existence of conclusion replication failure [47, 36, 39, 26, 19].”
A Style-Based Generator Architecture for Generative Adversarial Networks (2018, methodology): “We do not modify the discriminator or the loss function in any way, and our work is thus orthogonal to the ongoing discussion about GAN loss functions, regularization, and hyper-parameters [24, 45, 5, 40, 44, 36].”
A U-Net Based Discriminator for Generative Adversarial Networks (2020, background): “While there are many variations of the GAN objective function and its network architectures [23, 30], in this paper we focus on improving the discriminator network.”
Self-Supervised GANs via Auxiliary Rotation Loss (2018, background): “Robustness across hyperparameters GANs are fragile; changes to the hyperparameter settings have a substantial impact to their performance [30, 32].”
Assessing Generative Models via Precision and Recall (2018, background): “Some proposed measures, such as Inception Score (IS) [20] and Fréchet Inception Distance (FID) [9], have shown promising results in practice.”
Recent Progress on Generative Adversarial Networks (GANs): A Survey (2019, methodology): “Reference [39] suggested that FID and IS can be used as the auxiliary evaluation metrics with MS-SSIM for testing sample diversity.”
AutoGAN: Neural Architecture Search for Generative Adversarial Networks (2019, background): “Referring to recent GAN benchmark studies [39, 32], we consider extending our search space with the attention/self-attention [64], the style-based generator [27], the relativistic discriminator [25] and various losses such as the Wasserstein loss [3], among others.”
On Self Modulation for Generative Adversarial Networks (2018, neutral): “We consider loss functions, architectures, discriminator regularization/normalization strategies, and a variety of hyperparameter settings collected from recent studies (Radford et al., 2016; Gulrajani et al., 2017; Miyato et al., 2018; Lucic et al., 2018; Kurach et al., 2018).”
High-Fidelity Image Generation With Fewer Labels (2019, background): “Architecture and hyperparameters GANs are notoriously unstable to train and their performance strongly depends on the capacity of the neural architecture, optimization hyperparameters, and appropriate regularization (Lucic et al., 2018; Kurach et al., 2019).”
Adversarial Latent Autoencoders (2020, methodology): “For instance, we can seamlessly include independent sources of stochasticity, which have proven essential for generating image details, or can leverage recent improvements on GAN loss functions, regularization, and hyperparameters tuning [2, 30, 38, 34, 36, 3].”
Implicit Generation and Generalization in Energy-Based Models (2019, background): “Since our generator is implicitly deﬁned by the original energy function, there is no need for an objective to train the generator, unlike in GANs.”
Generative Adversarial Network in Medical Imaging: A Review (2018, background): “For a more indepth review and empirical evaluation of these different variants of GAN, we refer the reader to (Huang et al., 2018; Creswell et al., 2018; Kurach et al., 2018).”
Data collection and quality challenges in deep learning: a data-centric AI perspective (2021, methodology): “For data augmentation, a popular method for generating data in the machine learning community is generative adversarial networks (GANs) [66, 67, 95].”
A Domain Agnostic Measure for Monitoring and Evaluating GANs (2018, methodology): “We use the optimal hyperparameters suggested in [16].”