A Large-Scale Study on Regularization and Normalization in GANs

By Karol Kurach, Mario Lucic, Xiaohua Zhai, Marcin Michalski, Sylvain Gelly. 2018

189 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

InfoMax-GAN: Improved Adversarial Image Generation via Information Maximization and Contrastive Learning (2020, background): “However, we note that FID can produce highly biased estimates [41], where using larger sample sizes can produce better scores, which can causes FID comparisons to be often mismatched [43] in practice.”
High-Fidelity Generative Image Compression (2020, background): “In pure generation, SpectralNorm was shown to reduce the variance of FID across multiple runs [24].”
Consistency Regularization for Generative Adversarial Networks (2019, methodology): “Consistency regularization has emerged as a gold-standard technique (Sajjadi et al., 2016; Laine & Aila, 2016; Zhai et al., 2019; Xie et al., 2019; Oliver et al., 2018; Berthelot et al., 2019) for semi-supervised learning on image data.”
Image Augmentations for GAN Training (2020, methodology): “The size of this dataset is suitable for a large scale study in GANs [27, 23].”
Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs (2020, neutral): “See [25, 22] for the survey of GANs.”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “…operation of BN between training and inference limits its usage in complex networks ( e.g . recurrent neural networks (RNNs) [20], [60], [61]) and tasks [29], [62], [63]; 2) BN suffers from the small-batch-size problem — its error increases rapidly as the batch size becomes smaller [22].”
Training GANs with Stronger Augmentations via Contrastive Discriminator (2021, methodology): “Due to the non-convex/stationary nature of the minimax game, however, training GANs in practice is often very unstable with an extreme sensitivity to many hyperparameters (Salimans et al., 2016; Lucic et al., 2018; Kurach et al., 2019).”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, background): “Karol Kurach [224] conducted a large-scale study on regularization and normalization in GANs.”
Investigating object compositionality in Generative Adversarial Networks (2018, methodology): “In both cases we conduct extensive grid searches covering on the order of 40-50 hyperparameter configurations for each dataset, using ranges that were previously found good for GANs (Lucic et al., 2018; Kurach et al., 2019).”
TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up (2021, background): “[17, 8] reported a large-scale study of GANs and observed that when serving as (generator) backbones, popular neural architectures perform comparably well across the considered datasets.”
Efficient Learning of Generative Models via Finite-Difference Score Matching (2020, background): “EBMs allow more flexible architectures [9, 11] with simpler compositionality [15, 41] compared to other explicit generative models [46, 16], and have better stability and mode coverage in training [30, 31, 71] compared to implicit generative models [10].”
Recent advances and clinical applications of deep learning in medical image analysis (2021, background): “For a more comprehensive and in-depth review, we refer interested readers to (Wiatrak et al., 2019; Saxena and Cao, 2020; Kurach et al., 2019).”
StyleSwin: Transformer-based GAN for High-resolution Image Generation (2021, background): “While early attempts focus on stabilizing the training dynamics via proper regularization [15,16,36,46,47] or adversarial loss designs [2, 25, 39, 45], remarkable performance leaps in recent prominent works mainly attribute to the architectural modifications that aim for stronger modeling capacity, ”
Improved Consistency Regularization for GANs (2020, methodology): “For data set preparation, we follow the detailed procedures in Kurach et al. (2019).”
Generative Adversarial Networks–Enabled Human–Artificial Intelligence Collaborative Applications for Creative and Design Industries: A Systematic Review of Current Approaches and Trends (2021, background): “…excellent recent surveys on the state of the art (Alqahtani et al., 2019; Hong et al., 2019; Pan et al., 2019; Khursheed et al., 2020), performance (Kurach et al., 2018), advances in image synthesis (Wu et al., 2017; Wang et al., 2019), and approaches to improving stability (Wiatrak et al., 2019).”
Deep Learning in Protein Structural Modeling and Design (2020, background): “The Wasserstein GAN (WGAN)(66) is a popular extension of GAN which introduces a Wasserstein-1 distance measure between distributions, leading to easier and more robust training.(67) An example of a GAN, in the context of protein modeling is learning the distribution of protein backbone distances to ”
TransGAN: Two Transformers Can Make One Strong GAN (2021, background): “(Lucic et al., 2018; Kurach et al., 2019) reported a large-scale study of GANs and observed that when serving as (generator) backbones, popular neural architectures perform comparably well across the considered datasets.”
Generative Adversarial Networks for Face Generation: A Survey (2022, background): “In addition, we review the proposed state-of-the-art regularization and normalization approaches [67, 74, 106, 108] for tackling them.”