On the Effects of Batch and Weight Normalization in Generative Adversarial Networks

By Sitao Xiang, Hao Li. 2017

90 citations recorded by the source corpus.

Generative adversarial networks (GANs) are highly effective unsupervised learning frameworks that can generate very sharp data, even for data such as images with complex, highly multimodal distributions. However GANs are known to be very hard to train, suffering from problems such as mode collapse and disturbing visual artifacts. Batch normalization (BN) techniques have been introduced to address the training. Though BN accelerates the training in the beginning, our experiments show that the use of BN can be unstable and negatively impact the quality of the trained model. The evaluation of BN and numerous other recent schemes for improving GAN training is hindered by the lack of an effective objective quality measure for GAN models. To address these issues, we first introduce a weight normalization (WN) approach for GAN training that significantly improves the stability, efficiency and the quality of the generated samples. To allow a methodical evaluation, we introduce squared Euclidean reconstruction error on a test set as a new objective measure, to assess training performance in terms of speed, stability, and quality of generated samples. Our experiments with a standard DCGAN architecture on commonly used datasets (CelebA, LSUN bedroom, and CIFAR-10) indicate that training using WN is generally superior to BN for GANs, achieving 10% lower mean squared loss for reconstruction and significantly better qualitative results than BN. We further demonstrate the stability of WN on a 21-layer ResNet trained with the CelebA data set. The code for this paper is available at this https URL

Subfields: Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Analyzing and Improving the Image Quality of StyleGAN (2019, background): “Prior work has identified weight normalization as beneficial in the context of GAN training [38].”
Spectral Normalization for Generative Adversarial Networks (2018, background): “However, recent works in the ﬁeld of GANs (Salimans et al., 2016; Xiang & Li, 2017) found their another raison d’etat as a regularizer of discriminators, and succeeded in improving the performance of the original.”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “Great research interest has been drawn on the evaluation of GANs (DeVries et al., 2019; Gurumurthy et al., 2017; Sajjadi et al., 2018; Snell et al., 2017; Xiang & Li, 2017).”
The Effectiveness of Data Augmentation in Image Classification using Deep Learning (2017, positive): “Batch normalization can be applied to any layer within the net and hence is very effective [12], even when used in generative adversarial networks, such as CycleGAN ([17].”
Pros and Cons of GAN Evaluation Measures (2018, methodology): “Since it is not possible to directly infer the optimal z from x, Xiang and Li [93] used the following alternative method.”
Freeze and Chaos for DNNs: an NTK view of Batch Normalization, Checkerboard and Boundary Effects (2019, background): “The impact of architecture is very apparent in GANs [10]: their results are heavily affected by the architecture of the generator and discriminator [23, 30, 4, 14] and the training may fail without BN [3, 26].”
Decorrelated Batch Normalization (2018, background): “Specifically, the learnable parameters can be merged into the following ReLU activation [36], resulting in the Translated ReLU (TReLU) [52].”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, background): “[254] leveraged weight normalization to effectively improve the training performance of GANs.”
Self-supervised GAN: Analysis and Improvement with Multi-class Minimax Game (2019, methodology): “We report the best FID attained in 300K iterations as in [45, 25, 42, 47].”
A Spectral Regularizer for Unsupervised Disentanglement (2018, background): “Li, 2017) to the ﬁnal convolutional features, followed by two fully-connected layers, one for each statistic.”
Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification (2017, methodology): “Weight normalization with translated ReLU was introduced by Xiang & Li (2017) in the context of generative adversarial networks (Goodfellow et al. (2014)) where it was shown to achieve superior results to BN.”
On the Evaluation of Conditional GANs (2019, background): “Most of these metrics either focus on the separability between generated images and real images [21, 30, 50, 16], compute the distance between distributions [10, 12, 2], assess sample quality and diversity from conditional or marginal distributions [33, 11, 55], measure the similarity between genera”
On Data Augmentation for GAN Training (2020, methodology): “The best FID attained in 300K iterations are reported as in [54], [55], [35], [56].”
Backpropagation for Implicit Spectral Densities (2018, methodology): “The generator and discriminator have ﬁve layers each, and use translated LeakyReLU activations [Xiang and Li, 2017].”