Is Generator Conditioning Causally Related to GAN Performance?

By Augustus Odena, Jacob Buckman, Catherine Olsson, Tom B. Brown, Chris Olah, Christopher Olah, Colin Raffel, Ian J. Goodfellow. 2018

124 citations recorded by the source corpus.

Recent work (Pennington et al, 2017) suggests that controlling the entire distribution of Jacobian singular values is an important design consideration in deep learning. Motivated by this, we study the distribution of singular values of the Jacobian of the generator in Generative Adversarial Networks (GANs). We find that this Jacobian generally becomes ill-conditioned at the beginning of training. Moreover, we find that the average (with z from p(z)) conditioning of the generator is highly predictive of two other ad-hoc metrics for measuring the 'quality' of trained GANs: the Inception Score and the Frechet Inception Distance (FID). We test the hypothesis that this relationship is causal by proposing a 'regularization' technique (called Jacobian Clamping) that softly penalizes the condition number of the generator Jacobian. Jacobian Clamping improves the mean Inception Score and the mean FID for GANs trained on several datasets. It also greatly reduces inter-run variance of the aforementioned scores, addressing (at least partially) one of the main criticisms of GANs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

Analyzing and Improving the Image Quality of StyleGAN (2019, background): “These gradients should have close to an equal length regardless of w or the image-space direction, indicating that the mapping from the latent space to image space is well-conditioned [28].”
Large Scale GAN Training for High Fidelity Natural Image Synthesis (2018, background): “Relatedly Odena et al. (2018) analyze the condition number of the Jacobian of G and ﬁnd that performance is dependent on G ’s conditioning.”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, methodology): “Typical normalization layers used in the modulation blocks include batch normalization (Chen et al., 2019; Odena et al., 2018), adaptive instance normalization (Huang & Belongie, 2017; Karras et al., 2019a), and weight demodulation (Karras et al., 2019b) referred to the weight normalization…”
EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning (2019, methodology): “heir respective largest singular values, effectively restricting the Lipschitz constant of the network to one. Although this was originally proposed to be used only on the discriminator, recent works [54,36] suggest that generator can also beneﬁt from SN by suppressing sudden changes of parameter an”
Negative Momentum for Improved Game Dynamics (2018, background): “From another perspective, Odena et al. (2018) show that in a GAN setup, the average conditioning of the Jacobian of the generator becomes ill-conditioned during training.”
Self-Attention Generative Adversarial Networks (2018, background): “We argue that the generator can also beneﬁt from spectral normalization, based on recent evidence that the conditioning of the generator is an important causal factor in GAN performance [18].”
On Self Modulation for Generative Adversarial Networks (2018, methodology): “This result corroborates the observations in (Odena et al., 2018).”
Discriminator Rejection Sampling (2018, background): “Addressing these issues is an active area of research (Mao et al., 2017; Arjovsky et al., 2017; Gulrajani et al., 2017; Odena et al., 2018; Li et al., 2017).”
A Survey of Unsupervised Deep Domain Adaptation (2018, methodology): “Many methods have been proposed to resolve these training challenges using a variety of tricks [81, 90, 179, 207, 219, 229], network architecture choices [116, 190, 207], objective modifications [5, 15, 66, 87, 112, 120, 161, 163, 165, 176–178, 283], mixtures or ensembles [6, 63, 78, 93, 117, 170, 1”
TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing (2018, background): “This property has arguably contributed to the recent ‘reproducibility crisis’ in machine learning [9, 16, 24, 28, 30, 33, 34] – it’s tricky to make robust experimental conclusions about techniques that are hard to debug.”
Improved Consistency Regularization for GANs (2020, methodology): “Though motivated differently, this can be seen as related to the Jacobian Clamping technique from Odena et al. (2018) and diversity increase technique in Yang et al. (2019).”
Diversity-Sensitive Conditional Generative Adversarial Networks (2019, methodology): “Our method shares the similar motivation with (Odena et al., 2018) but employs a different objective function that simply maximizes the norm of the generator gradient with an optional upper-bound, which we found that works much more stable over a wide range of tasks with less number of hyper-paramet”
Orthogonal Jacobian Regularization for Unsupervised Disentanglement in Image Generation (2021, methodology): “Many recent studies have been given to incorporate the orthogonality for improving deep network training [5, 19, 31, 36, 43, 44].”
On the Learning Dynamics of Deep Neural Networks (2018, background): “Their paper is part of a larger trend focusing on the spectra of neural network weight matrices and their evolution during learning (Vorontsov et al., 2017; Odena et al., 2018; Pennington et al., 2017).”
AGAN: Towards Automated Design of Generative Adversarial Networks (2019, background): “Another line of works focus on regularization and normalization techniques, especially the Lipschitz continuity of the discriminator and the conditioning of the generator [8].”
Whitening and Coloring Batch Transform for GANs (2018, background): “Specifically, Odena et al. (2018) show the relation between stability of GAN training and conditioning of the Jacobian.”