Spectral Normalization for Generative Adversarial Networks
By Takeru Miyato, Toshiki Kataoka, Masanori Koyama, Yuichi Yoshida. 2018
5,083 citations recorded by the source corpus.
One of the challenges in the study of generative adversarial networks is the instability of its training. In this paper, we propose a novel weight normalization technique called spectral normalization to stabilize the training of the discriminator. Our new normalization technique is computationally light and easy to incorporate into existing implementations. We tested the efficacy of spectral normalization on CIFAR10, STL-10, and ILSVRC2012 dataset, and we experimentally confirmed that spectrally normalized GANs (SN-GANs) is capable of generating images of better or equal quality relative to the previous training stabilization techniques.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- Mohit Bansal
- Quoc V. Le
- Jianfeng Gao
- Dacheng Tao
- Percy Liang
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Andrea Vedaldi
In Generative Models
- Stefano Ermon
- Jun-Yan Zhu
- Eli Shechtman
- Richard Zhang
- Arash Vahdat
- Yoshua Bengio
- Jiaming Song
- Karsten Kreis
Citations in the corpus
- Denoising Diffusion Probabilistic Models (2020, neutral): “al Diffusion (original) [50] 5:40 Gated PixelCNN [56] 4:60 65:93 3:03 (2:90) Sparse Transformer [7] 2:80 PixelIQN [40] 5:29 49:46 EBM [11] 6:78 38:2 NCSNv2 [53] 31:75 NCSN [52] 8:87 0:12 25:32 SNGAN [36] 8:22 0:05 21:7 SNGAN-DDLS [4] 9:09 0:10 15:42 StyleGAN2 + ADA [28] 9:74 0:05 3:26 Ours (L, fixed ”
- Evaluating Lossy Compression Rates of Deep Generative Models (2020, methodology): “4b), we experimented with different GAN models such as DCGAN (Rad-ford et al., 2015), DCGAN with Gradient Penalty (GP-GAN) (Gulrajani et al., 2017), Spectral Normalization (SN-GAN) (Miyato et al., 2018), and DCGAN with Binarized Representation Entropy Regularization (BRE-GAN) (Cao et al., 2018).”
- Improved Techniques for Training Score-Based Generative Models (2020, background): “4 SNGAN [20] 8.”
- Training Generative Adversarial Networks with Limited Data (2020, methodology): “The increasingly impressive results of generative adversarial networks (GAN) [14, 32, 31, 5, 19, 20, 21] are fueled by the seemingly unlimited supply of images available online.”
- Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, methodology): “On CIFAR-10, DDLS substantially improves the Inception Score of an off-the-shelf pre-trained SN-GAN (Miyato et al., 2018) from 8.”
- Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One (2019, neutral): “, 2019), Glow (Kingma & Dhariwal, 2018), IGEBM (Du & Mordatch, 2019), SNGAN (Miyato et al., 2018), NCSN (Song & Ermon, 2019) Figure 2: CIFAR10 classconditional samples.”
- Analyzing and Improving the Image Quality of StyleGAN (2019, background): “Skip connections [29, 19], residual networks [15, 14, 26], and hierarchical methods [6, 41, 42] have proven highly successful also in the context of generative methods.”
- Generative Modeling by Estimating Gradients of the Data Distribution (2019, methodology): “32 on CIFAR-10 is also comparable to top existing models, such as SNGAN [36].”
- A Style-Based Generator Architecture for Generative Adversarial Networks (2018, methodology): “We do not modify the discriminator or the loss function in any way, and our work is thus orthogonal to the ongoing discussion about GAN loss functions, regularization, and hyper-parameters [23, 41, 4, 37, 40, 33].”
- Large Scale GAN Training for High Fidelity Natural Image Synthesis (2018, methodology): “Of particular relevance to our work is Spectral Normalization (Miyato et al., 2018), which enforces Lipschitz continuity on D by normalizing its parameters with running estimates of their first singular values, inducing backwards dynamics that adaptively regularize the top singular direction.”
- FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models (2018, background): “Various forms of regularization such as weight decay and spectral normalization (Miyato et al., 2018) can be used to reduce the this quantity, but their use tends to hurt performance slightly.”
- Generative Adversarial Networks (2021, neutral): “SNGAN [46] proposes to add weight normalization to stabilize the training of the discriminator.”
- Projected GANs Converge Faster (2021, methodology): “Each discriminator Dl uses a simple convolutional architecture with spectral normalization [37] at each convolutional layer.”
- Diffusion Models Beat GANs on Image Synthesis (2021, background): “Furthermore, GANs are often difficult to train, collapsing without carefully selected hyperparameters and regularizers [8, 47, 7].”
- Dual Contradistinctive Generative Autoencoder (2020, neutral): “3 SNGAN† [52] (from [7]) 16.”
- VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, methodology): “we do not use spectral normalization (Miyato et al., 2018); instead, we use weight normalization with data-dependent initialization (Salimans & Kingma, 2016).”
- NVAE: A Deep Hierarchical Variational Autoencoder (2020, methodology): “Formally, we add the term L SR = λ i s ( i ) to L VAE , where s ( i ) is the largest singular value of the i th conventional layer, estimated using a single power iteration update [56, 57].”