The Unusual Effectiveness of Averaging in GAN Training

By Yasin Yazici, Chuan-Sheng Foo, Stefan Winkler, Kim-Hui Yap, Georgios Piliouras, Vijay Ramaseshan Chandrasekhar. 2018

183 citations recorded by the source corpus.

We examine two different techniques for parameter averaging in GAN training. Moving Average (MA) computes the time-average of parameters, whereas Exponential Moving Average (EMA) computes an exponentially discounted sum. Whilst MA is known to lead to convergence in bilinear settings, we provide the -- to our knowledge -- first theoretical arguments in support of EMA. We show that EMA converges to limit cycles around the equilibrium with vanishing amplitude as the discount parameter approaches one for simple bilinear games and also enhances the stability of general GAN training. We establish experimentally that both techniques are strikingly effective in the non-convex-concave GAN setting as well. Both improve inception and FID scores on different architectures and for different GAN objectives. We provide comprehensive experimental results across a range of datasets -- mixture of Gaussians, CIFAR-10, STL-10, CelebA and ImageNet -- to demonstrate its effectiveness. We achieve state-of-the-art results on CIFAR-10 and produce clean CelebA face images.\footnote{~The code is available at \url{this https URL}}

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Large Scale GAN Training for High Fidelity Natural Image Synthesis (2018, methodology): “For evaluation, we employ moving averages of G ’s weights following Karras et al. (2018); Mescheder et al. (2018), with a decay of 0 .”
A Closer Look at the Optimization Landscapes of Generative Adversarial Networks (2019, neutral): “, 2019a), averaging (Yazıcı et al., 2019), optimism (Daskalakis et al.”
Which Training Methods for GANs do actually Converge? (2018, methodology): “Similarly to prior work (Karras et al., 2017; Yazici et al., 2018; Gidel et al., 2018), we use an exponential moving average 16 with decay 0 .”
A Variational Inequality Perspective on Generative Adversarial Networks (2018, neutral): “…t ← ∇ (cid:96) t ( ω t ) Option 2: Extrapolation from the past Load previously saved stochastic gradient: Sample new mini-batch and compute stochastic gradient: Update estimate of second moment: Compute bias corrected for ﬁrst and second moment: 2 ) Perform update step from the iterate at time t :”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, background): “SWA and related ideas have been successfully applied to a wide range of applications (see e.g. Athiwaratkun et al., 2019; Yazici et al., 2019; Nikishin et al., 2018).”
StarGAN v2: Diverse Image Synthesis for Multiple Domains (2019, methodology): “For evaluation, we employ exponential moving averages over parameters [21, 49] of all modules except D.”
There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average (2018, methodology): “We believe that application-specific analysis of the geometric properties of the training objective and optimization trajectories will further improve results over a wide range of application specific areas, including reinforcement learning with sparse rewards, generative adversarial networks (Yazıc”
Analyzing and Improving the Training Dynamics of Diffusion Models (2023, unknown): “Model averaging [27, 49, 57, 68, 72] is an indispensable technique in all high-quality image synthesis methods [2, 11, 23, 29, 31, 33, 46, 48, 54, 60, 63, 65].”
Dist-GAN: An Improved GAN Using Distance Constraints (2018, background): “Several GAN variants have been proposed [24, 22, 26, 4, 29] to solve these problems.”
MSG-GAN: Multi-Scale Gradients for Generative Adversarial Networks (2019, methodology): “previous works [28, 22, 2, 11, 19, 33, 14, 15, 37, 25].”
ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring for Minimax Problems and GANs (2019, unknown): “For GANs, however, last iterates or exponentially averaged iterates [69] (which assigns more weight to newer iterates) are used in practice.”
DeepPrivacy: A Generative Adversarial Network for Face Anonymization (2019, methodology): “We use an exponential running average for the weights of the generator as this improves overall image quality [28].”
MSG-GAN: Multi-Scale Gradient GAN for Stable Image Synthesis (2019, methodology): “This is a fundamental issue with GANs, and has been widely reported by previous works [24, 19, 2, 9, 16, 28, 12, 13, 31, 22].”
On Solving Minimax Optimization Locally: A Follow-the-Ridge Approach (2019, methodology): “To overcome this rotation behaviour of GDA, various modifications have been proposed, including averaging (Yazıcı et al., 2019), negative momentum (Gidel et al.”