Reducing Noise in GAN Training with Variance Reduced Extragradient

By Tatjana Chavdarova, Gauthier Gidel, Franccois Fleuret, Simon Lacoste-Julien. 2019

153 citations recorded by the source corpus.

We study the effect of the stochastic gradient noise on the training of generative adversarial networks (GANs) and show that it can prevent the convergence of standard game optimization methods, while the batch version converges. We address this issue with a novel stochastic variance-reduced extragradient (SVRE) optimization algorithm, which for a large class of games improves upon the previous convergence rates proposed in the literature. We observe empirically that SVRE performs similarly to a batch method on MNIST while being computationally cheaper, and that SVRE yields more stable GAN training on standard datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Optimization & Training

Citations in the corpus

Stochastic Hamiltonian Gradient Methods for Smooth Games (2020, methodology): “Similarly to Chavdarova et al. (2019), and 0 otherwise, and We show the convergence of the different algorithms in Fig.”
Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (2019, methodology): “[15] propose a challenging stochastic bilinear game as follows:”
First Analysis of Local GD on Heterogeneous Data (2019, methodology): “Second, there exists at least one method, whose convergence is guaranteed under bounded gradients [6], but in practice the method diverges [3, 12].”
Tighter Theory for Local SGD on Identical and Heterogeneous Data (2019, methodology): “Second, there exists at least one method, whose convergence is guaranteed under bounded variance (Juditsky et al., 2011), but in practice the method diverges (Chavdarova et al., 2019; Mishchenko et al., 2019).”
Revisiting Stochastic Extragradient (2019, methodology): “Since it uses two independent samples, it will rarely approximate the implicit update, so it is rather not surprisingly that it fails even on bilinear problems, as it was observed in [1].”
ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring for Minimax Problems and GANs (2019, unknown): “We note that there are other mechanisms similar to optimism and anchoring such as “prediction” [68], “negative momentum” [19], and “extragradient” [27, 65, 8].”
Last Iterate is Slower than Averaged Iterate in Smooth Convex-Concave Saddle Point Problems (2020, background): “The EG and OGDA algorithms have additionally received signiﬁcant recent attention due to their ability to improve the training dynamics in GANs (Chavdarova et al. (2019); Gidel et al. (2018a,b); Liang and Stokes (2018); Yadav et al. (2017); Daskalakis et al. (2017)).”
Stochastic Recursive Gradient Descent Ascent for Stochastic Nonconvex-Strongly-Concave Minimax Problems (2020, methodology): “[9], Palaniappan and Bach [34], extend SVRG [20, 51] and SAGA [11] to solving strongly-convex-strongly-concave minimax problem in the finite-sum case, and established a linear convergence.”
Adaptive and Universal Algorithms for Variational Inequalities with Optimal Convergence (2020, unknown): “…x ∗ ) ; x ∗ − x i ≤ 0 ∀ x ∈ X More recently, algorithms developed in this framework are also applied to non-convex problems including optimizing generative adversarial networks (GANs) (Daskalakis et al. 2018; Yadav et al. 2018; Chavdarova et al. 2019; Gidel et al. 2019; Mertikopoulos et al. 2019).”
Explore Aggressively, Update Conservatively: Stochastic Extragradient Methods with Variable Stepsize Scaling (2020, methodology): “[2] showed that if EG is run with a constant stepsize and noise with unbounded variance, the method’s iterates actually diverge at a geometric rate.”
Tight last-iterate convergence rates for no-regret learning in multi-player games (2020, background): “Moreover, this shortcoming is not merely an artifact of the analysis of [LZMJ20]: it has been observed (e.g. [DISZ17, GBV + 18]) that in bilinear games, the players’ actions in online gradient descent not only fail to converge, but diverge to inﬁnity.”
Variance Reduction for Matrix Games (2019, background): “In contrast, only a few works apply variance reduction to finite-sum minimax problems [3, 30, 6, 23], and the potential gains from variance reduction are not well-understood.”
On the convergence of single-call stochastic extra-gradient methods (2019, methodology): “…techniques are no longer applicable because the contraction property cannot be maintained in the presence of noise; in fact, unless the problem at hand is amenable to variance reduction – e.g., as in [6, 9, 21] – geometric convergence is not possible if the noise process is even weakly isotropic.”
A Unified Analysis of Extra-gradient and Optimistic Gradient Methods for Saddle Point Problems: Proximal Point Approach (2019, methodology): “More recently Palaniappan & Bach (2016) uses a variance reduced version of the proximal gradient method and Chavdarova et al. (2019) uses a variance reduced version of the EG method to solve Problem (1) when the function is strongly convex in x and strongly concave in y and the function has a ﬁnite…”