A Variational Inequality Perspective on Generative Adversarial Networks

By Gauthier Gidel, Hugo Berard, Pascal Vincent, Simon Lacoste-Julien. 2018

409 citations recorded by the source corpus.

Generative adversarial networks (GANs) form a generative modeling approach known for producing appealing samples, but they are notably difficult to train. One common way to tackle this issue has been to propose new formulations of the GAN objective. Yet, surprisingly few studies have looked at optimization methods designed for this adversarial training. In this work, we cast GAN optimization problems in the general variational inequality framework. Tapping into the mathematical programming literature, we counter some common misconceptions about the difficulties of saddle point optimization and propose to extend techniques designed for variational inequalities to the training of GANs. We apply averaging, extrapolation and a computationally cheaper variant that we call extrapolation from the past to the stochastic gradient method (SGD) and Adam.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

Stochastic Gradient Descent-Ascent and Consensus Optimization for Smooth Games: Convergence Analysis under Expected Co-coercivity (2021, methodology): “Also, we do not use the full power of variational inequalities that usually have constraints [Harker and Pang, 1990], but standard algorithms for (1) are coming from this literature [Gidel et al., 2018].”
Stochastic Hamiltonian Gradient Methods for Smooth Games (2020, neutral): “…of the problem has been studied in a number of classic (Korpelevich, 1976; Nemirovski, 2004) and recent results (Mescheder et al., 2017; Ibrahim et al., 2019; Gidel et al., 2018; Daskalakis et al., 2018; Gidel et al., 2019; Mokhtari et al., 2020; Azizian et al., 2020a;b) in various settings.”
Adversarial Example Games (2020, methodology): “This is unsurprising, due to the inherent difﬁculties in saddle point optimization [26, 5]”
A Tight and Unified Analysis of Extragradient for a Whole Spectrum of Differentiable Games (2019, methodology): “Gidel et al. (2019a) interpreted GANs as a variational inequality problem and derive a variant of extragradient which avoids "wasting" a gradient.”
Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (2019, methodology): “Note that using the same sample ik and step-size ⌘k for both steps [24] is important for the subsequent theoretical results.”
Reducing Noise in GAN Training with Variance Reduced Extragradient (2019, methodology): “Gidel et al. (2019b) formalize the notion of “adversarial component” of a game, which yields a rotational dynamics in gradients methods (oscillations in parameters), as illustrated by the gradient field of Fig.”
The Unusual Effectiveness of Averaging in GAN Training (2018, methodology): “Averaging at minimax objective: Gidel et al. (2018) used uniform averaging similar to our work, however we show its shortcomings and how exponential decay averaging performs better.”
Revisiting Stochastic Extragradient (2019, methodology): “One validation of this guess is that in (Gidel et al., 2019b), the networks were trained with negative momentum only on discriminator, while generator was trained with constant momentum +0.5.”
A Tight and Unified Analysis of Gradient-Based Methods for a Whole Spectrum of Differentiable Games (2020, methodology): “Note that this rate is similar to the one derived by Gidel et al. (2019b) for alternating gradient descent with negative momentum.”
Adaptive and Universal Algorithms for Variational Inequalities with Optimal Convergence (2020, methodology): “A different line of work focused on reducing the number of gradient computation to one per iteration (Popov 1980; Gidel et al. 2019; Hsieh et al. 2019; Chambolle and Pock 2011; Malit-sky 2015; Cui and Shanbhag 2016; Daskalakis et al. 2018; Mokhtari, Ozdaglar, and Pattathil 2020).”
Last Iterate is Slower than Averaged Iterate in Smooth Convex-Concave Saddle Point Problems (2020, background): “Finally, in the case that f is strongly convex-strongly concave, linear convergence rates on the distance between the last iterate and the global min-max point (namely, ( x ( T ) , y ( T ) ) − ( x ( ) , y ( ) ) ) are known (Tseng (1995); Gidel et al. (2018a); Liang and Stokes (2018); Mokhtari et al.”
A Domain Agnostic Measure for Monitoring and Evaluating GANs (2018, background): “Some exceptions are [4, 9, 7, 12] but these works do not address the problem of evaluation.”
Linear Lower Bounds and Conditioning of Differentiable Games (2019, background): “…number to be representative of a ﬁnite-dimensional bound as was the case in convex optimisation, we easily see an apparent contradiction from the upper bound on the rate of convergence of alternating gradient descent with negative momentum for bilinear games given by (Gidel et al., 2019b).”
Solving Weakly-Convex-Weakly-Concave Saddle-Point Problems as Successive Strongly Monotone Variational Inequalities (2018, methodology): “However, the analysis in [31] are for convex-concave problems or monotone variational inequalities.”
On Finding Local Nash Equilibria (and Only Local Nash Equilibria) in Zero-Sum Games (2019, methodology): “ly sound methods of ﬁnding the Nash equilibria of zero-sum games has approached the issue from the perspective of variational inequalities 5 MAZUMDAR, JORDAN, AND SASTRY (Mertikopoulos et al., 2018a; Gidel et al., 2018). In Mertikopoulos et al. (2018a) extragradient methods were used to solve cohere”
A Unified Analysis of Extra-gradient and Optimistic Gradient Methods for Saddle Point Problems: Proximal Point Approach (2019, methodology): “In this connection, the OGDA iterates are the “midpoints" whereas (Gidel et al., 2019) provides a convergence of the original points (not the OGDA iterates) to an error of in O(κ log(1/ )).”
An introduction to deep generative modeling (2021, background): “Recent works that propose to base the training on variational inequalities [15, 12] promise a more reliable solution, but we have not included these in our experiments.”