Stabilizing Adversarial Nets With Prediction Methods

By Abhay Kumar Yadav, Sohil Shah, Zheng Xu, David W. Jacobs, Tom Goldstein. 2017

92 citations recorded by the source corpus.

Adversarial neural networks solve many important problems in data science, but are notoriously difficult to train. These difficulties come from the fact that optimal weights for adversarial nets correspond to saddle points, and not minimizers, of the loss function. The alternating stochastic gradient methods typically used for such problems do not reliably converge to saddle points, and when convergence does happen it is often highly sensitive to learning rates. We propose a simple modification of stochastic gradient descent that stabilizes adversarial networks. We show, both in theory and practice, that the proposed method reliably converges to saddle points, and is stable with a wider range of training parameters than a non-prediction method. This makes adversarial networks less likely to "collapse," and enables faster training with larger learning rates.

Subfields: Generative Models, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

In Optimization & Training

Citations in the corpus

Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (2019, methodology): “Furthermore, the interpolation property can be used to improve the convergence for a bilinear saddle-point problem [24, 25, 54, 85].”
A Variational Inequality Perspective on Generative Adversarial Networks (2018, methodology): “One has to be careful though when extending properties from the minimization setting to the saddle point setting (e.g. the merit function used by Yadav et al. (2018) is vacuous for a bilinear game as explained in App C.2).”
Loss-Sensitive Generative Adversarial Networks on Lipschitz Densities (2017, background): “Finally, we note that the study of generalization performances is not directly related with the stability of training methods (Nagarajan and Kolter 2017; Yadav et al. 2017).”
Interaction Matters: A Note on Non-asymptotic Local Convergence of Generative Adversarial Networks (2018, methodology): “Several attempts to ﬁx GAN optimization have been put forth by independent researchers, which modify the dynamics [Mescheder et al., 2017, Daskalakis et al., 2017, Yadav et al., 2017] using very diﬀerent insights.”
Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile (2018, background): “where, in obvious notation, ĝn and ĝn+1/2 represent gradient oracle queries at the incumbent and intermediate states Xn and Xn+1/2 respectively. For a pseudocode implementation, see Algorithm 2; see also Rakhlin & Sridharan [2013] and Daskalakis et al.”
Adaptive and Universal Algorithms for Variational Inequalities with Optimal Convergence (2020, unknown): “…x ∗ ) ; x ∗ − x i ≤ 0 ∀ x ∈ X More recently, algorithms developed in this framework are also applied to non-convex problems including optimizing generative adversarial networks (GANs) (Daskalakis et al. 2018; Yadav et al. 2018; Chavdarova et al. 2019; Gidel et al. 2019; Mertikopoulos et al. 2019).”
Few-Shot Adaptation of Generative Adversarial Networks (2020, background): “While there have been signiﬁcant advancements in improving training stability (Karras et al., 2018; Miyato et al., 2018), adversarial optimization remains challenging because the optimal solutions lie at saddle points rather than a minimum of a loss function (Yadav et al., 2017).”
Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks (2017, unknown): “First, the adversarial training process is difficult[43].”
Competitive Gradient Descent (2019, background): “, 2017), fictitious play (Brown, 1951), predictive updates (Yadav et al., 2017), opponent learning awareness (Foerster et al.”
Medical image data augmentation: techniques, comparisons and interpretations (2023, background): “The common problems in those GAN based augmentations are mode collapsing and saddle point optimization (Yadav et al. 2017).”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, unknown): “[180] stabilized GANs with prediction methods.”
On the convergence of single-call stochastic extra-gradient methods (2019, methodology): “The well-known primal-dual algorithm of Chambolle and Pock [8] can be seen as a one-sided, alternating variant of the method for saddle-point problems; see also [44] for a more recent take.”
Last Iterate is Slower than Averaged Iterate in Smooth Convex-Concave Saddle Point Problems (2020, background): “The EG and OGDA algorithms have additionally received signiﬁcant recent attention due to their ability to improve the training dynamics in GANs (Chavdarova et al. (2019); Gidel et al. (2018a,b); Liang and Stokes (2018); Yadav et al. (2017); Daskalakis et al. (2017)).”
Generative Adversarial Networks (GANs) (2020, methodology): “Arjovsky, et al. [29] proposed a loss function which also acts as a measure of convergence, called Wasserstein Generative Adversarial Networks (WGAN).”
Training Student Networks for Acceleration with Conditional Adversarial Networks (2018, background): “First, the adversarial training process is difﬁcult [38].”
Tight last-iterate convergence rates for no-regret learning in multi-player games (2020, background): “Such saddle point problems have received much attention recently since they can be viewed as a simpliﬁed model of generative adversarial networks (e.g., [GBV + 18, DISZ17, CGFLJ19, GHP + 18, YSX + 17]).”
ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring for Minimax Problems and GANs (2019, unknown): “We note that there are other mechanisms similar to optimism and anchoring such as “prediction” [68], “negative momentum” [19], and “extragradient” [27, 65, 8].”