ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring for Minimax Problems and GANs
By Ernest K. Ryu, Kun Yuan, Wotao Yin. 2019
43 citations recorded by the source corpus.
Despite remarkable empirical success, the training dynamics of generative adversarial networks (GAN), which involves solving a minimax game using stochastic gradients, is still poorly understood. In this work, we analyze last-iterate convergence of simultaneous gradient descent (simGD) and its variants under the assumption of convex-concavity, guided by a continuous-time analysis with differential equations. First, we show that simGD, as is, converges with stochastic sub-gradients under strict convexity in the primal variable. Second, we generalize optimistic simGD to accommodate an optimism rate separate from the learning rate and show its convergence with full gradients. Finally, we present anchored simGD, a new method, and show convergence with stochastic subgradients.
Subfields: Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Optimization & Training
- Nicolas Loizou
- Simon Lacoste-Julien
- Ioannis Mitliagkas
- Hugo Berard
- Pascal Vincent
- Peter Richtárik
- Franck Iutzeler
- Jérôme Malick
Citations in the corpus
- Stochastic Hamiltonian Gradient Methods for Smooth Games (2020, methodology): “In another line of work, Carmon et al. (2019) analyze variance reduction methods for constrained finite-sum problems and Ryu et al. (2019) provide an ODE-based analysis and guarantees in the monotone but potentially non-smooth case.”
- Explore Aggressively, Update Conservatively: Stochastic Extragradient Methods with Variable Stepsize Scaling (2020, methodology): “Going one step further, we conduct experiments and observe similar phenomena for the generalized optimistic gradient method [24, 30] when the output vector is appropriately chosen.”