A Closer Look at the Optimization Landscapes of Generative Adversarial Networks
By Hugo Berard, Gauthier Gidel, Amjad Almahairi, Pascal Vincent, Simon Lacoste-Julien. 2019
66 citations recorded by the source corpus.
Generative adversarial networks have been very successful in generative modeling, however they remain relatively challenging to train compared to standard deep neural networks. In this paper, we propose new visualization techniques for the optimization landscapes of GANs that enable us to study the game vector field resulting from the concatenation of the gradient of both players. Using these visualization techniques we try to bridge the gap between theory and practice by showing empirically that the training of GANs exhibits significant rotations around Local Stable Stationary Points (LSSP), similar to the one predicted by theory on toy examples. Moreover, we provide empirical evidence that GAN training converge to a stable stationary point which is a saddle point for the generator loss, not a minimum, while still achieving excellent performance.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Guodong Zhang
- Jimmy Ba
- Yuanhao Wang
- Sergey Levine
- Ruslan Salakhutdinov
- Roger Baker Grosse
- Quoc V. Le
- Yoshua Bengio
In Computer Vision and Pattern Recognition
- Guodong Zhang
- Jimmy Ba
- Yuanhao Wang
- Luc Van Gool
- Xiaogang Wang
- Dacheng Tao
- Trevor Darrell
- Alan L. Yuille
In Generative Models
In Optimization & Training
- Guodong Zhang
- Jimmy Ba
- Yuanhao Wang
- Yoshua Bengio
- Roger Baker Grosse
- Kyunghyun Cho
- Sergey Levine
- Mohit Bansal
Citations in the corpus
- On Solving Minimax Optimization Locally: A Follow-the-Ridge Approach (2019, background): “As noted by Mescheder et al. (2017); Gidel et al. (2019); Balduzzi et al. (2018), the rotational behaviour (instability) of GDA is caused by eigenvalues with large imaginary part.”
- Convergence of Learning Dynamics in Stackelberg Games (2019, unknown): “Moreover, some very recent results show that a number of approaches to training GANs are not converging to local minima (stable Nash), but rather non-Nash attractors of the dynamics [8].”