Connections between Support Vector Machines, Wasserstein distance and gradient-penalty GANs

By Alexia Jolicoeur-Martineau, Ioannis Mitliagkas. 2019

12 citations recorded by the source corpus.

We generalize the concept of maximum-margin classifiers (MMCs) to arbitrary norms and non-linear functions. Support Vector Machines (SVMs) are a special case of MMC. We find that MMCs can be formulated as Integral Probability Metrics (IPMs) or classifiers with some form of gradient norm penalty. This implies a direct link to a class of Generative adversarial networks (GANs) which penalize a gradient norm. We show that the Discriminator in Wasserstein, Standard, Least-Squares, and Hinge GAN with Gradient Penalty is an MMC. We explain why maximizing a margin may be helpful in GANs. We hypothesize and confirm experimentally that $L^\infty$-norm penalties with Hinge loss produce better GANs than $L^2$-norm penalties (based on common evaluation metrics). We derive the margins of Relativistic paired (Rp) and average (Ra) GANs.

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

Simple and Scalable Epistemic Uncertainty Estimation Using a Single Deep Deterministic Neural Network (2020, neutral): “Follow up work has analysed the penalty in more detail and concluded that, contrary to our case, for training Wasserstein GANS the one-sided penalty is preferable theoretically and practically (Jolicoeur-Martineau & Mitliagkas, 2019; Petzka et al., 2017).”
Uncertainty Estimation Using a Single Deep Deterministic Neural Network (2020, result): “Follow up work has analysed the penalty in more detail and concluded that, contrary to our case, for training Wasserstein GANS the one-sided penalty is preferable theoretically and practically (Jolicoeur-Martineau & Mitliagkas, 2019; Petzka et al., 2017).”
Image Inpainting with Learnable Feature Imputation (2020, methodology): “Furthermore, imposing the constrain ||∇D(x̂)|| ≤ 1 is shown to properly estimate the Wasserstein distance [18].”
Gradient Starvation: A Learning Proclivity in Neural Networks (2020, background): “It should be stressed that these work refer to the margin in the hidden space and not in the input space as pointed out in [55].”