Wasserstein Introspective Neural Networks

By Kwonjoon Lee, Weijian Xu, Fan Fan, Zhuowen Tu. 2017

61 citations recorded by the source corpus.

We present Wasserstein introspective neural networks (WINN) that are both a generator and a discriminator within a single model. WINN provides a significant improvement over the recent introspective neural networks (INN) method by enhancing INN's generative modeling capability. WINN has three interesting properties: (1) A mathematical connection between the formulation of the INN algorithm and that of Wasserstein generative adversarial networks (WGAN) is made. (2) The explicit adoption of the Wasserstein distance into INN results in a large enhancement to INN, achieving compelling results even with a single classifier - e.g., providing nearly a 20 times reduction in model size over INN for unsupervised generative modeling. (3) When applied to supervised classification, WINN also gives rise to improved robustness against adversarial examples in terms of the error reduction. In the experiments, we report encouraging results on unsupervised learning problems including texture, face, and object modeling, as well as a supervised classification task against adversarial attacks. Our code is available online1.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Probabilistic & Bayesian Methods

Citations in the corpus

Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One (2019, background): “Also related are Introspective Networks (Jin et al., 2017; Lee et al., 2018) which have drawn a similar connection between discriminative classifiers and generative models.”
Flow Contrastive Estimation of Energy-Based Models (2019, methodology): “Generalizing from [69], [27, 40, 43] developed an intro-spective parameter estimation method, where the EBM is discriminatively learned and composed of a sequence of discriminative models obtained through the learning process.”
Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, methodology): “Our model in (2) directly corresponds to a classifier, specifically a spatial-temporal discriminative ConvNet, in the following sense [22], [51], [52], [53], [54], [55].”
On the Anatomy of MCMC-based Maximum Likelihood Learning of Energy-Based Models (2019, methodology): “Several works studying the energy (2) recruit Langevin Dynamics to obtain the negative samples [38, 23, 37, 8, 22].”
Dual Contradistinctive Generative Autoencoder (2020, neutral): “Energy-based generative models [56, 66, 31, 46] — which aim to directly model data density — are making a steady improvement for a simultaneously generative and discriminative single model.”
Learning Energy-Based Models by Diffusion Recovery Likelihood (2020, background): “EBMs [27, 33, 21, 47, 10, 43, 6, 7, 25, 35, 5, 11, 4, 8, 2, 12] are an appealing class of probabilistic models, which can be viewed as generative versions of discriminators [18, 26, 28, 12], yet can be learned from unlabeled data.”
Cooperative Training of Descriptor and Generator Networks (2016, methodology): “Building on the pioneering work of [34], recently [35], [36], [37] have developed an introspective learning method to learn the energy-based model, where the energy function is discriminatively learned, and the learned energy function is used to generate synthesized examples via Langevin dynamics.”
On Learning Non-Convergent Short-Run MCMC Toward Energy-Based Model (2019, methodology): “Generalizing [45], [20, 26, 30] developed an introspective learning method where the EBM is discriminatively learned, and the EBM is both a generative model and a discriminative model.”
Deep Set Prediction Networks (2019, methodology): “Our inspiration for using backpropagation through an encoder as a decoder comes from the line of introspective neural networks [9; 10] for image modeling.”
Improved Contrastive Divergence Training of Energy Based Models (2020, background): “, 2018b) , out of distribution and adversarial robustness (Lee et al., 2018; Du & Mordatch, 2019; Grathwohl et al., 2019), compositional generation (Hinton, 1999; Du et al.”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “Energy-based Models (EBMs) [26, 48, 58, 64, 67, 68, 75, 78, 79, 120, 129, 132, 165, 170, 182, 196, 254, 281] can be viewed as one generative version of discriminators [79, 104, 131, 134], while can be learned from unlabeled input data.”
Sliced Wasserstein Discrepancy for Unsupervised Domain Adaptation (2019, background): “The Wasserstein distance has also recently raised interest in stabilizing generative modeling [1, 14, 73], learning introspective neural networks [32], and obtaining Gaussian mixture models [29] thanks to its geometrically meaningful distance measure even when the supports of the distributions do no”