Training generative neural networks via Maximum Mean Discrepancy optimization

By Gintare Karolina Dziugaite, Daniel M. Roy, Zoubin Ghahramani. 2015

591 citations recorded by the source corpus.

We consider training a deep neural network to generate samples from an unknown distribution given i.i.d. data. We frame learning as an optimization minimizing a two-sample test statistic—informally speaking, a good generator network produces samples that cause a two-sample test to fail to reject the null hypothesis. As our two-sample test statistic, we use an unbiased estimate of the maximum mean discrepancy, which is the centerpiece of the nonparametric kernel two-sample test proposed by Gretton et al. [2]. We compare to the adversarial nets framework introduced by Goodfellow et al. [1], in which learning is a two-player game between a generator network and an adversarial discriminator network, both trained to outwit the other. From this perspective, the MMD statistic plays the role of the discriminator. In addition to empirical comparisons, we prove bounds on the generalization error incurred by optimizing the empirical MMD.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Improved Techniques for Training GANs (2016, neutral): “3.1, is similar in spirit to approaches that use maximum mean discrepancy [7, 8, 9] to train generator networks [10, 11].”
Generative Adversarial Networks (2021, neutral): “By providing a thorough overview of GANs, we hope to make this complex and challenging topic more accessible to researchers and practitioners in the field, and inspire new innovations and applications.”
Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields (2017, methodology): “In a similar fashion, Generative Moment Matching Networks [30] and MMD nets [11] directly optimize a generator network to match a training distribution by using a loss function based on the maximum mean discrepancy (MMD) criterion [16].”
On the Quantitative Analysis of Decoder-Based Generative Models (2016, methodology): “…autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014), generative adversarial networks (GANs) (Goodfellow et al., 2014), generative moment matching networks (GMMNs) (Li & Swersky, 2015; Dziugaite et al., 2015), and nonlinear independent components estimation (Dinh et al., 2014).”
f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization (2016, methodology): “The advantage of VAEs is in the encoder model which allows efficient inference from observation to latent representation and overall they are a compelling alternative to f -GANs and recent work has studied combinations of the two approaches [23] As an alternative to the GAN training objective the wo”
A note on the evaluation of generative models (2015, methodology): “Maximum mean discrepancy (MMD) has been used with generative moment matching networks (Li et al., 2015; Dziugaite et al., 2015) and Jensen-Shannon divergence (JSD) has connections to the objective function optimized by generative adversarial networks (Goodfellow et al., 2014) (see box for a…”
Generative Moment Matching Networks (2015, background): “In recent independent work, Dziugaite et al. (2015) proposed an idea very similar to our’s, which trains a feed-forward neural network generative model by optimizing MMD.”
Importance Weighted Autoencoders (2015, background): “They can be viewed as a kind of adversarial net where the adversary simply looks at the set of pre-chosen statistics (Dziugaite et al., 2015).”
NIPS 2016 Tutorial: Generative Adversarial Networks (2016, methodology): “At the time of their introduction, GANs were the only notable member of this family, but since then they have been joined by additional models based on kernelized moment matching (Li et al., 2015; Dziugaite et al., 2015).”
Demystifying MMD GANs (2018, methodology): “But if we hold an MMD GAN’s critic ﬁxed and optimize the generator, we obtain the GMMN model (Li et al., 2015; Dziugaite et al., 2015).”
Geometric GAN (2017, methodology): “Moreover, Maximum Mean Discrepancy objective (MMD) for GAN training was also proposed in [4, 5].”
How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary? (2015, background): “Recently we have seen promising new training strategies such as those based on adversarial networks (Goodfellow et al., 2014; Denton et al., 2015) and kernel moment matching (Li et al., 2015; Dziugaite et al., 2015) which are not — at least on the surface — related to maximum likelihood.”
Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, methodology): “…et al. 2011, Muandet et al. 2013), selection bias correction (Huang et al. 2007), herding (Chen et al. 2010, Huszar and Duvenaud 2012), Markov chain Monte Carlo (Sejdinovic et al. 2014), moment matching for training deep generative models (Li et al. 2015, Dziugaite et al. 2015), statistical model”
DISCO Nets : DISsimilarity COefficients Networks (2016, background): “[2], the MMD test can been seen as playing the role of an adversary.”
A Test of Relative Similarity For Model Selection in Generative Models (2015, methodology): “Recently, the MMD has been used in training generative models adversarially, (Li et al., 2015b; Dziugaite et al., 2015), where the MMD measures the distance of the generated samples to some reference target set; it has been used for statistical model criticism (Lloyd and Ghahramani, 2015); and to…”
A Spectral Energy Distance for Parallel Speech Synthesis (2020, positive): “(4) Loss functions of this type were used by [8, 23] and [4] to train generative models without requiring a tractable likelihood function.”
Variational Inference of Disentangled Latent Concepts from Unlabeled Observations (2017, background): “, 2012) is also an option, however it has its own challenges when combined with stochastic optimization (Dziugaite et al., 2015; Li et al., 2015).”
Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations (2017, methodology): “Examples of implicit models include Generative Adversarial Networks (GANs) developed by Goodfellow et al. [2014] and kernel based models, see Li et al. [2015], Dziugaite et al. [2015], Bouchacourt et al. [2016].”
Learning in Implicit Generative Models (2016, background): “The role of the MMD for learning in implicit generative models deﬁned by neural networks has been explored by both Li et al. (2015) and Dziugaite et al. (2015).”