Generative Moment Matching Networks

By Yujia Li, Kevin Swersky, Richard S. Zemel. 2015

936 citations recorded by the source corpus.

We consider the problem of learning deep generative models from data. We formulate a method that generates an independent sample via a single feedforward pass through a multilayer perceptron, as in the recently proposed generative adversarial networks (Goodfellow et al., 2014). Training a generative adversarial network, however, requires careful optimization of a difficult minimax program. Instead, we utilize a technique from statistical hypothesis testing known as maximum mean discrepancy (MMD), which leads to a simple objective that can be interpreted as matching all orders of statistics between a dataset and samples from the model, and can be trained by backpropagation. We further boost the performance of this approach by combining our generative network with an auto-encoder network, using MMD to learn to generate codes that can then be decoded to produce samples. We show that the combination of these techniques yields excellent generative models compared to baseline approaches as measured on MNIST and the Toronto Face Database.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

Generating Sentences from a Continuous Space (2015, methodology): “…Drawing inspiration from adversarial training methods for generative models as well as non-parametric two-sample tests (Goodfellow et al., 2014; Li et al., 2015b; Denton et al., 2015; Gretton et al., 2012), we evaluate the imputed sentence completions by examining their distinguishability from…”
Neural Autoregressive Distribution Estimation (2016, methodology): “Li et al. (2015) instead propose a training objective derived from Maximum Mean Discrepancy (MMD; Gretton et al., 2007).”
A-NICE-MC: Adversarial Training for MCMC (2017, neutral): “(3)The results are not very sensitive to changes in ; we also tried Maximum Mean Discrepancy (MMD, see [32] for details) and achieved similar results.”
Improved Techniques for Training GANs (2016, neutral): “3.1, is similar in spirit to approaches that use maximum mean discrepancy [7, 8, 9] to train generator networks [10, 11].”
Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields (2017, methodology): “In a similar fashion, Generative Moment Matching Networks [30] and MMD nets [11] directly optimize a generator network to match a training distribution by using a loss function based on the maximum mean discrepancy (MMD) criterion [16].”
Unsupervised Image-to-Image Translation Networks (2017, background): “, 2014), moment matching networks (Li et al., 2015), PixelCNN (van den Oord et al.”
Wasserstein GAN (2017, methodology): “However, in the original GMMN paper [10] they indeed used a minibatch of size 1000, much larger than the standard 32 or 64 (even when this incurred in quadratic computational cost).”
On the Quantitative Analysis of Decoder-Based Generative Models (2016, methodology): “…autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014), generative adversarial networks (GANs) (Goodfellow et al., 2014), generative moment matching networks (GMMNs) (Li & Swersky, 2015; Dziugaite et al., 2015), and nonlinear independent components estimation (Dinh et al., 2014).”
Coupled Generative Adversarial Networks (2016, background): “Several approaches, including generative adversarial networks[5], variational autoencoders (VAE)[22], attention models[23], moment matching[24], stochastic back-propagation[25], and diffusion processes[26], have shown that a deep network can learn an image distribution from samples.”
f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization (2016, methodology): “The advantage of VAEs is in the encoder model which allows efficient inference from observation to latent representation and overall they are a compelling alternative to f -GANs and recent work has studied combinations of the two approaches [23] As an alternative to the GAN training objective the wo”
Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks (2015, methodology): “Log-likelihood GMMN (Li et al., 2015) 147 ± 2 GSN (Bengio et al., 2014) 214 ± 1 GAN (Goodfellow et al., 2014) 225 ± 2 CatGAN 237 ± 6 GMMN + AE (Li et al., 2015) 282 ± 2 In Figure 4, 4 and 6 we show the results of training k-means, RIM and CatGAN models on the three synthetic datasets from the main…”
Adversarial Autoencoders (2015, methodology): “Generative moment matching networks (GMMN) [Li et al., 2015] use the maximum mean discrepancy (MMD) objective to shape the distribution of the output layer of a neural network.”
A note on the evaluation of generative models (2015, methodology): “…on the log-likelihood (Bishop, 2006), noise-contrastive estimation (Gutmann & Hyv¨arinen, 2010), probability ﬂow (Sohl-Dickstein et al., 2011), maximum mean discrepancy (MMD) (Gretton et al., 2007; Li et al., 2015), or approximations to the Jensen-Shannon divergence (JSD) (Goodfellow et al., 2014).”
Importance Weighted Autoencoders (2015, background): “Maximum mean discrepancy (MMD) networks (Li et al., 2015; Dziu-gaite et al., 2015) attempt to generate samples which match a certain set of statistics of the training data.”
NIPS 2016 Tutorial: Generative Adversarial Networks (2016, methodology): “At the time of their introduction, GANs were the only notable member of this family, but since then they have been joined by additional models based on kernelized moment matching (Li et al., 2015; Dziugaite et al., 2015).”
Demystifying MMD GANs (2018, methodology): “But if we hold an MMD GAN’s critic ﬁxed and optimize the generator, we obtain the GMMN model (Li et al., 2015; Dziugaite et al., 2015).”
Training generative neural networks via Maximum Mean Discrepancy optimization (2015, methodology): “In order to ﬁnd a minimax so-lution, they propose taking alternating gradient steps along D φ and G θ .”