MMD GAN: Towards Deeper Understanding of Moment Matching Network

By Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, Barnab'as P'oczos. 2017

807 citations recorded by the source corpus.

Generative moment matching network (GMMN) is a deep generative model that differs from Generative Adversarial Network (GAN) by replacing the discriminator in GAN with a two-sample test based on kernel maximum mean discrepancy (MMD). Although some theoretical guarantees of MMD have been studied, the empirical performance of GMMN is still not as competitive as that of GAN on challenging and large benchmark datasets. The computational efficiency of GMMN is also less desirable in comparison with GAN, partially due to its requirement for a rather large batch size during the training. In this paper, we propose to improve both the model expressiveness of GMMN and its computational efficiency by introducing adversarial kernel learning techniques, as the replacement of a fixed Gaussian kernel in the original GMMN. The new approach combines the key ideas in both GMMN and GAN, hence we name it MMD GAN. The new distance measure in MMD GAN is a meaningful loss that enjoys the advantage of weak topology and can be optimized via gradient descent with relatively small batch sizes. In our evaluation on multiple benchmark datasets, including MNIST, CIFAR- 10, CelebA and LSUN, the performance of MMD-GAN significantly outperforms GMMN, and is competitive with other representative GAN works.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, background): “For GAN, perform Langevin inference latent variables [19, 50].”
Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields (2017, methodology): “These approaches were later expanded to include an MMD criterion with learnable kernels (Li et al., 2017).”
Demystifying MMD GANs (2018, positive): “In terms of KID and FID, MMD GANs outperform Cramér and WGAN-GP for each critic size.”
GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium (2017, neutral): “g and unrealistic assumptions or a restriction to a linear discriminator. Recent convergence proofs for GANs hold for expectations over training samples or for the number of examples going to inﬁnity [37, 45, 40, 4], thus do not consider mini-batch learning which leads to a stochastic gradient [57,2”
GenDICE: Generalized Offline Estimation of Stationary Values (2020, methodology): “We use a learned Gaussian kernel in MMD, similar to Li et al. (2017). As we can see in Figure 13(a), with these different divergences, the proposed GenDICE estimator can always achieve significantly better performance compared with the model-based estimator, showing that the GenDICE estimator is com”
Moment Matching for Multi-Source Domain Adaptation (2018, background): “McGAN [29] utilizes a GAN to match the mean and covariance of feature distributions.”
Wasserstein Auto-Encoders (2017, methodology): “quire larger mini-batches during training: estimating MMD k(P X;P G) requires number of samples roughly proportional to the dimensionality of the input space X[23] which is typically larger than 103. [24] take a similar approach but further train kadversarially so as to arrive at a meaningful loss f”
The Cramer Distance as a Solution to Biased Wasserstein Gradients (2017, methodology): “An independent work by Li et al. (2017) combine Generative Moment Matching Networks with a learned transformation.”
Learning Generative Models with Sinkhorn Divergences (2017, methodology): “The experimental setting is the same as in [24] and we used the same parameters to carry out a fair comparison.”
Discriminator Rejection Sampling (2018, background): “Addressing these issues is an active area of research (Mao et al., 2017; Arjovsky et al., 2017; Gulrajani et al., 2017; Odena et al., 2018; Li et al., 2017).”
PacGAN: The Power of Two Samples in Generative Adversarial Networks (2017, background): “We note that there is a family of GANs whose discriminators take pairs of images [65, 37, 5], albeit for very diﬀerent reasons.”
Improving MMD-GAN Training with Repulsive Loss Function (2018, methodology): “Thus, the objective functions for G and D could be (Li et al. (2017a); Bińkowski et al. (2018)): min G LmmdG = M 2 k◦D(PX, PG) = EPG [kD(y,y′)]− 2EPX,PG [kD(x,y)] + EPX [kD(x,x′)] (2) min D LattD = −M2k◦D(PX, PG) = 2EPX,PG [kD(x,y)]− EPX [kD(x,x′)]− EPG [kD(y,y′)] (3) MMD-GAN has been shown to be…”
Pros and Cons of GAN Evaluation Measures (2018, methodology): “For example, the Generative Moment Matching Network (GMMN) [49, 22, 48] replaces the discriminator in GAN with a two-sample test based on kernel MMD.”
TransGAN: Two Transformers Can Make One Strong GAN (2021, background): “…introducing various regularization terms (Kurach et al., 2019; Roth et al., 2017; Zhang et al., 2019b; Mescheder et al., 2018), better losses (Gul-rajani et al., 2017; Mao et al., 2017; Jolicoeur-Martineau, 2018; Li et al., 2017), and training recipes (Salimans et al., 2016; Karras et al., 2017).”
Towards the High-quality Anime Characters Generation with Generative Adversarial Networks (2017, methodology): “[13] Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos.”
Causal Generative Domain Adaptation Networks (2018, methodology): “As done in [32], we add a discriminator network f to transform the images into high-level features and compare the distributions of the learned features using MMD.”
Learning deep kernels for exponential family densities (2018, background): “These types of kernels have seen success in supervised learning (Wilson et al. 2016; Jean et al. 2018) and critic functions for implicit generative models (Li et al. 2017; Bellemare et al. 2017; Bi´nkowski et al. 2018; Arbel et al. 2018).”