Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy

By Danica J. Sutherland, Hsiao-Yu Fish Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alex Smola, Arthur Gretton. 2016

293 citations recorded by the source corpus.

We propose a method to optimize the representation and distinguishability of samples from two probability distributions, by maximizing the estimated power of a statistical test based on the maximum mean discrepancy (MMD). This optimized MMD is applied to the setting of unsupervised learning by generative adversarial networks (GAN), in which a model attempts to generate realistic samples, and a discriminator attempts to tell these apart from data samples. In this context, the MMD may be used in two roles: first, as a discriminator, either directly on the samples, or on features of the samples. Second, the MMD can be used to evaluate the performance of a generative model, by testing the model's samples against a reference data set. In the latter role, the optimized MMD is particularly helpful, as it gives an interpretable indication of how the model and data distributions differ, even in cases where individual model samples are not easily distinguished either by eye or by classifier.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Wasserstein GAN (2017, background): “However, even Gaussian kernels are able to detect tiny noise patterns as recently evidenced by [20].”
Demystifying MMD GANs (2018, methodology): “Without such constraints, the convolutional features are unstable and difﬁcult to train (Sutherland et al., 2017).”
Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks (2017, methodology): “To measure the statistical distance between in-and out-of-distribution datasets, we adopt a commonly used metric, maximum mean discrepancy (MMD) with Gaussian RBF kernel (Sriperumbudur et al., 2010; Gretton et al., 2012; Sutherland et al., 2016).”
Learning in Implicit Generative Models (2016, methodology): “Other approaches for evaluation include reporting the value of the density ratio, a quantity we can always compute using the approaches described, the use of annealed importance sampling Wu et al. (2016), use of empirical distance metric such as the MMD (Sutherland et al., 2016) or the Wasserstein d”
Boundary-Seeking Generative Adversarial Networks (2017, background): “, 2017) or by constraining its output distribution (Mroueh & Sercu, 2017; Mroueh et al., 2017; Sutherland et al., 2016).”
MMD GAN: Towards Deeper Understanding of Moment Matching Network (2017, methodology): “[14] considers the variance of empirical MMD score during the training.”
Revisiting Classifier Two-Sample Tests (2016, methodology): “ls include the pioneering work of Box (1980), the use of Maximum Mean Discrepancy (MMD) criterion (Bengio et al., 2013; Dziugaite et al., 2015; Lloyd & Ghahramani, 2015; Bounliphone et al., 2015; Sutherland et al., 2016), and the connections to density-ratio estimation (Kanamori et al., 2010; Wo”
Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs (2017, methodology): “To select an appropriate kernel bandwidth σ we maximise the estimator of the t-statistic of the power of the MMD test between two distributions [24]; ˆ t = MMD 2 √ ˆ V , where V is the asymptotic variance of the estimator of MMD 2 .”
PacGAN: The Power of Two Samples in Generative Adversarial Networks (2017, background): “We note that there is a family of GANs whose discriminators take pairs of images [65, 37, 5], albeit for very diﬀerent reasons.”
Pros and Cons of GAN Evaluation Measures (2018, methodology): “This metric measures the dissimilarity between two probability distributions4 Pr and Pg using samples drawn independently from each [25, 72, 100, 8].”
Approximation and Convergence Properties of Generative Adversarial Learning (2017, background): “[13] focus on optimizing the estimating power of kernel distances.”
Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models (2017, background): “sed in [9]. The generators and discriminators are typically deep neural networks. Recent work has generalized the objective in Eq. (1) to arbitrary f-divergences [18] and integral probability metrics [28, 24]. In particular, Wasserstein GAN (WGAN, [1]), where the minimax is over the Wasserstein dist”
Learning Deep Kernels for Non-Parametric Two-Sample Tests (2020, methodology): “(All kernels used by Sutherland et al. (2017), for instance, were of this type.)”
Generative Adversarial Networks in Time Series: A Systematic Literature Review (2022, methodology): “Quantitative metrics compare statistical properties between generated and real images: two-sample tests such as maximum mean discrepancy (MMD) [93], inception score (IS) [88], and Fréchet inception distance (FID) [51].”
Synthesis of Realistic ECG using Generative Adversarial Networks (2019, methodology): “We use the Gaussian RBF kernel with bandwidth α equal to the pairwise distance between the joint data [41].”
Flow-GAN: Bridging implicit and prescribed learning in generative models (2017, background): “(1) to arbitrary f -divergences [18] and integral probability metrics [28, 24].”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, methodology): “MMD has been used for deep generative models [166]–[168] and model criticism [169].”