A Test of Relative Similarity For Model Selection in Generative Models

By Wacha Bounliphone, Eugene Belilovsky, Matthew B. Blaschko, Ioannis Antonoglou, Arthur Gretton. 2015

88 citations recorded by the source corpus.

Probabilistic generative models provide a powerful framework for representing data that avoids the expense of manual annotation typically needed by discriminative approaches. Model selection in this generative setting can be challenging, however, particularly when likelihoods are not easily accessible. To address this issue, we introduce a statistical test of relative similarity, which is used to determine which of two models generates samples that are significantly closer to a real-world reference dataset of interest. We use as our test statistic the difference in maximum mean discrepancies (MMDs) between the reference dataset and each model dataset, and derive a powerful, low-variance test based on the joint asymptotic distribution of the MMDs between each reference-model pair. In experiments on deep generative models, including the variational auto-encoder and generative moment matching network, the tests provide a meaningful ranking of model performance as a function of parameter and training settings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistics and Probability, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistics and Probability

In Generative Models

In Signal Processing

Citations in the corpus

On the Quantitative Analysis of Decoder-Based Generative Models (2016, background): “While alternative quantitative criteria have been proposed [4, 13, 25], log-likelihood of held-out test data remains one of the most important measures of a generative model’s performance.”
Demystifying MMD GANs (2018, methodology): “To avoid setting an explicit threshold on the change in the numerical value of the score, we use a p -value obtained from the relative similarity test of Bounliphone et al. (2016).”
Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, background): “criticism (Lloyd and Ghahramani 2015, Kim et al. 2016), approximate Bayesian computation (Park et al. 2016), and model selection in generative models (Bounliphone et al. 2016), for example.”
Revisiting Classifier Two-Sample Tests (2016, methodology): “The use of two-sample tests to evaluate the sample quality of generative models include the pioneering work of Box (1980), the use of Maximum Mean Discrepancy (MMD) criterion (Bengio et al., 2013; Dziugaite et al., 2015; Lloyd & Ghahramani, 2015; Bounliphone et al., 2015; Sutherland et al., 2016), a”
Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs (2017, methodology): “We generate these nearest neighbours by minimising the reconstruction error between target y and the generated point; L recon( y ) ( Z ) = 1 − K ( G ( Z ) , y ) where K is the RBF kernel described in 3.1.1, with bandwidth σ chosen using the median heuristic [5].”
An empirical study on evaluation metrics of generative adversarial networks (2018, methodology): “Bounliphone et al. (2015) propose to use maximum mean discrepancies (MMDs) for model selection in generative models.”
Pros and Cons of GAN Evaluation Measures (2018, methodology): “This metric measures the dissimilarity between two probability distributions4 Pr and Pg using samples drawn independently from each [25, 72, 100, 8].”
A Non-Parametric Test to Detect Data-Copying in Generative Models (2020, methodology): “Similarly, Bounliphone et al. (2016) explores three-sample testing, but for comparing the performance of different models, not for detecting overfitting.”
Learning to generate images with perceptual similarity metrics (2015, methodology): “We then utilized the test statistic of [2] to determine for each loss the value of C that produced samples with smallest squared MMD compared to the STL-10 train set.”