Revisiting Classifier Two-Sample Tests

By David Lopez-Paz, Maxime Oquab. 2016

527 citations recorded by the source corpus.

The goal of two-sample tests is to assess whether two samples, $S_P \sim P^n$ and $S_Q \sim Q^m$, are drawn from the same distribution. Perhaps intriguingly, one relatively unexplored method to build two-sample tests is the use of binary classifiers. In particular, construct a dataset by pairing the $n$ examples in $S_P$ with a positive label, and by pairing the $m$ examples in $S_Q$ with a negative label. If the null hypothesis "$P = Q$" is true, then the classification accuracy of a binary classifier on a held-out subset of this dataset should remain near chance-level. As we will show, such Classifier Two-Sample Tests (C2ST) learn a suitable representation of the data on the fly, return test statistics in interpretable units, have a simple null distribution, and their predictive uncertainty allow to interpret where $P$ and $Q$ differ. The goal of this paper is to establish the properties, performance, and uses of C2ST. First, we analyze their main theoretical properties. Second, we compare their performance against a variety of state-of-the-art alternatives. Third, we propose their use to evaluate the sample quality of generative models with intractable likelihoods, such as Generative Adversarial Networks (GANs). Fourth, we showcase the novel application of GANs together with C2ST for causal discovery.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, methodology): “Commonly-used ones include log-likelihood [28, 41], density estimate with Parzen window [28], Inception Score [64], Perceptual Path Length [37], Fréchet Inception Distance (FID) [30], Classification Accuracy Score and its early variants [60, 64], Classifier Twosample Tests [44, 47], precision and re”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, methodology): “Our proposed metric is also related to the classifier-based tests (Blau & Michaeli, 2018; Lopez-Paz & Oquab, 2016).”
Improved Precision and Recall Metric for Assessing Generative Models (2019, neutral): “[17] suffers from the same problem.”
Assessing Generative Models via Precision and Recall (2018, methodology): “compared based on statistical regularities in the images rather than random artifacts resulting from the generative process [17, 19].”
Recent Progress on Generative Adversarial Networks (GANs): A Survey (2019, background): “1-Nearest Neighbor classiﬁer (1-NN) [41] is the variant of C2ST which does not require special training and much hyper-parameter tuning.”
A theory of independent mechanisms for extrapolation in generative models (2020, background): “Deep neural network have been leveraged in causal inference for learning causal graphs between observed variables (Lopez-Paz and Oquab 2016) and associated causal effects (Louizos et al. 2017; Shalit, Johansson, and Sontag 2017; Kocaoglu et al. 2017; Lachapelle et al. 2019; Zhu, Ng, and Chen 2019).”
Conditional variance penalties and domain shift robustness (2017, background): “the problems of causeeffect inference and generative adversarial networks (Chalupka et al., 2014; Lopez-Paz et al., 2017; Lopez-Paz and Oquab, 2017; Goudet et al., 2017; Bahadori et al., 2017; Besserve et al., 2018; Kocaoglu et al., 2018).”
Learning in Implicit Generative Models (2016, background): “…et al. (2015) explore this approach for high-energy physics applications, Gutmann and Hyvärinen (2012) exploit it for learning un-normalised models, Lopez-Paz and Oquab (2016) for causal discovery, and Goodfellow et al. (2014) for learning in implicit generative models speciﬁed by neural networks.”
An empirical study on evaluation metrics of generative adversarial networks (2018, methodology): “Lopez-Paz & Oquab (2016) considered the 1-NN accuracy primarily as a statistic for two-sample testing.”
CausalGAN: Learning Causal Implicit Generative Models with Adversarial Training (2017, background): “In [26], authors observe the connection between conditional GAN layers, and structural equation models.”
Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting (2019, methodology): “Following the observation by [38], we found that training a binary classiﬁer on top of the feature space of any pretrained image classiﬁer was useful for removing the low-level artifacts in the generated images in classifying an image as real or fake.”
Pros and Cons of GAN Evaluation Measures (2018, background): “Lopez-Paz and Oquab [83] offer a revisit of classifier two-sample tests in [83].”
LTF: A Label Transformation Framework for Correcting Label Shift (2020, methodology): “…methods estimate importance sample weights P TX /P SX (Zadrozny, 2004; Huang et al., 2007; Sugiyama et al., 2008; Gretton et al., 2009) via kernel methods (Huang et al., 2007; Gretton et al., 2009; Zhang et al., 2013) or using a discriminative classiﬁer (Lopez-Paz & Oquab, 2016; Liu et al., 2017).”
Learning Deep Kernels for Non-Parametric Two-Sample Tests (2020, methodology): “• Classiﬁer two-sample tests, including C2STS-S (Lopez-Paz & Oquab, 2017) and C2ST-L (Chen & Cloninger, 2019) as described in Section 4.”
Image Manipulation with Perceptual Discriminators (2018, methodology): “To get objective performance measure, we have used the classifier two-sample test (C2ST) [26] to quantitatively compare GANs with the proposed discriminators to other methods.”
Explainable and Interpretable Models in Computer Vision and Machine Learning (2017, unknown): “…conditional distributions, and addresses the problem of learning FCMs (Section 2): An other inspiration of our framework is the recent method of Lopez-Paz and Oquab (2016), where a conditional generative adversarial network is trained to model X → Y and Y → X in order to infer the causal…”
PointFlow: 3D Point Cloud Generation With Continuous Normalizing Flows (2019, methodology): “We therefore introduce another metric that we believe is better suited for evaluating generative models of point clouds: • 1-nearest neighbor accuracy (1-NNA) is proposed by Lopez-Paz and Oquab [32] for two-sample tests, assessing whether two distributions are identical.”