A Kernel Test of Goodness of Fit

By Kacper P. Chwialkowski, Heiko Strathmann, Arthur Gretton. 2016

370 citations recorded by the source corpus.

We propose a nonparametric statistical test for goodness-of-fit: given a set of samples, the test determines how likely it is that these were generated from a target density function. The measure of goodness-of-fit is a divergence constructed via Stein's method using functions from a Reproducing Kernel Hilbert Space. Our test statistic is based on an empirical estimate of this divergence, taking the form of a V-statistic in terms of the log gradients of the target density and the kernel. We derive a statistical test, both for i.i.d. and non-i.i.d. samples, where we estimate the null distribution quantiles using a wild bootstrap procedure. We apply our test to quantifying convergence of approximate Markov Chain Monte Carlo methods, statistical model criticism, and evaluating quality of fit vs model complexity in nonparametric density estimation.

Subfields: Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, methodology): “Liu et al. (2016) and Chwialkowski et al. (2016) independently and simultaneously develop a goodness-of-fit test based on Stein’s identity applied to functions in RKHS (Oates et al. 2016).”
A Kernelized Stein Discrepancy for Goodness-of-fit Tests (2016, background): “The same idea was independently proposed by Chwialkowski et al. (2016) that appears simultaneously in this proceeding.”
Control functionals for Monte Carlo integration (2014, positive): “A similar bound forms the basis for recent work on two-sample testing by Chwialkowski et al. (2016) and Liu et al. (2016).”
Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning (2016, methodology): “One exception is a very recent paper (Ranganath et al., 2016) that avoids calculating q ( x ) using an idea related to Stein discrepancy (Gorham & Mackey, 2015; Liu et al., 2016; Oates et al., 2014; Chwialkowski et al., 2016).”
Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm (2016, background): “smooth variable transforms and a recently introduced kernelized Stein discrepancy [10, 8, 9], which allows us to derive a closed form solution for the optimal smooth perturbation direction that gives the steepest descent on the KL divergence within the unit ball of a reproducing kernel Hilbert space”
A Kernel Loss for Solving the Bellman Equation (2019, methodology): “In fact, by using methods from kernel-based hypothesis testing (e.g., Gretton et al., 2012; Liu et al., 2016; Chwialkowski et al., 2016), one can design statistically calibrated methods to test if V = V π has been achieved, which may be useful for designing efﬁcient exploration strategies.”
Measuring Sample Quality with Diffusions (2016, methodology): “While we have focused on the graph and classical Stein sets of [35], our diffusion Stein operators can also be paired with the reproducing kernel Hilbert space unit balls advocated in [69, 68, 15, 58, 36] to form tractable kernel diffusion Stein discrepancies.”
Convergence rates for a class of estimators based on Stein’s method (2016, background): “The assumption (A2̄) was first discussed in [10]; note in particular that (A2̄) implies (A2).”
Robust generalised Bayesian inference for intractable likelihoods (2021, positive): “The following result is a generalisation of the original construction of KSD (Chwialkowski et al., 2016; Liu et al., 2016) to general Stein operators.”
Learning deep kernels for exponential family densities (2018, background): “Another similar performance measure is the kernel Stein discrepancy (KSD) (Chwialkowski et al., 2016), where the model’s goodness-of-fit is evaluated at all test data points rather than at random test locations.”
Stein Variational Gradient Descent as Moment Matching (2018, background): “We may deﬁne a kernelized Stein discrepancy (KSD) (Liu et al., 2016; Chwialkowski et al., 2016; Oates et al., 2017): The optimal φ in (1) can be solved in closed form: (2) which yields a simple kernel-based representation of KSD: where x and x (cid:48) are i.i.d. draws from q , and κ p ( x , x…”
Black-box Importance Sampling (2016, methodology): “…a recently proposed kernelized Stein discrepancy that measures the goodness of a sample ﬁt with an unnor-malized distribution (Oates et al., 2017; Chwialkowski et al., 2016; Liu et al., 2016); this makes our method widely applicable for unnormalized distributions that widely appear in Bayesian…”
Action-depedent Control Variates for Policy Optimization via Stein's Identity (2017, methodology): “An incomplete list includes Gorham & Mackey (2015), Oates et al. (2017), Oates et al. (2016), Chwialkowski et al. (2016), Liu et al. (2016), Sedghi et al. (2016), Liu & Wang (2016), Feng et al. (2017), Liu & Lee (2017).”
Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences (2018, methodology): “On the other hand, if one considers a sample path of the GP prior (the OrnsteinUhlenbeck process in the quotation, which is the GP of the Matérn kernel with α = 1/2 and d = 1; see also Example 2.5), it is almost surely less smooth than functions in the RKHS (which is norm-equivalent to the first-ord”
A Spectral Approach to Gradient Estimation for Implicit Distributions (2018, methodology): “Recent developments in Stein discrepancy and its kernelized extensions (Gorham & Mackey, 2015; Chwialkowski et al., 2016; Liu et al., 2016; Liu & Wang, 2016) have renewed the interests in Stein’s method, which is a classic tool in statistics.”
Gradient Estimators for Implicit Models (2017, methodology): “For universal kernels satisfying the boundary condition, KSD is indeed a discrepancy measure: S(2)(q, q̂) = 0⇔ q = q̂ [6].”
Two Methods for Wild Variational Inference (2016, methodology): “, 2016) that also avoids calculating qη(z) and hence works for general inference networks; their method is based on a similar idea related to Stein discrepancy (Liu et al., 2016; Oates et al., 2017; Chwialkowski et al., 2016; Gorham & Mackey, 2015), for which we provide a more detailed discussion in”