Nonparametric independence testing via mutual information

By Thomas B. Berrett, Richard J. Samworth. 2017

119 citations recorded by the source corpus.

We propose a test of independence of two multivariate random vectors, given a sample from the underlying population. Our approach is based on the estimation of mutual information, whose decomposition into joint and marginal entropies facilitates the use of recently developed efficient entropy estimators derived from nearest neighbour distances. The proposed critical values may be obtained by simulation in the case where an approximation to one marginal is available or by permuting the data otherwise. This facilitates size guarantees, and we provide local power analyses, uniformly over classes of densities whose mutual information satisfies a lower bound. Our ideas may be extended to provide new goodness-of-fit tests for normal linear models based on assessing the independence of our vector of covariates and an appropriately defined notion of an error vector. The theory is supported by numerical studies on both simulated and real data.

Subfields: Economics and Econometrics, Food Science, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Food Science

In Statistics and Probability

Citations in the corpus

The hardness of conditional independence testing and the generalised covariance measure (2018, background): “For the multivariate case, Berrett and Samworth [6] consider a test based on mutual information and prove level guarantees, as well as uniform power results against a wide class of alternatives.”
High-dimensional consistent independence testing with maxima of rank correlations (2018, background): “This leads to notions of mutual information whose estimates are different from and usually more challenging to handle than the rank correlation measures we consider here; see Berrett and Samworth (2019) and references therein.”
Distribution-Free Consistent Independence Tests via Center-Outward Ranks and Signs (2019, background): “…the sample space (Heller, Heller, and Gorfine 2013; Heller et al. 2016; Ma and Mao 2019; Zhang 2019), (ii) mutual information (Kraskov, Stögbauer, and Grassberger 2004; Kinney and Atwal 2014; Berrett and Samworth 2019), and (iii) the maximal information coefficient (Reshef et al. 2011, 2016, 2018).”
Statistical Dependence: Beyond Pearson’s ρ (2018, background): “A number of authors have considered such an approach; both in the iid and time series case, see e.g. Rosenblatt (1975), Robinson (1991), Skaug and Tjøstheim (1993b; 1996), Granger, Maasoumi, and Racine (2004), Hong and White (2005), Su and White (2007) and Berrett and Samworth (2017).”
Multivariate Rank-Based Distribution-Free Nonparametric Testing Using Measure Transportation (2019, methodology): “…testing procedures have been proposed for this problem in the multivariate setting; see, for example, Székely, Rizzo, and Bakirov (2007), Berrett and Samworth (2019), Gretton, Fukumizu et al. (2008), Heller, Heller, and Gorfine (2013), Biswas, Sarkar, and Ghosh (2016), Gieser and…”
The conditional permutation test for independence while controlling for confounders (2018, methodology): “of independence, for example based on ranks, kernel methods, copulas, and nearest neighbours [32, 17, 16, 6].”
The conditional permutation test (2018, background): “Simulta-neously, a great deal of research eﬀort has gone into developing several diﬀerent tests of independence, for example based on ranks, kernel methods, copulas, and nearest neighbours [26, 13, 12, 4].”
On boosting the power of Chatterjee’s rank correlation (2021, background): “Notably speaking, similar ideas were already pursued in Deb et al. (2020, Equations (3.3) and (8.5)); see also Berrett and Samworth (2019) for a related proposal that approximates the mutual information (Berrett et al., 2019) using M -NN-based statistics.”