High-dimensional consistent independence testing with maxima of rank correlations

By Mathias Drton, Fang Han, Hongjian Shi. Annals of Statistics, 2018

56 citations recorded by the source corpus.

Testing mutual independence for high-dimensional observations is a fundamental statistical challenge. Popular tests based on linear and simple rank correlations are known to be incapable of detecting non-linear, non-monotone relationships, calling for methods that can account for such dependences. To address this challenge, we propose a family of tests that are constructed using maxima of pairwise rank correlations that permit consistent assessment of pairwise independence. Built upon a newly developed Cram\'{e}r-type moderate deviation theorem for degenerate U-statistics, our results cover a variety of rank correlations including Hoeffding's $D$, Blum-Kiefer-Rosenblatt's $R$, and Bergsma-Dassios-Yanagimoto's $\tau^*$. The proposed tests are distribution-free in the class of multivariate distributions with continuous margins, implementable without the need for permutation, and are shown to be rate-optimal against sparse alternatives under the Gaussian copula model. As a by-product of the study, we reveal an identity between the aforementioned three rank correlation statistics, and hence make a step towards proving a conjecture of Bergsma and Dassios.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/0135745f1b0a63d167d3e1078241e106fb24d11a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

On boosting the power of Chatterjee’s rank correlation (2021, background): “The statistic ξn,M is thus a rank correlation coefficient and accordingly enjoys all the nice properties shared by rank correlations, including in particular the distribution-freeness (i.e., of a distribution that is not dependent on P(X,Y )) under H0 (Hájek et al., 1999; Drton et al., 2020).”
On the power of Chatterjee’s rank correlation (2020, methodology): “…X (2) i2 ,X (2) i4 < X (2) i1 ,X (2) i3 ) − 8pr ( X (1) i1 ,X (1) i3 < X (1) i2 ,X (1) i4 , X (2) i1 ,X (2) i4 < X (2) i2 ,X (2) i3 ) , which coincides with 12D + 24R given continuity (not necessarily absolute continuity) of F , via page 62 of Yanagimoto (1970), as conveyed in Drton et al. (2020).”