High Dimensional Statistical Inference and Random Matrices

By Iain M. Johnstone. 2006

229 citations recorded by the source corpus.

Multivariate statistical analysis is concerned with observations on several variables which are thought to possess some degree of inter-dependence. Driven by problems in genetics and the social sciences, it first flowered in the earlier half of the last century. Subsequently, random matrix theory (RMT) developed, initially within physics, and more recently widely in mathematics. While some of the central objects of study in RMT are identical to those of multivariate statistics, statistical theory was slow to exploit the connection. However, with vast data collection ever more common, data sets now often have as many or more variables than the number of individuals observed. In such contexts, the techniques and results of RMT have much to offer multivariate statistics. The paper reviews some of the progress to date.

Subfields: Artificial Intelligence, Discrete Mathematics and Combinatorics, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/ecc621996465139fa68d9f3846c525d3a83b0571

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Discrete Mathematics and Combinatorics

In Statistics and Probability

Citations in the corpus

Eigenvalue Distribution of Large Random Matrices (2011, unknown): “, Two-dimensional quantum gravity, matrix models and string theory, Inter. Journ. of Mod. Phys. A8 (1993), 1185–1244.”
Financial Applications of Random Matrix Theory: a short review (2009, background): “(for reviews of other applications and techniques, see [2, 3, 4]) We will denote by T the total number of observations of each of the N quantities.”
Substitution principle for CLT of linear spectral statistics of high-dimensional sample covariance matrices with applications to hypothesis testing (2014, background): “It has been demonstrated that such limiting scheme has a wide application scope for real-life high-dimensional data analysis [13].”
On estimation of the noise variance in high dimensional probabilistic principal component analysis (2013, methodology): “Kritchman and Nadler (2008a) proposes an estimator by solving a system of implicit equations; Ulfarsson and Solo (2008) introduces an estimator using the median of the sample eigenvalues {λn,k}; and Johnstone and Lu (2009) uses the median of the sample variances.”
One-Pass AUC Optimization (2013, methodology): “Then, we study the parameter influence in Section 6.3.”
Denoising of diffusion MRI using random matrix theory (2016, unknown): “[5] (Johnstone, 2006).”
On the principal components of sample covariance matrices (2014, unknown): “physics of mixtures, and statistical learning [3, 26, 33].”
The Distributions of Random Matrix Theory and their Applications (2009, methodology): “With this choice of constants, F1 provides a good approximation for rather small values of p. (See Johnstone’s comparisons with the tables of Chen [21].)”
On testing for high-dimensional white noise (2018, methodology): “For a review on such progress, we refer to [21], [30] and monograph [40].”
Limits of spiked random matrices I (2010, background): “In general, asymptotic distributions in the non-null cases are relevant when evaluating the power of a statistical test (Johnstone [37]).”
On the Marčenko-Pastur law for linear time series (2013, background): “Comprehensive reviews of this topic can be found in Johnstone [22] and Paul and Aue [30].”
Determining the number of components in a factor model from limited noisy data (2008, methodology): “As described in [12], for real-valued observations the following expressions”
A necessary and sufficient condition for edge universality at the largest singular values of covariance matrices (2016, methodology): “e largest eigenvalue is important both in ﬁnding the second order asymptotics for the estimate in PCA and in computing the power of test in hypothesis testing. For a detailed review, one can refer to [13, 22, 30]. In this paper, we study an M ×N matrix X and the covariance matrix XX∗ in the regime w”