Spectrum estimation for large dimensional covariance matrices using random matrix theory

By Noureddine El Karoui. 2006

324 citations recorded by the source corpus.

Estimating the eigenvalues of a population covariance matrix from a sample covariance matrix is a problem of fundamental importance in multivariate statistics; the eigenvalues of covariance matrices play a key role in many widely used techniques, in particular in principal component analysis (PCA). In many modem data analysis problems, statisticians are faced with large datasets where the sample size, n, is of the same order of magnitude as the number of variables p. Random matrix theory predicts that in this context, the eigenvalues of the sample covariance matrix are not good estimators of the eigenvalues of the population covariance. We propose to use a fundamental result in random matrix theory, the Marcenko-Pastur equation, to better estimate the eigenvalues of large dimensional covariance matrices. The Marcenko-Pastur equation holds in very wide generality and under weak assumptions. The estimator we obtain can be thought of as "shrinking" in a nonlinear fashion the eigenvalues of the sample covariance matrix to estimate the population eigenvalues. Inspired by ideas of random matrix theory, we also suggest a change of point of view when thinking about estimation of high-dimensional vectors: we do not try to estimate directly the vectors but rather a probability measure that describes them. We think this is a theoretically more fruitful way to think about these problems. Our estimator gives fast and good or very good results in extended simulations. Our algorithmic approach is based on convex optimization. We also show that the proposed estimator is consistent.

Subfields: Artificial Intelligence, Mathematical Physics, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/fc517ab2afc62dd2feb34194de9e19a2e9287b3d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Mathematical Physics

In Statistics and Probability

Citations in the corpus

Covariance Estimation: The GLM and Regularization Perspectives (2011, methodology): “The concept of thresholding originally developed in nonparametric function estimation has been used in the estimation of large covariance matrices by Bickel and Levina (2008b), El Karoui (2008a, 2008b) and Rothman, Levina and Zhu (2009).”
Operator norm consistent estimation of large-dimensional sparse covariance matrices (2008, background): “…matrix, when in the asymptotics considered here, we would not, in general (e.g., as soon as p/n → l 6 = 0), have consistency even at the level of the vector of eigenvalues; in the case of Σ p = Id, this is a consequence of the results of [13] or [21] and we refer to [12] for a thorough discussion.”
Independence test for high dimensional data based on regularized canonical correlation coefficients (2015, methodology): “We use an estimator developed by [8] for H .”
Optimal estimation of a large-dimensional covariance matrix under Stein’s loss (2018, unknown): “The assumption that Supp(Hn) is uniformly bounded away from infinity is even more widespread and made by such authors as Bai and Silverstein [2–4], Johnstone [22], Bickel and Levina [8], Mestre [32], El Karoui [14], Won et al. [44], and Khare, Oh and Rajaratnam [23], among others.”
Eigenvectors of some large sample covariance matrix ensembles (2009, unknown): “Therefore the algorithm in [13] is bette r suited for estimating the population eigenvalues themselves, whereas our approach i s better suited for estimating the population covariance matrix as a whole.”
High-dimensionality effects in the Markowitz problem and other quadratic programs with linear constraints: Risk underestimation (2010, methodology): “Perhaps surprisingly, we did not need to make very strong assumptions about the covariance matrix at stake or the mean, whereas recent statistical work focused on estimation of covariance matrices [see El Karoui (2008) or Bickel and Levina (2008b)] tends to do so.”
Nonlinear Shrinkage Estimation of Large-Dimensional Covariance Matrices (2012, methodology): “The price to pay is that our optimization program is not convex, whereas the one of El Karoui (2008) is.”
Cleaning large correlation matrices: tools from random matrix theory (2016, methodology): “El Karoui [33] who proposed to solve an approximate form of the Marčenko–Pastur inverse problem.”
Statistical eigen-inference from large Wishart matrices (2007, methodology): “A semiparametric, grid-based technique for inferring the empirical distribution function of the population eigenvalues from the sample eigenspectrum was proposed by El Karoui (2006).”
On the estimation of integrated covariance matrices of high dimensional diffusion processes (2010, methodology): “In particular, applying Theorem 2 of El Karoui (2008) to our case yields COROLLARY 1.”
Efficient Estimation of Approximate Factor Models (2012, background): “…estimation include Cai and Zhou (2012), Fan et al. (2008), Jung and Marron (2009), Witten, Tibshirani and Hastie (2009), Deng and Tsui (2010), Yuan (2010), Ledoit and Wolf (2012), El Karoui (2008), Pati et al. (2012), Rohde and Tsybakov (2011), Zhou et al. (2011), Ravikumar et al. (2011) etc.”
High-dimensional covariance matrix estimation with missing observations (2012, background): “In random matrix theory, an important line of work, [15, 18, 19] and the references cited therein studied the asymptotic distribution of the sample covariance matrix eigenvalues for different settings of n and p.”
Random matrix theory (2019, background): “El Karoui, [17], “Spectrum estimation for large dimensional covariance matrices using random matrix theory”, Annals of Statistics 36:6 (2008), pp.”
Large Dynamic Covariance Matrices (2019, methodology): “El Karoui (2008) andMestre (2008) were the first to make attempts in this direction. The solutions they proposed suffered from some limitations that made them unsuitable for general use. Subsequently, Ledoit and Wolf (2015) introduced an effective method based on numerical inversion of what they cal”
Random matrix theory in statistics: A review (2014, methodology): “El Karoui (2008b) used the Marčenko–Pastur equation (18) and employed a linear programming method to minimize the sup-norm difference between z and the expression on the righthand side with the Stieltjes transform of the dual sample covariance matrix as input.”