On Consistency and Sparsity for Principal Components Analysis in High Dimensions

By Iain M. Johnstone, A. Lu. Journal of the American Statistical Association, 2009

936 citations recorded by the source corpus.

Principal components analysis (PCA) is a classic method for the reduction of dimensionality of data in the form of n observations (or cases) of a vector with p variables. Contemporary datasets often have p comparable with or even much larger than n. Our main assertions, in such settings, are (a) that some initial reduction in dimensionality is desirable before applying any PCA-type search for principal modes, and (b) the initial reduction in dimensionality is best achieved by working in a basis in which the signals have a sparse representation. We describe a simple asymptotic model in which the estimate of the leading principal component vector via standard PCA is consistent if and only if p(n)/n → 0. We provide a simple algorithm for selecting a subset of coordinates with largest sample variances, and show that if PCA is done on the selected subset, then consistency is recovered, even if p(n) ≫ n.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/6b0345fe5dbf7a8551edd7ae3f56f803fc21378a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Principal component analysis: a review and recent developments (2016, methodology): “A number of authors have investigated versions of sparse PCA for this situation using models for the data in which the vast majority of the variables are completely unstructured noise [18,19,32].”
Global Testing and Large-Scale Multiple Testing for High-Dimensional Covariance Structures (2017, background): “See, e.g., Johnstone (2001), Paul (2007), and Johnstone and Lu (2009).”
Fast Principal Component Analysis of Large-Scale Genome-Wide Data (2014, methodology): “It has been shown that standard PCA may be an inconsistent estimator of the true principal components in certain high dimensional settings [10], and there may be benefit from utilizing other approaches such as sparse (‘1-penalized) PCA [11,12]; however, standard PCA remains widely used in practice a”
PROJECTED PRINCIPAL COMPONENT ANALYSIS IN FACTOR MODELS (2014, neutral): “As shown in Johnston and Lu (2009), in the high-dimensional setting, the spectrum of the sample covariance matrix is not consistent.”
Principal Component Analysis of High-Frequency Data (2016, neutral): “In this scenario, Johnstone and Lu (2009) and Paul (2007) showed that the estimated eigenvector corresponding to the largest eigenvalue is inconsistent unless the sample size grows at a rate faster than the one at which the cross-sectional dimension increases.”
Statistical inference for principal components of spiked covariance matrices (2020, methodology): “For instance, sparse principal component analysis [15, 41] is proposed to estimate the spiked eigenvalues and eigenvectors assuming some sparsity structure in the population eigenvectors; factor-model based estimators [2, 3, 66] for the eigenvectors are constructed if the population covariance matri”
Using Principal Component Analysis to Estimate a High Dimensional Factor Model with High-Frequency Data (2017, methodology): “Moreover, our model is distinct from others in the class of spiked eigenvalue models discussed by Paul (2007) and Johnstone and Lu (2009), in which all eigenvalues are of the same order, and are bounded as the dimension grows.”
A Sparse Singular Value Decomposition Method for High-Dimensional Data (2014, methodology): “3 (bi-cross-validation or BCV according to Owen and Perry 2009). Indeed, a version of cross-validation for threshold selection is used by one of the two competing proposals with which we compare ours: Witten, Tibshirani, and Hastie (2009) left out random subsets of the entries in the data matrix, me”
Asymptotics and Concentration Bounds for Bilinear Forms of Spectral Projectors of Sample Covariance (2014, methodology): “Concerning the estimation of spectral projectors, Johnstone and Lu [13] proved that the classical PCA approach could fail to produce a consistent estimator when p n → c > 0 as n → ∞.”
A Unified Feature Selection Framework for Graph Embedding on High Dimensional Data (2015, background): “Simply incorporating these noisy features could make graph embedding unreliable and noisy [9], [10].”
Covariance Regression Analysis (2017, neutral): “…few years, various sparsity constraints have been imposed on either (Huang et al. 2006; Bickel and Levina 2008a, 2008b; Cai and Liu 2011; Leng and Li 2011), its inverse −1 (Dempster 1972; Friedman, Hastie, and Tibshirani 2008; Cai, Liu, and Luo 2011), or its eigenvalues (Johnstone and Lu 2009).”
Efficient Estimation of Approximate Factor Models (2012, background): “, Bai and Ng (2002), Kapetanios (2010), Onatski (2010), Alessi et al.”
Covariance Matrix Estimation under Total Positivity for Portfolio Selection* (2019, background): “However, this estimator is known to have poor properties [MP67, Wac78, BY93, Joh01, JLN + 09], as can be seen by the following degrees-of-freedom argument (see also [ELW17, Section 3.1]): as is common when daily or monthly returns are used, the number of historical data points T is of the order of 1”
Robust covariance and scatter matrix estimation under Huber’s contamination model (2015, background): “We adopt the spiked covariance model [27, 4].”
Large Covariance Estimation by Thresholding Principal Orthogonal Complements (2011, neutral): “More literature on sparse PCA is found in Johnstone and Lu (2009), Amini and Wainwright (2009), Zhang and El Ghaoui (2011), Birnbaum et al. (2012), among others.”