Sparse Principal Components Analysis

By Iain M. Johnstone, A. Lu. 2009

215 citations recorded by the source corpus.

Principal components analysis (PCA) is a classical method for the reduction of dimensionality of data in the form of n observations (or cases) of a vector with p variables. For a simple model of factor analysis type, it is proved that ordinary PCA can produce a consistent (for n large) estimate of the principal factor if and only if p(n) is asymptotically of smaller order than n. There may be a basis in which typical signals have sparse representations: most co-ordinates have small signal energies. If such a basis (e.g. wavelets) is used to represent the signals, then the variation in many coordinates is likely to be small. Consequently, we study a simple "sparse PCA" algorithm: select a subset of coordinates of largest variance, estimate eigenvectors from PCA on the selected subset, threshold and reexpress in the original basis. We illustrate the algorithm on some exercise ECG data, and prove that in a single factor model, under an appropriate sparsity assumption, it yields consistent estimates of the principal factor.

Subfields: Analytical Chemistry, Computational Mechanics, Signal Processing

View Paper: https://www.semanticscholar.org/paper/39ac586c519ef8569b4f401a257c29b2039d5d25

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Analytical Chemistry

In Computational Mechanics

In Signal Processing

Citations in the corpus

On consistency and sparsity for sliced inverse regression in high dimensions (2015, methodology): “The following lemma is borrowed from Johnstone and Lu [2004].”
Regularized estimation of large covariance matrices (2008, unknown): “19 09 v1 [ m at h. ST ] 1 3 M ar 2 00 8 The Annals of Statistics 2008, Vol. 36, No. 1, 199–227 DOI: 10.1214/009053607000000758 c© Institute of Mathematical Statistics, 2008”
0-1 Phase Transitions in Sparse Spiked Matrix Estimation (2019, methodology): “trix) and a Bernoulli-Rademacher distribution for V i˘P V;n= (1 ˆ V;n) 0+1 2 ˆ V;n( 1+ 1). This estimation problem is equivalent to the important “spiked covariance model” or “gaussian sparse-PCA” [3,4] which amounts to estimate a sparse binary matrix V Vfrom samples generated by the normal law N(0;”
High-dimensional covariance estimation by minimizing ℓ1-penalized log-determinant divergence (2008, background): “from a multivariate Gaussian distribution, neither the eigenvalues nor the eigenvectors of the sample covariance matrix are consistent estimators of the population versions [16, 17].”
MINIMAX ESTIMATION OF LARGE COVARIANCE MATRICES UNDER ℓ1-NORM (2012, background): “See, for example, Wu and Pourahmadi (2003), Zou, Hastie, and Tibshirani (2006), Bickel and Levina (2008a, b), El Karoui (2008), Lam and Fan (2009), Johnstone and Lu (2009), Cai, Zhang, and Zhou (2010), and Cai and Liu (2011).”
Asymptotics and Concentration Bounds for Bilinear Forms of Spectral Projectors of Sample Covariance (2014, background): “as n→ ∞.To overcome this diﬃculty, several authors proposed alternative estimators of the covariance matrix Σ and studied their performance under various sparsity assumptions on Σ. See, for instance, [12, 20, 23, 27, 32] and the references cited therein. V. Koltchinskii and K. Lounici/Asymptotics an”
Regression on manifolds: Estimation of the exterior derivative (2011, methodology): “We consider a linear case with errors-in-variables where the noise variance is identifiable [31, 28], and this distinguishes our setup from that of other linear regression setups [54, 30, 13, 60, 59].”
Estimating structured high-dimensional covariance and precision matrices: Optimal rates and adaptive estimation (2016, methodology): “Covariance and precision matrix estimation with incomplete data has been investigated in a few recent papers Lounici (2013, 2014); Kolar and Xing (2012); Loh and Wainwright (2012); Cai and Zhang (2015).”
PCA CONSISTENCY IN HIGH DIMENSION, LOW SAMPLE SIZE CONTEXT (2009, background): “see e.g. Paul [13], Johnstone and Lu [ 11 ], and Baik et al. [2].”
Optimal rates of convergence for covariance matrix estimation (2010, background): “…tapering in Furrer and Bengtsson (2007), thresholding in Bickel and Levina (2008b) and El Karoui (2008), penalized estimation in Huang et al. (2006), Lam and Fan (2007) and Rothman et al. (2008), regularizing principal components in Johnstone and Lu (2009) and Zou, Hastie and Tibshirani (2006).”
Covariance regularization by thresholding (2009, unknown): “30 79 v1 [ m at h. ST ] 2 0 Ja n 20 09 The Annals of Statistics 2008, Vol. 36, No. 6, 2577–2604 DOI: 10.1214/08-AOS600 c© Institute of Mathematical Statistics, 2008”
The spectral norm of random inner-product kernel matrices (2015, methodology): “Computationally-efficient procedures for estimating sparse principal components include diagonal thresholding [31, 32, 9], `1- and model-selection-penalization approaches [33, 62, 21, 49, 60, 12], iterative thresholding via the QR method [40], approximate message passing [22], and covariance thresho”
Compressed and Privacy-Sensitive Sparse Regression (2009, background): “The proof of this result uses inequalities from [31] and [32] on random variables and the sum of products of normals.”