Sparsistency and Rates of Convergence in Large Covariance Matrix Estimation

By Clifford Lam, Jianqing Fan. Annals of Statistics, 2007

624 citations recorded by the source corpus.

This paper studies the sparsistency and rates of convergence for estimating sparse covariance and precision matrices based on penalized likelihood with nonconvex penalty functions. Here, sparsistency refers to the property that all parameters that are zero are actually estimated as zero with probability tending to one. Depending on the case of applications, sparsity priori may occur on the covariance matrix, its inverse or its Cholesky decomposition. We study these three sparsity exploration problems under a unified framework with a general penalty function. We show that the rates of convergence for these problems under the Frobenius norm are of order (sn log pn/n)1/2, where sn is the number of nonzero elements, pn is the size of the covariance matrix and n is the sample size. This explicitly spells out the contribution of high-dimensionality is merely of a logarithmic factor. The conditions on the rate with which the tuning parameter λn goes to 0 have been made explicit and compared under different penalties. As a result, for the L1-penalty, to guarantee the sparsistency and optimal rate of convergence, the number of nonzero elements should be small: sn′=O(pn) at most, among O(pn2) parameters, for estimating sparse covariance or correlation matrix, sparse precision or inverse correlation matrix or sparse Cholesky factor, where sn′ is the number of the nonzero elements on the off-diagonal entries. On the other hand, using the SCAD or hard-thresholding penalty functions, there is no such a restriction.

Subfields: Artificial Intelligence, Computational Mechanics, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/c8d4be093cd4da26f0402b6b9b2680d7ffae2959

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Statistics and Probability

Citations in the corpus

Asymptotic normality and optimalities in estimation of large Gaussian graphical models (2013, background): “See also El Karoui (2008) and Lam and Fan (2009).”
Latent variable graphical model selection via convex optimization (2010, background): “The consistency properties of such an estimator have been studied [24], [22], [18], and under suitable conditions [18], [22] this estimator is also “sparsistent”, i.”
Sub-Gaussian estimators of the mean of a random matrix with heavy-tailed entries (2016, background): “r which achieves strong deviation guarantees in the Frobenius norm. Estimation of the covariance matrix with respect to this norm has been previously investigated in the literature, for instance, see [30], [9] and references therein; Frobenius norm is a natural choice when one wants to understand th”
STRONG ORACLE OPTIMALITY OF FOLDED CONCAVE PENALIZED ESTIMATION (2012, methodology): “…sparse generalized linear model (Fan and Lv, 2011), sparse Cox’s proportional hazards model (Bradic et al., 2012), sparse precision matrix estimation (Lam and Fan, 2009), sparse Ising model (Xue et al., 2012), and sparse quantile regression (Wang et al., 2012; Fan et al., 2013), among others.”
Covariance and precision matrix estimation for high-dimensional time series (2013, background): “…estimator has a convergence rate established in (9) with α = γ , ˜ α = γ ∧ (1 / 2 − 1 /q ) and ˜ β . assumed Note that our moment condition relaxes the commonly sub-Gaussian condition in previous literature [Rothman et al. (2008), Lam and Fan (2009), Zhou, Laf-ferty and Wasserman (2010)].”
An Overview on the Estimation of Large Covariance and Precision Matrices (2015, positive): “To further reduce the estimation bias, Lam and Fan (2009); Shen et al. (2012) proposed non-convex penalties for sparse precision matrix estimation and studied their theoretical properties.”
Estimating structured high-dimensional covariance and precision matrices: Optimal rates and adaptive estimation (2016, methodology): “Lam and Fan (2009) studied a generalization of (27) and replace the Lasso penalty by general non-convex penalties such as SCAD to overcome the bias issue.”
A unified framework for high-dimensional analysis of $M$-estimators with decomposable regularizers (2009, result): “Here there are a range of results, including convergence rates in Frobenius, operator and other matrix norms [35, 60, 64, 82], as well as results on model selection consistency [35, 45, 60].”
Efficient Estimation of Approximate Factor Models (2012, methodology): “To establish the consistency in the high dimensional literature, one usually constructs a neighborhood of the true parameters (Λ0,Σu0) ∈ U (e.g., Rothman et al. 2008, Lam and Fan 2009), and show that with probability approaching one, L2(Λ0,Σu0) > sup(Λ,Σu)/∈U L2(Λ,Σu).”
MINIMAX ESTIMATION OF LARGE COVARIANCE MATRICES UNDER ℓ1-NORM (2012, methodology): “See, for example, Wu and Pourahmadi (2003), Zou, Hastie, and Tibshirani (2006), Bickel and Levina (2008a, b), El Karoui (2008), Lam and Fan (2009), Johnstone and Lu (2009), Cai, Zhang, and Zhou (2010), and Cai and Liu (2011).”
Random matrix theory in statistics: A review (2014, background): “Lam and Fan (2009) compared different penalization and thresholding methods for sparse precision matrix estimation and derived rates of convergence under the Frobenius norm, while Cai et al. (2011) obtained rates of convergence of the l1-constrained estimator under the spectral norm.”
Gaussian and bootstrap approximations for high-dimensional U-statistics and their applications (2016, neutral): “non-zero off-diagonal entries in Σ [44].”
Non-Concave Penalized Likelihood with NP-Dimensionality (2009, background): “See, for example, Huang et al. (2006), Meinshausen and Bühlmann (2006), Levina et al. (2008), Rothman et al. (2008), and Lam and Fan (2009), among others.”
Generalized Thresholding of Large Covariance Matrices (2009, background): “The implicit regularizing assumption here is that variables far apart are only weakly correlated, and estimators that take advantage of this have been proposed by Wu and Pourahmadi (2003), Bickel and Levina (2004), Huang et al.”
Global Testing and Large-Scale Multiple Testing for High-Dimensional Covariance Structures (2017, background): “Lam, C. and J. Fan (2009).”
The Graphical Horseshoe Estimator for Inverse Covariance Matrices (2017, background): “t have a Bayesian representation, although Polson and Scott (2012) provide an understanding of how priors and penalty functions are related even when some penalty functions lack Bayesian equivalents. Lam and Fan (2009) showed that under certain conditions, both frequentist graphical lasso and graphi”