Covariance Estimation: The GLM and Regularization Perspectives

By Mohsen Pourahmadi. 2011

240 citations recorded by the source corpus.

Finding an unconstrained and statistically interpretable reparameterization of a covariance matrix is still an open problem in statistics. Its solution is of central importance in covariance estimation, particularly in the recent high-dimensional data environment where enforcing the positive-definiteness constraint could be computationally expensive. We provide a survey of the progress made in modeling covariance matrices from two relatively complementary perspectives: (1) generalized linear models (GLM) or parsimony and use of covariates in low dimensions, and (2) regularization or sparsity for high-dimensional data. An emerging, unifying and powerful trend in both perspectives is that of reducing a covariance estimation problem to that of estimating a sequence of regression problems. We point out several instances of the regression-based formulation. A notable case is in sparse estimation of a precision matrix or a Gaussian graphical model leading to the fast graphical LASSO algorithm. Some advantages and limitations of the regression-based Cholesky decomposition relative to the classical spectral (eigenvalue) and variance-correlation decompositions are highlighted. The former provides an unconstrained and statistically interpretable reparameterization, and guarantees the positive-definiteness of the estimated covariance matrix. It reduces the unintuitive task of covariance estimation to that of modeling a sequence of regressions at the cost of imposing an a priori order among the variables. Elementwise regularization of the sample covariance matrix such as banding, tapering and thresholding has desirable asymptotic properties and the sparse estimated covariance matrix is positive definite with probability tending to one for large samples and dimensions.

Subfields: Applied Mathematics, Artificial Intelligence, Computational Mathematics

View Paper: https://www.semanticscholar.org/paper/f48e7b996b1e96bc4e2a6117dcb2e6518c16acad

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Mathematics

In Artificial Intelligence

In Computational Mathematics

Citations in the corpus

Dynamic connectivity regression: Determining state-related changes in brain connectivity (2012, background): “However, the process of estimating the covariance matrix can at times be difficult (Stein, 1956; Pourahmadi, 2011), due to the positive definite constraint on the matrix and the high-dimensional nature of the data.”
Optimal Whitening and Decorrelation (2015, methodology): “This can be useful speciﬁcally in time course analysis to account for auto-correlation (cf. Pourahmadi, 2011, and references therein).”
A New Parametrization of Correlation Matrices (2020, methodology): “This makes the transformation potentially useful for regularization, see Pourahmadi (2011), and inference.”
The Gaussian Graphical Model in Cross-Sectional and Time-Series Data (2016, background): “It can then be shown (Pourahmadi, 2011) that 7 (3) Thus, κ ij is proportional to both γ ij and γ ji ; a zero in the inverse variance-covariance matrix indicates that one variable does not predict another.”
Optimizing Neural Networks with Kronecker-factored Approximate Curvature (2015, background): “Pourahmadi (2011) showed that the inverse covariance ma-Optimizing trix can be expressed as Σ Intuitively, this result says that each row of the inverse co-variance Σ − 1 is given by the coefﬁcients of the optimal linear predictor of the i -th variable from the others, up to a scaling factor.”
NETS: Network estimation for time series (2019, unknown): “The model captures an important fraction of total variability, on top of what is explained by volatility factors, and improves out-of-sample forecasting.”
Model Selection in Linear Mixed Models (2013, methodology): “One open issue with both Bondell, Krishna and Ghosh (2010) and Ibrahim et al. (2011) is that the Cholesky decompositions are dependent on the order in which the random effects appear and are not permutation invariant (Pourahmadi (2011)).”
A Tutorial on Regularized Partial Correlation Networks (2016, background): “…partial correlation coefficient between yi and y j is proportional to either the regression slope predicting yi from y j or the regression slope predicting y j from yi (Pourahmadi, 2011): Cor ( yi, y j | y−(i, j) ) = βi jSD(ε j) SD(εi) = β jiSD(εi) SD(ε j) , in which SD stands for the…”
Maximum likelihood estimation for linear Gaussian covariance models (2014, methodology): “haves as if it were a convex optimization problem. 1. Introduction. In many statistical analyses, the covariance matrix possesses a specic structure and must satisfy certain constraints. We refer to [Pou11] for a comprehensive review of covariance estimation in general and a discussion of numerous s”
Multivariate covariance generalized linear models (2015, methodology): “The first issue was discussed by Pinheiro and Bates (1996) and Pourahmadi (2011).”
The Graphical Horseshoe Estimator for Inverse Covariance Matrices (2017, methodology): “Pourahmadi (2011) provided a survey of some of the most popular methods in high-dimensional covariance and inverse covariance estimation.”