A well-conditioned estimator for large-dimensional covariance matrices

By Olivier Ledoit, Michael Wolf. 2004

2,952 citations recorded by the source corpus.

Many applied problems require a covariance matrix estimator that is not only invertible, but also well-conditioned (that is, inverting it does not amplify estimation error). For large-dimensional covariance matrices, the usual estimator--the sample covariance matrix--is typically not well-conditioned and may not even be invertible. This paper introduces an estimator that is both well-conditioned and more accurate than the sample covariance matrix asymptotically. This estimator is distribution-free and has a simple explicit formula that is easy to compute and interpret. It is the asymptotically optimal convex linear combination of the sample covariance matrix with the identity matrix. Optimality is meant with respect to a quadratic loss function, asymptotically as the number of observations and the number of variables go to infinity together. Extensive Monte Carlo confirm that the asymptotic results tend to hold well in finite sample.

Subfields: Computational Theory and Mathematics, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Statistics and Probability

Citations in the corpus

Nonparametric Independence Testing for Small Sample Sizes (2014, methodology): “ge Estimation We now develop a new family of linear shrinkage estimators for the cross covariance operator XY (estimators for XX follows naturally) by extending a now classical technique described by [10]. We derive the optimal (oracle) shrinkage towards the zero operator 0 (which maps each RKHS fun”
TESTS FOR PARAMETER INSTABILITY IN DYNAMIC FACTOR MODELS (2013, background): “…in probability; (2) taking the inverse of a high dimensional matrix will amplify the estimation error dramatically and lead to very poor results (Ledoit and Wolf, 2004); and (3) the number of time periods, T , can be even smaller than the dimension of the estimated covariance matrix, so the…”
Improving Mean Variance Optimization through Sparse Hedging Restrictions (2015, methodology): “We then solve the optimal weight (shrinkage intensity) by minimizing the loss function of Ledoit and Wolf (2004a), which is represented by the Frobenius norm of the distance between the true covariance and shrinkage estimator. standard deviations, the latter denoted by σΨ̂ρ , σŜ−1 , σEW, σJM, σLW,…”
Large-Scale Correlation Screening (2011, background): “The correlation screening problem treated here is not related to inverse covariance and covariance selection problems studied by many authors (see [6, 13, 19, 9, 8, 17] to name just a few from an increasing literature).”
Multivariate Shrinkage for Optimal Portfolio Weights (2007, methodology): “Jorion (1986), and Ledoit and Wolf (2003) propose portfolio procedure improvements via shrinkage estimators, optimizing the trade-off between sample moments and shrinkage targets.”
Covariance Estimation: The GLM and Regularization Perspectives (2011, methodology): “Now, one may choose α1 and α2 to optimize certain criterion (Ledoit and Wolf, 2004).”
An Overview on the Estimation of Large Covariance and Precision Matrices (2015, neutral): “Note that this paper is not an exhaustive survey, and many other regularization methods are also commonly used in the literature, e.g., the shrinkage method (Ledoit and Wolf, 2003, 2004).”
Risk Reduction in Large Portfolios: Why Imposing the Wrong Constraints Helps (2003, neutral): “For comparison purposes, we also reported the row average covariances of the Ledoit estimate of the covariance matrix S Ledoit , which is a shrinkage estimator of the covariance matrix (Ledoit (1996, 1999)).”
Using Principal Component Analysis to Estimate a High Dimensional Factor Model with High-Frequency Data (2017, background): “A factor model can serve as the reference point for shrinkage estimation (see Ledoit and Wolf (2012) and Ledoit and Wolf (2004)).”
A Nodewise Regression Approach to Estimating Large Portfolios (2016, methodology): “Ledoit and Wolf (2004) proposed an estimator for highdimensional covariance matrices that is invertible and well- conditioned.”
Improving Portfolio Selection Using Option-Implied Volatility and Skewness (2013, methodology): “…daily rather than monthly returns (Jagannathan and Ma, 2003), using Bayesian methods (Jobson, Korkie, and Ratti, 1979; Jorion, 1986; Pástor, 2000; Ledoit and Wolf, 2004b), constraining shortsales (Jagannathan and Ma, 2003), constraining the norm of the vector of portfolio weights (DeMiguel,…”
Maximum likelihood estimation for linear Gaussian covariance models (2014, methodology): “The suggested initiation is motivated by shrinkage estimators that are used for high dimensional covariance matrix estimation (Ledoit and Wolf, 2004; Schäfer and Strimmer, 2005).”
High-dimensional covariance estimation by minimizing ℓ1-penalized log-determinant divergence (2008, methodology): “We illustrate our theoretical results via simulations forva ious graphs and problem parameters, showing good correspondences between the theoretical predictionsand behavior in simulations.”
Bayesian Synthetic Likelihood (2017, background): “Lee and Latuszyński (2014) show that the ABC MCMC kernel that we use in this paper is not geometrically ergodic.”