Analytical nonlinear shrinkage of large-dimensional covariance matrices

By Olivier Ledoit, Michael Wolf. Annals of Statistics, 2020

392 citations recorded by the source corpus.

This paper establishes the first analytical formula for nonlinear shrinkage estimation of large-dimensional covariance matrices. We achieve this by identifying and mathematically exploiting a deep connection between nonlinear shrinkage and nonparametric estimation of the Hilbert transform of the sample spectral density. Previous nonlinear shrinkage methods were of numerical nature: QuEST requires numerical inversion of a complex equation from random matrix theory whereas NERCOME is based on a sample-splitting scheme. The new analytical method is more elegant and also has more potential to accommodate future variations or extensions. Immediate benefits are (i) that it is typically 1000 times faster with basically the same accuracy as QuEST and (ii) that it accommodates covariance matrices of dimension up to 10,000 and more. The difficult case where the matrix dimension exceeds the sample size is also covered.

Subfields: Electrical and Electronic Engineering, Statistics and Probability, Signal Processing

View Paper: https://www.semanticscholar.org/paper/a426196935a5b5cb2397c96078ef307e29d96cbe

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Electrical and Electronic Engineering

In Statistics and Probability

In Signal Processing

Citations in the corpus

Optimal Shrinkage-Based Portfolio Selection in High Dimensions (2016, background): “In the seminal paper of Markowitz (1952) the author suggests to determine the optimal composition of a portfolio of financial assets by minimizing the portfolio variance assuming that the expected portfolio return attains some prespecified fixed value. By varying this value we obtain the whole effic”
Covariance Matrix Estimation under Total Positivity for Portfolio Selection* (2019, methodology): “…known as non-linear shrinkage , considers non-linear transforms of the eigenvalues according to the Marcenko-Pastur distribution, which describes the asymptotic distribution of eigenvalues of random matrices, and has been shown to out-perform the linear-shrinkage estimator empirically [LW12].”
A Nodewise Regression Approach to Estimating Large Portfolios (2016, background): “Ledoit and Wolf (2012) proposed a nonlinear shrinkage of the sample © 2019 American Statistical Association eigenvalues for the case where p < n, and Ledoit and Wolf (2015) extended this approach to the case where p > n.”
Sample Path Generation for Probabilistic Demand Forecasting (2018, background): “While there is a plethora of literature related to shrinkage estimators [37] [26] [27] - none have been applied to the problem of producing consistent demand forecasts.”
High‐dimensional covariance matrix estimation (2019, methodology): “Its calculation involves data splitting and eigenanalysis, which can be faster than the algorithm in Ledoit and Wolf (2012) when p is small to moderate in size (e.g. p of order of hundreds).”
Quadratic Shrinkage for Large Covariance Matrices (2020, background): “…(2001), Dey and Srinivasan (1985), Donoho, Gavish and Johnstone (2018), Efron and Morris (1976), Haff (1980, 1991), Krishnamoorthy and Gupta (1989), Ledoit and Wolf (2004, 2012), Lin and Perlman (1985), Loh (1991), Pal (1993), Won et al. (2013), Yang and Berger (1994), and the references therein.”