Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequences

By Iain M. Johnstone, Bernard Walter Silverman. 2004

517 citations recorded by the source corpus.

An empirical Bayes approach to the estimation of possibly sparse sequences observed in Gaussian white noise is set out and investigated. The prior considered is a mixture of an atom of probability at zero and a heavy-tailed density y, with the mixing weight chosen by marginal maximum likelihood, in the hope of adapting between sparse and dense sequences. If estimation is then carried out using the posterior median, this is a random thresholding procedure. Other thresholding rules employing the same threshold can also be used. Probability bounds on the threshold chosen by the marginal maximum likelihood approach lead to overall risk bounds over classes of signal sequences of length n, allowing for sparsity of various kinds and degrees. The signal classes considered are nearly black sequences where only a proportion η is allowed to be nonzero, and sequences with normalized p norm bounded by η, for η > 0 and 0 1. Simulations show excellent performance. For appropriately chosen functions y, the method is computationally tractable and software is available. The extension to a modified thresholding method relevant to the estimation of very sparse sequences is also considered.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/ac2ebdb7fc572b75f4f8a8afb6abd104bb9a233b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Empirical Bayes vs. Fully Bayes Variable Selection (2008, background): “Of particular interest for this paper, are the empirical Bayes (EB) criteria proposed by George and Foster (2000) which were further developed and extended by Clyde and George (2000) and Johnstone and Silverman (2004).”
Adaptive posterior contraction rates for the horseshoe (2017, methodology): “Those most directly related to this work are [30, 20, 9, 8, 18, 16, 19, 14, 5, 3, 2, 25].”
The Horseshoe Estimator: Posterior Concentration around Nearly Black Vectors (2014, methodology): “By adapting the approach in Paragraph 6.2 in (Johnstone and Silverman, 2004), we can ﬁnd conditions under which the horseshoe estimator with an empirical Bayes estimate of τ will still attain the minimax ℓ 2 risk.”
Shrink Globally, Act Locally: Sparse Bayesian Regularization and Prediction (2012, unknown): “For example, β may be of small α norm for some suitably small α, or its entries may decay in absolute value according to some power law [e.g. Johnstone and Silverman, 2004].”
Mean field variational Bayes for continuous sparse signal shrinkage: Pitfalls and remedies (2014, unknown): “[17] Johnstone, I.M. and Silverman, B.W. (2004).”
High Dimensional Statistical Inference and Random Matrices (2006, background): “Bounds such as (23) are reminiscent of those for estimation of sparse mean sequences in white Gaussian noise [12, 32, 59].”
Sparse Solution Of Underdetermined Linear Equations By Stagewise Orthogonal Matching Pursuit (2006, background): “The problem of separating ‘signal’ from ‘noise’ in such problems has generated a large literature including the papers [24, 27, 26, 1, 23, 37], which influenced our way of thinking.”
BAYESIAN LINEAR REGRESSION WITH SPARSE PRIORS (2014, background): “In the sequence model considered in [19] and [10] the observation is a vector (Y1, .”
The Spike-and-Slab LASSO (2018, background): “However, instead of estimating θ from the marginal likelihood (as in Johnstone and Silverman (2004)), here it is estimated from the global mode functional of the data.”
Asymptotically minimax empirical Bayes estimation of a sparse normal mean vector (2013, methodology): “…hard thresholding estimators (Donoho and Johnstone 1994), Bayes and empirical Bayes estimators based on priors with a two-groups structure (Castillo and van der Vaart 2012; Johnstone and Silverman 2004), and a new estimator based on the one-group Dirichlet–Laplace prior (Bhattacharya et al. 2014).”
Sparse Solution of Underdetermined Systems of Linear Equations by Stagewise Orthogonal Matching Pursuit (2012, background): “The problem of separating “signal” from “noise” in such problems has generated a large literature including the papers [1], [23], [24], [26], [27], [37], which influenced our way of thinking.”
Rao–Blackwellization for Bayesian Variable Selection and Model Averaging in Linear and Binary Regression: A Novel Data Augmentation Approach (2011, background): “Johnstone and Silverman (2004, 2005) also demonstrate the superior performance of mixtures of a point mass and heavy-tailed prior distributions compared to mixtures of a point mass and Gaussian prior distributions.”
NONPARAMETRIC EMPIRICAL BAYES AND COMPOUND DECISION APPROACHES TO ESTIMATION OF A HIGH-DIMENSIONAL VECTOR OF NORMAL MEANS (2009, methodology): “Many recent statistical results have focused on the importance of treating situations involving “sparse” models [see, e.g., Donoho and Johnstone (1994), Johnstone and Silverman (2004) and Efron (2003)].”
Real nonparametric regression using complex wavelets (2004, unknown): “In (5) recall that t/τ < |μ|2 and soφ0,σ(|μ|) 6 φ0,σ( √ t/τ) so control by the equivalent of tφ(t) is possible in our version (as in Johnstone and Silverman (2002b)).”
Generalized Beta Mixtures of Gaussians (2011, methodology): “An alternative and widely applied Bayesian framework relies on variable selection priors and Bayesian model selection/averaging [18, 9, 16, 15].”