Spike and slab variable selection: Frequentist and Bayesian strategies

By BY Hemant Ishwaran, Sunil Rao. 2005

1,174 citations recorded by the source corpus.

Variable selection in the linear regression model takes many apparent faces from both frequentist and Bayesian standpoints. In this paper we introduce a variable selection method referred to as a rescaled spike and slab model. We study the importance of prior hierarchical specifications and draw connections to frequentist generalized ridge regression estimation. Specifically, we study the usefulness of continuous bimodal priors to model hypervariance parameters, and the effect scaling has on the posterior mean through its relationship to penalization. Several model selection strategies, some frequentist and some Bayesian in nature, are developed and studied theoretically. We demonstrate the importance of selective shrinkage for effective variable selection in terms of risk misclassification, and show this is achieved using the posterior from a rescaled spike and slab model. We also show how to verify a procedure’s ability to reduce model uncertainty in finite samples using a specialized forward selection strategy. Using this tool, we illustrate the effectiveness of rescaled spike and slab models in reducing model uncertainty.

Subfields: Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/17a076aa8b9451757ab36488891c3962c4f9563b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Scale-Dependent Priors for Variance Parameters in Structured Additive Distributional Regression (2016, background): “For example, the parameters of spike and slab priors for variable or function selection (George and Mc Culloch, 1993; Ishwaran and Rao, 2005; Scheipl et al., 2012) could also be based on analogous considerations for the scaling of effects under both the spike and the slab component.”
Robustly representing inferential uncertainty in deep neural networks through sampling (2016, methodology): “In addition to Bernoulli and Gaussian distributions, spike-and-slab distributions (Ishwaran & Rao, 2005), a combination of the two have been discussed.”
Learning Weight Uncertainty with Stochastic Gradient MCMC for Shape Classification (2016, methodology): “Similarly, binary DropConnect can be viewed as using the Spike-andSlab prior [30] on the weights without updating the prior parameters during inference.”
Graph Learning From Data Under Laplacian and Structural Constraints (2016, background): “Also, the connectivity and Laplacian constraints in Problems 1, 2 and 3 can be incorporated in a Bayesian setting by choosing spike-and-slab prior and improper prior distributions [62] on v and w, so that spike priors correspond to zero edge weights, and slab priors allow nonnegative edge weights.”
Uncertainty in Deep Learning (2016, methodology): “Spike and slab distributions return realisations which are identically zero with some probability (the spike) or sampled from a wide Gaussian otherwise (the slab), and have been used in the context of variable selection for example [George and McCulloch, 1997; Ishwaran and Rao, 2005; Madigan and Raf”
Bayesian Regression Using a Prior on the Model Fit: The R2-D2 Shrinkage Prior (2016, background): “s: two component discrete mixture prior (also referred as spike and slab prior) and continuous shrinkage priors. The discrete mixture priors (Mitchell and Beauchamp, 1988; George and McCulloch, 1993; Ishwaran and Rao, 2005; Narisetty et al., 2014) put a point mass (spike) at j = 0 and a continuous p”
BAYESIAN HYPER‐LASSOS WITH NON‐CONVEX PENALIZATION (2011, background): “George & McCulloch (1997), West (2003), Ishwaran & Rao (2005) and Fahrmeir, Kneib & Konrath (2010) explore their use for univariate responses, and Brown, Vannucci & Fearn (1998) consider multivariate ∗Author to whom correspondence should be addressed.”
Bayesian forecasting using stochastic search variable selection in a VAR subject to breaks (2010, background): “Such spike and slab priors have become popular for model selection or model averaging in regression models [see, e.g., Ishwaran and Rao (2005)].”
Smart Regularization of Deep Architectures (2015, background): “We refer any interested reader to [19] for more information.”
The Rhetoric and Reality of Anthropomorphism in Artificial Intelligence (2019, background): “Several sparsity-inducing Bayesian priors have been proposed to similar effect (Carvalho et al. 2010; Ishwaran and Rao 2005).”
Optimization with Sparsity-Inducing Penalties (2011, methodology): “Another line of work using Bayesian inference considers using priors on the loading vectors wi that put non-zero mass on exact zeros, leading to so-called “spike-and-slab” priors [54].”
Structured sparsity through convex optimization (2011, neutral): “, “spike and slab” (Ishwaran and Rao, 2005) • Less theory than frequentist methods”
Generalized Beta Mixtures of Gaussians (2011, methodology): “An alternative and widely applied Bayesian framework relies on variable selection priors and Bayesian model selection/averaging [18, 9, 16, 15].”
Modeling User Exposure in Recommendation (2015, background): “and-slab model [10] where the “spike” comes from the exposure variables and the matrix factorization component forms the flat “slab” part.”
Review and evaluation of penalised regression methods for risk prediction in low‐dimensional data with few events (2015, background): “[37] suggested moving the mixture element one level down, that is, on the variances rather than on the regression coefficients directly.”
The Bayesian Lasso (2008, methodology): “A better summary would be separate measures of the centres of each mode, along with the approximate amount of probability associated with each, in the spirit of “spike and slab” models (Ishwaran and Rao, 2005), but this would require an entirely different methodology.”