Sparsity information and regularization in the horseshoe and other shrinkage priors

By Juho Piironen, Aki Vehtari. 2017

464 citations recorded by the source corpus.

The horseshoe prior has proven to be a noteworthy alternative for sparse Bayesian estimation, but has previously suffered from two problems. First, there has been no systematic way of specifying a prior for the global shrinkage hyperparameter based on the prior information about the degree of sparsity in the parameter vector. Second, the horseshoe prior has the undesired property that there is no possibility of specifying separately information about sparsity and the amount of regularization for the largest coefficients, which can be problematic with weakly identified parameters, such as the logistic regression coefficients in the case of data separation. This paper proposes solutions to both of these problems. We introduce a concept of effective number of nonzero parameters, show an intuitive way of formulating the prior for the global hyperparameter based on the sparsity assumptions, and argue that the previous default choices are dubious based on their tendency to favor solutions with more unshrunk parameters than we typically expect a priori. Moreover, we introduce a generalization to the horseshoe prior, called the regularized horseshoe, that allows us to specify a minimum level of regularization to the largest values. We show that the new prior can be considered as the continuous counterpart of the spike-and-slab prior with a finite slab width, whereas the original horseshoe resembles the spike-and-slab with an infinitely wide slab. Numerical experiments on synthetic and real world data illustrate the benefit of both of these theoretical advances.

Subfields: Computational Mechanics, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/52e96e0132e632c82d504f4b4afdf19c1048f81a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Statistics and Probability

Citations in the corpus

Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors (2020, methodology): “In this section, we view HMC/NUTS sampling in Stan as the gold standard, which is slow (several hours per 1000 iterations) but mixes well in this dataset (Piironen and Vehtari, 2017b).”
The Kernel Interaction Trick: Fast Bayesian Discovery of Pairwise Interactions in High Dimensions (2019, methodology): “SKIM is based closely on the regularized horseshoe prior Piironen & Vehtari (2017) and the model proposed in Grifﬁn & Brown (2017): where s, α i , and β i are user-speciﬁed hyperparameters, C + (0 , 1) is a half-Cauchy distribution, and N + is a half-normal distribution.”
Uncertainty Quantification for the Horseshoe (with Discussion) (2016, methodology): “Piironen and Vehtari (2017b; 2017c) demonstrate that the effects of τ in horseshoe priors are intimately related to the measurement variability σ, even for the simple normal-linear measurement model.”
Pathfinder: Parallel quasi-Newton variational inference (2021, unknown): “The model has 3075 parameters and a horseshoe prior that performs soft variable selection (Piironen and Vehtari, 2017).”
Using Stacking to Average Bayesian Predictive Distributions (with Discussion) (2017, methodology): “For high-dimensional variable selection we recommend a projection predictive approach (Piironen and Vehtari, 2016, 2017a), which has a smaller variance in selection process due to the use of the encompassing model as a reference model and has better predictive performance due to making the…”
Bayesian Workflow (2020, methodology): “In contrast, projection predictive variable selection has been shown to be stable and reliable in finding smaller models with good predictive performance (Piironen and Vehtari, 2017, Piironen et al., 2020, Pavone et al., 2020).”