Model Selection in Bayesian Neural Networks via Horseshoe Priors

By Soumya Ghosh, Jiayu Yao, Finale Doshi-Velez. 2017

133 citations recorded by the source corpus.

Bayesian Neural Networks (BNNs) have recently received increasing attention for their ability to provide well-calibrated posterior uncertainties. However, model selection---even choosing the number of nodes---remains an open question. In this work, we apply a horseshoe prior over node pre-activations of a Bayesian neural network, which effectively turns off nodes that do not help explain the data. We demonstrate that our prior prevents the BNN from under-fitting even when the number of nodes required is grossly over-estimated. Moreover, this model selection over the number of nodes doesn't come at the expense of predictive or computational performance; in fact, we learn smaller networks with comparable predictive performance to current approaches.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors (2020, background): “Section 3.2 details this and differences from other hierarchical priors (Louizos et al., 2017; Ghosh & Doshi-Velez, 2017).”
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors (2018, methodology): “Recent works analyzing priors have been specific to compression and model selection (Ghosh and Doshi-Velez, 2017; Louizos et al., 2017); this is with the exception of Flam-Shepherd et al.”
Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors (2018, background): “More closely related to us, are the recent works of (Ghosh & Doshi-Velez, 2017) and (Louizos et al., 2017).”
Multi-period Time Series Modeling with Sparsity via Bayesian Variational Inference (2017, methodology): “Then following [28,29], we use fully factorized variational family for νl and τkl, and structured variational family for βl, shown as below,”
Mapping Gaussian Process Priors to Bayesian Neural Networks (2017, background): “What defines a reasonable prior to use when forming and training Bayesian models? Recent work, such as [1] seek to improve priors over network parameters, by applying a horseshoe prior over preactivations of a Bayesian neural network (BNN) to effectively turn off weights that do not explain the data”
Posterior Concentration for Sparse Deep Learning (2018, background): “The second approach was explored by Ghosh and Doshi-Velez (2017), who suggested assigning a Horseshoe prior on the node preactivators, diminishing inﬂuence of individual neurons.”
Action and Perception as Divergence Minimization (2020, methodology): “This includes hierarchical priors (Louizos and Welling, 2016; Ghosh and Doshi-Velez, 2017), data priors (Louizos and Welling, 2016; Hafner et al., 2019b; Sun et al., 2019), ﬂexible posteriors (Louizos and Welling, 2016; Sun et al., 2017; Louizos and Welling, 2017; Zhang et al., 2018; Chang et al.,…”
Noise Contrastive Priors for Functional Uncertainty (2018, unknown): “Other works have analyzed priors for compression and model selection (Ghosh and Doshi-Velez, 2017; Louizos et al., 2017).”
A survey of uncertainty in deep neural networks (2021, background): “Notable examples include Bayesian model selection [109], [110], [111], [112], model compression [76], [113], [114], active learning [115], [23], [116], continual learning [117],”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, methodology): “[369] exploited the horseshoe, continuous shrinkage priors and the regularized”