Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors

By Soumya Ghosh, Jiayu Yao, Finale Doshi-Velez. 2018

85 citations recorded by the source corpus.

Bayesian Neural Networks (BNNs) have recently received increasing attention for their ability to provide well-calibrated posterior uncertainties. However, model selection---even choosing the number of nodes---remains an open question. Recent work has proposed the use of a horseshoe prior over node pre-activations of a Bayesian neural network, which effectively turns off nodes that do not help explain the data. In this work, we propose several modeling and inference advances that consistently improve the compactness of the model learned while maintaining predictive performance, especially in smaller-sample settings including reinforcement learning.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Bayesian Structure Adaptation for Continual Learning (2019, background): “Adapting other sparse Bayesian structure learning methods, e.g. (Ghosh et al., 2018) to the continual learning setting is also a promising avenue.”
Continual Learning with Adaptive Weights (CLAW) (2019, methodology): “To automate adaptation, we perform inference on pi,j, which would have otherwise been a hyperparameter of the prior (Louizos et al., 2017; Molchanov et al., 2017; Ghosh et al., 2018).”
Model Selection in Bayesian Neural Networks via Horseshoe Priors (2017, methodology): “Finally, we note that aspects of this work have appeared in (Ghosh and Doshi-Velez, 2017a,b; Ghosh et al., 2018); here we organize these results and provide a cohesive picture on priors for model selection in Bayesian neural networks.”
Functional Variational Bayesian Neural Networks (2019, background): “[7] Soumya Ghosh, Jiayu Yao, and Finale Doshi-Velez.”
Continual Learning with Bayesian Neural Networks for Non-Stationary Data (2020, background): “This is because VB yields poor posterior approximations if the dataset is too small or the neural network architecture has too much capacity (cf. Ghosh et al. (2018), Fig.”
Multi-period Time Series Modeling with Sparsity via Bayesian Variational Inference (2017, methodology): “Then following [28,29], we use fully factorized variational family for νl and τkl, and structured variational family for βl, shown as below,”
Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights (2020, background): “Imposing structure over the weights with a carefully chosen prior has been observed to improve predictive performance [Ghosh et al., 2018, Neal, 2012, Blundell et al., 2015], but it has remained elusive how to express prior knowledge or handle interpolation or extrapolation in such models.”