Posterior Concentration for Sparse Deep Learning

By Nicholas G. Polson, Veronika Rocková. 2018

106 citations recorded by the source corpus.

We introduce Spike-and-Slab Deep Learning (SS-DL), a fully Bayesian alternative to dropout for improving generalizability of deep ReLU networks. This new type of regularization enables provable recovery of smooth input-output maps with {\sl unknown} levels of smoothness. Indeed, we show that the posterior distribution concentrates at the near minimax rate for alpha-Holder smooth maps, performing as well as if we knew the smoothness level alpha ahead of time. Our result sheds light on architecture design for deep neural networks, namely the choice of depth, width and sparsity level. These network attributes typically depend on unknown smoothness in order to be optimal. We obviate this constraint with the fully Bayes construction. As an aside, we show that SS-DL does not overfit in the sense that the posterior concentrates on smaller networks with fewer (up to the optimal number of) nodes and links. Our results provide new theoretical justifications for deep ReLU networks from a Bayesian point of view.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sanjog Misra
2. Max H. Farrell
3. Tengyuan Liang
4. Simon Lacoste-Julien

In Computer Vision and Pattern Recognition

1. Max H. Farrell
2. Sanjog Misra
3. Tengyuan Liang

Citations in the corpus

• Deep Neural Networks for Estimation and Inference: Application to Causal Effects and Other Semiparametric Estimands (2018, background): “Given the renewed interest in deep networks, this is an area of study already (Hartford et al., 2017; Polson and Rockova, 2018) and we expect this to continue and that implementations will rapidly evolve.”