Avoiding pathologies in very deep networks

By David Kristjanson Duvenaud, Oren Rippel, Ryan P. Adams, Zoubin Ghahramani. 2014

163 citations recorded by the source corpus.

Choosing appropriate architectures and regularization strategies of deep networks is crucial to good predictive performance. To shed light on this problem, we analyze the analogous problem of constructing useful priors on compositions of functions. Specifically, we study the deep Gaussian process, a type of infinitely-wide, deep neural network. We show that in standard architectures, the representational capacity of the network tends to capture fewer degrees of freedom as the number of layers increases, retaining only a single degree of freedom in the limit. We propose an alternate network architecture which does not suffer from this pathology. We also examine deep covariance functions, obtained by composing infinitely many feature transforms. Lastly, we characterize the class of models obtained by performing dropout on Gaussian processes.

Subfields: Artificial Intelligence, Control and Systems Engineering, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

Calibrating Deep Convolutional Gaussian Processes (2018, methodology): “Combining this with CNN s leads to a particular form of Bayesian CNN s, much like GP s and DGP s are particular forms of Bayesian DNN s [6, 9, 22].”
Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks (2017, background): “People have built more expressive functions by automatically combining kernels together [14] as well as by composing GPs together [13, 11, 15].”
Deep Neural Networks as Gaussian Processes (2017, background): “Moreover, a body of work on other constructions of GP analogs for multilayer networks can be found in [9, 3, 8, 5, 1, 17].”
Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors (2016, background): “A similar scenario was also considered theoretically in (Duvenaud et al., 2014).”
Deep Gaussian Processes for Regression using Approximate Expectation Propagation (2016, background): “An extension of (Damianou & Lawrence, 2013) that has skip links from the inputs to every hidden layer in the network was proposed by Dai, Damianou, González, and Lawrence (2016), based on suggestions provided in (Duvenaud et al., 2014).”
Gaussian Process Behaviour in Wide Deep Neural Networks (2018, background): “Duvenaud et al. (2014) oﬀer an analysis that largely applies to DGPs though they also study the Cho and Saul recursion with the motivating argument from the original paper.”
On the energy landscape of deep networks (2015, background): “, 2013), deep Gaussian processes (Duvenaud et al., 2014), spin glasses (Choromanska et al.”
Random Feature Expansions for Deep Gaussian Processes (2016, methodology): “The connection between DGP s and DNN s has been pointed out in several papers, such as Neal (1996) and Duvenaud et al. (2014), where pathologies with deep nets are investigated.”
NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations (2018, background): “Note that this is not the same as the full input-to-output Jacobian, for instance as the one defined in (Duvenaud et al., 2014).”
Doubly Stochastic Variational Inference for Deep Gaussian Processes (2017, methodology): “A pathology with the DGP with zero mean function for the inner layers was identiﬁed in Duvenaud et al. (2014).”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, background): “Deep Gaussian processes (DGPs) [136], [137], [138], [139], [140], [141], [142] are effective multi-layer decision making models that can accurately model the uncertainty.”
Variational Auto-encoded Deep Gaussian Processes (2015, methodology): “Furthermore, we incorporate the deep GP modiﬁcation of Duvenaud et al. (2014) so that the observed input layer has an additional connection to the output layer.”
Deep convolutional Gaussian processes (2018, background): “While more exible and powerful than shallow GPs, deep Gaussian processes result in degenerate models if the individual GP layers are not invertible, which limits their potential (Duvenaud et al., 2014).”
Tensor Switching Networks (2016, background): “(5)Interestingly, a similar kernel is also observed by [21] for models with explicit skip connections.”