On the Expressive Power of Deep Neural Networks
By Maithra Raghu, Ben Poole, Jon M. Kleinberg, Surya Ganguli, Jascha Narain Sohl-Dickstein. 2016
897 citations recorded by the source corpus.
We propose a new approach to the problem of neural network expressivity, which seeks to characterize how structural properties of a neural network family affect the functions it is able to compute. Our approach is based on an interrelated set of measures of expressivity, unified by the novel notion of trajectory length, which measures how the output of a network changes as the input sweeps along a one-dimensional path. Our findings can be summarized as follows: (1) The complexity of the computed function grows exponentially with depth. (2) All weights are not equal: trained networks are more sensitive to their lower (initial) layer weights. (3) Regularizing on trajectory length (trajectory regularization) is a simpler alternative to batch normalization, with the same performance.
Subfields: Artificial Intelligence, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Quoc V. Le
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Sergey Levine
- Kyunghyun Cho
In Scaling & Learning Theory
- Greg Yang
- Sam Schoenholz
- Yuanzhi Li
- Jeffrey Pennington
- Boris Hanin
- Jascha Narain Sohl-Dickstein
- Simon Shaolei Du
- Quanquan Gu
Citations in the corpus
- Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “, 2018), characterizes the expressivity of deep models (Raghu et al., 2017; Lu et al., 2017), and bounds the neural network generalization error (Du et al.”
- Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “A better theoretical understanding of the generalization ability of deep and wide models (Raghu et al., 2017; Arora et al., 2018), trainability challenges including exploding and vanishing gradients (Glo-rot and Bengio, 2010; Balduzzi et al., 2017) and empirical approaches to counter these…”
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, background): “Although prior studies (Raghu et al., 2017; Lin & Jegelka, 2018; Sharir & Shashua, 2018; Lu et al., 2018) have shown that network deep and width are both important for ConvNets’ expressive power, it still remains an open question of how to effectively scale a ConvNet to achieve better efficiency and”
- A Closer Look at Memorization in Deep Networks (2017, methodology): “Multiple techniques for analyzing the training of DNNs have been proposed before, including looking at generalization error, trajectory length evolution (Raghu et al., 2016), analyzing Jacobians associated to different layers (Wang; Saxe et al.”
- The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, background): “their ability to build complex functions with a much smaller number of parameters than needed for shallow networks [9–13].”
- High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “However, being overly parsimonious with early stages (such as through aggressive downsampling) can hurt performance, since the model needs enough capacity to extract good local features (Raghu et al., 2017b).”
- Swish: a Self-Gated Activation Function (2017, methodology): “…of Glorot & Bengio (2010) that is used widely for ReLU networks, Dinh et al. (2017) used the linearity of ReLU to form an equivalent sharp repa-rameterization of networks, and Raghu et al. (2016) measured expressivity by counting the number of piecewise linear regions learned by a ReLU network.”
- Sharp Minima Can Generalize For Deep Nets (2017, background): “From a representational point of view, it has been argued that deep networks can efficiently approximate certain functions (e.g. Montufar et al., 2014; Raghu et al., 2016).”
- On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization (2018, background): “Superior expressiveness of deeper networks, long suspected, is now confirmed by theory, albeit for fairly limited learning problems (Eldan & Shamir, 2015; Raghu et al., 2016; Lee et al., 2017; Cohen et al., 2017; Daniely, 2017; Arora et al., 2018).”
- Stable and Fair Classification (2019, background): “Moreover, several recent works studied stability in deep neural networks (Raghu et al., 2017; Vidal et al., 2017).”
- fPINNs: Fractional Physics-Informed Neural Networks (2018, methodology): “In this paper, we focus on the NN approaches due to the high expressive power of NNs in function approximation [24, 25, 26, 27].”
- Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations (2018, background): “, number of layers/neurons and form of activation functions) is crucial and in many cases still remains an art that relies on one’s ability to balance the trade off between expressivity and trainability of the neural network (Raghu et al., 2016).”
- Gradient Descent Finds Global Minima of Deep Neural Networks (2018, neutral): “Our analysis of the Gram matrices at random initialization is closely related to prior work on the analysis of infinite-width networks as Gaussian Processes (Raghu et al., 2016; Matthews et al., 2018; Lee et al., 2017; Schoenholz et al., 2016).”
- Exponential expressivity in deep neural networks through transient chaos (2016, neutral): “In a companion paper [18], we study several related measures of expressivity in deep random neural networks with piecewise linear activations.”
- Recovery Guarantees for One-hidden-layer Neural Networks (2017, background): “(Raghu et al., 2016; Poole et al., 2016; Montufar et al., 2014; Arora et al., 2017) design some measures of expressivity that display an exponential dependence on the depth of the network.”
- On the importance of single directions for generalization (2018, positive): “Interestingly, in all three networks, ablations in early layers were more impactful than ablations in later layers, consistent with theoretical observations (Raghu et al., 2016).”
- Mean Field Residual Networks: On the Edge of Chaos (2017, background): “A series of papers [9, 10, 11] investigated the “average behavior” of random neural networks sampled via w(l)”
- How to Start Training: The Effect of Initialization and Architecture (2018, background): “[8] already mentioned above, as well as [14, 22, 23, 25].”
- Deep Information Propagation (2016, background): “A pair of recent papers (Poole et al., 2016; Raghu et al., 2016) demonstrated that random neural networks are exponentially expressive in their depth.”