Exponential expressivity in deep neural networks through transient chaos

By Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Narain Sohl-Dickstein, Surya Ganguli. 2016

683 citations recorded by the source corpus.

We combine Riemannian geometry with the mean field theory of high dimensional chaos to study the nature of signal propagation in generic, deep neural networks with random weights. Our results reveal an order-to-chaos expressivity phase transition, with networks in the chaotic phase computing nonlinear functions whose global curvature grows exponentially with depth but not width. We prove this generic class of deep random functions cannot be efficiently computed by any shallow network, going beyond prior work restricted to the analysis of single functions. Moreover, we formalize and quantitatively demonstrate the long conjectured idea that deep networks can disentangle highly curved manifolds in input space into flat manifolds in hidden space. Our theoretical analysis of the expressive power of deep networks broadly applies to arbitrary nonlinearities, and provides a quantitative underpinning for previously abstract notions about the geometry of deep functions.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Statistical and Nonlinear Physics, Model Architectures, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Statistical and Nonlinear Physics

In Model Architectures

In Scaling & Learning Theory

Citations in the corpus

A Closer Look at Memorization in Deep Networks (2017, methodology): “…techniques for analyzing the training of DNNs have been proposed before, including looking at generalization error, trajectory length evolution (Raghu et al., 2016), analyzing Jacobians associated to different layers (Wang; Saxe et al., 2013), or the shape of the loss minima found by SGD (Im…”
ReZero is All You Need: Fast Convergence at Large Depth (2020, background): “The expressivity of neural networks typically grows exponentially with depth [5], enabling strong generalization performance, but often induces vanishing/exploding gradients and poor signal propagation through the model [6].”
Neural-net-induced Gaussian process regression for function approximation and PDE solution (2018, background): “More recently, the authors of [4] extended [3]’s work to the case of deep NNs having generic activation functions, with reference to the mean ﬁeld theory of signal propagation [1, 5].”
Representing smooth functions as compositions of near-identity functions with implications for deep network optimization (2018, background): “A number of authors have investigated how using deep architecture affects the set of functions computed by a network (see Montufar et al., 2014; Telgarsky, 2015; Poole et al., 2016; Mhaskar et al., 2016).”
PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos (2019, background): “Notably, Poole et al. (2016) suggested that such properties lead to ”exponential expressivity”, but we believe that this phenomenon may instead be a curse.”
Deep Neural Networks as Gaussian Processes (2017, background): “They are exactly the relations derived in the mean ﬁeld theory of signal propagation in fully-connected random neural networks (Poole et al. (2016); Schoenholz et al. (2017)) and also appear in the literature on compositional kernels (Cho & Saul (2009); Daniely et al. (2016)).”
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation (2019, neutral): “This evolution is referred to as (forward and backward) signal propagation in the literature (Poole et al., 2016; Schoenholz et al., 2017; Yang & Schoenholz, 2017; 2018; Hanin & Rolnick, 2018; Chen et al., 2018; Yang et al., 2018; Pennington et al., 2017).”
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10, 000-Layer Vanilla Convolutional Neural Networks (2018, methodology): “We will follow the general methodology established in Poole et al. (2016); Schoenholz et al. (2017); Yang & Schoen-holz (2017).”
Recovery Guarantees for One-hidden-layer Neural Networks (2017, background): “The remarkable results of NNs on complex tasks in computer vision and natural language processing inspired works on the expressive power of NNs (Co-hen et al., 2016; Cohen & Shashua, 2016; Raghu et al., 2016; Daniely et al., 2016; Poole et al., 2016; Montufar et al., 2014; Telgarsky, 2016).”
Neural Tangents: Fast and Easy Infinite Neural Networks in Python (2019, background): “…NNGP kernel has additionally been extensively studied in the context of mean ﬁeld theory and initialization (Cho & Saul, 2009; Daniely et al., 2016; Poole et al., 2016; Schoenholz et al., 2016; Yang & Schoenholz, 2017; Xiao et al., 2018; Li & Nguyen, 2019; Pretorius et al., 2018; Hayou et al.,…”
On the Convergence Rate of Training Recurrent Neural Networks (2018, methodology): “There is huge literature on using the mean-ﬁeld theory to study neural networks [10, 14, 25, 30, 34, 38–40, 42, 47, 56–58].”
Biologically inspired protection of deep networks from adversarial attacks (2017, background): “An extensive analysis of this geometry, performed by Poole et al. (2016), revealed the existence of two phases in deep networks: a chaotic (ordered) phase when the random weights have high (low) variance.”
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice (2017, background): “Indeed, the mean squared singular value of J was shown simply to be L in [12, 13], so = 1 is a critical line of initializations with neither vanishing nor exploding gradients.”
How to Start Training: The Effect of Initialization and Architecture (2018, background): “An interesting systematic approach to predicting the correct multiplicative constant in the variance of weights as a function of the non-linearity is proposed in [22, 25] (e.g., the deﬁnition of χ 1 around (7) in Poole et al. [22]).”
A Mean Field Theory of Batch Normalization (2019, background): “Recently, progress has been made (Poole et al., 2016; Schoenholz et al., 2016; Daniely et al., 2016; Pennington et al., 2017; Hanin & Rolnick, 2018; Yang, 2019) in this direction by considering neural networks at initialization, before any training has occurred.”
Gaussian Process Behaviour in Wide Deep Neural Networks (2018, background): “A less obvious connection is to the recent series of papers studying deep networks using a mean field approximation (Poole et al., 2016; Schoenholz et al., 2017).”