Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity

By Amit Daniely, Roy Frostig, Yoram Singer. 2016

362 citations recorded by the source corpus.

We develop a general duality between neural networks and compositional kernels, striving towards a better understanding of deep learning. We show that initial representations generated by common random initializations are sufficiently rich to express all functions in the dual kernel space. Hence, though the training objective is hard to optimize in the worst case, the initial weights form a good starting point for optimization. Our dual view also reveals a pragmatic and aesthetic perspective of neural networks and underscores their expressive power.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Scaling & Learning Theory

Citations in the corpus

Neural Tangent Kernel: Convergence and Generalization in Neural Networks (2018, background): “ restriction of the limiting NTK n(L) 1 to the unit sphere S 0 1 = fx2Rn 0 : xTx= 1gis positivedeﬁnite if L2. A key ingredient for the proof of Proposition 2 is the following Lemma, which comes from [4]. Lemma 2 (Lemma 12(a) in suppl. mat. of [4]). Let ^ : [ 1;1] !R denote the dual of a Lipschitz fu”
Feature Purification: How Adversarial Training Performs Robust Deep Learning (2020, neutral): “[4, 7, 8, 13, 14, 26–29, 38, 45, 50, 57, 61, 104, 115]).”
What Can ResNet Learn Efficiently, Going Beyond Kernels? (2019, methodology): “Most of these works rely on the so-called Neural Tangent Kernel (NTK) technique [12, 20], by relating the training process of sufficiently over-parameterized (or even infinite-width) neural networks to the learning process over a kernel whose features are defined by the randomly initialized weights ”
Deep Learning with Differential Privacy (2016, neutral): “22 , 1e-5 ).”
Deriving Neural Architectures from Sequence and Graph Kernels (2017, neutral): “Similar results have been made for convolutional neural nets (Anselmi et al., 2015), and general computational graphs (Daniely et al., 2016).”
On the Expressive Power of Deep Neural Networks (2016, background): “Given a neural network with piecewise linear activations (such as ReLU or hard tanh), the function it computes is also piecewise linear, a consequence of the fact that composing piecewise linear functions results in a piecewise linear function.”
Convergence Analysis of Two-layer Neural Networks with ReLU Activation (2017, background): “It is well known that neural networks have great expressive power [22, 7, 3, 8, 31].”
Deep Neural Networks as Gaussian Processes (2017, background): “Daniely et al. (2016) extends the construction of compositional kernels to neural networks whose underlying directed acyclic graph (which they term a “computation skeleton”) is of general form.”
Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks (2019, methodology): “Lastly, our work is related to kernel methods, especially recent discoveries of the connection between deep learning and kernels (Jacot et al., 2018; Chizat & Bach, 2018b; Daniely et al., 2016; Daniely, 2017).”
On Exact Computation with an Infinitely Wide Neural Net (2019, background): “Daniely et al. [2016] proposed a general framework to transform a neural network to a compositional kernel and later Daniely [2017] showed for suﬃciently wide neural networks, stochastic gradient descent can learn functions that lie in the corresponding reproducing kernel Hilbert space.”
Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers (2018, background): “On a separate note, if one treats all but the last layer of neural networks as generating a random feature map, then training only the last layer is a convex task, so one can learn the class of linear functions in this implicit feature space [15, 16].”
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation (2019, methodology): “, 2016), phase retrieval (Schniter & Rangan, 2015), and community detection in graphs (Deshpande et al., 2017). The behavior of the AMP algorithm is accurately described by a formalism called “stated evolution” (SE), as n,N →∞ with constant ratio n/N → δ ∈ (0,∞), that bears some resemblance to the e”
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10, 000-Layer Vanilla Convolutional Neural Networks (2018, background): “…(cid:63) denotes 2D circular cross-correlation, i.e. for any matrix C , A (cid:63) C is deﬁned as, (2.4) The function C : PSD n → PSD n is related to the C -map deﬁned in Poole et al. (2016) (see also (Daniely et al., 2016)) and is given by, (2.5) All but the two dimensions α and α (cid:48) in eqn.”
Recovery Guarantees for One-hidden-layer Neural Networks (2017, background): “The remarkable results of NNs on complex tasks in computer vision and natural language processing inspired works on the expressive power of NNs (Co-hen et al., 2016; Cohen & Shashua, 2016; Raghu et al., 2016; Daniely et al., 2016; Poole et al., 2016; Montufar et al., 2014; Telgarsky, 2016).”
Neural Tangents: Fast and Easy Infinite Neural Networks in Python (2019, background): “The recurrence relationship defining the NNGP kernel has additionally been studied separately from the GP correspondence in (Cho and Saul, 2009; Daniely et al., 2016; Poole et al., 2016; Schoenholz et al., 2016; Yang and Schoenholz, 2017; Xiao et al., 2018).”
On the Convergence Rate of Training Recurrent Neural Networks (2018, methodology): “There is huge literature on using the mean-ﬁeld theory to study neural networks [10, 14, 25, 30, 34, 38–40, 42, 47, 56–58].”
Learning One-hidden-layer Neural Networks with Landscape Design (2017, methodology): “Livni et al. [LSSS14], Zhang et al. [ZLJ16, ZLWJ17], and Daniely et al. [DFS16] study the learnability of special cases of neural networks using ideas from kernel methods.”