Identity Matters in Deep Learning

By Moritz Hardt, Tengyu Ma. 2016

421 citations recorded by the source corpus.

An emerging design principle in deep learning is that each layer of a deep artificial neural network should be able to easily express the identity transformation. This idea not only motivated various normalization techniques, such as \emph{batch normalization}, but was also key to the immense success of \emph{residual networks}. In this work, we put the principle of \emph{identity parameterization} on a more solid theoretical footing alongside further empirical progress. We first give a strikingly simple proof that arbitrarily deep linear residual networks have no spurious local optima. The same result for linear feed-forward networks in their standard parameterization is substantially more delicate. Second, we show that residual networks with ReLu activations have universal finite-sample expressivity in the sense that the network can represent any function of its sample provided that the model has more parameters than the sample size. Directly inspired by our theory, we experiment with a radically simple residual architecture consisting of only residual convolutional layers and ReLu activations, but no batch normalization, dropout, or max pool. Our model improves significantly on previous all-convolutional networks on the CIFAR10, CIFAR100, and ImageNet classification benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Convergence Theory for Deep Learning via Over-Parameterization (2018, background): “…the main purpose here is to illustrate the generality of our techniques but not to attack each speciﬁc setting, in Appendix C, we choose to consider the simplest residual setting h (cid:96) = φ ( h (cid:96) − 1 + W h (cid:96) − 1 ) (that was also studied for instance by theoretical work [26]).”
Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data (2018, background): “It is also observed that overparameterization and proper random initialization can help the optimization [28, 12, 26, 18], but it is also not well understood why a particular initialization can improve learning.”
ReZero is All You Need: Fast Convergence at Large Depth (2020, methodology): “The original Transformer [21] implementation can be seen as a residual network with post-normalization (row 5 in Table 1).”
Gradient Descent Provably Optimizes Over-parameterized Neural Networks (2018, methodology): “Landscape analyses of ReLU activated neural networks for other settings have also been studied in many previous works (Ge et al., 2017; Safran and Shamir, 2016; Zhou and Liang, 2017; Freeman and Bruna, 2016; Hardt and Ma, 2016; Nguyen and Hein, 2018).”
Fixup Initialization: Residual Learning Without Normalization (2019, background): “Srivastava et al. (2015); Hardt & Ma (2016); Goyal et al. (2017); Kingma & Dhariwal (2018) found that initializing the residual branches at (or close to) zero helped optimization.”
On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization (2018, background): “Relevant theoretical guarantees, in the context of regret minimization, were given in Hazan et al. (2007); Duchi et al.”
Visualizing the Loss Landscape of Neural Nets (2017, background): “Theoretical results usually make restrictive assumptions about the sample distributions, non-linearity of the architecture, or loss functions [16, 31, 40, 36, 9, 39].”
How Does Batch Normalization Help Optimization? (No, It Is Not About Internal Covariate Shift) (2018, methodology): “We choose to not experiment with ResNets [7] since they seem to provide several similar benefits to BatchNorm [6] and would make our study less clean.”
Architecture Matters in Continual Learning (2022, background): “Many works have been done to explain why skip connections are useful: Hardt and Ma (2016) show that skip connection tends to eliminate spurious local optima; Bartlett et al. (2018b) study the function expressivity of residual architectures; Bartlett et al. (2018a) show that gradient descent…”
Representing smooth functions as compositions of near-identity functions with implications for deep network optimization (2018, methodology): “Finally, Theorem 2.1 in Hardt and Ma (2017) shows that we can decompose the Jacobian”
A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks (2018, background): “One recent progress is the analysis of Bartlett et al. (2018) for linear residual networks — a particular subclass of linear neural networks in which the input, output and all hidden dimensions are equal, and all layers are initialized to be the identity matrix ( cf. Hardt and Ma (2016)).”
Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks (2019, background): “Many papers tried to characterize geometric landscapes of objective functions (Safran & Shamir, 2017; Zhou & Liang, 2017; Freeman & Bruna, 2016; Hardt & Ma, 2016; Nguyen & Hein, 2017; Kawaguchi, 2016; Venturi et al., 2018; Soudry & Carmon, 2016; Du & Lee, 2018; Soltanolkotabi et al., 2018; Haeﬀele…”
The Shattered Gradients Problem: If resnets are the answer, then what is the question? (2017, background): “A recent paper showed resnets have universal ﬁnite-sample expressivity and may lack spurious local optima (Hardt & Ma, 2017) but this does not explain why deep feedforward networks are harder to train than resnets.”
Gradient Descent Finds Global Minima of Deep Neural Networks (2018, methodology): “This architecture was previously studied in Hardt and Ma [2016]. We study this architecture for the ease of presentation and analysis.”
Understanding Batch Normalization (2018, neutral): “Under our assumption E (cid:2) (cid:107)∇ (cid:96) i ( x ) − ∇ (cid:96) ( x ) 2 ≤ C we now have”
Training Deeper Neural Machine Translation Models with Transparent Attention (2018, background): “Residual connections have been shown, in theory and practice, to improve training stability and performance of deeper networks (see (He et al., 2015; Philipp et al., 2017; Hardt and Ma, 2017; Orhan, 2017)).”
Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks (2018, background): “Deep linear networks Beyond shallow one-hidden-layer neural networks, a series of recent work (Hardt and Ma, 2016; Kawaguchi, 2016; Bartlett et al., 2018; Arora et al., 2018a,b) focus on the optimization landscape of deep linear networks.”
Recovery Guarantees for One-hidden-layer Neural Networks (2017, background): “…setting, where the global optima or local minima close to the global optima will be achieved when the number of parameters is large enough, including (Freeman & Bruna, 2016; Haeffele & Vidal, 2015; Livni et al., 2014; Dauphin et al., 2014; Safran & Shamir, 2016; Hardt & Ma, 2017).”
Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs (2017, background): “Another interesting case is linear dynamical systems, where (Hardt et al., 2016) show that under independence assumptions maximum likelihood is quasi-concave and hence solvable with gradient ascent.”