How to Start Training: The Effect of Initialization and Architecture

By Boris Hanin, David Rolnick. 2018

282 citations recorded by the source corpus.

We investigate the effects of initialization and architecture on the start of training in deep ReLU nets. We identify two common failure modes for early training in which the mean and variance of activations are poorly behaved. For each failure mode, we give a rigorous proof of when it occurs at initialization and how to avoid it. The first failure mode, exploding/vanishing mean activation length, can be avoided by initializing weights from a symmetric distribution with variance 2/fan-in. The second failure mode, exponentially large variance of activation length, can be avoided by keeping constant the sum of the reciprocals of layer widths. We demonstrate empirically the effectiveness of our theoretical results in predicting when networks are able to start training. In particular, we note that many popular initializations fail our criteria, whereas correct initialization and architecture allows much deeper networks to be trained.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Training data-efficient image transformers & distillation through attention (2020, methodology): “After testing several options in preliminary experiments, some of them not converging, we follow the recommendation of Hanin and Rolnick [18] to initialize the weights with a truncated normal distribution.”
High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “This biases the signal towards the skip path, which ensures that the network has well-behaved gradients early in training, enabling efficient optimization (Balduzzi et al., 2017; Hanin & Rolnick, 2018; Yang et al., 2019).”
Fixup Initialization: Residual Learning Without Normalization (2019, background): “…variance of each residual branch Var[ F l ( x l ) x l ] will be about the same as its input variance Var[ x l ] , and thus Var | ≈ This causes the output variance to explode exponentially with depth without normalization (Hanin & Rolnick, 2018) for positively homogeneous blocks (see Deﬁnition 1).”
Salvaging Federated Learning by Local Adaptation (2020, methodology): “As a consequence, the federated model is so bad for these participants than when it is used to initialize local adaptation, the ﬁnal adapted model still has poor accuracy (Grosse et al., 2019; Hanin & Rolnick, 2018).”
On Lazy Training in Differentiable Programming (2018, background): “Moreover, for very large neural networks of more than 2 layers, this choice of initialization is essentially mandatory to avoid exploding or vanishing initial gradients [15, 14] if the weights are independent with zero mean.”
Segmenter: Transformer for Semantic Segmentation (2021, methodology): “The decoders, described in Section 3.2 are initialized with random weights from a truncated normal distribution [25].”
Multiscale Vision Transformers (2021, methodology): “We use truncated normal distribution initialization [42] and adopt synchronized AdamW [76] optimization with a base learning rate of 0.”
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation (2019, neutral): “This evolution is referred to as (forward and backward) signal propagation in the literature (Poole et al., 2016; Schoenholz et al., 2017; Yang & Schoenholz, 2017; 2018; Hanin & Rolnick, 2018; Chen et al., 2018; Yang et al., 2018; Pennington et al., 2017).”
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10, 000-Layer Vanilla Convolutional Neural Networks (2018, methodology): “…work has used mean ﬁeld theory to build a theoretical understanding of neural networks with random parameters (Poole et al., 2016; Schoenholz et al., 2017; Yang & Schoenholz, 2017; Schoenholz et al., 2017; Karakida et al., 2018; Hayou et al., 2018; Hanin & Rolnick, 2018; Yang & Schoenholz, 2018).”
A Mean Field Theory of Batch Normalization (2019, background): “Recently, progress has been made (Poole et al., 2016; Schoenholz et al., 2016; Daniely et al., 2016; Pennington et al., 2017; Hanin & Rolnick, 2018; Yang, 2019) in this direction by considering neural networks at initialization, before any training has occurred.”
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes (2018, background): “The line of work examining signal propagation in random deep networks [33, 37, 47, 16, 9] is related to the construction of the GPs we consider.”
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks (2018, background): “…theory to explore questions of trainability and expressivity in fully-connected networks and residual networks (Poole et al., 2016; Schoen-holz et al., 2017; Yang & Schoenholz, 2017; Schoenholz et al., 2017; Karakida et al., 2018; Hayou et al., 2018; Hanin & Rolnick, 2018; Yang & Schoenholz, 2018).”
Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction (2021, background): “This may be due to the non-convexity of the loss surface during optimization [Goodfellow et al., 2015], the high dimensionality of the optimization space, or the sensitivity to hyperparameters [Hanin and Rolnick, 2018], which causes models to converge to different regions in weight space.”
Batch Normalization Biases Deep Residual Networks Towards Shallow Paths (2020, methodology): “Although weaker than the factor of d proposed by Hanin & Rolnick (2018), we will ﬁnd empirically that it is sufﬁciently strong to train networks with 1000 layers.”
Collapse of Deep and Narrow Neural Nets (2018, background): “A parallel work by Hanin & Rolnick (2018) shows that initializing weights from a symmetric distribution with variance 2/fan-in (fan-in is the dimension of the input of each layer) avoids the problem of exploding/vanishing mean activation length.”
StyleSwin: Transformer-based GAN for High-resolution Image Generation (2021, unknown): “The transformers are initialized with a truncated normal distribution [5] with zero mean and standard deviation of 0.”
Liquid Time-constant Networks (2020, background): “By establishing a lower-bound for the growth of the trajectory length, one can set a barrier between neural networks of shallow and deep architectures, regardless of any assumptions on the network’s weight conﬁguration [46], unlike many other measures of expressivity [40, 37, 51, 17, 19, 20, 36].”
One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers (2019, result): “This is consistent with work which suggests that theoretically-grounded initialization schemes can enable the training of extremely deep networks with hundreds of layers [13, 27, 29, 33, 35].”