All you need is a good init

By Dmytro Mishkin, Jiri Matas. 2015

655 citations recorded by the source corpus.

Layer-sequential unit-variance (LSUV) initialization - a simple method for weight initialization for deep net learning - is proposed. The method consists of the two steps. First, pre-initialize weights of each convolution or inner-product layer with orthonormal matrices. Second, proceed from the first to the final layer, normalizing the variance of the output of each layer to be equal to one. Experiment with different activation functions (maxout, ReLU-family, tanh) show that the proposed initialization leads to learning of very deep nets that (i) produces networks with test accuracy better or equal to standard methods and (ii) is at least as fast as the complex schemes proposed specifically for very deep nets such as FitNets (Romero et al. (2015)) and Highway (Srivastava et al. (2015)). Performance is evaluated on GoogLeNet, CaffeNet, FitNets and Residual nets and the state-of-the-art, or very close to it, is achieved on the MNIST, CIFAR-10/100 and ImageNet datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neurology, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Neurology

In Optimization & Training

Citations in the corpus

Gaussian Error Linear Units (GELUs) (2016, background): “Weights are initialized with unit norm rows, as this has positive impact on each nonlinearity’s performance (Hendrycks & Gimpel, 2016; Mishkin & Matas, 2016; Saxe et al., 2014).”
Identity Mappings in Deep Residual Networks (2016, neutral): “mini-batch size of 64 (code available at https://github.com/KaimingHe/resnet-1k-layers). CIFAR-10 error (%) NIN [13] 8.81 DSN [14] 8.22 FitNet [15] 8.39 Highway [7] 7.72 ELU [12] 6.55 FitResNet, LSUV [16] 5.84 ResNet-110 [1] (1.7M) 6.61 ResNet-1202 [1] (19.4M) 7.93 ResNet-164 [ours] (1.7M) 5.46 ResN”
Initialization and Regularization of Factorized Neural Layers (2021, background): “Initialization is a major focus of deep learning research (He et al., 2015; Mishkin & Matas, 2016; Yang & Schoenholz, 2017).”
ReZero is All You Need: Fast Convergence at Large Depth (2020, methodology): “Researchers have relied on careful initialization [7, 8] and normalization techniques such as BatchNorm [9] and LayerNorm [10] to mitigate this issue, but these techniques can be costly and limited.”
Temporal Ensembling for Semi-Supervised Learning (2016, methodology): “When all labels are used for training, our network approximately matches the state-of-the-art error rate for a single model in CIFAR-10 with augmentation [7, 10] at 6.”
Colorization as a Proxy Task for Visual Understanding (2017, background): “The uniform scale however is only statistically predicted given broad data assumptions, so this idea can be taken one step further by looking at the activations of actual data and normalizing [23].”
Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning (2017, methodology): “Recent work (Kr¨ahenb¨uhl et al 2016; Mishkin and Matas 2015) has addressed these optimization issues by rescaling the weights of a pretrained network using a data-driven procedure.”
FractalNet: Ultra-Deep Neural Networks without Residuals (2016, background): “Good initialization can also partially ameliorate this problem [4, 23].”
Fixup Initialization: Residual Learning Without Normalization (2019, background): “However, our use of positive homogeneity for lower bounding the gradient norm of a neural network is novel, and applies to a broad class of neural network architectures (e.g., ResNet, DenseNet) and initialization methods (e.g., Xavier, LSUV) with simple assumptions and proof.”
Recent advances in convolutional neural networks (2015, background): “To achieve a fast convergency in training and avoid the vanishing gradient problem, a proper network initialization is one of the most important prerequisites [53], [54].”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “Additionally, optimizing a neural network system like HRL [20] is sensitive to weight initialization [64, 55], due to its complexity and lack of clear supervision.”
Convolutional Neural Fabrics (2016, background): “84% [22] was obtained using residual connections, without residual connections they report an error of 6.”
An Empirical Investigation of the Role of Pre-training in Lifelong Learning (2021, background): “3 Analyzing the influence of task-agnostic favorable initializations on forgetting Initialization is widely recognized as a crucial factor in a model’s learning dynamics and overall performance (Glorot and Bengio, 2010; LeCun et al., 2015; Mishkin and Matas, 2016).”
Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs (2017, neutral): “Interestingly, we had no success with other feature normalization techniques such as data-dependent initialization [26] or layer normalization [3].”
The Shattered Gradients Problem: If resnets are the answer, then what is the question? (2017, background): “Orthogonalizing weight matrices can yield further improvements albeit at a computational cost (Saxe et al., 2014; Mishkin & Matas, 2016).”
Understanding Batch Normalization (2018, background): “With such an architecture, one gets only somewhat weaker performance than a completely batch normalized network, using a learning rate of 0 .”