Fixup Initialization: Residual Learning Without Normalization

By Hongyi Zhang, Yann Dauphin, Tengyu Ma. 2019

406 citations recorded by the source corpus.

Normalization layers are a staple in state-of-the-art deep neural network architectures. They are widely believed to stabilize training, enable higher learning rate, accelerate convergence and improve generalization, though the reason for their effectiveness is still an active research topic. In this work, we challenge the commonly-held beliefs by showing that none of the perceived benefits is unique to normalization. Specifically, we propose fixed-update initialization (Fixup), an initialization motivated by solving the exploding and vanishing gradient problem at the beginning of training via properly rescaling a standard initialization. We find training residual networks with Fixup to be as stable as training with normalization -- even for networks with 10,000 layers. Furthermore, with proper regularization, Fixup enables residual networks without normalization to achieve state-of-the-art performance in image classification and machine translation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “…et al., 2017; Arora et al., 2018), trainability challenges including exploding and vanishing gradients (Glo-rot and Bengio, 2010; Balduzzi et al., 2017) and empirical approaches to counter these challenges (Bapna et al., 2018; Zhang et al., 2019) are critical to further scaling of model capacity.”
Jukebox: A Generative Model for Music (2020, methodology): “The rate of progress in this ﬁeld has been rapid, where only a few years ago we had algorithms producing blurry faces (Kingma & Welling, 2014; Goodfellow et al., 2014) but now we now can generate high-resolution faces indistinguishable from real ones (Zhang et al., 2019b).”
Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images (2020, background): “The one exception is the final convolutional layer in each residual bottleneck block, which we scale by 1√ N , where N is the depth (similar to Radford et al. (2019); Child et al. (2019); Zhang et al. (2019)).”
Measuring the Algorithmic Efficiency of Neural Networks (2020, methodology): “We suspect we would have needed to carefully initialize the network to do so [Zhang et al., 2019].”
Stable and low-precision training for large-scale vision-language models (2023, background): “Consequently, researchers have proposed initialization and scaling schemes which remedy this issue [1, 72, 4, 19].”
High-Performance Large-Scale Image Recognition Without Normalization (2021, methodology): “Most of these works suppress the scale of the activations on the residual branch at initialization, by introducing either small constants or learnable scalars (Hanin & Rolnick, 2018; Zhang et al., 2019a; De & Smith, 2020; Shao et al., 2020).”
ReZero is All You Need: Fast Convergence at Large Depth (2020, background): “, 2017, Hardt and Ma, 2016], FixUp [Zhang et al., 2019], ReZero and Pre-Act ResNet [He et al.”
Do Transformer Modifications Transfer Across Implementations and Applications? (2021, methodology): “However, we found that certain architectural (Rezero and Fixup) variants achieved signiﬁcantly lower negative log perplexity than the baseline model with the Adafactor optimizer.”
On the Variance of the Adaptive Learning Rate and Beyond (2019, neutral): “…al., 2013), smoothing the adaptive learning rate ( i.e. , increasing (cid:15) , applying geometric mean ﬁlter (Chen et al., 2018), or adding range constraints (Luo et al., 2019)), initialization (Balduzzi et al., 2017; Zhang et al., 2019) and normalization (Ba et al., 2016; Ioffe & Szegedy, 2015).”
AutoML: A Survey of the State-of-the-Art (2019, methodology): “GPIPE [130] achieves the best result on CIFAR10 based on AmoebaNet-B [17] and one can easily see that the models generated automatically have already outperformed the handcrafted models (SENet [131] and WRN [132]).”
What is being transferred in transfer learning? (2020, methodology): “For all our training and transfer experiments we use ResNet-50 [16] with fixup initialization [46] which eliminate batch normalization layers from the ResNet architecture.”
Going deeper with Image Transformers (2021, methodology): “As pointed in our paper, in that respect our LayerScale approach is more related to Rezero [2] and Skipinit [16], Fixup [76], and T-Fixup [34].”
FlauBERT: Unsupervised Language Model Pre-training for French (2019, methodology): “Similar to previous approaches to create multilingual corpora, Yang et al. (2019a) used machine translation to create the training set for each target language in PAWS-X from the English training set in PAWS.”
Four Things Everyone Should Know to Improve Batch Normalization (2019, background): “Even in situations where training without normalization layers is possible, their usage can still aid generalization [36].”