Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

By Tim Salimans, Diederik P. Kingma. 2016

2,132 citations recorded by the source corpus.

We present weight normalization: a reparameterization of the weight vectors in a neural network that decouples the length of those weight vectors from their direction. By reparameterizing the weights in this way we improve the conditioning of the optimization problem and we speed up convergence of stochastic gradient descent. Our reparameterization is inspired by batch normalization but does not introduce any dependencies between the examples in a minibatch. This means that our method can also be applied successfully to recurrent models such as LSTMs and to noise-sensitive applications such as deep reinforcement learning or generative models, for which batch normalization is less well suited. Although our method is much simpler, it still provides much of the speed-up of full batch normalization. In addition, the computational overhead of our method is lower, permitting more optimization steps to be taken in the same amount of time. We demonstrate the usefulness of our method on applications in supervised image recognition, generative modelling, and deep reinforcement learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

Convolutional Sequence to Sequence Learning (2017, methodology): “We also use weight normalization for all layers except for lookup tables (Salimans & Kingma, 2016).”
Layer Normalization (2016, background): “Our work is also related to weight normalization [Salimans and Kingma, 2016].”
Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “We modified the base ResNet-50 with the anti-alias improvements from (Zhang, 2019) and used weight norm (Salimans & Kingma, 2016) instead of batch norm (Ioffe & Szegedy, 2015) to avoid leaking information about duplicates via batch statistics - a problem previously noted in (Henaff, 2020).”
Denoising Diffusion Probabilistic Models (2020, methodology): “We replaced weight normalization [46] with group normalization [62] to make the implementation simpler.”
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017, background): “Most of these works have considered good random initial-izations, though a number of papers have addressed data-dependent initializers (Kr¨ahenb¨uhl et al., 2016; Salimans & Kingma, 2016), including learned initializations (Husken & Goerick, 2000; Maclaurin et al., 2015).”
Gaussian Error Linear Units (GELUs) (2016, methodology): “Our shallower convolutional neural network is a 9-layer network with the architecture and training procedure from Salimans & Kingma (2016) while using batch normalization to speed up training.”
Root Mean Square Layer Normalization (2019, methodology): “Although Euclidean norm which only differs from RMS by a factor of √ n has been successfully explored [22], we empirically find that it does not work for layer normalization.”
Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering (2019, methodology): “Motivated by Salimans & Kingma (2016), we normalize the RNN states to control the scale of logits in Equation (3) and allow the model to learn multiple reasoning paths.”
Weight Standardization (2019, methodology): “Weight Normalization [36] is close to our method in that it also considers weights instead of activations for normalization.”
Group Normalization (2018, methodology): “4 For completeness, we have also trained ResNet-50 with WN (Salimans and Kingma 2016), which is filter (instead of feature) normalization.”
Emerging Properties in Self-Supervised Vision Transformers (2021, positive): “Then we apply a `2 normalization and a weight normalized fully connected layer [16, 61] with K dimensions.”
An Empirical Study of Training Self-Supervised Vision Transformers (2021, methodology): “We have also tried BatchNorm (BN) [23], WeightNorm (WN) [39], or gradient clip on patch projection.”
Exploring Simple Siamese Representation Learning (2020, background): “(ii) The prototypes were weight-normalized outside of gradient propagation in [7]; we instead implement by full gradient computation [33].”
Analyzing and Improving the Image Quality of StyleGAN (2019, methodology): “Our demodulation is also related to weight normalization [32] that performs the same calculation as a part of reparameterizing the weight tensor.”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, background): “Weight normalization [46] is applied in all neural network layers.”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, methodology): “Our experiments employed weight normalization and data-dependent initialization (Salimans & Kingma, 2016).”
Waveglow: A Flow-based Generative Network for Speech Synthesis (2018, methodology): “…network was trained on 8 Nvidia GV100 GPU’s using randomly chosen clips of 16,000 samples for 580,000 iterations using weight normalization [19] and the Adam optimizer [18], with a batch size of 24 and a step size of 1 × 10 − 4 When training appeared to plateau, the learning rate was…”