Layer Normalization
By Jimmy Ba, Jamie Ryan Kiros, Geoffrey E. Hinton. 2016
12,877 citations recorded by the source corpus.
Training state-of-the-art, deep neural networks is computationally expensive. One way to reduce the training time is to normalize the activities of the neurons. A recently introduced technique called batch normalization uses the distribution of the summed input to a neuron over a mini-batch of training cases to compute a mean and variance which are then used to normalize the summed input to that neuron on each training case. This significantly reduces the training time in feed-forward neural networks. However, the effect of batch normalization is dependent on the mini-batch size and it is not obvious how to apply it to recurrent neural networks. In this paper, we transpose batch normalization into layer normalization by computing the mean and variance used for normalization from all of the summed inputs to the neurons in a layer on a single training case. Like batch normalization, we also give each neuron its own adaptive bias and gain which are applied after the normalization but before the non-linearity. Unlike batch normalization, layer normalization performs exactly the same computation at training and test times. It is also straightforward to apply to recurrent neural networks by computing the normalization statistics separately at each time step. Layer normalization is very effective at stabilizing the hidden state dynamics in recurrent networks. Empirically, we show that layer normalization can substantially reduce the training time compared with previously published techniques.
Subfields: Artificial Intelligence, Materials Chemistry, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yoshua Bengio
- Jianfeng Gao
- Quoc V. Le
- Percy Liang
- Luke S. Zettlemoyer
- Sergey Levine
- Graham Neubig
In Materials Chemistry
- Tommi S. Jaakkola
- Jure Leskovec
- Yoshua Bengio
- Regina Barzilay
- William L. Hamilton
- Max Welling
- Rex Ying
- Peter W. Battaglia
In Optimization & Training
- Yoshua Bengio
- Quoc V. Le
- Sergey Levine
- Frank Hutter
- Soham De
- Samuel L. Smith
- Simon Lacoste-Julien
- Ruslan Salakhutdinov
Citations in the corpus
- Attention is All you Need (2017, methodology): “We employ a residual connection [11] around each of the two sub-layers, followed by layer normalization [1].”
- Neural Machine Translation in Linear Time (2016, methodology): “In both cases, we use layer normalization (Ba et al., 2016) before the activation function, as it is well suited to sequence processing where computing the activation statistics over the following future tokens (as would be done by batch normalization) must be avoided.”
- Deep Contextualized Word Representations (2018, methodology): “Considering that the activations of each biLM layer have a different distribution, in some cases it also helped to apply layer normalization (Ba et al., 2016) to each biLM layer before weighting.”
- QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (2018, background): “put xand a given operation f, the output is f(layernorm(x))+x, meaning there is a full identity path from the input to output of each block, where layernorm indicates layer-normalization proposed in (Ba et al., 2016). The total number of encoder blocks is 1. Note that the input of this layer is a ve”
- Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, neutral): “They further obtained better results using layer-norm regularization of their model in (Ba et al., 2016).”
- Masked Autoencoders Are Scalable Vision Learners (2021, background): “It has a stack of Transformer blocks [57], and each block consists of a multi-head self-attention block and an MLP block, both having LayerNorm (LN) [1].”
- Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, methodology): “We move the model’s layer normalization layers (Ba et al., 2016) to the input of every sub-layer (often called pre-norm ).”
- Generating Long Sequences with Sparse Transformers (2019, background): “First, we use the pre-activation residual block of (He et al., 2016), defining a network of N layers in the following way: where embed is a function we describe in the next section, W out is a weight matrix, and resblock( h ) normalizes the input to the attention block and a positionwise feedforward ”
- Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding (2019, background): “…a process of distilling or transferring the knowledge from a (set of) large, cumbersome model(s) to a lighter, easier-to-deploy single model, without significant loss in performance (Bucilu et al., 2006; Hinton et al., 2015; Balan et al., 2015; Ba et al., 2016; Chen et al., 2015; Tan et al., 2019).”
- Universal Transformers (2018, methodology): “We also add residual connections around each of these function blocks and apply dropout and layer normalization (Srivastava et al., 2014; Ba et al., 2016) (see Fig.”
- The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “We then use projected, residual feedforward networks (FFN) with ReLU activations [Nair and Hinton, 2010, Vaswani et al., 2017] and layer normalization [Ba et al., 2016] on the inputs and outputs.”
- Simple, Scalable Adaptation for Neural Machine Translation (2019, methodology): “We first apply layer-normalization (Ba et al., 2016) to the inputs of the adapter corresponding to task T .”
- CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “…with ReLU activation (Nair & Hinton, 2010) that projects inputs to an inner dimension f , with parameters U ∈ R d × f and V ∈ R f × d : Each block precedes core functionality with layer normalization (Ba et al., 2016; Child et al., 2019) and follows it with a residual connection (He et al., 2016).”
- SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, methodology): “We implement the representation function f(·) as a 2-layer feed-forward network with GeLU activations (Hendrycks and Gimpel, 2016) and layer normalization (Ba et al., 2016):”
- Unifying Question Answering and Text Classification via Span Extraction (2019, background): “…) by first applying self-attention with k heads over the previous layer’s outputs: where A residual connection (He et al., 2016) and layer normalization (Ba et al., 2016) merge information from the input and the multi-head attention: This is followed by a feedforward network with ReLU activation…”
- Fine-tune BERT for Extractive Summarization (2019, methodology): “To stabilize the training, pergate layer normalization (Ba et al., 2016) is applied within each LSTM cell.”
- Rethinking ImageNet Pre-Training (2018, neutral): “Successful forms of normalization include normalized parameter initialization [11, 16] and activation normalization layers [20, 2, 46, 48].”
- Self-Attention with Relative Position Representations (2018, methodology): “It uses residual connections around each of the sublayers, followed by layer normalization (Ba et al., 2016).”