Recurrent Batch Normalization

By Tim Cooijmans, Nicolas Ballas, César Laurent, Aaron C. Courville. 2016

418 citations recorded by the source corpus.

We propose a reparameterization of LSTM that brings the benefits of batch normalization to recurrent neural networks. Whereas previous works only apply batch normalization to the input-to-hidden transformation of RNNs, we demonstrate that it is both possible and beneficial to batch-normalize the hidden-to-hidden transition, thereby reducing internal covariate shift between time steps. We evaluate our proposal on various sequential problems such as sequence classification, language modeling and question answering. Our empirical results show that our batch-normalized LSTM consistently leads to faster convergence and improved generalization.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Layer Normalization (2016, methodology): “In order to compare layer normalization to the recently proposed recurrent batch normalization [Cooijmans et al., 2016], we train an unidirectional attentive reader model on the CNN corpus both introduced by Hermann et al. [2015].”
Character-Level Language Modeling with Deeper Self-Attention (2018, background): “Much of the progress in this area has been made by mitigating the vanishing gradients problem [14] by architectures such as LSTMs [15], GRU [3], Recurrent Highway Networks [51], Uni-tary RNNs [1] and others.”
Root Mean Square Layer Normalization (2019, methodology): “The model is trained on the CNN corpus based on the public source code in Theano [8].”
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks (2016, background): “Some other potential solutions were recently proposed in [4, 2].”
HyperNetworks (2016, methodology): “ME n-gram (Mikolov et al., 2012) 1.37 Batch Norm LSTM (Cooijmans et al., 2016) 1.32 Recurrent Dropout LSTM (Semeniuta et al., 2016) 1.301 1.338 Zoneout RNN (Krueger et al., 2016) 1.27 HM-LSTM 3 (Chung et al., 2016) 1.27 (Sutskever et al., 2011) 1.60 GF-RNN (Chung et al., 2015) 1.58 20.0 M Grid-LSTM ”
Light Gated Recurrent Units for Speech Recognition (2018, background): “nique is effective both to improve the system performance and to speed-up the training procedure [20], [22], [37], [49], [50].”
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset (2017, methodology): “We position an LSTM layer with batch normalization (as proposed by Cooijmans et al. [4]) after the last average pooling layer of Inception-V1, with 512 hidden units.”
Regularizing and Optimizing LSTM Language Models (2017, background): “Other forms of regularization explicitly act upon activations such as batch normalization (Ioffe & Szegedy, 2015), recurrent batch normalization (Cooijmans et al., 2016), and layer normalization (Ba et al., 2016).”
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization (2017, background): “Recently, several alternative normalization schemes have been proposed to extend BN’s effectiveness to recurrent architectures [35, 2, 47, 8, 29, 44].”
Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses (2017, methodology): “We also use layer normalization (Ba et al., 2016) for the hierarchical encoder, which we found worked better at the task of dialogue generation than the related recurrent batch normalization (Ioffe and Szegedy, 2015; Cooijmans et al., 2016).”
Hierarchical Multiscale Recurrent Neural Networks (2016, methodology): “Published as a conference paper at ICLR 2017 Text8 Model BPC td -LSTM (Zhang et al., 2016) 1.63 HF-MRNN (Mikolov et al., 2012) 1.54 MI-RNN (Wu et al., 2016) 1.52 Skipping-RNN (Pachitariu & Sahani, 2013) 1.48 MI-LSTM (Wu et al., 2016) 1.44 BatchNorm LSTM (Cooijmans et al., 2016) Ba, 2014) with an ini”
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations (2016, methodology): “We achieve competitive results with relatively simple models in character- and word-level language modelling on the Penn Treebank and Text8 datasets, and combining with recurrent batch normalization (Cooijmans et al., 2016) yields state-of-the-art results on permuted sequential MNIST.”
Decoupled Neural Interfaces using Synthetic Gradients (2016, methodology): “Multi-Network Implementation Details The two RNNs in this experiment, Network A and Network B, are both LSTMs with 256 units which use batchnormalisation as described in (Cooijmans et al., 2016).”
Entropy-SGD: biasing gradient descent into wide valleys (2016, background): “…(Tieleman & Hinton, 2012), as well as architectural modiﬁcations like dropout (Srivastava et al., 2014), batch-normalization (Ioffe & Szegedy, 2015; Cooijmans et al., 2016), weight scaling (Salimans & Kingma, 2016) etc. are different ways of tackling this issue by making the underlying landscape…”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, background): “%) based on recurrent networks with Zoneout and Recurrent BatchNorm (Cooijmans et al., 2016; Krueger et al., 2017).”
Adding Gradient Noise Improves Learning for Very Deep Networks (2015, background): “, 2015), recent work has had success in this area (Cooijmans et al., 2016; Ba et al., 2016).”