Batch normalized recurrent neural networks

By César Laurent, Gabriel Pereyra, Philemon Brakel, Y. Zhang, Yoshua Bengio. 2015

224 citations recorded by the source corpus.

Recurrent Neural Networks (RNNs) are powerful models for sequential data that have the potential to learn long-term dependencies. However, they are computationally expensive to train and difficult to parallelize. Recent work has shown that normalizing intermediate representations of neural networks can significantly improve convergence rates in feed-forward neural networks [1]. In particular, batch normalization, which uses mini-batch statistics to standardize features, was shown to significantly reduce training time. In this paper, we investigate how batch normalization can be applied to RNNs. We show for both a speech recognition task and language modeling that the way we apply batch normalization leads to a faster convergence of the training criterion but doesn't seem to improve the generalization performance.

Subfields: Artificial Intelligence, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Optimization & Training

Citations in the corpus

Layer Normalization (2016, methodology): “Batch normalization has been previously extended to recurrent neural networks [Laurent et al., 2015, Amodei et al., 2015, Cooijmans et al., 2016].”
Root Mean Square Layer Normalization (2019, background): “Unfortunately, the reliance across training cases deprives BatchNorm of the capability in handling variable-length sequences, though several researchers develop different strategies to enable it in RNNs [16, 8].”
Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin (2015, methodology): “As in (Laurent et al., 2015), there are two ways of applying BatchNorm to the recurrent operation.”
Light Gated Recurrent Units for Speech Recognition (2018, methodology): “nique is effective both to improve the system performance and to speed-up the training procedure [20], [22], [37], [49], [50].”
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization (2017, background): “Recently, several alternative normalization schemes have been proposed to extend BN’s effectiveness to recurrent architectures [35, 2, 47, 8, 29, 44].”
Recurrent Batch Normalization (2016, background): “Contrary to previous work [14, 1], we leverage batch normalization in both the input-to-hidden and”
Adding Gradient Noise Improves Learning for Very Deep Networks (2015, methodology): “While there has been some difficulty in combining batch normalization-type algorithms with recurrent networks (Laurent et al., 2015), recent work has had success in this area (Cooijmans et al.”
Jasper: An End-to-End Convolutional Neural Acoustic Model (2019, methodology): “Further, we computed mean and variance over both the time dimension and channels similar to the sequence-wise normalization proposed by Laurent et al. [20].”
Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex (2016, neutral): “This may be consistent with the observations from [18].”
Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems (2017, background): “Each convolutional or recurrent layer is followed by batch normalization [29, 30] and a ReLU nonlinearity.”
Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes (2016, background): “Moreover, application to recurrent neural networks (RNNs) is not straightforward and leads to poor performance (Laurent et al., 2015).”
The Pytorch-kaldi Speech Recognition Toolkit (2018, methodology): “weights were initialized with orthogonal matrices [39]. Recurrent dropout was used as a regularization technique [40]. Batch normalization was adopted for feedforward connections only, as proposed in [41, 42]. The optimization was done using the RMSprop algorithm running for 24 epochs. The performan”
Advances in Very Deep Convolutional Neural Networks for LVCSR (2016, methodology): “When applying batch normalization to sequence data, stacking multiple utterances as one batch for computing the mean and variance stastistics, is identical to how BN was applied to recurrent neural networks in [33, 27].”
Improving Speech Recognition by Revising Gated Recurrent Units (2017, methodology): “In [33], authors suggest to apply it to feed-forward connections only, while in [34] the normalization step is extended to recurrent connections, using separate statistics for each time-step.”
DEEP CLUSTERING AND CONVENTIONAL NETWORKS FOR MUSIC SEPARATION: STRONGER TOGETHER (2016, methodology): “2 were added between each feed-forward connection, and sequence-wise batch normalization [20] was applied in the input-to-hidden transformation in each BLSTM layer.”
On-chip training of recurrent neural networks with limited numerical precision (2017, background): “This paper presents limited precision training of RNNs for both forward pass and backward pass (Fig.”
Decorrelated Batch Normalization (2018, methodology): “There have also been efforts to adapt Batch Normalization to Recurrent Neural Networks [33, 12].”