Light Gated Recurrent Units for Speech Recognition

By Mirco Ravanelli, Philemon Brakel, Maurizio Omologo, Yoshua Bengio. 2018

369 citations recorded by the source corpus.

A field that has directly benefited from the recent advances in deep learning is automatic speech recognition (ASR). Despite the great achievements of the past decades, however, a natural and robust human–machine speech interaction still appears to be out of reach, especially in challenging environments characterized by significant noise and reverberation. To improve robustness, modern speech recognizers often employ acoustic models based on recurrent neural networks (RNNs) that are naturally able to exploit large time contexts and long-term speech modulations. It is thus of great interest to continue the study of proper techniques for improving the effectiveness of RNNs in processing speech signals. In this paper, we revise one of the most popular RNN models, namely, gated recurrent units (GRUs), and propose a simplified architecture that turned out to be very effective for ASR. The contribution of this work is twofold: First, we analyze the role played by the reset gate, showing that a significant redundancy with the update gate occurs. As a result, we propose to remove the former from the GRU design, leading to a more efficient and compact single-gate model. Second, we propose to replace hyperbolic tangent with rectified linear unit activations. This variation couples well with batch normalization and could help the model learn long-term dependencies without numerical issues. Results show that the proposed architecture, called light GRU, not only reduces the per-epoch training time by more than 30% over a standard GRU, but also consistently improves the recognition accuracy across different tasks, input features, noisy conditions, as well as across different ASR paradigms, ranging from standard DNN-HMM speech recognizers to end-to-end connectionist temporal classification models.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Speech & Audio

In Signal Processing

Citations in the corpus

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, neutral): “2 Li-GRU + fMLLR [46] – 14.”
Unsupervised Speech Recognition (2021, methodology): “Model LM core-dev core-test all-test Supervised learning LiGRU (Ravanelli et al., 2018) - 14.”
vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations (2019, methodology): “…BERT models which require a relatively small vocabulary. dev PER test PER CNN + TD-ﬁlterbanks (Zeghidour et al., 2018) 15.6 18.0 Li-GRU + fMLLR (Ravanelli et al., 2018) – 14.9 wav2vec (Schneider et al., 2019) 12.9 14.7 Baseline Next, we compare Gumbel-Softmax to k-means for vector…”
wav2vec: Unsupervised Pre-training for Speech Recognition (2019, neutral): “26 Li-GRU + FBANK (Ravanelli et al., 2018) – 15.”
Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks (2019, background): “%) is a competitive performance as well, especially when compared to state-of-the-art results that do not use complex techniques as system combination, speaker adaptation, or multiple steps of lattice rescoring and decoding [37,38,44,45].”
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech (2020, methodology): “performance. On the other hand, pre-training with FBANK yields better performance for phoneme and speaker Models Pre-train PER CNN + TD-ﬁlterbanks [55] None 18.0 CNN + HMM [56] None 16.5 liGRU + MFCC [57] None 16.7 liGRU + FBANK [57] None 15.8 liGRU + fMLLR [57] None 14.9 wav2vec [3] 80 hr 17.6 wav2”
Twin Regularization for online speech recognition (2018, methodology): “Beyond standard LSTM [36] and GRU [37], we also considered recently proposed architectural variations [38]: MGRU [39] is the minimal GRU architecture based on replacing reset gate with updated gate activations, while light GRU (LiGRU) [40] directly avoids the reset gate and exploits ReLU activations”
Quaternion Recurrent Neural Networks (2018, methodology): “Deep RNNs have been recently used to obtain hidden representations of speech unit sequences (Ravanelli et al., 2018a) or text word sequences (Conneau et al., 2018), and to achieve state-of-the-art performances in many speech recognition tasks (Graves et al., 2013a;b; Amodei et al., 2016; Povey et…”
An Experimental Review on Deep Learning Architectures for Time Series Forecasting (2020, methodology): “66 The GRU cell uses an update gate and a re-set gate that will learn to decide which information should be kept without vanishing it through time and which information is irrelevant for the problem.”
Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps (2020, methodology): “(Ravanelli et al., 2018) 13 .”
The Pytorch-kaldi Speech Recognition Toolkit (2018, methodology): “Recurrent models signiﬁcantly outperform the standard MLP one, especially when using LSTM, GRU, and Li-GRU architecture, that effectively address gradient vanishing through multiplicative gates.”
SpeechBrain: A General-Purpose Speech Toolkit (2021, methodology): “A considerable improvement in PER is observed when Light-GRUs [80] are used instead of GRUs [79] or LSTMs [78] in the CRDNN encoder.”
Unsupervised Adaptation of Acoustic Models for ASR Using Utterance-Level Embeddings from Squeeze and Excitation Networks (2019, methodology): “Compared to LSTM and GRU, it does not have a reset gate since it was shown that the information in the update and reset gates are redundant [34].”
Speaker Recognition from Raw Waveform with SincNet (2018, methodology): “Multiple standard convolutional, fully-connected or recurrent layers [37–40] can then be stacked together to finally perform a speaker classification with a softmax classifier.”
SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts (2021, methodology): “Various types of neural network architectures have been employed in ASR systems, such as convolutional neural networks (CNNs) [3, 4], long short-term memory (LSTM) [5], gated recurrent unit[6], time-delayed neural network [7], feedforward sequential memory networks (FSMN) [8], etc.”
Interpretable Convolutional Filters with SincNet (2018, methodology): “Multiple standard convolutional, fully-connected or recurrent layers [15, 16, 17, 45] can then be stacked together to finally perform a classification with a softmax classifier.”
Attention Is All You Need In Speech Separation (2020, neutral): “Especially when coupled with multiplicative gate mechanisms (like LSTM [1] and GRU [2, 3]), their recurrent connections are essential to learn long-term dependencies and properly manage speech contexts.”