Residual LSTM: Design of a Deep Recurrent Architecture for Distant Speech Recognition

By Jaeyoung Kim, Mostafa El-Khamy, Jungwon Lee. 2017

193 citations recorded by the source corpus.

In this paper, a novel architecture for a deep recurrent neural network, residual LSTM is introduced. A plain LSTM has an internal memory cell that can learn long term dependencies of sequential data. It also provides a temporal shortcut path to avoid vanishing or exploding gradients in the temporal domain. The residual LSTM provides an additional spatial shortcut path from lower layers for efficient training of deep networks with multiple LSTM layers. Compared with the previous work, highway LSTM, residual LSTM separates a spatial shortcut path with temporal one by using output layers, which can help to avoid a conflict between spatial and temporal-domain gradient flows. Furthermore, residual LSTM reuses the output projection matrix and the output gate of LSTM to control the spatial information flow instead of additional gate networks, which effectively reduces more than 10% of network parameters. An experiment for distant speech recognition on the AMI SDM corpus shows that 10-layer plain and highway LSTM networks presented 13.7% and 6.2% increase in WER over 3-layer aselines, respectively. On the contrary, 10-layer residual LSTM networks provided the lowest WER 41.0%, which corresponds to 3.3% and 2.8% WER reduction over plain and highway LSTM networks, respectively.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Speech & Audio

In Signal Processing

Citations in the corpus

The CAPIO 2017 Conversational Speech Recognition System (2017, methodology): “In speech recognition, residual or highway connection have been applied to LSTMs, only between adjacent layers [19, 20, 21, 22].”
Federated Learning of N-Gram Language Models (2019, methodology): “Residual LSTM (Kim et al., 2017) and layer normalization (Lei Ba et al., 2016) are used throughout experiments, as these techniques were observed to improve convergence.”
Contextualized Word Representations for Reading Comprehension (2017, methodology): “It consists of an initial layer which produces character-based word representations, followed by two stacked LSTM layers and a softmax prediction layer.”
Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform (2018, methodology): “SNN (Klambauer et al., 2017), Residual LSTM (Kim et al., 2017), and Transformer (Vaswani et al., 2017) (Wu et al., 2017; Keskar et al., 2016).”
AdaDurIAN: Few-shot Adaptation for Neural Text-to-Speech with DurIAN (2020, methodology): “Different from DurIAN, we adopt the residual LSTM [26] layers for its efﬁcient training which is of great importance in few-shot learning.”
Recent progresses in deep learning based acoustic models (2017, background): “Residual LSTM [30], [31] uses shortcut connections be-tween LSTM layers, and hence also provides a way to alleviating the gradient vanishing problem.”
Residual Convolutional CTC Networks for Automatic Speech Recognition (2017, unknown): “Residual LSTM architecture was proposed in (Kim et al., 2017).”
Forecasting stock prices with long-short term memory neural network based on attention mechanism (2020, background): “Long short-term memory (LSTM) neural networks have performed well in speech recognition[3, 4] and text processing.”
Residual Recurrent Highway Networks for Learning Deep Sequence Prediction Models (2018, background): “Recently, the problem is rectified by using LSTM with an additional shortcut connection between hierarchical layers [14, 15].”
Pyraformer: Low-Complexity Pyramidal Attention for Long-Range Time Series Modeling and Forecasting (2022, background): “Another line of works aim to shorten the signal traversing path by adding residual connections (Kim et al., 2017) or dilated connections to LSTMs (Chang et al., 2017).”
A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting (2020, unknown): “The residual shortcuts around the blocks, or, in the special case of LSTMs as per Kim et al. (2017 • stands for level variability penalty, and is the multiplier that is applied to the level wiggliness penalty.”