Highway long short-term memory RNNS for distant speech recognition

By Yu Zhang, Guoguo Chen, Dong Yu, Kaisheng Yao, Sanjeev Khudanpur, James R. Glass. 2015

297 citations recorded by the source corpus.

In this paper, we extend the deep long short-term memory (DL-STM) recurrent neural networks by introducing gated direct connections between memory cells in adjacent layers. These direct links, called highway connections, enable unimpeded information flow across different layers and thus alleviate the gradient vanishing problem when building deeper LSTMs. We further introduce the latency-controlled bidirectional LSTMs (BLSTMs) which can exploit the whole history while keeping the latency under control. Efficient algorithms are proposed to train these novel networks using both frame and sequence discriminative criteria. Experiments on the AMI distant speech recognition (DSR) task indicate that we can train deeper LSTMs and achieve better improvement from sequence training with highway LSTMs (HLSTMs). Our novel model obtains 43.9/47.7% WER on AMI (SDM) dev and eval sets, outperforming all previous works. It beats the strong DNN and DLSTM baselines with 15.7% and 5.3% relative improvement respectively.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Speech & Audio

In Signal Processing

Citations in the corpus

Pixel Recurrent Neural Networks (2016, methodology): “In subsequent layers, the channels are also connected to themselves. previous approaches that use gating along the depth of the recurrent network (Kalchbrenner et al., 2015; Zhang et al., 2016), but has the advantage of not requiring additional gates.”
Training Deep Nets with Sublinear Memory Cost (2016, background): “Through the discussion with our colleagues [18], we know that the idea of dropping computation has been applied in some limited specific use-cases.”
Light Gated Recurrent Units for Speech Recognition (2018, background): “On the other hand, computational efﬁciency is a crucial issue for RNNs and considerable research efforts have recently been devoted to the development of alternative architectures [7], [30], [46].”
Recurrent Highway Networks (2016, methodology): “These problems have been recently addressed by architectures utilizing LSTM-based transformations for stacking (Zhang et al., 2016; Kalchbrenner et al., 2015).”
Residual LSTM: Design of a Deep Recurrent Architecture for Distant Speech Recognition (2017, methodology): “However, [20] also showed that word error rate (WER) degraded when the number of layers in the highway LSTM network increases from 3 to 8.”
Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data (2017, methodology): “(2) train an LSTM acoustic model [12, 35, 42] on the set that only covers partial domains using mean and log variance of the latent segment variable z1 extracted from the trained Seq2Seq-FHVAE.”
Very deep convolutional networks for end-to-end speech recognition (2016, background): “Several architectures have been proposed recently to enable training of very deep networks [27, 23, 28, 29].”
Deep Semantic Role Labeling: What Works and What’s Next (2017, methodology): “Highway Connections To alleviate the vanishing gradient problem when training deep BiLSTMs, we use gated highway connections (Zhang et al., 2016; Srivastava et al., 2015).”
Shortcut-Stacked Sentence Encoders for Multi-Domain Inference (2017, background): “…to the widely used idea of residual connections in CNNs for computer vision (He et al., 2016), highway networks for RNNs in speech processing (Zhang et al., 2016), and shortcut connections in hierarchical multitasking learning (Hashimoto et al., 2016); but in our case we feed in all the…”
The CAPIO 2017 Conversational Speech Recognition System (2017, methodology): “In speech recognition, residual or highway connection have been applied to LSTMs, only between adjacent layers [19, 20, 21, 22].”
Transformer-Based Acoustic Modeling for Hybrid Speech Recognition (2019, methodology): “6Note that [24] used LC-BLSTM [40] instead of full-context BLSTM.”
Neural Motifs: Scene Graph Parsing with Global Context (2017, background): “Alternating Highway LSTMs To mitigate vanishing gradient problems as information flows upward, we add highway connections to all LSTMs [14, 41, 58].”
Contextualized Word Representations for Reading Comprehension (2017, methodology): “Our use of a Highway layer with RNNs is re-lated to Highway LSTM (Zhang et al., 2016) and Residual LSTM (Kim et al., 2017).”
Extracting Domain Invariant Features by Unsupervised Learning for Robust Automatic Speech Recognition (2018, methodology): “For all experiments, the same LSTM acoustic model [23] with the architecture proposed in [24] is applied, which has 1,024 memory cells and a 512-node projection layer for each LSTM layer, and 3 LSTM layers in total.”
Fast, Simpler and More Accurate Hybrid ASR Systems Using Wordpieces (2020, background): “To strike a balance between recognition latency and accuracy, LC-BLSTM was ﬁrst introduced in [5].”
Scaling Up Online Speech Recognition Using ConvNets (2020, methodology): “Baseline 1 The ﬁrst baseline [24] is a hybrid system based on context-dependent graphemes (chenones) and uses multi-layer Latency Controlled Bidirectional Long Short-Term Memory layers (LC-BLSTM)[26] in the acoustic model.”
From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognition (2019, methodology): “The LC-BLSTM operates on 1.28s chunks with 200ms lookahead.”
Semi-Orthogonal Low-Rank Matrix Factorization for Deep Neural Networks (2018, background): “This is somewhat related to the shortcut connections of residual learning [7] and highway connections [8, 9], but consists of appending/concatenating the output of previous layers rather than summing them.”
A Study of Enhancement, Augmentation, and Autoencoder Methods for Domain Adaptation in Distant Speech Recognition (2018, methodology): “Following [34], we use 80-dimensional log Mel filterbank features, and train two speaker-adaptive hidden Markov models (HMM), one for IHM and one for SDM.”
Gradual DropIn of Layers to Train Very Deep Neural Networks (2015, methodology): “[32] applied highway networks to LSTM recurrent neural netFigure 1: Diagram of traditional vs DropIn training method.”