Memory visualization for gated recurrent neural networks in speech recognition
By Zhiyuan Tang, Ying Shi, Dong Wang, Yang Feng, Shiyue Zhang. 2016
56 citations recorded by the source corpus.
Recurrent neural networks (RNNs) have shown clear superiority in sequence modeling, particularly the ones with gated units, such as long short-term memory (LSTM) and gated recurrent unit (GRU). However, the dynamic properties behind the remarkable performance remain unclear in many applications, e.g., automatic speech recognition (ASR). This paper employs visualization techniques to study the behavior of LSTM and GRU when performing speech recognition tasks. Our experiments show some interesting patterns in the gated memory, and some of them have inspired simple yet effective modifications on the network structure. We report two of such modifications: (1) lazy cell update in LSTM, and (2) shortcut connections for residual learning. Both modifications lead to more comprehensible and powerful networks.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yonatan Belinkov
- James R. Glass
- Hermann Ney
- Ralf Schlüter
- Albert Zeyer
- Grzegorz Chrupała
- Hinrich Schütze
- Afra Alishahi
Citations in the corpus
- The emergence of number and syntax units in LSTM language models (2019, background): “Other studies are closer to our approach in that they attempt to attribute function to spe-cific network cells, often by means of visualization (Karpathy et al., 2016; Li et al., 2016; Tang et al., 2017).”
- A comprehensive analysis on attention models (2019, background): “Understanding the inner working of a speech recognition system is also subject in [Tang et al., 2017], where the authors examine activation distribution and temporal patterns, focussing on the comparison between LSTM and GRU systems.”
- Representations of language in a model of visually grounded speech signal (2017, background): “Adi et al. (2016) introduce prediction tasks to analyze information encoded in sentence embeddings about word order, sentence length, and the presence of individual words.”
- Understanding Hidden Memories of Recurrent Neural Networks (2017, background): “[47] studied the behavior of LSTM and GRU in speech recognition by projecting sequence history.”
- LISA: Explaining Recurrent Neural Network Judgments via Layer-wIse Semantic Accumulation and Example to Pattern Transformation (2018, background): “Tang et al. (2017) visualized the memory vectors to understand the behavior of LSTM and gated recurrent unit (GRU) in speech recognition task.”