A Time-Restricted Self-Attention Layer for ASR

By Daniel Povey, Hossein Hadian, Pegah Ghahremani, Ke Li, Sanjeev Khudanpur. 2018

163 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Music Transformer: Generating Music with Long-Term Structure (2018, background): “Dot-product self-attention is the mechanism at the core of the Transformer, and several recent works have focused on applying and improving it for image generation, speech, and summarization (Parmar et al., 2018; Povey et al., 2018; Liu et al., 2018).”
Fast Transformer Decoding: One Write-Head is All You Need (2019, background): “Another is to reduce the number of positions being attended-to, either by attending to a local neighborhood, or by otherwise compressing the number of memory positions, as in [Liu et al., 2018], [Zhang et al., 2018], [Povey et al., 2018].”
Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss (2020, methodology): “Transformer-based alternatives to RNNs have recently been explored for use in ASR [9, 10, 11, 12].”
Transformers with convolutional context for ASR (2019, methodology): “For a standard hybrid ASR system, [16] introduced a time-constrained key-value self-attention layer to be used in tandem with other TDNN and recurrent layers.”
Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting (2019, background): “Several recent works apply it to translation, speech, music and image generation [1, 21, 22, 23].”
Transformer-Based Acoustic Modeling for Hybrid Speech Recognition (2019, methodology): “Self-attention and transformer models were also used for ASR, mostly in the sequence-to-sequence architecture [17–19] with notable exceptions of [20,21].”
WENETSPEECH: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition (2021, methodology): “Then, we train a chain model, which stacks 6 convolutional neural network (CNN) layers, 9 factored time-delay neural network (TDNN-F) layers [30], 1 time-restricted attention layer [31] (H = 30), 2 projected long short-term memory (LSTMP) with TDNN-F blocks.”
Attention mechanism in neural networks: where it comes and where it goes (2022, background): “It is shown that self-attention can lead to improvements to discriminative constituency parser [81], and speech recognition as well [82, 83].”
Simplified Self-Attention for Transformer-Based end-to-end Speech Recognition (2020, methodology): “Recently, the transformer models have been applied to speech recognition tasks with competitive performance [16–19].”
Self-Attentional Acoustic Models (2018, background): “Notice the similar idea explored concurrently in independent work [8].”
Enhancing Transformer for End-to-end Speech-to-Text Translation (2019, background): “…the modeling side, Transformer’s performance is limited by the absence of a bias to capture short-range dependencies along time (Sperber et al., 2018; Povey et al., 2018), as well as the 2D joint dependencies over the time and frequency dimensions that characterize a spectrogram (Li et al., 2016).”
Self-Attentive Sequential Recommendation (2018, background): “A few options are promising to investigate in the future: 1) using restricted self-attention [42] which only attends on recent actions rather than all actions, and distant actions can be considered in higher layers; 2) splitting long sequences into short segments as in [22].”
Self-attention Networks for Connectionist Temporal Classification in Speech Recognition (2019, methodology): “Inspired by [26], we further plot the attention weights relative to the current time position (here, per head).”
Faster Transformer Decoding: N-gram Masked Self-Attention (2020, methodology): “It is used in LM and NMT experiments reported in (Shazeer, 2019), text summarization work in (Liu et al., 2018), image processing in (Parmar et al., 2018) and as well as automatic speech recognition as in (Povey et al., 2018).”
Self-attention Aligner: A Latency-control End-to-end Model for ASR Using Self-attention Network and Chunk-hopping (2019, methodology): “Besides, we utilize multiple vanilla self-attention networks in [9], contrasting to a single time-restricted self-attention layer in [12], which is placed towards the end of the TDNN or TDNNLSTM and provides latency-control by attending to limited future high-level context.”
State-of-the-Art Speech Recognition Using Multi-Stream Self-Attention with Dilated 1D Convolutions (2019, methodology): “With the huge success in NLP, self-attention has been actively investigated for speech recognition as well [5, 6, 7, 8, 9, 10].”