Self-attention Networks for Connectionist Temporal Classification in Speech Recognition

By Julian Salazar, Katrin Kirchhoff, Zhiheng Huang. 2019

125 citations recorded by the source corpus.

The success of self-attention in NLP has led to recent applications in end-to-end encoder-decoder architectures for speech recognition. Separately, connectionist temporal classification (CTC) has matured as an alignment-free, non-autoregressive approach to sequence transduction, either by itself or in various multitask and decoding frameworks. We propose SAN-CTC, a deep, fully self-attentional network for CTC, and show it is tractable and competitive for end-to-end speech recognition. SAN-CTC trains quickly and outperforms existing CTC models and most encoder-decoder models, with character error rates (CERs) of 4.7% in 1 day on WSJ eval92 and 2.8% in 1 week on LibriSpeech test-clean, with a fixed architecture and one GPU. Similar improvements hold for WERs after LM decoding. We motivate the architecture for speech, evaluate position and down-sampling approaches, and explore how label alphabets (character, phoneme, subword) affect attention heads and performance.

Subfields: Model Architectures, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Speech & Audio

Citations in the corpus

Transformers with convolutional context for ASR (2019, methodology): “There has been recent research interest in using transformer networks for end-to-end ASR both with CTC loss [13] and in an encoder-decoder framework [14, 15] with modest performance compared to baseline systems.”
Transformers without Tears: Improving the Normalization of Self-Attention (2019, methodology): “V S QRT D ECAY (Dong et al., 2018; Salazar et al., 2019).”
Transformer-Based Acoustic Modeling for Hybrid Speech Recognition (2019, methodology): “Self-attention and transformer models were also used for ASR, mostly in the sequence-to-sequence architecture [17–19] with notable exceptions of [20,21].”
Language Modeling with Deep Transformers (2019, neutral): “, [11, 23–25]).”
Towards Online End-to-end Transformer Automatic Speech Recognition (2019, methodology): “Transformer [15] has been successfully introduced into E2E ASR by replacing RNNs [16–20], and it outperforms bidirectional RNN models in most tasks [21].”
Interactive Speech and Noise Modeling for Speech Enhancement (2020, background): “Recently, speech-related tasks have also beneﬁted from self-attention, e.g., speech recognition (Salazar, Kirchhoff, and Huang 2019) and speech enhancement (Kim, El-Khamy, and Lee 2020; Koizumi et al. 2020).”
A Further Study of Unsupervised Pretraining for Transformer Based Speech Recognition (2020, methodology): “Experiments were conducted on HKUST with Transformer-CTC model and the results of various pretraining ob- jectives are presented in Table 5.”
A General Survey on Attention Mechanisms in Deep Learning (2022, methodology): “In [78], self-attention is used to create a speech recognition model.”
Speech-XLNet: Unsupervised Acoustic Model Pretraining For Self-Attention Networks (2019, methodology): “For automatic speech recognition (ASR), SANs have been introduced for acoustic modeling in either attention framework (also known as speech transformer [4]) or with CTC loss [5, 6].”
State-of-the-Art Speech Recognition Using Multi-Stream Self-Attention with Dilated 1D Convolutions (2019, methodology): “With the huge success in NLP, self-attention has been actively investigated for speech recognition as well [5, 6, 7, 8, 9, 10].”
Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “One way to relax that assumption is to use the attention mechanism [23, 24] which can introduce implicit language modeling across speech frames.”
A Comparison of Transformer and LSTM Encoder Decoder Models for ASR (2019, methodology): “More recent work suggested to use the Transformer directly as the acoustic model [34] or as an end-to-end ASR model [34–42].”