Very Deep Self-Attention Networks for End-to-End Speech Recognition

By Ngoc-Quan Pham, Thai Son Nguyen, Jan Niehues, Markus Müller, A. Waibel. 2019

172 citations recorded by the source corpus.

Recently, end-to-end sequence-to-sequence models for speech recognition have gained significant interest in the research community. While previous architecture choices revolve around time-delay neural networks (TDNN) and long short-term memory (LSTM) recurrent neural networks, we propose to use self-attention via the Transformer architecture as an alternative. Our analysis shows that deep Transformer networks with high learning capacity are able to exceed performance from previous end-to-end approaches and even match the conventional hybrid systems. Moreover, we trained very deep models with up to 48 Transformer layers for both encoder and decoders combined with stochastic residual connections, which greatly improve generalizability and training efficiency. The resulting models outperform all previous end-to-end ASR approaches on the Switchboard benchmark. An ensemble of these models achieve 9.9% and 17.7% WER on Switchboard and CallHome test sets respectively. This finding brings our end-to-end models to competitive levels with previous hybrid systems. Further, with model ensembling the Transformers can outperform certain hybrid systems, which are more complicated in terms of both structure and training procedure.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Speech & Audio

In Signal Processing

Citations in the corpus

Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss (2020, methodology): “As typically used in sequence-to-sequence transduction tasks [4, 5, 6, 7, 8], Transformerbased models attend over encoder features using decoder features, implying that the decoding has to be done in a label-synchronous way, thereby posing a challenge for streaming speech recognition applications.”
Reducing Transformer Depth on Demand with Structured Dropout (2019, methodology): “Concurrently to this work, Pham et al. (2019) applied Stochastic Depth to train very deep Transformers for speech and show the benefits of its regularization effect.”
Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders (2019, methodology): “As direct addition of acoustic features to positional encoding may lead to potential training failure [14], the input frames are ﬁrst projected linearly to the hidden dimension of H dim before adding with positional encoding [15].”
Efficient Attention: Attention with Linear Complexities (2018, background): “They hold the state-ofthe-art records on virtually all tasks in natural language processing [7, 20, 26] and is highly competitive on end-to-end speech recognition [8, 18].”
FlauBERT: Unsupervised Language Model Pre-training for French (2019, background): “Recently, a regularization technique called stochastic depths (Huang et al., 2016) has been demonstrated to be very effective for training deep Transformers, by e.g. Pham et al. (2019) and Fan et al. (2019) who successfully trained architectures of more than 40 layers.”
A Survey of Transformers (2021, background): “, speech recognition [15, 31, 41, 97], speech synthesis [57, 76, 168], speech enhancement [65, 161] and music generation [56].”
Adaptive Adapters: An Efficient Way to Incorporate BERT Into Neural Machine Translation (2021, methodology): “On the Transformer based networks, related works have studied to prune the multi-head attention components [50] or the entire layer [20], as well as dynamically selecting layers in very deep models [51], [52].”
Learning Light-Weight Translation Models from Deep Transformer (2020, methodology): “On the other hand, our Skipping Sub-Layer method and Stochastic Layers (Pham et al. 2019) trained by the finetuning schema both beat the strong baseline.”
Simplified Self-Attention for Transformer-Based end-to-end Speech Recognition (2020, background): “For example, reported in [21], 48 SAN encoder layers plus another 48 SAN decoder layers totally constitute to 252M model parameters.”
MetaFormer Baselines for Vision (2022, background): “Transformer, since being introduced in [72], has become a popular backbone for various tasks in NLP [53, 3, 44, 78, 55], computer vision [5, 17, 68, 82, 73, 45] and other domains [18, 51, 39, 32, 29, 38, 7].”
FINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN (2020, methodology): “For the end-to-end system they applied a deep Transformer with stochastic layers (Pham et al., 2019b).”
Don’t Shoot Butterfly with Rifles: Multi-Channel Continuous Speech Separation with Early Exit Transformer (2020, background): “Prior work shows that a deeper structure yields superior performance [14].”
Multiattention Network for Semantic Segmentation of Fine-Resolution Remote Sensing Images (2020, background): “Moreover, attention also attained state-of-the-art performance in speech recognition [56, 57].”
End-to-End Speech Recognition: A Survey (2023, background): “…weight decay [166], [182]; weight noise [183]; adaptive mean L1/L2 [184]; gradient noise [185]; dropout [186], [187], [188], layer dropout [189], [190], [191]; dropcon-nect [192]; zoneout [193]; smoothing of attention scores [15]; label smoothing [194]; scheduled sampling [195]; auxiliary loss…”