Improved training of end-to-end attention models for speech recognition

By Albert Zeyer, Kazuki Irie, Ralf Schlüter, Hermann Ney. 2018

281 citations recorded by the source corpus.

Sequence-to-sequence attention-based models on subword units allow simple open-vocabulary end-to-end speech recognition. In this work, we show that such models can achieve competitive results on the Switchboard 300h and LibriSpeech 1000h tasks. In particular, we report the state-of-the-art word error rates (WER) of 3.54% on the dev-clean and 3.82% on the test-clean evaluation subsets of LibriSpeech. We introduce a new pretraining scheme by starting with a high time reduction factor and lowering it during training, which is crucial both for convergence and final performance. In some experiments, we also use an auxiliary CTC loss function to help the convergence. In addition, we train long short-term memory (LSTM) language models on subword units. By shallow fusion, we report up to 27% relative improvements in WER over the attention baseline without a language model.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Speech & Audio

Citations in the corpus

RWTH ASR Systems for LibriSpeech: Hybrid vs Attention - w/o Data Augmentation (2019, methodology): “An end-to-end attention-based encoder-decoder approach with a pretraining scheme is presented in [5, 6].”
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition (2019, background): “These models, being end-to-end, are simple to train and have the added benefit of having well-documented benchmarks [24, 25] that we are able to build upon to get our results.”
Transformers with convolutional context for ASR (2019, methodology): “her potential improvements to the WER using an LM trained on much larger text corpus. Model LM on extra text dev clean dev other test clean test other CAPIO spk adpt[22] RNNLM 3.12 8.28 3.51 8.58 LSTM[23] 4gramLM 4.79 14.31 4.82 15.30 Gated Cnv[24] 4gramLM 4.6 13.8 4.8 14.5 Tsf w/sin pos embd[13] 4g”
Self-Training for End-to-End Speech Recognition (2019, methodology): “[1] use the sequence-tosequence model proposed in [22] and to our knowledge, produce the best prior result when limited to “train-clean-100.”
Sequence-to-Sequence Speech Recognition with Time-Depth Separable Convolutions (2019, methodology): “Compared with the RNN-based encoder [23], the TDS model improves WER by 14.”
Fully Convolutional Speech Recognition (2018, methodology): “In end-to-end speech recognition however, recurrent architectures are still prevalent for acoustic and/or language modeling [5, 6, 7, 8, 9].”
Jasper: An End-to-End Convolutional Neural Acoustic Model (2019, methodology): “Traditional N-gram LMs have been augmented with neural LMs in recent work [21, 22, 23].”
Letter-Based Speech Recognition with Gated ConvNets (2017, methodology): “Competitive end-to-end approaches leverage acoustic models (often ConvNet-based) topped with RNN layers as in (Hannun et al., 2014a; Miao et al., 2015; Saon et al., 2015; Amodei et al., 2016; Zhou et al., 2018; Zeyer et al., 2018) (e.”
A Comparative Study on Transformer vs RNN in Speech Applications (2019, methodology): “For RNN, we followed our existing best architecture configured on each corpus as in previous studies [17], [42].”
Optimal Completion Distillation for Sequence Learning (2018, methodology): “Our MLE baseline achieves 5.7% WER, while OCD achieves 4.5% WER on test-clean (21% improvement) and improves the state-of-the-art results over Zeyer et al. (2018). test-other is the more challenging test split ranked by the WER of a model trained on WSJ (Panayotov et al., 2015) mainly because…”
Model Unit Exploration for Sequence-to-Sequence Speech Recognition (2019, methodology): “Standard LAS models use character-based output units: grapheme [12], word-piece [9] or BPE [21].”
Language Modeling with Deep Transformers (2019, methodology): “t experiments on the LibriSpeech automatic speech recognition (ASR) task [16] for both word-level conventional speech recognition and byte-pair encoding (BPE) [17] level end-to-end speech recognition [18,19]. We apply our word-level models to hybrid speech recognition by lattice rescoring [20], and ”
A novel pyramidal-FSMN architecture with lattice-free MMI for speech recognition (2018, unknown): “Inspired by [26], the combination of LF-MMI and end-to-end on our architecture will be further explored in recent future.”
One-to-Many Multilingual End-to-End Speech Translation (2019, background): “The state-of-the art results obtained by encoder-decoder models [1] with sequence-to-sequence learning in fields like ASR [2, 3, 4, 5] and, most importantly, machine translation [6, 7], have led to the recent proposal of sequence-to-sequence learning for direct speech-to-text translation [8, 9], tha”
Survey on Deep Neural Networks in Speech and Vision Systems (2019, methodology): “Using a pretraining technique [133] with attention and LSTM model, speech recognition performance has been improved to a new state-of-the-art level.”
Enhancing Transformer for End-to-end Speech-to-Text Translation (2019, background): “…results on machine translation (MT) (Bentivogli et al., 2016; Gehring et al., 2017; Vaswani et al., 2017; Chen et al., 2018) and obtained increasingly good performance in automatic speech recognition (Chan et al., 2016; Chiu et al., 2018; Zhang et al., 2017; Zeyer et al., 2018; Dong et al., 2018).”
Who Needs Words? Lexicon-Free Speech Recognition (2019, background): “In recent works [18, 19], authors achieved competitive results with acoustic models (AM) and LMs operating on word pieces, and a lexicon-free decoder.”
Fully Neural Network Based Speech Recognition on Mobile and Embedded Devices (2018, unknown): “7 4-gram LM Encoder-decoder [32] 150M 3.”