A Comparison of Transformer and LSTM Encoder Decoder Models for ASR

By Albert Zeyer, Parnia Bahar, Kazuki Irie, Ralf Schlüter, Hermann Ney. 2019

217 citations recorded by the source corpus.

We present competitive results using a Transformer encoder-decoder-attention model for end-to-end speech recognition needing less training time compared to a similarly performing LSTM model. We observe that the Transformer training is in general more stable compared to the LSTM, although it also seems to overfit more, and thus shows more problems with generalization. We also find that two initial LSTM layers in the Transformer encoder provide a much better positional encoding. Data-augmentation, a variant of SpecAugment, helps to improve both the Transformer by 33% and the LSTM by 15% relative. We analyze several pretraining and scheduling schemes, which is crucial for both the Transformer and the LSTM models. We improve our LSTM model by additional convolutional layers. We perform our experiments on Lib-riSpeech 1000h, Switchboard 300h and TED-LIUM-v2 200h, and we show state-of-the-art performance on TED-LIUM-v2 for attention based end-to-end models. We deliberately limit the training on LibriSpeech to 12.5 epochs of the training data for comparisons, to keep the results of practical interest, although we show that longer training time still improves more. We publish all the code and setups to run our experiments.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Yu Zhang
3. Tara N. Sainath
4. Shinji Watanabe
5. Yonghui Wu
6. Dong Yu
7. Chung-Cheng Chiu
8. Gabriel Synnaeve

In Model Architectures

1. Jinyu Li
2. Shinji Watanabe
3. Tara N. Sainath
4. Yu Zhang
5. Yonghui Wu
6. Ruoming Pang
7. Hermann Ney
8. Ralf Schlüter

In Speech & Audio

1. Jinyu Li
2. Yu Zhang
3. Tara N. Sainath
4. Yonghui Wu
5. Gabriel Synnaeve
6. Chung-Cheng Chiu
7. Shinji Watanabe
8. Ronan Collobert

In Signal Processing

1. Jinyu Li
2. Yu Zhang
3. Tara N. Sainath
4. Shinji Watanabe
5. Dong Yu
6. Chung-Cheng Chiu
7. Yonghui Wu
8. Ralf Schlüter

Citations in the corpus

• ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context (2020, methodology): “This is a big improvement over previous CNN based architectures such as QuartzNet[4], and it outperforms transformer and LSTM based models [11, 10, 24, 9, 25].”
• CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings (2021, background): “Besides these datasets we use two other sets for evaluation only: i) LibriSpeech (LS) [36], read speech from audiobook recordings (we use only test sets with clean, test-clean , and noisy, test-other , speech); ii) Robust Video (RV), our in-house English video dataset, which is sampled from public s”
• The Rwth Asr System for Ted-Lium Release 2: Improving Hybrid Hmm With Specaugment (2020, background): “With increased model size and training time, end-to-end systems benefit strongly from SpecAugment [11, 12], and large improvements are also reported for end-to-end speech translation [13].”
• Recent Advances in End-to-End Automatic Speech Recognition (2021, methodology): “The encoder can be either a multi-layer unidirectional LSTM-RNN as Eq.”
• End-to-End Speech Recognition: A Survey (2023, background): “The subsequent gains mostly come from the exploration of the new E2E architectures, including transformer [75], [307], conformer [73], [76], and contextnet [70] on top of SpecAugment, as discussed in Section III-C.”