A Comparative Study on Transformer vs RNN in Speech Applications

By Shigeki Karita, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Yalta, Ryuichi Yamamoto, Xiao-fei Wang and 3 more. 2019

796 citations recorded by the source corpus.

Sequence-to-sequence models have been widely used in end-to-end speech processing, for example, automatic speech recognition (ASR), speech translation (ST), and text-to-speech (TTS). This paper focuses on an emergent sequence-to-sequence model called Transformer, which achieves state-of-the-art performance in neural machine translation and other natural language processing applications. We undertook intensive studies in which we experimentally compared and analyzed Transformer and conventional recurrent neural networks (RNN) in a total of 15 ASR, one multilingual ASR, one ST, and two TTS benchmarks. Our experiments revealed various training tips and significant performance benefits obtained with Transformer for each task including the surprising superiority of Transformer in 13/15 ASR benchmarks in comparison with RNN. We are preparing to release Kaldi-style reproducible recipes using open source and publicly available datasets for all the ASR, ST, and TTS tasks for the community to succeed our exciting outcomes.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Speech & Audio

Citations in the corpus

End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures (2019, neutral): “, 2012), and many of state-of-theart models still rely on force alignment (Han et al., 2017; Lüscher et al., 2019; Karita et al., 2019).”
GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10, 000 Hours of Transcribed Audio (2021, methodology): “The Athena baseline implements an encoder-decoder based transformer model, which is similar to [18], with parameters e = 12, d = 6, dmodel = 1024, dff = 2048 and dhead = 8.”
Iterative Pseudo-Labeling for Speech Recognition (2020, positive): “Some of the most promising are from the Transformer family [1, 2, 3, 4, 5], which give state-of-the-art results on many ASR benchmarks and close the gap between end-to-end and hybrid systems.”
Conformer: Convolution-augmented Transformer for Speech Recognition (2020, methodology): “The distinctive feature of our model is the use of Conformer blocks in the place of Transformer blocks as in [7, 19].”
You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentation (2020, neutral): “Sun et al. [19] suggest modeling style using an autoregressive prior over representations from quantized fine-grained VAE and perform evaluation by synthesizing utterances for training ASR on LibriSpeech.”
ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context (2020, methodology): “This is a big improvement over previous CNN based architectures such as QuartzNet[4], and it outperforms transformer and LSTM based models [11, 10, 24, 9, 25].”
Improving Transformer-based Speech Recognition Using Unsupervised Pre-training (2019, methodology): “Originally proposed as a better replacement of Recurrent Neural Network (RNN) for neural machine translation, Transformer based models got a lot of traction in the field of speech recognition recently [11, 12, 13, 14].”
Going deeper with Image Transformers (2021, background): “They became a staple in speech recognition too [45, 36], being it in encoder and sequence criterion or encoder-decoder seq2seq [61] conformations, and hold the state of the art to this day [74, 78] with models 36 blocks deep.”
Transformer-Based Acoustic Modeling for Hybrid Speech Recognition (2019, methodology): “In [30], transformers are compared with RNNs in the sequence-to-sequence architecture.”
Fast, Simpler and More Accurate Hybrid ASR Systems Using Wordpieces (2020, result): “Recently, Transformer [9] architectures have shown superior results in ASR tasks compared to BLSTMs [10–13].”
Don’t Shoot Butterfly with Rifles: Multi-Channel Continuous Speech Separation with Early Exit Transformer (2020, methodology): “For example, for automatic speech recognition (ASR) tasks, a common setting is to use twelve [15] or more layers [16] in the encoder.”
Speech Recognition via Large-Scale Weak Supervision (neutral): “Karita et al. (2019) is a nice comparison of RNNs vs Trans-formers in encoder-architectures for ASR, TTS, and speech-to-speech translation.”
A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (2022, background): “…on various application ﬁelds such as natural language processing (NLP) (Kalyan et al., 2021), computer vision (CV) (Khan et al., 2021), speech (Karita et al., 2019), and more recently time series (Wen et al., 2022), beneﬁting from its attention mechanism which can automatically learn the…”
Attention Is All You Need In Speech Separation (2020, positive): “For this reason, Transformers are gaining considerable popularity for speech processing and recently showed competitive performance in speech recognition [6], synthesis [7], enhancement [8], diarization [9], as well as speaker recognition [10].”
Large scale weakly and semi-supervised learning for low-resource video ASR (2020, background): “Transformers [29, 30, 31, 32], may lead to better absolute WER numbers, we believe that the conclusions reached in our experiments are invariant to such modeling decisions.”
Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “8 ESPnet recipe: SpecAugment, dropout [86] 2019 AED Transformer 2.”