Improving Transformer-Based End-to-End Speech Recognition with Connectionist Temporal Classification and Language Model Integration

By Shigeki Karita, Nelson Yalta, Shinji Watanabe, Marc Delcroix, Atsunori Ogawa, Tomohiro Nakatani. 2019

231 citations recorded by the source corpus.

The state-of-the-art neural network architecture named Trans-former has been used successfully for many sequence-to-sequence transformation tasks. The advantage of this architecture is that it has a fast iteration speed in the training stage because there is no sequential operation as with recurrent neural networks (RNN). However, an RNN is still the best option for end-to-end automatic speech recognition (ASR) tasks in terms of overall training speed (i.e., convergence) and word error rate (WER) because of effective joint training and decoding meth-ods. To realize a faster and more accurate ASR system, we combine Transformer and the advances in RNN-based ASR. In our experiments, we found that the training of Transformer is slower than that of RNN as regards the learning curve and integration with the naive language model (LM) is difficult. To address these problems, we integrate connectionist temporal classification (CTC) with Transformer for joint training and decoding. This approach makes training faster than with RNNs and assists LM integration. Our proposed ASR system realizes significant improvements in various ASR tasks. For example, it reduced the WERs from 11.1% to 4.5% on the Wall Street Journal and from 16.1% to 11.6% on the TED-LIUM by introducing CTC and LM integration into the Transformer baseline.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Shinji Watanabe
2. Jinyu Li
3. Takaaki Hori
4. Dong Yu
5. Tomoki Hayashi
6. Yoshua Bengio
7. Quoc V. Le
8. Tie-Yan Liu

In Natural Language Processing

1. Jinyu Li
2. Shinji Watanabe
3. Barret Zoph
4. William Fedus
5. Takaaki Hori
6. Daniel Garcia-Romero
7. Jing Shi
8. Jiatong Shi

In Model Architectures

1. Jinyu Li
2. Shinji Watanabe
3. Joseph Dean
4. William Fedus
5. Barret Zoph
6. Tomoki Hayashi
7. Daniel Garcia-Romero
8. Takaaki Hori

Citations in the corpus

• A Review of Sparse Expert Models in Deep Learning (2022, background): “Some examples are image classification (Dosovitskiy et al., 2020), object detection (Carion et al., 2020), recommendation systems (Chen et al., 2019), speech recognition (Dong et al., 2018; Nakatani, 2019; Gulati et al., 2020).”
• A Comparative Study on Transformer vs RNN in Speech Applications (2019, methodology): “We adopted the same architecture for Transformer in [41] (e = 12, d = 6, d = 2048, d = 4, d = 256) for every corpus except for the largest, LibriSpeech (d = 8, d = 512).”
• Recent Developments on Espnet Toolkit Boosted By Conformer (2020, methodology): “For each corpus, the detail configurations of our Conformer model are same as ESPnet Transformer recipes [27] (Enc = 12 , Dec = 6 , d ff = 2048 , H = 4 , d att = 256 ).”
• Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “It becomes the standard training recipe for most AED models [52, 95, 121, 127, 166, 277, 284, 300].”
• Simplified Self-Attention for Transformer-Based end-to-end Speech Recognition (2020, unknown): “In [19], the authors integrate CTC with transformer for joint training and decoding, which leads to significant improvements in various ASR tasks.”
• Large-Scale End-to-End Multilingual Speech Recognition and Language Identification with Multi-Task Learning (2020, methodology): “[8] employed the multi-head selfattention mechanism in a similar Transformer architecture [9].”