Joint CTC-attention based end-to-end speech recognition using multi-task learning

By Suyoun Kim, Takaaki Hori, Shinji Watanabe. 2016

1,000 citations recorded by the source corpus.

Recently, there has been an increasing interest in end-to-end speech recognition that directly transcribes speech to text without any predefined alignments. One approach is the attention-based encoder-decoder framework that learns a mapping between variable-length input and output sequences in one step using a purely data-driven method. The attention model has often been shown to improve the performance over another end-to-end approach, the Connectionist Temporal Classification (CTC), mainly because it explicitly uses the history of the target character without any conditional independence assumptions. However, we observed that the performance of the attention has shown poor results in noisy condition and is hard to learn in the initial training stage with long input sequences. This is because the attention model is too flexible to predict proper alignments in such cases due to the lack of left-to-right constraints as used in CTC. This paper presents a novel method for end-to-end speech recognition to improve robustness and achieve fast convergence by using a joint CTC-attention model within the multi-task learning framework, thereby mitigating the alignment issue. An experiment on the WSJ and CHiME-4 tasks demonstrates its advantages over both the CTC and attention-based encoder-decoder baselines, showing 5.4–14.6% relative improvements in Character Error Rate (CER).

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Shinji Watanabe
3. Dong Yu
4. Tara N. Sainath
5. Yu Zhang
6. Hung-yi Lee
7. Yoshua Bengio
8. Yonghui Wu

In Speech & Audio

1. Jinyu Li
2. Shinji Watanabe
3. Tara N. Sainath
4. Yu Zhang
5. Yonghui Wu
6. Hung-yi Lee
7. Dong Yu
8. Chung-Cheng Chiu

In Signal Processing

1. Jinyu Li
2. Shinji Watanabe
3. Dong Yu
4. Tara N. Sainath
5. Yu Zhang
6. Rohit Prabhavalkar
7. Takaaki Hori
8. Bo Li

Citations in the corpus

• Imputer: Sequence Modelling via Imputation and Dynamic Programming (2020, neutral): “. Model CER WER Iterations seq2seq Bahdanau et al. (2016a) 6.4 18.6 n Bahdanau et al. (2016b) 5.9 18.0 n Chorowski & Jaitly (2017) - 10.6 n Zhang et al. (2017) - 10.5 n Chan et al. (2017) - 9.6 n Kim et al. (2017) 7.4 - n Serdyuk et al. (2018) 6.2 - n Tjandra et al. (2018) 6.1 - n Sabour et al. ”
• Multitask Training with Text Data for End-to-End Speech Recognition (2020, background): “Various model architectures [6, 7, 8, 9, 10] and training schemes [11, 12, 13] were proposed.”
• Streaming End-to-end Speech Recognition for Mobile Devices (2018, background): “More recent work has demonstrated that performance can be improved further using either the recurrent neural network transducer (RNN-T) model [12, 20, 21] or attention-based encoder-decoder models [10, 13, 14, 22].”
• Model Unit Exploration for Sequence-to-Sequence Speech Recognition (2019, background): “onemes, sequence-to-sequence 1. INTRODUCTION Sequence-to-sequence learning [1] based on encoder-decoder attention models [2] has become popular for both machine translation [3] and speech recognition [4, 5, 6, 7, 8]. Such models are typically trained to output character-based units: graphemes, byte-”
• WENETSPEECH: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition (2021, background): “On the one hand, the development of neural networks has increased the capacity of models, pushing the dominant framework from the hybrid hidden Markov models [1, 2] to end-to-end models, like CTC [3, 4], RNNT [5, 6], and encoder-decoder based models [7, 8, 9, 10, 11].”
• Learning Disentangled Phone and Speaker Representations in a Semi-Supervised VQ-VAE Paradigm (2020, methodology): “We used the ESPNet[18] Kaldi recipe for CMU AN4 ASR [19] and a single-layer LSTM model with 64 units, trained to 100 epochs, selected the best model based on CTC loss without attention (mtlalpha = 1.0), and decoded with beam size 20.”
• Large-Scale End-to-End Multilingual Speech Recognition and Language Identification with Multi-Task Learning (2020, methodology): “Similar to [18], we adopt a hybrid CTC/attention architecture, where the model is composed of three components: a shared encoder, an attention decoder, and a CTC module.”
• Didispeech: A Large Scale Mandarin Speech Corpus (2020, methodology): “CTC-joint decoding is performed as proposed in [34] with CTC weight 0.”
• End-to-end Feedback Loss in Speech Chain Framework via Straight-through Estimator (2018, methodology): “models in the proposed section were trained with a combination from two losses: L ASR + L rec TTS , 6 Table 1: ASR experiment result on WSJ dataset test eval92. Baseline (L ASR) Model CER (%) Att MLP [20] 11.08 Att MLP + Location [20] 8.17 Att MLP [21] 7.12 Att MLP-MA (ours) [19] 6.43 Proposed (L AS”
• Recent progresses in deep learning based acoustic models (2017, methodology): “In [93], the attention training is combined with CTC training in a multi-task learning way by using CTC objective function as the auxiliary function.”
• Adversarial Training of End-to-end Speech Recognition Using a Criticizing Language Model (2018, methodology): “ht is also projected to y̌t with linear layer Transform as output to help in learning of the encoder as shown in previous work [12].”
• Improving Attention-Based End-to-End ASR Systems with Sequence-Based Loss Functions (2018, background): “The performance improvement by decoding with extra LMs trained on the same corpus suggests that certain knowledge has yet to be exploited in E2E systems[12].”
• Gram-CTC: Automatic Unit Selection and Target Decomposition for Sequence Labelling (2017, background): “Joint training of multiple objectives for sequence labelling has also been explored in previous works (Kim et al., 2016; Kim and Rush, 2016).”
• A Streaming On-Device End-To-End Model Surpassing Server-Side Conventional Model Quality and Latency (2020, background): “End-to-end (E2E) models [2, 3, 4, 5, 6, 7, 8, 9] have gained large popularity in the automatic speech recognition (ASR) community over the last few years.”
• Any-to-Many Voice Conversion With Location-Relative Sequence-to-Sequence Modeling (2020, methodology): “2) Model Structure and Training Objective: Following [30], we regard the CTC objective as an auxiliary task to train the attention model encoder, which contains a VGG-Prenet and a bidirectional LSTM (BiLSTM) encoder, as shown in Fig.”