Advances in Joint CTC-Attention Based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM

By Takaaki Hori, Shinji Watanabe, Yu Zhang, William Chan. 2017

310 citations recorded by the source corpus.

We present a state-of-the-art end-to-end Automatic Speech Recognition (ASR) model. We learn to listen and write characters with a joint Connectionist Temporal Classification (CTC) and attention-based encoder-decoder network. The encoder is a deep Convolutional Neural Network (CNN) based on the VGG network. The CTC network sits on top of the encoder and is jointly trained with the attention-based decoder. During the beam search process, we combine the CTC predictions, the attention-based decoder predictions and a separately trained LSTM language model. We achieve a 5-10\% error reduction compared to prior systems on spontaneous Japanese and Chinese speech, and our end-to-end model beats out traditional hybrid ASR systems.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

A Comparison of Modeling Units in Sequence-to-Sequence Speech Recognition with the Transformer on Mandarin Chinese (2018, methodology): “0% by the joint CTC-attention based encoder-decoder network with a separate RNN-LM integration [11].”
Syllable-Based Sequence-to-Sequence Speech Recognition with the Transformer in Mandarin Chinese (2018, methodology): “0% by the joint CTCattention based encoder-decoder network [9].”
ESPnet: End-to-End Speech Processing Toolkit (2018, methodology): “The Chainer backend also supports convolutional neural networks based on initial two blocks of VGG layer (VGG2()) [32] followed by BLSTM layers inspired by [33, 34], that is”
A Comparative Study on Transformer vs RNN in Speech Applications (2019, methodology): “First, EncPre(·) transforms the source sequence X into a subsampled sequence X0 ∈ R sub×datt by using two-layer CNN with 256 channels, stride size 2 and kernel size 3 in [2], or VGG-like max pooling in [21], where n is the length of the output sequence of the CNN.”
Monotonic Chunkwise Attention (2017, background): “Further, Hori et al. (2017) recently proposed jointly training a speech recognition model with both a CTC loss and an attention mechanism.”
Performance Evaluation of Deep Neural Networks Applied to Speech Recognition: RNN, LSTM and GRU (2019, methodology): “Generative models are typically composed of Maximum-A-Posteriori (MAP) estimation, Gaussian Mixture Models (GMMs), and Hidden Markov Models (HMM) [11, 12].”
Extending Recurrent Neural Aligner for Streaming End-to-End Speech Recognition in Mandarin (2018, methodology): “[23] 28.0 Extended-RNA (speed perturb.)”
Effective Sentence Scoring Method Using BERT for Speech Recognition (2019, methodology): “Third, LAS is trained with additional CTC objective function because the left-to-right constraint of CTC helps LAS learn alignments between speech-text pairs Hori et al. (2017).”
Improving Transformer-Based End-to-End Speech Recognition with Connectionist Temporal Classification and Language Model Integration (2019, methodology): “In addition, language model (LM) integration [2], [6] that has been widely used for the HMM based systems [8] is still effective for end-to-end ASR.”
Adversarial Training of End-to-end Speech Recognition Using a Criticizing Language Model (2018, methodology): “With many recent results [9, 10, 11, 12, 13] approaching the stateof-the-art, end-to-end deep learning has definitely been a very important direction for speech recognition.”
Attention-Based End-to-End Speech Recognition on Voice Search (2017, background): “Recently, some works on end-to-end neural network ASR systems such as Connectionist Temporal Classification (CTC) [4, 5, 6] and attention-based encoderdecoder [7, 8, 9, 10, 11, 12, 13, 14] have emerged.”
Pretraining by Backtranslation for End-to-End ASR in Low-Resource Settings (2018, methodology): “Bolstering the decoder with a language model (LM) trained on supplemental text data is one such method that improves end-to-end ASR performance [4, 5]; however, more significant gains can be obtained by training on additional synthetically perturbed speech [6, 7, 8], or by multilingual training, whi”
Back-Translation-Style Data Augmentation for end-to-end ASR (2018, methodology): “Finally, the results of shallow fusion with a characterbased language model (LM) [14] are shown in Table 6, where the LM was trained using text of 360 hours of clean speech, and the balancing weight parameter between two models was decided to achieve the best recognition performance on CER.”
Language Independent End-to-End Architecture For Joint Language and Speech Recognition (methodology): “End-to-end automatic speech recognition (ASR) has shown the effectiveness recently by greatly reducing ASR building procedure, and by reaching the state-of-the-art performance obtained by conventional hybrid systems [1, 2, 3, 4].”
Cycle-consistency Training for End-to-end Speech Recognition (2018, methodology): “Finally, we combined the ASR model with a character-based language model (LM) in a shallow fusion technique [25].”
Massively Multilingual Adversarial Speech Recognition (2019, methodology): “…ASR has recently become popular, with approaches such as attention-based encoder-decoder models (Chorowski et al., 2015; Chan et al., 2015), the connectionist temporal classi-ﬁcation (CTC) objective of Graves et al. (2006, 2013), or a combination of both (Kim et al., 2016; Hori et al., 2017).”