Hybrid CTC/Attention Architecture for End-to-End Speech Recognition

By Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R. Hershey, Tomoki Hayashi. 2017

882 citations recorded by the source corpus.

Conventional automatic speech recognition (ASR) based on a hidden Markov model (HMM)/deep neural network (DNN) is a very complicated system consisting of various modules such as acoustic, lexicon, and language models. It also requires linguistic resources, such as a pronunciation dictionary, tokenization, and phonetic context-dependency trees. On the other hand, end-to-end ASR has become a popular alternative to greatly simplify the model-building process of conventional ASR systems by representing complicated modules with a single deep network architecture, and by replacing the use of linguistic resources with a data-driven learning method. There are two major types of end-to-end architectures for ASR; attention-based methods use an attention mechanism to perform alignment between acoustic frames and recognized symbols, and connectionist temporal classification (CTC) uses Markov assumptions to efficiently solve sequential problems by dynamic programming. This paper proposes hybrid CTC/attention end-to-end ASR, which effectively utilizes the advantages of both architectures in training and decoding. During training, we employ the multiobjective learning framework to improve robustness and achieve fast convergence. During decoding, we perform joint decoding by combining both attention-based and CTC scores in a one-pass beam search algorithm to further eliminate irregular alignments. Experiments with English (WSJ and CHiME-4) tasks demonstrate the effectiveness of the proposed multiobjective learning over both the CTC and attention-based encoder–decoder baselines. Moreover, the proposed method is applied to two large-scale ASR benchmarks (spontaneous Japanese and Mandarin Chinese), and exhibits performance that is comparable to conventional DNN/HMM ASR systems based on the advantages of both multiobjective learning and joint decoding without linguistic resources.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Speech & Audio

In Signal Processing

Citations in the corpus

Earnings-21: A Practical Benchmark for ASR in the Wild (2021, methodology): “The second system is an ESPnet V2 [17] based hybrid CTC/Attention encoder-decoder [18] model trained on 10,000 hours of out-of-domain proprietary audio.”
Privacy-Preserving Adversarial Representation Learning in ASR: Reality or Illusion? (2019, positive): “As argued in [32], attention works well in most cases because it does not assume conditional independence between the output labels (unlike CTC).”
Libriheavy: A 50,000 Hours ASR Corpus with Punctuation Casing and Context (2023, methodology): “In this section, we present the baseline systems and experimental results for two popular models, namely CTC-Attention [18] and neural transducer [2].”
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, methodology): “For ASR, we add an extra linear layer to calculate the CTC loss at the top of the encoder (Watanabe et al., 2017).”
ESPnet: End-to-End Speech Processing Toolkit (2018, methodology): “This method corresponds to a shallow fusion of a decoder network and RNNLM originally proposed in neural machine translation [41] and applied to end-to-end speech recognition [34].”
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization (2020, methodology): “This ASR system can be either a conventional acoustic model in a hybrid-based ASR system, or an end-to-end speech recognition such as RNN-Transducers [34, 35] or Encoder-Decoder based model [36, 37].”
Massively Multilingual Adversarial Speech Recognition (2019, methodology): “Our experiments are conducted within the framework of a hybrid CTC-attention end-to-end neural model using ESPnet (Watanabe et al., 2017b), which uses an encoder-decoder architecture implemented in PyTorch (Paszke et al., 2017).”
The Pytorch-kaldi Speech Recognition Toolkit (2018, background): “Features K i rch K i K i Fig. 1: An overview of the PyTorch-Kaldi architecture. of the PyKaldi does not provide several pre-implemented and readyto-use neural models. Another python project is ESPnet [23]. ESPnet is an end-to-end speech processing toolkit, mainly focuses on end-to-end speech recogni”
Towards Online End-to-end Transformer Automatic Speech Recognition (2019, methodology): “For comparison, unidirectional and bidirectional LSTM models were also trained as in [11].”
Multi-talker ASR for an unknown number of sources: Joint training of source counting, separation and ASR (2020, methodology): “The FE is either a TasNet or an OR-PIT and the backend is a CTC/attention speech recognizer similar to [3,19] taken from the ESPnet toolkit [21].”
ESPnet-ST: All-in-One Speech Translation Toolkit (2020, methodology): “5https://github.com/NVIDIA/apex Automatic speech recognition (ASR) We build ASR components with the Transformer-based hybrid CTC/attention framework (Watanabe et al., 2017), which has been shown to be more effective than RNN-based models on various speech corpora (Karita et al., 2019).”
End-To-End Audio-Visual Speech Recognition with Conformers (2021, methodology): “In this work, we adopt a hybrid CTC/Attention architecture [32] to force monotonic alignments and at the same time get rid of the conditional independence assumption.”
AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario (2021, methodology): “The model is trained with joint CTC and cross-entropy objective function [33] to accelerate the convergence of the training.”
Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining (2020, methodology): “E2E ASR is optimized with hybrid CTC/attention losses [30] with label smoothing.”
Investigation of Practical Aspects of Single Channel Speech Separation for ASR (2021, methodology): “In this work, during training, we employ encoder-decoder based E2E speech recognition model in the experiments, with a hybrid CTC & attention objective function [33]:”
The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines (2018, neutral): “A hybrid CTC/attention-based encoder-decoder network [42] is trained by using either the Chainer or PyTorch backend.”
MAM: Masked Acoustic Modeling for End-to-End Speech-to-Text Translation (2020, positive): “In our baseline setting, we also hybrid CTC/Attention framework (Watanabe et al., 2017) on the encoder side.”
SpeechBrain: A General-Purpose Speech Toolkit (2021, methodology): “CTC [24] Transducers [25] CTC+Attention [26] Shallow fusion [27] LibriSpeech [28] Common Voice [29] AISHELL [30] TIMIT [31] separation, pyannote [17] and sidekit [18] for speaker diarization, and s3prl [19] for self-supervised speech representations.”
Large-Scale End-to-End Multilingual Speech Recognition and Language Identification with Multi-Task Learning (2020, background): “[7] proposed a hybrid CTC/attention architecture, which benefits from both architectures in training and decoding.”
Have Best of Both Worlds: Two-Pass Hybrid and E2E Cascading Framework for Speech Recognition (2021, methodology): “The joint CTC and attention structure follows the work in [30], we find it works well especially for long utterances.”