Joint CTC/attention decoding for end-to-end speech recognition

By Takaaki Hori, Shinji Watanabe, John R. Hershey. 2017

150 citations recorded by the source corpus.

End-to-end automatic speech recognition (ASR) has become a popular alternative to conventional DNN/HMM systems because it avoids the need for linguistic resources such as pronunciation dictionary, tokenization, and context-dependency trees, leading to a greatly simplified model-building process. There are two major types of end-to-end architectures for ASR: attention-based methods use an attention mechanism to perform alignment between acoustic frames and recognized symbols, and connectionist temporal classification (CTC), uses Markov assumptions to efficiently solve sequential problems by dynamic programming. This paper proposes joint decoding algorithm for end-to-end ASR with a hybrid CTC/attention architecture, which effectively utilizes both advantages in decoding. We have applied the proposed method to two ASR benchmarks (spontaneous Japanese and Mandarin Chinese), and showing the comparable performance to conventional state-of-the-art DNN/HMM ASR systems without linguistic resources.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, methodology): “We select the model with the highest accuracy on dev-other set for inference and apply the joint CTC/attention decoding (Hori et al., 2017) to further improve the performance.”
WENETSPEECH: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition (2021, methodology): “For decoding, ESPNet system follows the joint CTC/Attention beam search strategy [35].”
Recent Advances in End-to-End Automatic Speech Recognition (2021, methodology): “In [42], a further improvement was proposed by combining the scores from both the AED model and the CTC model during decoding.”
Improved Knowledge Distillation from Bi-Directional to Uni-Directional LSTM CTC for End-to-End Speech Recognition (2018, background): “Watson Research Center, USA 1We do not focus on other major approaches to E2E ASR in this paper, namely attention-based sequence-to-sequence systems [1] and joint CTC/sequence-to-sequence systems [2].”
SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training (2022, methodology): “ASR inference We select the model with the highest accuracy on dev-other set as the final model and apply the joint CTC/ED decoding (Hori et al., 2017).”
Unsupervised Domain Adaptation for Speech Recognition via Uncertainty Driven Self-Training (2020, background): “This success is driven by improved neural network architectures and training frameworks [1–5], increasingly large amounts of labeled data [6,7], and increased computational resources for training complex models.”
Recent progresses in deep learning based acoustic models (2017, methodology): “In [94], it was further advanced with jointly decoding with the scores from both attention-based model and CTC model.”
Cycle-consistency Training for End-to-end Speech Recognition (2018, background): “End-to-end ASR has the goal of simplifying the module-based architecture into a single-network architecture within a deep learning framework, in order to address these issues [2–6].”
Hybrid CTC/Attention Architecture for End-to-End Speech Recognition (2017, methodology): “During decoding, we propose a joint decoding approach, which combines both attention-based and CTC scores in a rescoring/one-pass beam search algorithm to eliminate the irregular alignments [20].”
Token-wise Training for Attention Based End-to-end Speech Recognition (2019, methodology): “During evaluation, we apply both regular decoding as well as a joint decoding method using an external LSTM LM [22, 11].”
Language Independent End-to-End Architecture For Joint Language and Speech Recognition (methodology): “This model compensates too flexible alignment properties in the attention-based method with CTC as a regularization during training and as a score correction during decoding [13].”
End-to-End Speech Recognition: A Survey (2023, methodology): “sentences in the label-synchronous beam search [190], [214].”