EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding

By Yajie Miao, Mohammad A. Gowayyed, Florian Metze. 2015

788 citations recorded by the source corpus.

The performance of automatic speech recognition (ASR) has improved tremendously due to the application of deep neural networks (DNNs). Despite this progress, building a new ASR system remains a challenging task, requiring various resources, multiple training stages and significant expertise. This paper presents our Eesen framework which drastically simplifies the existing pipeline to build state-of-the-art ASR systems. Acoustic modeling in Eesen involves learning a single recurrent neural network (RNN) predicting context-independent targets (phonemes or characters). To remove the need for pre-generated frame labels, we adopt the connectionist temporal classification (CTC) objective function to infer the alignments between speech and label sequences. A distinctive feature of Eesen is a generalized decoding approach based on weighted finite-state transducers (WFSTs), which enables the efficient incorporation of lexicons and language models into CTC decoding. Experiments show that compared with the standard hybrid DNN systems, Eesen achieves comparable word error rates (WERs), while at the same time speeding up decoding significantly.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin (2015, methodology): “The CTC-RNN model also works well in predicting phonemes (Miao et al., 2015; Sak et al., 2015), though a lexicon is still needed in this case.”
Unsupervised Pre-Training of Bidirectional Speech Encoders via Masked Reconstruction (2020, methodology): “[31], with the extended 4-gram language model built by the Kaldi recipe.”
Wav2Letter++: A Fast Open-source Speech Recognition System (2018, background): “[4] Yajie Miao, Mohammad Gowayyed, and Florian Metze, “Eesen: End-to-end speech recognition using deep rnn models and wfst-based decoding,” in Automatic Speech Recognition and Understanding (ASRU), 2015 IEEE Workshop on.”
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition (2015, methodology): “CTC has also been applied to end-to-end training with phoneme targets and n-gram language models using FSTs in [26, 27, 6].”
Wav2Letter: an End-to-End ConvNet-based Speech Recognition System (2016, methodology): “There are now competitive end-to-end approaches of acoustic models toppled with RNNs layers as in [8, 13, 21, 1], trained with a sequence criterion [6].”
Towards Better Decoding and Language Model Integration in Sequence to Sequence Models (2016, methodology): “Our best result surpasses CTCbased networks [6] and matches the results of a DNN-HMM and CTC ensemble [23].”
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations (2016, methodology): “Recurrent nets have recently been used to achieve state-of-the-art results in many machine learning domains including speech recognition [Miao et al., 2015], computer vision [Yao et al., 2015], language modeling [Kim et al., 2015] and machine translation [Bahdanau et al., 2014].”
The zero resource speech challenge 2017 (2017, background): “Recent work has begun to mitigate the need for expert knowledge, having succeeded at training supervised systems using textual transcripts only [1, 2].”
On Multiplicative Integration with Recurrent Neural Networks (2016, methodology): “We follow the same data preparation process and model setting as in [18], and we use 59 characters as the targets for the acoustic modelling.”
ESPnet: End-to-End Speech Processing Toolkit (2018, methodology): “• CTC-based: EESEN [11], Stanford CTC [28], Baidu Deepsppech [12],”
Deep Contextualized Acoustic Representations for Semi-Supervised Speech Recognition (2019, methodology): “[26] Yajie Miao, Mohammad Gowayyed, and Florian Metze, “EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding,” in ASRU.”
Letter-Based Speech Recognition with Gated ConvNets (2017, methodology): “Competitive end-to-end approaches are acoustic models toppled with RNNs layers as in (Hannun et al., 2014; Miao et al., 2015; Saon et al., 2015; Amodei et al., 2016) trained with a sequence criterion Tang et al. (2017) (the most popular ones being CTC (Graves et al., 2006) and MMI (Bahl et al.,…”
Streaming End-to-end Speech Recognition for Mobile Devices (2018, background): “∗Equal contribution Early E2E work examined connectionist temporal classification (CTC) [15] with grapheme or word targets [16, 17, 18, 19].”
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization (2020, methodology): “For decoding, we used WFST-based decoding using EESEN [38].”
WENETSPEECH: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition (2021, methodology): “After defining the force alignment graph L, it is further composed with the CTC topology graph T[27] to build the final force decoding graph F = T ◦ L for label error detection and validation.”
English Conversational Telephone Speech Recognition by Humans and Machines (2017, methodology): “One possibility was to train an end-to-end LSTM using CTC as in [14, 15, 16, 17] and append the features from the last LSTM layer before the softmax to our existing features.”
Exploring neural transducers for end-to-end speech recognition (2017, methodology): “…is artiﬁcially broken by the inclusion of a language model, and decoding is then the task of ﬁnding the argmax of log ( P CTC ( y | x )) + α log ( P LM ( y )) + β wordcount ( y ) (4) This decoding is approximate, and performed using beam search, typically with a large beam or lattice [15, 19].”
Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems (2017, background): “Two main types of end-to-end models for speech recognition have been proposed in the literature: connectionist temporal classification (CTC) [1, 2] and sequence-to-sequence learning (seq2seq) [3, 4].”