On training the recurrent neural network encoder-decoder for large vocabulary end-to-end speech recognition

By Liang Lu, Xingxing Zhang, S. Renals. 2016

125 citations recorded by the source corpus.

Recently, there has been an increasing interest in end-to-end speech recognition using neural networks, with no reliance on hidden Markov models (HMMs) for sequence modelling as in the standard hybrid framework. The recurrent neural network (RNN) encoder-decoder is such a model, performing sequence to sequence mapping without any predefined alignment. This model first transforms the input sequence into a fixed length vector representation, from which the decoder recovers the output sequence. In this paper, we extend our previous work on this model for large vocabulary end-to-end speech recognition. We first present a more effective stochastic gradient decent (SGD) learning rate schedule that can significantly improve the recognition accuracy. We then extend the decoder with long memory by introducing another recurrent layer that performs implicit language modelling. Finally, we demonstrate that using multiple recurrent layers in the encoder can reduce the word error rate. Our experiments were carried out on the Switchboard corpus using a training set of around 300 hours of transcribed audio data, and we have achieved significantly higher recognition accuracy, thereby reduced the gap compared to the hybrid baseline.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Dong Yu
3. Shinji Watanabe
4. Tara N. Sainath
5. Yu Zhang
6. Li Deng
7. Takaaki Hori
8. Rohit Prabhavalkar

In Signal Processing

1. Jinyu Li
2. Dong Yu
3. Shinji Watanabe
4. Tara N. Sainath
5. Yu Zhang
6. Rohit Prabhavalkar
7. Li Deng
8. Bo Li

Citations in the corpus

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition (2019, neutral): “42] 9.3 18.9 Zeyer et al., (2018) [24] 8.3 17.3 CTC Zweig et al., (2017) [43] 24.7 37.1 14.0 25.3 Audhkhasi et al., (2018) [44] 20.8 30.4 Audhkhasi et al., (2018) [45] 14.6 23.6 LAS Lu et al., (2016) [46] 26.8 48.2 25.8 46.0 Toshniwal et al., (2017) [47] 23.1 40.8 Zeyer et al., (2018) [24] 13.1 26.1”
• ESPnet: End-to-End Speech Processing Toolkit (2018, methodology): “1Language modeling is often performed by external language model toolkits, for example SRILM [6] network [13, 14, 15, 16].”
• Joint CTC-attention based end-to-end speech recognition using multi-task learning (2016, background): “End-to-end speech recognition is a recently proposed approach that directly transcribes speech to text without requiring predefined alignment between acoustic frames and characters [1, 2, 3, 4, 5, 6, 7, 8, 9].”
• End-to-End Speech Recognition: A Survey (2023, background): “Another prominent feature of E2E systems is their direct character-level modeling avoiding phoneme-based modeling and pronunication lexica [18], [216], [194], [15], [259], [187], [260], [261], [262], [112], [263], with some even heading for whole-word modeling [259], [25].”
• Recurrent Neural Aligner: An Encoder-Decoder Neural Network Model for Sequence to Sequence Mapping (2017, methodology): “Recurrent neural network models within encoder-decoder framework with attention mechanism have been commonly used for machine translation [4, 5] and speech recognition [6, 7, 8, 9, 10, 11].”
• A Comparison of Sequence-to-Sequence Models for Speech Recognition (2017, background): “However, examining the performance of these models on the voice-search test sets reveals that these systems are 13–35% worse than the baseline in this case (although we note that the gap is significantly less than has been reported in previous work, when decoding without an LM [6, 11]).”
• Towards Online End-to-end Transformer Automatic Speech Recognition (2019, background): “In recent years, various models have been studied, including connectionist temporal classification (CTC) [1–4], attention-based encoder–decoder models [5–9], their hybrid models [10, 11], and the RNN-transducer [12–14].”
• Token-wise Training for Attention Based End-to-end Speech Recognition (2019, background): “Attention based end-to-end (A-E2E) speech recognition systems map speech signals directly to tokens (characters/subwords/words) [1, 2, 3, 4, 5].”
• Improving Attention Based Sequence-to-Sequence Models for End-to-End English Conversational Speech Recognition (2018, background): “E nd -t oen d Attention Seq2Seq + Trigram [37] 25.”
• Deep Learning for Audio Signal Processing (2019, background): “systems that directly map the input audio signal to the target sequences [42]–[47].”
• Advances in Joint CTC-Attention Based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM (2017, methodology): “There are two major types of end-to-end architectures for ASR: attention-based methods use an attention mechanism to perform alignment between acoustic frames and recognized symbols [5, 6, 7, 8, 9], and Connectionist Temporal Classification (CTC), uses Markov assumptions to efficiently solve sequent”
• Improving Attention-Based End-to-End ASR Systems with Sequence-Based Loss Functions (2018, methodology): “Compare with other E2E speech recognition systems Systems SWB CH Attention Seq2Seq + Trigram[17] 25.”
• Advances in all-neural speech recognition (2016, methodology): “e need to quickly build systems in new languages without a dictionary, and kept the rest of a standard HMM system, in particular the use of a decision tree. In contrast to this, we follow recent work [22, 18, 23, 19] where a neural network learns context-dependence implicitly. Our approach is most s”
• Language Independent End-to-End Architecture For Joint Language and Speech Recognition (methodology): “There are two main streams for end-to-end ASR systems by using connectionist temporal classification (CTC) [5, 6, 7] and attention-based encoder-decoder approaches [8, 9, 10, 11].”
• Large Margin Training for Attention Based End-to-End Speech Recognition (2019, methodology): “The application of attention based end-to-end systems [1, 2, 3] to automatic speech recognition tasks [4, 5, 6, 7] has been actively investigated in the past few years.”
• Recent Advances in End-to-End Automatic Speech Recognition (2021, methodology): “The attention-based encoder-decoder (AED) model is another type of E2E ASR model [4, 6, 7, 32, 33].”
• Multitask Learning with Low-Level Auxiliary Tasks for Encoder-Decoder Based Speech Recognition (2017, methodology): “Our baseline model has better performance than the most similar previous encoder-decoder result [7].”
• Multi-Dialect Speech Recognition with a Single Sequence-to-Sequence Model (2017, background): “Sequence-to-sequence models provide a simple and elegant solution to the ASR problem by learning and optimizing a single neural network for the AM, PM and LM [15, 16, 17, 18, 19, 20].”