End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results

By Jan Chorowski, Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio. 2014

488 citations recorded by the source corpus.

We replace the Hidden Markov Model (HMM) which is traditionally used in in continuous speech recognition with a bi-directional recurrent neural network encoder coupled to a recurrent neural network decoder that directly emits a stream of phonemes. The alignment between the input and output sequences is established using an attention mechanism: the decoder emits each symbol based on a context created with a subset of input symbols elected by the attention mechanism. We report initial results demonstrating that this new approach achieves phoneme error rates that are comparable to the state-of-the-art HMM-based decoders, on the TIMIT dataset.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Yoshua Bengio
3. Mohit Bansal
4. Quoc V. Le
5. Dong Yu
6. Shinji Watanabe
7. Graham Neubig
8. Jianfeng Gao

In Speech & Audio

1. Jinyu Li
2. Shinji Watanabe
3. Yu Zhang
4. Takaaki Hori
5. Dong Yu
6. Wei-Ning Hsu
7. Xu Tan
8. James R. Glass

In Signal Processing

1. Jinyu Li
2. Shinji Watanabe
3. Dong Yu
4. Yu Zhang
5. Yoshua Bengio
6. Takaaki Hori
7. Xu Tan
8. Tara N. Sainath

Citations in the corpus

• Effective Approaches to Attention-based Neural Machine Translation (2015, background): “…image objects and agent actions in the dynamic control problem (Mnih et al., 2014), between speech frames and text in the speech recognition task (Chorowski et al., 2014), or between visual features of a picture and its text description in the image caption generation task (Xu et al., 2015).”
• Semi-supervised Sequence Learning (2015, methodology): “[31], which has been successfully used f or machine translation [20, 10], text parsing [32], image captioning [ 34], video analysis [30], speech recognition [4, 3] and conversational modeling [27, 33].”
• Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin (2015, methodology): “The RNN encoder-decoder with attention performs well in predicting phonemes (Chorowski et al., 2015) or graphemes (Bahdanau et al.”
• Listen, attend and spell: A neural network for large vocabulary conversational speech recognition (2015, methodology): “At each step the RNN uses its internal state to guide an attention mechanism [10, 11, 12] to compute a “context” vector from the high level features of the listener.”
• Attention-Based Models for Speech Recognition (2015, methodology): “The (1)An early version of this work was presented at the NIPS 2014 Deep Learning Workshop [5].”
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks (2015, methodology): “A model such as the attention model of [26] which predicts phone sequences directly, instead of the highly redundant HMM state sequences, would not suffer from this problem because it would need to exploit both the acoustic signal and the language model sufficiently to make predictions.”
• End-to-end attention-based large vocabulary speech recognition (2015, background): “Systems of this type have been successfully applied to a variety of tasks including machine translation [6], caption generation [7], handwriting synthesis [8], visual object classification [9] and phoneme recognition [10, 11].”
• Reward Augmented Maximum Likelihood for Neural Structured Prediction (2016, methodology): “Finally, we note that previous work [6, 7] suggests several refinements to improve sequence to sequence models on TIMIT by adding noise to the weights and using more focused forward-moving attention mechanism.”
• ESPnet: End-to-End Speech Processing Toolkit (2018, methodology): “1Language modeling is often performed by external language model toolkits, for example SRILM [6] network [13, 14, 15, 16].”
• EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding (2015, background): “Meanwhile, another line of work [6, 7, 8, 9, 10, 11, 12] has focused on end-to-end ASR, i.”
• ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs (2015, methodology): “Chorowski et al. (2014) and Chorowski et al. (2015) use attention in RNN for speech recognition.”
• GRAM: Graph-based Attention Model for Healthcare Representation Learning (2016, methodology): “The attention mechanism is a general framework for neural network learning (Bahdanau et al., 2014), and has been since used in many areas such as speech recognition (Chorowski et al., 2014), computer vision (Ba et al., 2014; Xu et al., 2015) and healthcare (Choi et al., 2016b).”
• Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems (2017, background): “Two main types of end-to-end models for speech recognition have been proposed in the literature: connectionist temporal classification (CTC) [1, 2] and sequence-to-sequence learning (seq2seq) [3, 4].”
• Joint CTC-attention based end-to-end speech recognition using multi-task learning (2016, background): “End-to-end speech recognition is a recently proposed approach that directly transcribes speech to text without requiring predefined alignment between acoustic frames and characters [1, 2, 3, 4, 5, 6, 7, 8, 9].”
• Listening while speaking: Speech chain by deep learning (2017, background): “With this newly emerging approach to sequence-to-sequence mapping tasks, a model with a common architecture can directly learn the mapping between variable-length representations of different modalities: text-to-text sequences [21, 22], speech-to-text sequences [23, 24], text-to-speech sequences [25”