Attention-Based Models for Speech Recognition
By Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, Yoshua Bengio. 2015
2,769 citations recorded by the source corpus.
Recurrent sequence generators conditioned on input data through an attention mechanism have recently shown very good performance on a range of tasks including machine translation, handwriting synthesis [1,2] and image caption generation [3]. We extend the attention-mechanism with features needed for speech recognition. We show that while an adaptation of the model used for machine translation in [2] reaches a competitive 18.7% phoneme error rate (PER) on the TIMET phoneme recognition task, it can only be applied to utterances which are roughly as long as the ones it was trained on. We offer a qualitative explanation of this failure and propose a novel and generic method of adding location-awareness to the attention mechanism to alleviate this issue. The new method yields a model that is robust to long inputs and achieves 18% PER in single utterances and 20% in 10-times longer (repeated) utterances. Finally, we propose a change to the attention mechanism that prevents it from concentrating too much on single frames, which further reduces PER to 17.6% level.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Speech & Audio
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Jianfeng Gao
- Quoc V. Le
- Percy Liang
- Kyunghyun Cho
- Luke S. Zettlemoyer
- Graham Neubig
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Ming-Hsuan Yang
- Andrew Zisserman
- Yi Yang
- Jitendra Malik
In Speech & Audio
Citations in the corpus
- Convolutional Sequence to Sequence Learning (2017, background): “Sequence to sequence learning has been successful in many tasks such as machine translation, speech recognition (Sutskever et al., 2014; Chorowski et al., 2015) and text summarization (Rush et al.”
- Structured Attention Networks (2017, background): “, 2015), speech recognition (Chorowski et al., 2015; Chan et al., 2015), question answering (Hermann et al.”
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks (2015, background): “terminology of neural networks with ‘attention’ [31] mechanisms, the RPN module tells the Fast R-CNNmodule where to look.”
- Deep Learning Scaling is Predictable, Empirically (2017, background): “Recent research has shifted from hand-engineered speech recognition pipelines over to end-to-end deep learning based methods that show promising results (Hannun et al. (2014); Chorowski et al. (2015); Amodei et al. (2016)).”
- Efficient Content-Based Sparse Attention with Routing Transformers (2020, background): “a fixed temporal context around the current prediction has also been explored in Chorowski et al. (2015), while ? dynamically segment the sequence into variable sized-chunks.”
- Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, background): “Attention shows its ability of modeling long range interactions in language modeling [78,83], speech recognition [20,10], and neural captioning [86].”
- Stand-Alone Self-Attention in Vision Models (2019, neutral): “We achieve this in two steps:”
- An Actor-Critic Algorithm for Sequence Prediction (2016, methodology): “The apostrophy and the asterisk identify results from (Bahdanau et al., 2015) and (Shen et al., 2015) To better understand the IWSLT 2014 results we provide the learning curves for the considered approaches in Figure 2.”
- Unsupervised Speech Recognition (2021, background): “Such models are often classified into two categories: hybrid (Juang et al., 1986; Young, 1996; Povey, 2005; Hinton et al., 2012; Abdel-Hamid et al., 2012; Bourlard and Morgan, 2012) or end-to-end (Graves et al., 2006; Graves, 2012; Chorowski et al., 2015; Rao et al., 2017).”
- Multitask Training with Text Data for End-to-End Speech Recognition (2020, background): “Attention-based end-to-end (E2E) speech recognition systems map audio features directly to text-level representations [1, 2, 3, 4, 5].”
- You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentation (2020, positive): “We used dynamic convolution attention [25] instead of location-sensitive attention [26].”
- Transformers with convolutional context for ASR (2019, background): “The success of sequence-to-sequence models in neural machine translation systems [7, 8] offered further simplification to ASR systems by integrating the acoustic and the language models into a single encoder-decoder architecture that is jointly optimized [9, 10].”
- Pay Less Attention with Lightweight and Dynamic Convolutions (2019, background): “Attention assigns context elements attention weights which define a weighted sum over context representations (Bahdanau et al., 2015; Sukhbaatar et al., 2015; Chorowski et al., 2015; Luong et al., 2015).”
- Wav2Letter++: A Fast Open-source Speech Recognition System (2018, methodology): “The CTC criterion does not have parameters whereas the ASG and S2S criteria both have parameters which can be learned.”
- Learning Filterbanks from Raw Speech for Phone Recognition (2017, neutral): “Features + Smooth Focus [18] MFSC + energy + ∆ + ∆∆ 15.”
- Listen, attend and spell: A neural network for large vocabulary conversational speech recognition (2015, methodology): “Unlike previous approaches, LAS does not make independence assumptions in the label sequence and it does not rely on HMMs. LAS is based on the sequence to sequence learning framework with attention [17, 18, 16, 14, 15].”
- Enhanced LSTM for Natural Language Inference (2016, background): “…based models have been shown to be effective in a wide range of tasks, including machine translation (Bahdanau et al., 2014), speech recognition (Chorowski et al., 2015; Chan et al., 2016), image caption (Xu et al., 2015), and text summarization (Rush et al., 2015; Chen et al., 2016), among…”
- Stacked Cross Attention for Image-Text Matching (2018, background): “and λ1 is the inversed temperature of the softmax function [5] (Eq.”