End-to-end attention-based large vocabulary speech recognition

By Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, Yoshua Bengio. 2015

1,207 citations recorded by the source corpus.

Many state-of-the-art Large Vocabulary Continuous Speech Recognition (LVCSR) Systems are hybrids of neural networks and Hidden Markov Models (HMMs). Recently, more direct end-to-end methods have been investigated, in which neural architectures were trained to model sequences of characters [1,2]. To our knowledge, all these approaches relied on Connectionist Temporal Classification [3] modules. We investigate an alternative method for sequence modelling based on an attention mechanism that allows a Recurrent Neural Network (RNN) to learn alignments between sequences of input frames and output labels. We show how this setup can be applied to LVCSR by integrating the decoding RNN with an n-gram language model and by speeding up its operation by constraining selections made by the attention mechanism and by reducing the source sequence lengths by pooling information over time. Recognition accuracies similar to other HMM-free RNN-based approaches are reported for the Wall Street Journal corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Speech & Audio

Citations in the corpus

Deep Learning Scaling is Predictable, Empirically (2017, background): “Consistent with prior work (Bahdanau et al. (2016); Battenberg et al. (2017)), larger attention models trained on larger data sets tend to be easier to optimize than DS2 models, whose generalization error tends away from the power-law trend on larger data sets.”
KERMIT: Generative Insertion-Based Modeling for Sequences (2019, background): “…many conditional generation applications, including machine translation (Bahdanau et al., 2015; Luong et al., 2015), speech recognition (Chan et al., 2016; Bahdanau et al., 2016), speech synthesis (Oord et al., 2016; Wang et al., 2017) and image captioning (Vinyals et al., 2015; Xu et al., 2015).”
Imputer: Sequence Modelling via Imputation and Dynamic Programming (2020, background): “Three popular types of generative models have been developed for end-to-end speech recognition applications: CTC (Graves et al., 2006; Graves & Jaitly, 2014), RNN-T (Graves, 2012; Graves et al., 2013) and seq2seq (Chan et al., 2016; Bahdanau et al., 2016a).”
Order Matters: Sequence to sequence for sets (2015, neutral): “…Bahdanau et al., 2015a), to image captioning (Vinyals et al., 2015c; Mao et al., 2015; Donahue et al., 2015), speech recognition (Chan et al., 2015; Bahdanau et al., 2015b), constituency parsing (Vinyals et al., 2015b) and learning to compute (Zaremba & Sutskever, 2014; Vinyals et al., 2015a).”
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition (2019, methodology): “Deep Learning has been applied successfully to Automatic Speech Recognition (ASR) [1], where the main focus of research has been designing better network architectures, for example, DNNs [2], CNNs [3], RNNs [4] and end-to-end models [5, 6, 7].”
Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin (2015, background): “, 2014), have been shown to be very effective on similar tasks (Bahdanau et al., 2015).”
Light Gated Recurrent Units for Speech Recognition (2018, background): “Attention models are based on an encoder-decoder architecture coupled with an attention mechanism [26] that decides which”
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition (2015, methodology): “In our experiments, we ﬁnd that these components are necessary for LAS to work well.”
State-of-the-Art Speech Recognition with Sequence-to-Sequence Models (2017, neutral): “There have been a variety of methods explored in the literature to address this issue in the context of sequence-to-sequence models [22, 23, 5, 12].”
Towards Better Decoding and Language Model Integration in Sequence to Sequence Models (2016, methodology): “Language Model Integration during decoding can be implemented by extending the beam search cost [12, 4, 15]:”
Neural Programmer: Inducing Latent Programs with Gradient Descent (2015, background): “…al., 2015; Xu et al., 2015), conversational modeling (Shang et al., 2015; Vinyals & Le, 2015), end-to-end Q&A (Sukhbaatar et al., 2015; Peng et al., 2015; Hermann et al., 2015), and end-to-end speech recognition (Graves & Jaitly, 2014; Hannun et al., 2014; Chan et al., 2015; Bahdanau et al., 2015).”
Professor Forcing: A New Algorithm for Training Recurrent Networks (2016, background): “Recurrent neural networks (RNNs) have become to be the generative models of choice for sequential data (Graves, 2012) with impressive results in language modeling (Mikolov, 2010; Mikolov and Zweig, 2012), speech recognition (Bahdanau et al., 2015; Chorowski et al., 2015), Machine Translation (Cho et”
Hierarchical Multiscale Recurrent Neural Networks (2016, methodology): “The most popular approach is to set the timescales as hyperparameters (El Hihi & Bengio, 1995; Koutník et al., 2014; Bahdanau et al., 2016) instead of treating them as dynamic variables that can be learned from the data (Schmidhuber, 1991; 1992; Chung et al., 2015; 2016).”
Regularizing Neural Networks by Penalizing Confident Output Distributions (2017, methodology): “The encoder of the network consisted of 4 bidirectional LSTM layers each having 256 units, interleaved with 3 time-subsampling layers, conﬁgured to drop every second frame (Bahdanau et al., 2016; Chan et al., 2015).”
Syllable-Based Sequence-to-Sequence Speech Recognition with the Transformer in Mandarin Chinese (2018, background): “Experts have shown significant interest in the area of sequenceto-sequence modeling with attention [1, 2, 3, 4] on ASR tasks in recent years.”
ESPnet: End-to-End Speech Processing Toolkit (2018, background): “• Attention-based: Attention-LVCSR [29], OpenNMT speech to text [30]”
Multilingual Language Processing From Bytes (2015, methodology): “Finally, recent work on Automatic Speech Recognition (ASR) uses a similar sequence-to-sequence LSTM framework to produce letter sequences directly from acoustic frame sequences (Chan et al., 2015; Bahdanau et al., 2015).”