Listen, attend and spell: A neural network for large vocabulary conversational speech recognition

By William Chan, Navdeep Jaitly, Quoc V. Le, Oriol Vinyals. 2015

2,461 citations recorded by the source corpus.

We present Listen, Attend and Spell (LAS), a neural speech recognizer that transcribes speech utterances directly to characters without pronunciation models, HMMs or other components of traditional speech recognizers. In LAS, the neural network architecture subsumes the acoustic, pronunciation and language models making it not only an end-to-end trained system but an end-to-end model. In contrast to DNN-HMM, CTC and most other models, LAS makes no independence assumptions about the probability distribution of the output character sequences given the acoustic sequence. Our system has two components: a listener and a speller. The listener is a pyramidal recurrent network encoder that accepts filter bank spectra as inputs. The speller is an attention-based recurrent network decoder that emits each character conditioned on all previous characters, and the entire acoustic sequence. On a Google voice search task, LAS achieves a WER of 14.1% without a dictionary or an external language model and 10.3% with language model rescoring over the top 32 beams. In comparison, the state-of-the-art CLDNN-HMM model achieves a WER of 8.0% on the same set.

Subfields: Artificial Intelligence, Natural Language Processing, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Signal Processing

Citations in the corpus

Structured Attention Networks (2017, background): “…machine translation (Bahdanau et al., 2015; Luong et al., 2015), image captioning (Xu et al., 2015), speech recognition (Chorowski et al., 2015; Chan et al., 2015), question answering (Hermann et al., 2015; Sukhbaatar et al., 2015), and algorithm-learning (Graves et al., 2014; Vinyals et al.,…”
KERMIT: Generative Insertion-Based Modeling for Sequences (2019, background): “…applied to many conditional generation applications, including machine translation (Bahdanau et al., 2015; Luong et al., 2015), speech recognition (Chan et al., 2016; Bahdanau et al., 2016), speech synthesis (Oord et al., 2016; Wang et al., 2017) and image captioning (Vinyals et al., 2015; Xu et…”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “For years, the fields have been continuously reporting new state of the art results using varieties of model architectures for computer vision tasks [57, 58, 7], for natural language understanding tasks [59, 60, 61], for speech recognition and synthesis tasks [62, 63, 64, 65, 66].”
Imputer: Sequence Modelling via Imputation and Dynamic Programming (2020, background): “Three popular types of generative models have been developed for end-to-end speech recognition applications: CTC (Graves et al., 2006; Graves & Jaitly, 2014), RNN-T (Graves, 2012; Graves et al., 2013) and seq2seq (Chan et al., 2016; Bahdanau et al., 2016a).”
RWTH ASR Systems for LibriSpeech: Hybrid vs Attention - w/o Data Augmentation (2019, background): “Recent investigations have shown promising results by applying the same approach for speech recognition [5,7,30–33].”
SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network (2021, neutral): “End-to-end speech recognition models [1, 2, 3] have seen remarkable success in recent years [4].”
Multitask Training with Text Data for End-to-End Speech Recognition (2020, background): “Various model architectures [6, 7, 8, 9, 10] and training schemes [11, 12, 13] were proposed.”
Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, methodology): “T-Md”) and RNN-based models (“B-Base”) on all the MuST-C language directions.”
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition (2019, methodology): “The three networks LAS-4-1024, LAS-6-1024 and LAS-6-1280 are trained on LibriSpeech 960h with a combination of augmentation policies (None, LB, LD) and training schedules (B/D).”
Multilingual Speech Recognition with a Single End-to-End Model (2017, methodology): “This downsampling enables us to use a simpler encoder architecture than [17].”
Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin (2015, neutral): “, 2015) or graphemes (Bahdanau et al., 2015; Chan et al., 2015).”
Improved Noisy Student Training for Automatic Speech Recognition (2020, methodology): “We introduce a coverage penalty term [32] in the fusion score with parameter c for LAS models [33], while we use a constant non-blank reward ρ to shift the logits produced by the LM for RNN-T [34] networks [35, 36].”
Improving Transformer-based Speech Recognition Using Unsupervised Pre-training (2019, background): “e masking pattern is generated every time a sequence is fed to the model. One unique characteristic of sequence-to-sequence with attention ASR models is it usually applies downsampling in the encoder [18]. Previous research shows temporal pooling encourages effective encoding in different temporal r”
Transformers with convolutional context for ASR (2019, background): “The success of sequence-to-sequence models in neural machine translation systems [7, 8] offered further simpliﬁcation to ASR systems by integrating the acoustic and the language models into a single encoder-decoder architecture that is jointly optimized [9, 10].”
Enhanced LSTM for Natural Language Inference (2016, background): “…based models have been shown to be effective in a wide range of tasks, including machine translation (Bahdanau et al., 2014), speech recognition (Chorowski et al., 2015; Chan et al., 2016), image caption (Xu et al., 2015), and text summarization (Rush et al., 2015; Chen et al., 2016), among others.”
Hierarchical Multiscale Recurrent Neural Networks (2016, methodology): “This model, called a hierarchical multiscale recurrent neural network (HM-RNN), does not assign ﬁxed update rates, but adaptively determines proper update times corresponding to different abstraction levels of the layers.”