Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin

By Dario Amodei, S. Ananthanarayanan, Rishita Anubhai, Jin Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski and 31 more. 2015

3,177 citations recorded by the source corpus.

We show that an end-to-end deep learning approach can be used to recognize either English or Mandarin Chinese speech-two vastly different languages. Because it replaces entire pipelines of hand-engineered components with neural networks, end-to-end learning allows us to handle a diverse variety of speech including noisy environments, accents and different languages. Key to our approach is our application of HPC techniques, enabling experiments that previously took weeks to now run in days. This allows us to iterate more quickly to identify superior architectures and algorithms. As a result, in several cases, our system is competitive with the transcription of human workers when benchmarked on standard datasets. Finally, using a technique called Batch Dispatch with GPUs in the data center, we show that our system can be inexpensively deployed in an online setting, delivering low latency when serving users at scale.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Layer Normalization (2016, methodology): “Batch normalization has been previously extended to recurrent neural networks [Laurent et al., 2015, Amodei et al., 2015, Cooijmans et al., 2016].”
Deep Learning Scaling is Predictable, Empirically (2017, methodology): “In speech recognition, Amodei et al. (2016) show word error rate improvement for a Deep Speech 2 model on varying sizes of training data.”
Mixed Precision Training (2017, methodology): “For example, the speech recognition model used in Hannun et al. (2014) was trained on 5,000 hours of data, more recent acoustic models trained on nearly 12,000 hours (Amodei et al., 2016) and even larger data sets like 125,000 hours used in Soltau et al. (2016).”
Beyond human-level accuracy: computational challenges in deep learning (2019, neutral): “, [6, 7, 36]).”
Unbounded cache model for online language modeling with open vocabulary (2017, background): “Language models are a core component of many natural language processing applications such as machine translation [3], speech recognition [2] or dialogue agents [50].”
End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures (2019, neutral): “Nonetheless, there have been increasingly competitive end-to-end results trained with CTC (Graves & Jaitly, 2014; Amodei et al., 2016), ASG (Collobert et al.”
VAIN: Attentional Multi-agent Predictive Modeling (2017, neutral): “Deep Neural Networks (DNNs) have had much success in machine perception e.g. Computer Vision [1, 2, 3], Natural Language Processing [4] and Speech Recognition [5, 6].”
Generative Language Modeling for Automated Theorem Proving (2020, background): “Artiﬁcial neural networks have enjoyed a spectacularly successful decade, having made considerable advances in computer vision [1, 2], translation [3, 4, 5], speech recognition [6, 7], image generation [8, 9, 10, 11, 12], game playing [13, 14, 15], and robotics [16, 17].”
Robust Speech Recognition via Large-Scale Weak Supervision (2022, background): “In 2015, Deep Speech 2 (Amodei et al., 2015) reported a speech recognition system matched human-level performance when transcribing the LibriSpeech test-clean split.”
A Convergence Theory for Deep Learning via Over-Parameterization (2018, background): “Neural networks have demonstrated a great success in numerous machine-learning tasks [6, 25, 30, 33, 36, 46, 47].”
The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage (2021, methodology): “For example, in 2018, Baidu tried to release publicly the dataset used to develop Deep Speech [14] as a resource to accelerate speech recognition research, similar to ImageNet for speech.”
Unsupervised Speech Recognition (2021, methodology): “Model Unlabeled LM dev test data clean other clean other 960h - Supervised learning DeepSpeech 2 (Amodei et al., 2016) 5-gram - - 5.”
Rethinking Evaluation in ASR: Are Our Models Robust Enough? (2020, background): “Finally, while DeepSpeech 2 [7] did not focus their study on transfer, we train a single acoustic model on multiple datasets at once, as they did.”
Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition (2020, background): “The papers corresponding to each data point have been collected in the bibliography [6, 8, 9, 13–21].”
TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation (2018, methodology): “The fully end-to-end architecture used in this study is similar to the Deep Speech 2 neural ASR system proposed by Baidu in [1].”
Learning Robust and Multilingual Speech Representations (2020, methodology): “As most of the dataset are small, we trained DeepSpeech2 small models with the Adam optimizer with a fixed learning rate of 0.0002 and applied gradient clipping with maximum norm 25.0, using the same configuration for all languages.”