Deep Speech: Scaling up end-to-end speech recognition
By Awni Y. Hannun, Carl Case, Jared Casper, Bryan Catanzaro, G. Diamos, Erich Elsen, Ryan J. Prenger, Sanjeev Satheesh, Shubho Sengupta, Vinay Rao and 2 more. 2014
2,263 citations recorded by the source corpus.
We present a state-of-the-art speech recognition system developed using end-to-end deep learning. Our architecture is significantly simpler than traditional speech systems, which rely on laboriously engineered processing pipelines; these traditional systems also tend to perform poorly when used in noisy environments. In contrast, our system does not need hand-designed components to model background noise, reverberation, or speaker variation, but instead directly learns a function that is robust to such effects. We do not need a phoneme dictionary, nor even the concept of a "phoneme." Key to our approach is a well-optimized RNN training system that uses multiple GPUs, as well as a set of novel data synthesis techniques that allow us to efficiently obtain a large amount of varied data for training. Our system, called Deep Speech, outperforms previously published results on the widely studied Switchboard Hub5'00, achieving 16.0% error on the full test set. Deep Speech also handles challenging noisy environments better than widely used, state-of-the-art commercial speech systems.
Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Quoc V. Le
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Kyunghyun Cho
- Sergey Levine
In Natural Language Processing
- Kyunghyun Cho
- Jianfeng Gao
- Mohit Bansal
- Yoshua Bengio
- Graham Neubig
- Quoc V. Le
- Richard Socher
- Luke S. Zettlemoyer
In Speech & Audio
- Jinyu Li
- Shinji Watanabe
- Yu Zhang
- Tara N. Sainath
- Wei-Ning Hsu
- Dong Yu
- Gabriel Synnaeve
- Abdel-rahman Mohamed
Citations in the corpus
- Unsupervised Data Augmentation for Consistency Training (2019, background): “In recent years, there have been significant advancements on the design of data augmentations for NLP [12], vision [10, 11] and speech [13, 14] in supervised settings.”
- A Simple Method for Commonsense Reasoning (2018, background): “Although deep neural networks have achieved remarkable successes (e.g., [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]), their dependence on supervised learning has been challenged as a significant weakness.”
- Deep Learning Scaling is Predictable, Empirically (2017, background): “Recent research has shifted from hand-engineered speech recognition pipelines over to end-to-end deep learning based methods that show promising results (Hannun et al. (2014); Chorowski et al. (2015); Amodei et al. (2016)).”
- Mixed Precision Training (2017, methodology): “For example, the speech recognition model used in Hannun et al. (2014) was trained on 5,000 hours of data, more recent acoustic models trained on nearly 12,000 hours (Amodei et al., 2016) and even larger data sets like 125,000 hours used in Soltau et al. (2016).”
- Beyond human-level accuracy: computational challenges in deep learning (2019, background): “Baidu’s prior work uses speech recognition datasets of multiple terabytes [15].”
- Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, methodology): “A common approach is to increase the source side data, for instance by adding modifications or sampling from generative models (Hannun et al., 2014; Jia & Liang, 2016; Krizhevsky et al., 2017; Antoniou et al., 2017; Yasunaga et al., 2018; Yu et al., 2018a; Lee et al., 2019; Berthelot et al., 2019; X”
- Randaugment: Practical automated data augmentation with a reduced search space (2019, methodology): “Data augmentation is a widely used method for generating additional data to improve machine learning systems, for image classification [43, 23, 7, 54], object detection [13], instance segmentation [10], and speech recognition [21, 16, 36].”
- Local Aggregation for Unsupervised Learning of Visual Embeddings (2019, background): “Deep convolutional neural networks (DCNNs) have achieved great success on many tasks across a variety of domains, such as vision [41, 64, 26, 25, 8], audition [29, 23, 12, 51], and natural language processing [73, 32, 10, 42].”
- Common Voice: A Massively-Multilingual Speech Corpus (2019, background): “0 — an open-source implementation of a variation of Baidu’s first DeepSpeech paper (Hannun et al., 2014).”
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition (2019, methodology): “Noisy audio has been synthesised via superimposing clean audio with a noisy audio signal in [12].”
- Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin (2015, methodology): “We not only find networks that produce much better predictions than those in previous work [26], but also find instances of recurrent models that can be deployed in a production setting with no significant loss in accuracy.”
- Transformers with convolutional context for ASR (2019, background): “Advances in alignment-free sequence-level loss functions like CTC and ASG [3, 4] enabled easier training with letters as output units [5, 6].”
- Wav2Letter++: A Fast Open-source Speech Recognition System (2018, methodology): “To minimize wait time between processes and improve the efficiency of a single process, we sort the dataset on input length prior to constructing batches for training [13].”
- Listen, attend and spell: A neural network for large vocabulary conversational speech recognition (2015, background): “There has recently been an explosion in methods for end-to-end trained speech models because of their inherent simplicity compared to current speech recognition systems [2, 24, 6, 25, 14].”
- Wav2Letter: an End-to-End ConvNet-based Speech Recognition System (2016, methodology): “We observe that we compare very well against Deep Speech 1 & 2 which were trained with much more data [8, 1].”
- On Calibration of Modern Neural Networks (2017, methodology): “As a result, neural networks are now entrusted with making complex decisions in applications, such as object detection (Girshick, 2015), speech recognition (Hannun et al., 2014), and medical diagnosis (Caruana et al.”
- Towards Better Decoding and Language Model Integration in Sequence to Sequence Models (2016, background): “Language Model Integration during decoding can be implemented by extending the beam search cost [12, 4, 15]:”
- Neural Programmer: Inducing Latent Programs with Gradient Descent (2015, background): “…al., 2015; Xu et al., 2015), conversational modeling (Shang et al., 2015; Vinyals & Le, 2015), end-to-end Q&A (Sukhbaatar et al., 2015; Peng et al., 2015; Hermann et al., 2015), and end-to-end speech recognition (Graves & Jaitly, 2014; Hannun et al., 2014; Chan et al., 2015; Bahdanau et al., 2015).”
- End-to-end attention-based large vocabulary speech recognition (2015, methodology): “In this work, we investigate the application of an Attentionbased Recurrent Sequence Generator (ARSG) as a part of an end-to-end LVCSR system.”