Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups

By Geoffrey E. Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew W. Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath and 1 more. 2012

3,180 citations recorded by the source corpus.

Most current speech recognition systems use hidden Markov models (HMMs) to deal with the temporal variability of speech and Gaussian mixture models (GMMs) to determine how well each state of each HMM fits a frame or a short window of frames of coefficients that represents the acoustic input. An alternative way to evaluate the fit is to use a feed-forward neural network that takes several frames of coefficients as input and produces posterior probabilities over HMM states as output. Deep neural networks (DNNs) that have many hidden layers and are trained using new methods have been shown to outperform GMMs on a variety of speech recognition benchmarks, sometimes by a large margin. This article provides an overview of this progress and represents the shared views of four research groups that have had recent successes in using DNNs for acoustic modeling in speech recognition.

Subfields: Artificial Intelligence, Natural Language Processing, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Signal Processing

Citations in the corpus

Adam: A Method for Stochastic Optimization (2014, methodology): “Objectives may also have other sources of noise than data subsampling, such as dropout (Hinton et al., 2012b) regularization.”
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, background): “This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015).”
Layer Normalization (2016, methodology): “Deep neural networks trained with some version of Stochastic Gradient Descent have been shown to substantially outperform previous approaches on various supervised learning tasks in computer vision [Krizhevsky et al., 2012] and speech processing [Hinton et al., 2012].”
Intriguing properties of neural networks (2013, background): “Deep neural networks are powerful learning models that achieve excellent performance on visual and speech recognition problems [9, 8].”
A Simple Method for Commonsense Reasoning (2018, background): “Although deep neural networks have achieved remarkable successes (e.g., [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]), their dependence on supervised learning has been challenged as a signiﬁcant weakness.”
Matching Networks for One Shot Learning (2016, background): “Deep learning has made major advances in areas such as speech [7], vision [13] and language [16], but is notorious for requiring large datasets.”
Distilling the Knowledge in a Neural Network (2015, background): “State-of-the-art ASR systems currently use DNNs to map a (short) temporal context of features derived from the waveform to a probability distribution over the discrete states of a Hidden Markov Model (HMM) [4].”
A Deep Relevance Matching Model for Ad-hoc Retrieval (2016, background): “Recently, deep models have been applied to a variety of applications in computer vision [16], speech recognition [10] and NLP [25, 17], and have yielded significant performance improvements.”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “For years, the fields have been continuously reporting new state of the art results using varieties of model architectures for computer vision tasks [57, 58, 7], for natural language understanding tasks [59, 60, 61], for speech recognition and synthesis tasks [62, 63, 64, 65, 66].”
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset (2016, background): “Such agents can have tremendous value for consumers because they can power personal assistants such as Cortana [3], Siri [6], Alexa [1], or Google Assistant [4] found on phones or headless devices like Amazon Echo [2], all of which have been facilitated by recent advances in deep speech recognition ”
Representation Learning with Contrastive Predictive Coding (2018, methodology): “These techniques made manually speciﬁed features largely redundant and have greatly improved state-of-the-art in several real-world applications [1, 2, 3].”
TensorFlow: A system for large-scale machine learning (2016, neutral): “In recent years, machine learning has driven advances in many different ﬁelds [3, 5, 23, 24, 30, 27, 40, 45, 48, 50, 55, 68, 69, 73, 76].”
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour (2017, background): “We are in an unprecedented era in AI research history in which the increasing data and model scale is rapidly improving accuracy in computer vision [22, 40, 33, 34, 35, 16], speech [17, 39], and natural language processing [7, 37].”
Deep Perm-Set Net: Learn to predict sets with unknown permutation and cardinality using deep neural networks (2018, background): “…(RNN) neural networks have enjoyed great success in many real-world problems, including scene classification Krizhevsky et al., semantic segmentation Papandreou et al. (2015), speech recognition Hinton et al. (2012), gaming Mnih et al. (2013; 2015), and image captioning Johnson et al. (2016).”
Parallel WaveNet: Fast High-Fidelity Speech Synthesis (2017, background): “Recent successes of deep learning go beyond achieving state-of-the-art results in research benchmarks, and push the frontiers in some of the most challenging real world applications such as speech recognition [10], image recognition [16, 25], and machine translation [29].”
VAIN: Attentional Multi-agent Predictive Modeling (2017, neutral): “rtant step towards machine intelligence. Deep Neural Networks (DNNs) have had much success in machine perception e.g. Computer Vision [1, 2, 3], Natural Language Processing [4] and Speech Recognition [5, 6]. These problems usually have temporal and/or spatial structure, which makes them amenable to ”