Acoustic Modeling Using Deep Belief Networks

By Abdel-rahman Mohamed, George E. Dahl, Geoffrey E. Hinton. 2012

1,824 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dong Yu
2. Yoshua Bengio
3. Li Deng
4. Jinyu Li
5. Quoc V. Le
6. Jianfeng Gao
7. Mohit Bansal
8. Kyunghyun Cho

In Natural Language Processing

1. Li Deng
2. Yoshua Bengio
3. Dong Yu
4. Kyunghyun Cho
5. Jianfeng Gao
6. Quoc V. Le
7. Mohit Bansal
8. Jinyu Li

In Signal Processing

1. Dong Yu
2. Jinyu Li
3. Li Deng
4. Shinji Watanabe
5. Yoshua Bengio
6. Tara N. Sainath
7. Yifan Gong
8. Deliang Wang

Citations in the corpus

• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (2013, background): “At the same time, recent results have obtained excellent performance starting from carefullyscaled random initializations, though interestingly, pretrained initializations still exhibit faster convergence [21, 13, 22, 3, 4, 1, 23] (see Supplementary Appendix D for discussion).”
• Improving neural networks by preventing co-adaptation of feature detectors (2012, methodology): “Recently, deep, pre-trained, feedforward neural networks that map a short sequence of frames into a probability distribution over HMM states have been shown to outperform traditional Gaussian mixture models on both TIMIT [12] and a variety of more realistic large vocabulary tasks [6, 10].”
• Speech recognition with deep recurrent neural networks (2013, background): “three main conclusions are that LSTM works much better than tanh for this architecture, that bidirectional LSTM has a slight advantage over unidirectional and that depth is more important than layer size (which supports previous findings for deep networks [3]).”
• Gaussian Error Linear Units (GELUs) (2016, methodology): “The system is a five-layer, 2048-neuron wide classifier as in (Mohamed et al., 2012) with 39 output phone labels and a dropout rate of 0.5 as in (Srivas-tava, 2013).”
• Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin (2015, methodology): “More recently DNNs have become a fixture in the ASR pipeline with almost all state of the art speech work containing some form of deep neural network (Mohamed et al., 2011; Hinton et al., 2012; Dahl et al., 2011; N. Jaitly & Vanhoucke, 2012; Seide et al., 2011).”
• Representation Learning: A Review and New Perspectives (2012, background): “For that reason, much of the actual effort in deploying machine learning algorithms goes into the design of preprocessing pipelines and data transformations that result in a representation of the data that can support effective machine learning.”
• Wav2Letter: an End-to-End ConvNet-based Speech Recognition System (2016, background): “The improvements brought by deep neural networks (DNNs) [14, 10] and convolutional neural networks (CNNs) [24, 25] for acoustic modeling only extend this training pipeline.”
• Deep Speech: Scaling up end-to-end speech recognition (2014, methodology): “The introduction of deep learning algorithms [27, 30, 15, 18, 9] has improved speech system performance, usually by improving acoustic models.”
• Do Deep Nets Really Need to be Deep? (2013, methodology): “Deep learning was first successfully applied to speech recognition in [9].”
• Towards Better Analysis of Deep Convolutional Neural Networks (2016, methodology): “ring. 1 INTRODUCTION Deep convolutional neural networks (CNNs) have demonstrated significant improvements over traditional approaches in many pattern recognition tasks [31], such as speech recognition [42, 43], image classification [17, 30], and video classification [27, 58]. More recently, deep CNNs h”
• Deep Learning: Methods and Applications (2014, background): “The collaborative work started in phone recognition tasks [89, 100, 135, 136, 257, 260, 258, 309, 311, 334], demonstrating the power of hybrid DNN architectures discussed in Section 5 and of subsequent new architectures with convolutional and recurrent structure.”
• Pre-training Graph Neural Networks with Kernels (2018, neutral): “Concerning sequences, earlier approaches to pre-training for sequences did not take into account temporal dependencies (e.g., Mohamed et al. [2012]; Boulanger-Lewandowski et al. [2012]) only involving input-to-hidden connections, i.e. assuming each item belonging to a sequence as independent of the…”
• DeepMPC: Learning Deep Latent Features for Model Predictive Control (2015, background): “In the recent past, feature learning methods have proven effective for learning latent task-specific features across many domains [3, 19, 24, 9, 28].”
• Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition (2014, methodology): “While DNNs have shown state of the art performance in both phone recognition and large vocabulary speech recognition [9, 10, 11], the application of LSTM networks has been limited to phone recognition on the TIMIT database, and it has required using additional techniques and models such as CTC and R”
• Theoretical Insights Into the Optimization Landscape of Over-Parameterized Shallow Neural Networks (2017, background): “RELATED WORK As mentioned earlier, neural networks have enjoyed great empirical success [15], [26], [35].”
• Deep learning for detecting robotic grasps (2013, neutral): “Recent methods based on deep learning [1] have demonstrated state-of-the-art performance in a wide variety of tasks, including visual recognition [35, 60], audio recognition [39, 41], and natural language processing [12].”