Improving deep neural networks for LVCSR using rectified linear units and dropout

By George E. Dahl, Tara N. Sainath, Geoffrey E. Hinton. 2013

1,431 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Optimization & Training, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Optimization & Training

In Signal Processing

Citations in the corpus

Neural Autoregressive Distribution Estimation (2016, background): “Deep neural networks (Bengio, 2009; LeCun et al., 2015) are at the core of state-of-the-art models for supervised tasks like image recognition (Krizhevsky et al., 2012) and speech recognition (Dahl et al., 2013).”
Stochastic Optimization of Plain Convolutional Neural Networks with Simple Methods (2020, background): “George Dahl et al. [19] used a deep neural network with rectified linear activation functions and dropout to achieve state-of-the-art results on a standard speech recognition task [20].”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, background): “The ReLU activation function is the identity for positive arguments and zero otherwise.”
Learning to Extract Coherent Summary via Deep Reinforcement Learning (2018, background): “ReLU is the nonlinear function proposed by (Dahl, Sainath, and Hinton 2013).”
Learning to Answer Questions from Image Using Convolutional Neural Network (2015, background): “…2014; Klein et al., 2015; Karpathy et al., 2014a; Ordonez et al., 2011) to automatic image captioning (Chen et al., 2014; Karpathy et al., 2014b; Donahue et al., 2014; Fang et al., 2014; Kiros et al., 2014a; Kiros et al., 2014b; Klein et al., 2015; Mao et al., 2014a; Mao et al., 2014b; Vinyals…”
Convolutional Neural Network Architectures for Matching Natural Language Sentences (2014, methodology): “We use ReLu [7] as the activation function for all of models (convolution and MLP), which yields comparable or better results to sigmoid-like functions, but converges faster.”
Dropout Improves Recurrent Neural Networks for Handwriting Recognition (2013, background): “It is shown to be effective with deep convolutional networks [12], [13], [14], feed-forward networks [15], [16], [17] but, to the best of our knowledge, has never been applied to RNNs.”
Deep learning in neural networks: An overview (2014, background): “, 2011), and helped to obtain best results on several benchmark problems across multiple domains (e.g., Krizhevsky et al., 2012; Dahl et al., 2013).”
Automated Machine Learning: State-of-The-Art and Open Challenges (2019, background): “Bayesian optimization has received huge attention from the machine learning community in tuning deep neural networks for different tasks including classification tasks [113, 112], speech recognition [24] and natural language processing [81].”
Deep Learning: Methods and Applications (2014, methodology): “Applications of the same “dropout” techniques are also successful for some speech recognition tasks [65, 81].”
SPOT-Disorder2: Improved Protein Intrinsic Disorder Prediction by Ensembled Deep Learning (2019, background): “Each FC layer’s size is denoted as NFC, is activated by a rectified linear unit (ReLU) [33] and regularized by dropout.”
Initializing Bayesian Hyperparameter Optimization via Meta-Learning (2015, methodology): “When manually optimizing hyperparameters for D N +1 , they would begin the search with hyperparameter configurations that were optimal for the most similar previous datasets (see, e.g., Dahl, Sainath, and Hinton (2013); Goodfellow et al. (2013)).”
Multimodal Convolutional Neural Networks for Matching Image and Sentence (2015, methodology): “• Image CNN: The image CNN is used to generate the image representation for matching the fragments composed from words, which is computed as follows: where σ ( · ) is the activation function (e.g., Sigmoid or ReLU [4]).”
On the Computational Efficiency of Training Neural Networks (2014, background): “…algorithmic advancements, as well as increasing computational power and data size, has led to a breakthrough in the effectiveness of neural networks, and they have been used to obtain very impressive practical performance on a variety of domains (a few recent examples include [17, 16, 24, 10, 7]).”
First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNs (2014, background): “Recent work shows that with rectiﬁer nonlinearities DNNs can perform well in DNN-HMM systems without suffering from vanishing gradient problems during optimization (Dahl et al., 2013; Zeiler et al., 2013; Maas et al., 2013).”
Fixed-point feedforward deep neural network design using weights +1, 0, and −1 (2014, methodology): “We also examine the performance of fixed-point DNNs with dropout [10], [11], which is a recently developed strong regularization technique.”
ℓ1-regularized Neural Networks are Improperly Learnable in Polynomial Time (2015, background): “Examples of recent work in this area include [16, 15, 22, 7, 9, 11].”
Web-scale training for face identification (2014, background): “The ReLU(a) = max(0, a) nonlinearity [4] is applied after every layer (except for F8) and optimization is done through stochastic gradient descent and standard back-propagation [17, 15], minimizing the cross-entropy loss.”