Improving neural networks by preventing co-adaptation of feature detectors
By Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov. 2012
8,038 citations recorded by the source corpus.
When a large feedforward neural network is trained on a small training set, it typically performs poorly on held-out test data. This "overfitting" is greatly reduced by randomly omitting half of the feature detectors on each training case. This prevents complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors. Instead, each neuron learns to detect a feature that is generally helpful for producing the correct answer given the combinatorially large variety of internal contexts in which it must operate. Random "dropout" gives big improvements on many benchmark tasks and sets new records for speech and object recognition.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Quoc V. Le
- Jianfeng Gao
- Sergey Levine
- Percy Liang
- Kyunghyun Cho
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Model Architectures
- Yoshua Bengio
- Quoc V. Le
- Kyunghyun Cho
- Kaiming He
- Ruslan Salakhutdinov
- Alan L. Yuille
- Aaron C. Courville
- Dacheng Tao
In Optimization & Training
- Yoshua Bengio
- Quoc V. Le
- Frank Hutter
- Aaron C. Courville
- Kyunghyun Cho
- Roger Baker Grosse
- Stefano Soatto
- Ruslan Salakhutdinov
Citations in the corpus
- Deep Residual Learning for Image Recognition (2015, methodology): “Strong regularization such as maxout [9] or dropout [13] is applied to obtain the best results ([9, 25, 24, 34]) on this dataset.”
- Adam: A Method for Stochastic Optimization (2014, methodology): “Objectives may also have other sources of noise than data subsampling, such as dropout (Hinton et al., 2012b) regularization.”
- Generative Adversarial Nets (2014, methodology): “Dropout [16] was applied in training the discriminator net.”
- Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “We apply dropout (Hinton et al., 2012) to the word embeddings and outputs of each Bi-LSTM.”
- How transferable are features in deep neural networks? (2014, methodology): “As in previous studies, including Krizhevsky et al. (2012), we use dropout (Hinton et al., 2012) on fully connected layers except for the softmax output layer.”
- Object Detection Networks on Convolutional Feature Maps (2015, methodology): “d divide by 10 twice after each 50k iterations. When the NoC weights are randomly initialized, we use the scaled Gaussian distribution [14]. All fc layers (expect the last 21-d fc) are with a dropout [15] ratio of 50%. Running Time. A prominent behavior of the NoC is that it benefits considerably fro”
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (2015, background): “techniques [13, 30, 10, 36], aggressive data augmentation [18, 14, 29, 33], and large-scale data [4, 26].”
- Deeply-Supervised Nets (2014, background): “The application of deep learning (DL) techniques to large training sets has yielded significant performance gains in image classification [11, 15] and speech recognition [4].”
- Going deeper with convolutions (2014, methodology): “For larger datasets such as Imagenet, the recent trend has been to increase the number of layers [12] and layer size [21, 14], while using dropout [7] to address the problem of overfitting.”
- ImageNet Large Scale Visual Recognition Challenge (2014, methodology): “They trained a large, deep convolutional neural network on RGB values, with 60 million parameters using an efficient GPU implementation and a novel hidden-unit dropout trick (Krizhevsky et al., 2012; Hinton et al., 2012).”
- OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks (2013, background): “DropOut [11] with a rate of 0 .”
- Network In Network (2013, background): “Furthermore, the fully connected layers are prone to overfitting and heavily depend on dropout regularization [4] [5], while global average pooling is itself a structural regularizer, which natively prevents overfitting for the overall structure.”
- Maxout Networks (2013, methodology): “Dropout applied to multilayer perceptrons and deep convolutional networks has improved the state of the art on tasks ranging from audio classification to very large scale object recognition (Hinton et al., 2012; Krizhevsky et al., 2012).”
- Pushing Stochastic Gradient towards Second-Order Methods -- Backpropagation Learning with Transformations in Nonlinearities (2013, background): “As hyperparameter selection and regularization are usually nuisance in practical use of neural networks, it would be interesting to see whether combining dropouts [4] and our transformations can provide a robust framework enabling training of robust neural networks in reasonable time.”
- Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods (2013, methodology): “We used a similar architecture to (Hinton et al., 2012).”
- Pylearn2: a machine learning research library (2013, methodology): “Other costs implemented in the library include dropout [27] , contrastive divergence [26], noise contrastive estimation [24], score matching [28], denoising s core matching [52], softmax log likelihood for classification with MLPs, and Gaussian log likelihood fo r regression with MLPs.”