Improving Neural Networks with Dropout

By Nitish Srivastava. 2013

314 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Network In Network (2013, background): “Test Error Stochastic Pooling [11] 2.80% Rectiﬁer + Dropout [18] 2.78% Rectiﬁer + Dropout + Synthetic Translation [18] 2.68% Conv. maxout + Dropout [8] 2.47% NIN + Dropout 2.35% Multi-digit Number Recognition [19] 2.16% DropConnect [15] 1.94% test error rate of 2.35%.”
Gaussian Error Linear Units (GELUs) (2016, neutral): “5 as in (Srivastava, 2013).”
Unsupervised Domain Adaptation by Backpropagation (2014, methodology): “The chosen architectures are fairly standard for these datasets in terms of feature extractor and label predictor parts: the “MNIST” design is inspired by the classical LeNet-5 [12], while the second CNN is adopted from [19].”
Recurrent Neural Network Regularization (2014, methodology): “Unfortunately, dropout (Srivastava, 2013), the most powerful regularization method for feed-forward neural networks, does not work well with RNNs.”
BinaryConnect: Training Deep Neural Networks with binary weights during propagations (2015, background): “They suggest that the stochastic version of BinaryConnect can be considered a regularizer, although a slightly less powerful one than Dropout, in this context.”
An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks (2013, background): “Dropout (Hinton et al., 2012; Srivastava, 2013) is a recently introduced training algorithm for neural networks.”
Understanding the Role of Training Regimes in Continual Learning (2020, background): “Dropout [25] is a well-established technique in deep learning, which is also well-studied theoretically [3, 67, 24, 17, 70].”
Dropout as an Implicit Gating Mechanism For Continual Learning (2020, background): “Dropout [10] is a well-established technique in deep learning, which is also well-studied theoretically [2, 24, 9, 25].”
Binarized Neural Networks (2016, methodology): “We regularize the model with Dropout (Srivastava, 2013; Srivastava et al., 2014).”
A Kronecker-factored approximate Fisher matrix for convolution layers (2016, methodology): “While cuda-convnet provides some better-performing architectures, we could not use these, since these included locally connected layers, which KFC can’t handle.) For SVHN, we used the architecture of Srivastava (2013). This architecture consists of three convolutional layers followed by three fully ”
Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors (2016, methodology): “For the regression task we found that our model over-all achieves better RMSE and predictive log-likelihoods than VI (Graves, 2011), PBP and Dropout uncertainty.”
Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networks (2013, background): “When the regularized network is running in the inference mode, dropout has been observed to h ave a sparsifying effect on the hidden activations [17].”
Avoiding pathologies in very deep networks (2014, methodology): “Dropout is a method for regularizing neural networks (Hinton et al., 2012; Srivastava, 2013).”
Change Detection in Synthetic Aperture Radar Images Based on Deep Neural Networks (2016, background): “to improve the performance of the networks [27]–[29].”
The Pitfalls of Simplicity Bias in Neural Networks (2020, methodology): “4 Effect of `2 Regularization and Dropout In this section, we use SGD-trained FCNs trained on LMS-7 data to examine the extent to which Dropout [63] and `2 regularization alters the extreme nature of SB.”
Improving Deep Neural Networks with Probabilistic Maxout Units (2013, unknown): “N ET + DROPOUT [26] 2.68 % 5 × C ONV .”