Dropout: a simple way to prevent neural networks from overfitting

By Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov. 2014

43,754 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Deep Contextualized Word Representations (2018, methodology): “To add ELMo to the supervised model, we ﬁrst freeze the weights of the biLM and then Finally, we found it beneﬁcial to add a moderate amount of dropout to ELMo (Srivastava et al., 2014) and in some cases to regularize the ELMo weights by adding λ (cid:107) w to the loss.”
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference (2017, positive): “use Dropout (Srivastava et al., 2014) for regularization.”
Bidirectional Attention Flow for Machine Comprehension (2016, methodology): “A dropout (Srivastava et al., 2014) rate of 0.2 is used for the CNN, all LSTM layers, and the linear transformation before the softmax for the answers.”
A Decomposable Attention Model for Natural Language Inference (2016, methodology): “Dropout regularization (Srivastava et al., 2014) was used for all ReLU layers, but not for the final linear layer.”
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (2015, background): “techniques [13, 30, 10, 36], aggressive data augmentation [18, 14, 29, 33], and large-scale data [4, 26].”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, positive): “Dropout [Srivastava et al., 2014] is applied within the feed-forward network, on the skip connection, on the attention weights, and at the input and output of the entire stack.”
Semi-Supervised Classification with Graph Convolutional Networks (2016, background): “Stochasticity in the training process is introduced via dropout (Srivastava et al., 2014).”
Graph Attention Networks (2017, neutral): “Furthermore, dropout (Srivastava et al., 2014) with p = 0 .”
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, background): “Techniques that utilize random number generation, such as dropout (Srivastava et al., 2014), are a staple of modern deep learning training.”
Universal Transformers (2018, background): “We also add residual connections around each of these function blocks and apply dropout and layer normalization [25, 2] (see Fig.”
Adaptive Computation Time for Recurrent Neural Networks (2016, background): “Secondly, neural networks have been successfully trained under a wide range of adversarial regularisation constraints, including sparse internal states [23], stochastically masked units [28] and randomly perturbed weights [1].”
Distilling the Knowledge in a Neural Network (2015, methodology): “The cumbersome model could be an ensemble of separately trained models or a single very large model trained with a very strong regularizer such as dropout [9].”
Simple, Scalable Adaptation for Neural Machine Translation (2019, methodology): “We tune different values of dropout (Srivastava et al., 2014), depending on the dataset size”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “dropout (Srivastava et al., 2014)) depending on the dataset size for each language pair.”