Adaptive dropout for training deep neural networks

By Jimmy Ba, Brendan J. Frey. 2013

397 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Gaussian Error Linear Units (GELUs) (2016, methodology): “It relates to stochastic regularizers in that it is the expectation of a modification to Adaptive Dropout (Ba & Frey, 2013).”
Random Erasing Data Augmentation (2017, background): “Various regularization methods have been proposed [14, 23, 1, 29, 26].”
Deep learning in neural networks: An overview (2014, background): “Dropout (Ba & Frey, 2013; Hinton, Srivastava, Krizhevsky, Sutskever, & Salakhutdinov, 2012) removes units from NNs during training to improve generalization.”
Recent advances in convolutional neural networks (2015, methodology): “following subsection, we introduce two effective regularization techniques: Dropout [16], [49], [50] and DropConnect [28].”
Deep Learning: Methods and Applications (2014, methodology): “The first is a powerful regularization technique called “dropout”; see details in [166] and a series of further analysis and improvement in [10, 13, 240, 381, 385].”
Variational Dropout and the Local Reparameterization Trick (2015, background): “In Ba and Frey (2013), standout was introduced, a variation of dropout where a binary belief network is learned for producing dropout rates.”
Concrete Dropout (2017, methodology): “Current methods include the optimisation of ↵ in Gaussian dropout following its variational interpretation [23], and overlaying a binary belief network to optimise the dropout probabilities as a function of the inputs [2].”
Discrete Variational Autoencoders (2016, background): “Nevertheless, Equation 17 of the VAE can be understood in analogy to dropout (Srivastava et al., 2014) or standout (Ba & Frey, 2013) regularization.”
Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networks (2013, methodology): “The adaptive dropout method [1] can furth e decrease the number of active neurons without degrading the performance of the network.”
Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Ordering (2017, background): “For example, instead of learning a weighting across layers at each depth, a probability of applying each module could be learned in a manner similar to adaptive dropout (Ba and Frey, 2013; Li et al., 2016) or a sparsely-gated mixture of experts (Shazeer et al., 2017).”
A Bayesian encourages dropout (2014, background): “The success of dropout attracts many researchers’ attention and several theoretical studies have been performed aiming for better understanding of dropout (Wager et al., 2013; Baldi & Sadowski, 2013; Ba & Frey, 2013).”
Curriculum Dropout (2017, background): “Ba and Frey [14] introduce a binary belief network which is overlaid on a neural network to selectively suppress hidden units.”
Ensemble Classification and Regression-Recent Developments, Applications and Future Directions [Review Article] (2016, methodology): “In [169], the author proposed ‘adaptive dropout’ where a stand out network was used to adaptively learn the probability of each node.”
Generalized Dropout (2016, background): “Standout [9] is a version of Dropout where the dropout rate depends on the output activations of a layer.”
Altitude Training: Strong Bounds for Single-Layer Dropout (2014, background): “Dropout is most commonly used for regularizing deep neural networks [2, 3, 4, 5], but it has also been found to improve the performance of logistic regression and other single-layer models for natural language tasks such as document classiﬁcation and named entity recognition [6, 7, 8].”
Deep learning for visual understanding: A review (2016, methodology): “This method was further improved in [40 – 45].”
Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization (2017, background): “For example, Ba et al. [3] proposed adaptive dropout, where dropout rate is determined by another neural network dynamically.”