Towards dropout training for convolutional neural networks

By Haibing Wu, Xiaodong Gu. 2015

333 citations recorded by the source corpus.

Recently, dropout has seen increasing use in deep learning. For deep convolutional neural networks, dropout is known to work well in fully-connected layers. However, its effect in convolutional and pooling layers is still not clear. This paper demonstrates that max-pooling dropout is equivalent to randomly picking activation based on a multinomial distribution at training time. In light of this insight, we advocate employing our proposed probabilistic weighted pooling, instead of commonly used max-pooling, to act as model averaging at test time. Empirical evidence validates the superiority of probabilistic weighted pooling. We also empirically show that the effect of convolutional dropout is not trivial, despite the dramatically reduced possibility of over-fitting due to the convolutional architecture. Elaborately designing dropout training simultaneously in max-pooling and fully-connected layers, we achieve state-of-the-art performance on MNIST, and very competitive results on CIFAR-10 and CIFAR-100, relative to other approaches without data augmentation. Finally, we compare max-pooling dropout and stochastic pooling, both of which introduce stochasticity based on multinomial distributions at pooling stage.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Improved Regularization of Convolutional Neural Networks with Cutout (2017, background): “Wu and Gu propose probabilistic weighted pooling [20], wherein activations in each pooling region are dropped with some probability.”
R-Drop: Regularized Dropout for Neural Networks (2021, background): “…cost as well as good compatibility with other regularization methods [46], which has been successfully applied to regularize a wide range of neural network architectures [51], e.g., convolutional neural network layers [68, 10], recurrent neural networks [17, 58, 43], Transformer [73, 79, 69].”
Deep Learning Advances in Computer Vision with 3D Data (2017, background): “Wu and Gu [2015] applied dropout to the input of the max-pooling layers of a CNN achieving enhanced performance, and showed experimentally that the simultaneous use of dropout in different layers can be beneﬁcial.”
Analysis on the Dropout Effect in Convolutional Neural Networks (2016, background): “Wu and Gu [10] proposed probabilistic weighted pooling which combines dropout in convolutional layers and max pooling together.”
Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks (2018, background): “, 2014), max pooling dropout (Wu and Gu, 2015) and batch normalization (Ioffe and Szegedy, 2015).”
Thin MobileNet: An Enhanced MobileNet Architecture (2019, methodology): “Dropout changes the variance of the layers output when the model is in testing phase after its training phase.”