Stochastic Pooling for Regularization of Deep Convolutional Neural Networks

By Matthew D. Zeiler, Rob Fergus. 2013

1,037 citations recorded by the source corpus.

We introduce a simple and effective method for regularizing large convolutional neural networks. We replace the conventional deterministic pooling operations with a stochastic procedure, randomly picking the activation within each pooling region according to a multinomial distribution, given by the activities within the pooling region. The approach is hyper-parameter free and can be combined with other regularization approaches, such as dropout and data augmentation. We achieve state-of-the-art performance on four image datasets, relative to other approaches that do not utilize data augmentation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Maxout Networks (2013, methodology): “We applied local contrast normalization preprocessing the same way as Zeiler & Fergus (2013).”
Striving for Simplicity: The All Convolutional Net (2014, methodology): “…techniques for improving class inference (Stollenga et al., 2014; Srivastava & Salakhutdinov, 2013) as well as procedures for improved regularization (Zeiler & Fergus, 2013; Springenberg & Riedmiller, 2013; Wan et al., 2013) and layer-wise pre-training using label information (Lee et al., 2014).”
Random Erasing Data Augmentation (2017, background): “Various regularization methods have been proposed [14, 23, 1, 29, 26].”
Do Deep Nets Really Need to be Deep? (2013, neutral): “6% (no augmentation) + dropout on lc CNN[21] 64c-p-64c-p-128c-p-fc ∼56k ∼120k 15.”
Fractional Max-Pooling (2014, background): “• Stochastic pooling, where the act of picking the maximum value in each pooling region is replaced by a form of size-biased sampling [13].”
Compressing Neural Networks with the Hashing Trick (2015, background): “The incorporation of pooling layers (Zeiler & Fergus, 2013) can reduce the number of connections between layers in domains exhibiting locality among input features, such as images.”
Recent advances in convolutional neural networks (2015, background): “Compared with ReLU, Leaky ReLU compresses the negative part rather than mapping it to constant zero, which makes it allow for a small, non-zero gradient when the unit is not active.”
Deep Learning: Methods and Applications (2014, methodology): “[437], Zeiler and Fergus [435, 436], and Zeiler ([434]).”
Robust Convolutional Neural Networks under Adversarial Noise (2015, methodology): “To clarify, we adopted the artiﬁcial noise distribution in order to improve the robustness of CNNs and it is unrelated to natural image statistics.”
Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree (2015, background): “In the current deep learning literature, popular pooling functions include max, average, and stochastic pooling [3, 2, 40].”
Convolutional Kernel Networks (2014, neutral): “Method [12] [27] [18] [13] [4] [17] [31] CKN-GM CKN-PM CKN-CO CIFAR-10 82.”
Deep Networks with Internal Selective Attention through Feedback Connections (2014, neutral): “tep. At the ﬁrst step, the classiﬁcation is ‘dog’, and the cat could indeed be mistaken for a puppy. Note that in the ﬁrst step, 7 Method CIFAR-10 CIFAR-100 Dropconnect [9] 9.32% - Stochastic Pooling [60] 15.13% - Multi-column CNN [7] 11.21% - Maxout [10] 9.38% 38.57% Maxout (our model) 9.61% 34.54%”
APAC: Augmented PAttern Classification with Neural Networks (2015, background): “[24] and the CNN with dropout in ﬁnal hidden units (15.6”