Fractional Max-Pooling

By Benjamin Graham. 2014

531 citations recorded by the source corpus.

Convolutional networks almost always incorporate some form of spatial pooling, and very often it is max-pooling with = 2. Max-pooling act on the hidden layers of the network, reducing their size by an integer multiplicative factor . The amazing by product of discarding 75% of your data is that you build into the network a degree of invariance with respect to translations and elastic distortions. However, if you simply alternate convolutional layers with max-pooling layers, performance is limited due to the rapid reduction in spatial size, and the disjoint nature of the pooling regions. We have formulated a fractional version of max-pooling where is allowed to take non-integer values. Our version of maxpooling is stochastic as there are lots of different ways of constructing suitable pooling regions. We find that our form of fractional max-pooling reduces overfitting on a variety of datasets: for instance, we improve on the state of the art for CIFAR-100 without even using dropout.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Wide Residual Networks (2016, neutral): “…# params time,s CIFAR-10 B(1,3,1) 40 1.4M 85.8 6.06 B(3,1) 40 1.2M 67.5 5.78 B(1,3) 40 1.3M 72.2 6.42 B(3,1,1) 40 1.3M 82.2 5.86 B(3,3) 28 1.5M 67.5 5.73 B(3,1,3) 22 1.1M 59.9 5.78 Table 2: Test error (%, median over 5 runs) on CIFAR-10 of residual networks with k = 2 and different block types.”
Deep Networks with Stochastic Depth (2016, background): “05%) [29] on CIFAR-10 prior to our submission, without resorting to massive data augmentation [6, 24].”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, background): “…compared to following recent successful CNN architectures: AlexNet (Krizhevsky et al., 2012), DSN (Lee et al., 2015), NiN (Lin et al., 2013), Maxout (Goodfellow et al., 2013), All-CNN (Springenberg et al., 2014), Highway Network (Srivastava et al., 2015) and Fractional Max-Pooling (Graham, 2014).”
Identity Mappings in Deep Residual Networks (2016, neutral): “Better results of [13,14] have been reported using stronger data augmentation and ensembling.”
Striving for Simplicity: The All Convolutional Net (2014, methodology): “It is only outperformed by the fractional max-pooling approach (Graham, 2015) which uses a much larger network (on the order of 50 M parameters).”
Deep Learning with Differential Privacy (2016, neutral): “5% [25].”
Temporal Ensembling for Semi-Supervised Learning (2016, background): “ is difﬁcult due to several reasons. They provide results only for a fairly extreme set of augmentations (translations, ﬂipping, rotations, stretching, and shearing) on top of fractional max pooling (Graham, 2014), which introduces random, local stretching inside the network, and is known to improve”
Grid Long Short-Term Memory (2015, methodology): “ Error (%) Wan et al. [38] 0.28 Graham [11] 0.31 Untied 3-LSTM 0.32 Ciresan et al. [5] 0.35 Untied 3-LSTM with ReLU (*) 0.36 Mairar et al. [29] 0.39 Lee et al. [27] 0.39 Simard et al. [32] 0.4 Graham [10] 0.44 Goodfellow et al. [9] 0.45 Visin et al. [37] 0.45 Lin et al. [28] 0.47 Figure 12: Test err”
A MultiPath Network for Object Detection (2016, methodology): “To maximize accuracy prior to submitting to the COCO leaderboard, we added validation data to training, employed horizontal flip and fractional max pooling [12] at inference, and ensembled 6 models.”
Stacked What-Where Auto-encoders (2015, background): “43% Fractional Max-pooling with large augmentation (Graham (2014)) 95.”
Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning (2016, methodology): “For this dataset, we create 10 labeled sets.”
All you need is a good init (2015, methodology): “For the FitNet-1 architecture, we have not experienced any difﬁculties training the network with any of the activation functions (ReLU, maxout, tanh), optimizers (SGD, RMSProp) or initialization (Xavier, MSRA, Ortho, LSUV), unlike the uniform initialization used in Romero et al. (2015).”
Communication-Efficient Learning of Deep Networks from Decentralized Data (2016, methodology): “5% [19] for CIFAR; nevertheless, the standard model we use is sufficient for our needs, as our goal is to evaluate our optimization method, not achieve the best possible accuracy on this task.”
Federated Learning with Non-IID Data (2018, background): “Note that the SGD accuracy reported in this paper are not state-of-the-art [6, 30, 31, 1] but the CNNs we train are sufficient for our goal to evaluate federated learning on non-IID data.”
Verifying Neural Networks with Mixed Integer Programming (2017, neutral): “In particular, within computer vision, they represent the state-of-the-art for classification of images [8, 17] and object localization in images [15].”
Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules (2019, background): “A key consideration when applying data augmentation is picking a good set of augmentation functions, since redundant or overly aggressive augmentation can slow down training and introduce biases into the dataset (Graham, 2014).”