Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree

By Chen-Yu Lee, Patrick W. Gallagher, Zhuowen Tu. 2015

519 citations recorded by the source corpus.

We seek to improve deep neural networks by generalizing the pooling operations that play a central role in current architectures. We pursue a careful exploration of approaches to allow pooling to learn and to adapt to complex and variable patterns. The two primary directions lie in (1) learning a pooling function via (two strategies of) combining of max and average pooling, and (2) learning a pooling function in the form of a tree-structured fusion of pooling filters that are themselves learned. In our experiments every generalized pooling operation we explore improves performance when used in place of average or max pooling. We experimentally demonstrate that the proposed pooling operations provide a boost in invariance properties relative to conventional pooling and set the state of the art on several widely adopted benchmark datasets; they are also easy to implement, and can be applied within various deep neural network architectures. These benefits come with only a light increase in computational overhead during training and a very modest increase in the number of model parameters.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Making Convolutional Networks Shift-Invariant Again (2019, methodology): “Interestingly, Lee et al. (2016) does not explore low-pass ﬁlters, but does propose to softly gate be-tween max and average pooling.”
Temporal Ensembling for Semi-Supervised Learning (2016, methodology): “Given that in a separate experiment our network matched the best published result for non-augmented SVHN when extra data is used (1.69% from Lee et al. (2015)), this gap is quite surprising, and leads us to conclude that fractional max pooling leads to a powerful augmentation of the dataset, well…”
EMNIST: an extension of MNIST to handwritten letters (2017, background): “7% [6]–[10], a classiﬁcation accuracy at which the dataset labeling can be called into question.”
Efficient Architecture Search by Network Transformation (2017, neutral): “d architectures. Model Depth Params Test Error Rate (%) Maxout [8] - - 9.38 Network in Network [17] - - 8.81 Highway Network [27] - - 7.72 All-CNN [26] - - 7.25 ResNet [10] 110 1.7M 6.61 Tree+Max-Avg [16] - - 6.05 Wide ResNet [34] 16 11.0M 4.81 DenseNet [11] 190 25.6M 3.46 MetaQNN [4] 8 11.18M 6.92 ”
Designing Neural Network Architectures using Reinforcement Learning (2016, methodology): “31 (Lee et al., 2016) Table 5: Prediction Error for the top MetaQNN (CIFAR-10) model trained for other tasks.”
Convolutional Neural Fabrics (2016, background): “[17] generalize the max and average pooling operators by computing both max and average pooling, and then fusing the result in a possibly data-driven manner.”
Group Equivariant Convolutional Networks (2016, neutral): “…7.04 1.22M ResNet44 Z2 9.45 5.61 2.64M p4m 6.46 4.94 2.62M To the best of our knowledge, thep4m-CNN outperforms all published results on plain CIFAR10 (Wan et al., 2013; Goodfellow et al., 2013; Lin et al., 2014; Lee et al., 2015b; Srivastava et al., 2015; Clevert et al., 2015; Lee et al., 2015a).”
Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units (2016, neutral): “We report the test set error rates Model CIFAR-10 CIFAR-100 (Rippel et al., 2015) 8.60 31.60 (Snoek et al., 2015) 6.37 27.40 (Liang & Hu, 2015) 7.09 31.75 (Lee et al., 2016) 6.05 32.37 (Srivastava et al., 2015) 7”
SkipNet: Learning Dynamic Routing in Convolutional Networks (2017, methodology): “We follow the common data augmentation scheme (mirroring/shifting) that is adopted for CIFAR and ImageNet datasets [7, 19, 20, 32].”
Large-Margin Softmax Loss for Convolutional Neural Networks (2016, background): “Error Rate FitNet (Romero et al., 2015) 35.04 NiN (Lin et al., 2014) 35.68 Maxout (Goodfellow et al., 2013) 38.57 DSN (Lee et al., 2015) 34.57 dasNet (Stollenga et al., 2014) 33.78 All-CNN (Springenberg et al., 2015) 33.71 R-CNN (Liang & Hu, 2015) 31.75 GenPool (Lee et al., 2016) 32.37 accuracy over”
Reinforcement Learning for Architecture Search by Network Transformation (2017, neutral): “Speciﬁcally, the number of ﬁlters is chosen from [6 , 8 , · · · , 32] and the number of units is chosen from [16 , 18 , · · · , 42] .”