Improving Deep Neural Networks with Probabilistic Maxout Units

By Jost Tobias Springenberg, Martin A. Riedmiller. 2013

102 citations recorded by the source corpus.

We present a probabilistic variant of the recently introduced maxout unit. The success of deep neural networks utilizing maxout can partly be attributed to favorable performance under dropout, when compared to rectified linear units. It however also depends on the fact that each maxout unit performs a pooling operation over a group of linear transformations and is thus partially invariant to changes in its input. Starting from this observation we ask the question: Can the desirable properties of maxout units be preserved while improving their invariance properties ? We argue that our probabilistic maxout (probout) units successfully achieve this balance. We quantitatively verify this claim and report classification performance matching or exceeding the current state of the art on three challenging image classification benchmarks (CIFAR-10, CIFAR-100 and SVHN).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Recent advances in convolutional neural networks (2015, background): “7) Probout: [42] proposes a probabilistic variant of max-out called probout.”
Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree (2015, neutral): “Maxout [35] - 11.”
APAC: Augmented PAttern Classification with Neural Networks (2015, neutral): “[19], Maxout (9.35”
Contractive Rectifier Networks for Nonlinear Maximum Margin Classification (2015, unknown): “85% Probablistic Maxout Network [35] 38.”
HD-CNN: Hierarchical Deep Convolutional Neural Networks for Large Scale Visual Recognition (2014, background): “Recently, there has been considerable interest in enhancing CNN components, including pooling layers [36], activation units [11, 28], and nonlinear layers [21].”
Recurrent convolutional neural network for object recognition (2015, unknown): “68 Prob maxout [47] >5 M 11.”
Deep convolutional neural networks for brain image analysis on magnetic resonance imaging: a review (2017, background): “ne major drawback of this technique is that the number of weights to be learned in each layer is increased by a factor of K. A workaround to this situation was proposed by Springenberg and Reidmiller [62] in which a probabilisticsamplingprocedureto computethemaximumacrossfeaturemaps,generalisingmaxo”
Striving for Simplicity: The All Convolutional Net (2014, methodology): “…techniques for improving class inference (Stollenga et al., 2014; Srivastava & Salakhutdinov, 2013) as well as procedures for improved regularization (Zeiler & Fergus, 2013; Springenberg & Riedmiller, 2013; Wan et al., 2013) and layer-wise pre-training using label information (Lee et al., 2014).”
DisturbLabel: Regularizing CNN on the Loss Layer (2016, background): “[33] explores the performance of discriminatively-trained CNNs on the noisy data, where there are some freely available labels for each image which may or may not be accurate.”
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review (2017, unknown): “-100 SV-HN DropConnect (Wan et al., 2013) Optimized DCNNs with dropped connections 18.7 9.32 - 1.94 Stochastic pooling (Zeiler & Fergus, 2013) Stochastic rather than deterministic pooling procedure 15.13 - 42.51 2.80 Maxout networks (Goodfellow et al., 2013) Maxout activation functions 11.68 9.38 38”
A survey on modern trainable activation functions (2020, background): “In (Springenberg and Riedmiller, 2014) the authors describe Probout, a stochastic generalization of the Maxout unit trying to improve its invariance replacing the maximum operation in Maxout with a probabilistic sampling procedure, i.”
Generalizing Pooling Functions in CNNs: Mixed, Gated, and Tree (2018, unknown): “Maxout [37] 11.”