Learned-Norm Pooling for Deep Feedforward and Recurrent Neural Networks

By Caglar Gulcehre, Kyunghyun Cho, Razvan Pascanu, Yoshua Bengio. 2013

184 citations recorded by the source corpus.

In this paper we propose and investigate a novel nonlinear unit, called L p unit, for deep neural networks. The proposed L p unit receives signals from several projections of a subset of units in the layer below and computes a normalized L p norm. We notice two interesting interpretations of the L p unit. First, the proposed unit can be understood as a generalization of a number of conventional pooling operators such as average, root-mean-square and max pooling widely used in, for instance, convolutional neural networks (CNN), HMAX models and neocognitrons. Furthermore, the L p unit is, to a certain degree, similar to the recently proposed maxout unit [13] which achieved the state-of-the-art object recognition results on a number of benchmark datasets. Secondly, we provide a geometrical interpretation of the activation function based on which we argue that the L p unit is more efficient at representing complex, nonlinear separating boundaries. Each L p unit defines a superelliptic boundary, with its exact shape defined by the order p. We claim that this makes it possible to model arbitrarily shaped, curved boundaries more efficiently by combining a few L p units of different orders. This insight justifies the need for learning different orders for each unit in the model. We empirically evaluate the proposed L p units on a number of datasets and show that multilayer perceptrons (MLP) consisting of the L p units achieve the state-of-the-art results on a number of benchmark datasets. Furthermore, we evaluate the proposed L p unit on the recently proposed deep recurrent neural networks (RNN).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling (2014, methodology): “This approach of comparing different types of hidden units in neural networks has been done before, for instance, by Gulcehre et al. [2014]. See Table 1 for the details of the model sizes.”
Striving for Simplicity: The All Convolutional Net (2014, methodology): “For a first step in that direction we study properties of our network using a deconvolutional approach similar to Zeiler & Fergus (2014).”
How to Construct Deep Recurrent Neural Networks (2013, methodology): “(*) We obtained these results using DOT(S)-RNN with Lp units in the deep transition, maxout units in the deep output function and dropout (Gulcehre et al., 2013).”
Recent advances in convolutional neural networks (2015, background): “The ReLU activation function is deﬁned as: where z i,j,k is the input of the activation function at location ( i, j ) on the k -th channel.”
Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree (2015, background): “Learning pooling functions is analogous to receptive ﬁeld learning [8, 11, 5, 15].”
Generalizing Pooling Functions in CNNs: Mixed, Gated, and Tree (2018, background): “Learning pooling functions is analogous to receptive field learning [17], [18], [19], [20].”
A review of convolutional neural network architectures and their optimizations (2022, background): “Thus, ReLU and its variants are superior to conventional activation functions such as sigmoid and tanh in overcoming the vanishing gradient problem (Gulcehre et al. 2014).”
Review of Image Classification Algorithms Based on Convolutional Neural Networks (2021, methodology): “GSoP-Net In 2019, Gao et al. [142] proposed Global Second-order Pooling Convolutional Networks (GSoP-Net), which introduced two-dimensional GAP into the middle part of CNNs.”
Using Neural Network Formalism to Solve Multiple-Instance Problems (2016, methodology): “Although the optimization of q is appealing, our experiments with the method of [13] did not yield any improvement.”
Learning Activation Functions to Improve Deep Neural Networks (2014, background): “Springenberg & Riedmiller (2013) replaced the max function with a probabilistic max function and Gulcehre et al. (2014) explored an activation function that replaces the max function with an L P norm.”
Speech Emotion Recognition Using Deep Convolutional Neural Network and Discriminant Temporal Pyramid Matching (2018, background): “In recent years, it has been found in [52], [53] that the p value has an important impact on the image classification accuracy.”
Mollifying Networks (2016, unknown): “15% accuracy (Gulcehre et al., 2014).”