Batch-normalized Maxout Network in Network

By Jia-Ren Chang, Yong-Sheng Chen. 2015

110 citations recorded by the source corpus.

This paper reports a novel deep architecture referred to as Maxout network In Network (MIN), which can enhance model discriminability and facilitate the process of information abstraction within the receptive field. The proposed network adopts the framework of the recently developed Network In Network structure, which slides a universal approximator, multilayer perceptron (MLP) with rectifier units, to exact features. Instead of MLP, we employ maxout MLP to learn a variety of piecewise linear activation functions and to mediate the problem of vanishing gradients that can occur when using rectifier units. Moreover, batch normalization is applied to reduce the saturation of maxout units by pre-conditioning the model and dropout is applied to prevent overfitting. Finally, average pooling is used in all pooling layers to regularize maxout MLP in order to facilitate information abstraction in every receptive field while tolerating the change of object position. Because average pooling preserves all features in the local patch, the proposed MIN model can enforce the suppression of irrelevant information during training. Our experiments demonstrated the state-of-the-art classification performance when the MIN model was applied to MNIST, CIFAR-10, and CIFAR-100 datasets and comparable performance for SVHN dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Optimization & Training

Citations in the corpus

Dynamic Routing Between Capsules (2017, neutral): “This model gives comparable results to deep convolutional networks (such as Chang and Chen [2015]).”
Convolutional Neural Fabrics (2016, methodology): “f 28 28 pixel images of the handwritten digits 0;:::;9. We use the standard split of the dataset into 50k training samples, 10k validation samples and 10k test samples. Pixel values are normalized to [0;1] by dividing them by 255. We augment the train data by randomly positioning the original image ”
Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning (2016, methodology): “We define the T j(xi) to be a random linear or non-linear transformation on the training sample xi before the j’th pass through the network.”
Striking the Right Balance With Uncertainty (2019, background): “[27] 99.7% Maxout NIN [6] 99.8% Baseline CNN [24] 97.1% CoSen CNN [24] 98.4% This paper 98.7% Table 3: Results for digit classiﬁcation on MNIST.”
Doing the Impossible: Why Neural Networks Can Be Trained at All (2018, background): “Recent neural networks have been able to accurately identify over 99.5% of the validation examples correctly [2].”
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review (2017, methodology): “-100 SV-HN DropConnect (Wan et al., 2013) Optimized DCNNs with dropped connections 18.7 9.32 - 1.94 Stochastic pooling (Zeiler & Fergus, 2013) Stochastic rather than deterministic pooling procedure 15.13 - 42.51 2.80 Maxout networks (Goodfellow et al., 2013) Maxout activation functions 11.68 9.38 38”
Oblivious Neural Network Predictions via MiniONN Transformations (2017, background): “Collectively they support the following activation functions: ReLU [39, 51, 57], leaky ReLU [32, 55], maxout [17, 43] and tanh [19].”
Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures (2016, background): “NIN Jia-Ren Chang (2015) 0.24% Max-out network (k=2) Goodfellow et al. (2013) 0.45% Network In Network Lin et al. (2013) 0.45% Deeply Supervised Network Lee et al. (2015) 0.39% RCNN-96 Liang & Hu (2015) 0.31% SimpleNet * 0.25% *Note that we didn’t intend on achieving the state of the art performance”
All you need is a good init (2015, background): “…of leaky ReLU ( Maas et al. (2013), with a large value of the negative slope 2 When preparing this submission we have found recent unreviewed paper MIN Chang & Chen (2015) paper, which uses a sophisticated combination of batch normalization, maxout and network-in-network nonlinearities and…”