Maxout Networks

By Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron C. Courville, Yoshua Bengio. 2013

2,259 citations recorded by the source corpus.

We consider the problem of designing models to leverage a recently introduced approximate model averaging technique called dropout. We define a simple new model called maxout (so named because its output is the max of a set of inputs, and because it is a natural companion to dropout) designed to both facilitate optimization by dropout and improve the accuracy of dropout's fast approximate model averaging technique. We empirically verify that the model successfully accomplishes both of these tasks. We use maxout and dropout to demonstrate state of the art classification performance on four benchmark datasets: MNIST, CIFAR-10, CIFAR-100, and SVHN.

Subfields: Computer Networks and Communications, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Model Architectures

Citations in the corpus

Deep Residual Learning for Image Recognition (2015, methodology): “en shortcut connections are used, they are connected to the pairs of 3 3 layers (totally 3nshortcuts). On this dataset we use identity shortcuts in all cases (i.e., option A), method error (%) Maxout [10] 9.38 NIN [25] 8.81 DSN [24] 8.22 # layers # params FitNet [35] 19 2.5M 8.39 Highway [42,43] 19 ”
Neural Machine Translation by Jointly Learning to Align and Translate (2014, methodology): “In both cases, we use a multilayer network with a single maxout (Goodfellow et al. , 2013) hidden layer to compute the conditional probability of each target word (Pascanu et al. , 2014).”
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (2014, methodology): “The computation from the hidden state in the decoder to the output is implemented as a deep neural network (Pascanu et al., 2014) with a single intermediate layer having 500 maxout units each pooling 2 inputs (Goodfellow et al., 2013).”
Learned in Translation: Contextualized Word Vectors (2017, methodology): “…representation for all inputs. x pool = max( X | y ); mean( X | y ); min( X | y ); x self (15) We feed this joined representation through a three-layer, batch-normalized [Ioffe and Szegedy, 2015] maxout network [Goodfellow et al., 2013] to produce a probability distribution over possible classes.”
Object Detection Networks on Convolutional Feature Maps (2015, methodology): “NoC families: MLPs of various depths, ConvNets of various depths, and ConvNets with maxout [16] for latent scale selection, where the latter two are unex-”
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (2015, background): “techniques [13, 30, 10, 36], aggressive data augmentation [18, 14, 29, 33], and large-scale data [4, 26].”
FitNets: Hints for Thin Deep Nets (2014, methodology): “To validate our approach, we trained a teacher network of maxout convolutional layers as reported in Goodfellow et al. (2013b) and used the same FitNet architecture as in CIFAR-10.”
Deeply-Supervised Nets (2014, methodology): “For a fair comparison and clear illustration of the effectiveness of DSN, we match the complexity of our model to the complexity of the network architectures used in [17] and [9] so that there are a comparable number of parameters.”
On the Number of Linear Regions of Deep Neural Networks (2014, methodology): “Artificial neural networks with several hidden layers, called deep neural networks, have become popular due to their unprecedented success in a variety of machine learning tasks (see, e.g., Krizhevsky et al. 2012, Ciresan et al. 2012, Goodfellow et al. 2013, Hinton et al. 2012).”
Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods (2013, positive): “This architecture was loosely based on [14].”
Pylearn2: a machine learning research library (2013, methodology): “Pylearn2 also contains models that were developed not just at LISA but originally developed using Pylearn2, such as spike-and-slab sparse coding with parallel variational inference [22] and maxout units for neural nets [21].”
Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation (2013, background): “…condition for exact gradient-based training methods to work well, recent successes of deep networks with rectiﬁers and other “non-smooth” non-linearities (Glorot et al. , 2011; Krizhevsky et al. , 2012a; Goodfellow et al. , 2013) clearly question that belief: see Section 2 for a deeper discussion.”
Wide Residual Networks (2016, background): “In very deep residual networks that should help deal with diminishing feature reuse problem enforcing learning in different residual blocks.”
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks (2016, neutral): “Test Error Maxout [6] 11.68% Network in Network [17] 10.41% Deeply Supervised [16] 9.6% ConvPool-CNN-C [26] 9.31% ALL-CNN-C [26] 9.08% our CNN, mean-only B.N. 8.52% our CNN, weight norm.”
Deep Networks with Stochastic Depth (2016, methodology): “Many follow up works of Dropout have been empirically successful, such as DropConnect [20] and Maxout [21].”
Instance-Aware Semantic Segmentation via Multi-task Network Cascades (2015, methodology): “ill be used for stage 2. During training, the forward/backward propagated signals of stages 2 and 3 only go through the “pathways” determined by these 300 boxes. NMS is similar to max pooling, maxout [11], or other local competing layers [28], which are implemented as routers of forward/backward pat”