Striving for Simplicity: The All Convolutional Net
By Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, Martin A. Riedmiller. 2014
5,049 citations recorded by the source corpus.
Most modern convolutional neural networks (CNNs) used for object recognition are built using the same principles: Alternating convolution and max-pooling layers followed by a small number of fully connected layers. We re-evaluate the state of the art for object recognition from small images with convolutional networks, questioning the necessity of different components in the pipeline. We find that max-pooling can simply be replaced by a convolutional layer with increased stride without loss in accuracy on several image recognition benchmarks. Following this finding -- and building on other recent work for finding simple network structures -- we propose a new architecture that consists solely of convolutional layers and yields competitive or state of the art performance on several object recognition datasets (CIFAR-10, CIFAR-100, ImageNet). To analyze the network we introduce a new variant of the "deconvolution approach" for visualizing features learned by CNNs, which can be applied to a broader range of network structures than existing approaches.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Quoc V. Le
- Sergey Levine
- Percy Liang
- Jianfeng Gao
- Ruslan Salakhutdinov
- Trevor Darrell
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Bernt Schiele
- Andrea Vedaldi
In Natural Language Processing
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Yejin Choi
- Graham Neubig
- Percy Liang
- Kyunghyun Cho
- Christopher D. Manning
In Safety, Robustness & Interpretability
- Wojciech Samek
- Been Kim
- K. Müller
- Nicholas Carlini
- Lars Kai Hansen
- Grgoire Montavon
- Quanshi Zhang
- Song-Chun Zhu
Citations in the corpus
- Neural Autoregressive Distribution Estimation (2016, background): “In fact, recent research suggests that these types of layers are not essential to obtain state-of-the-art results (Springenberg et al., 2015).”
- Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks (2016, methodology): “The model we are using is based on the ConvPool-CNN-C architecture of [26], with some small modifications: we replace the first dropout layer by a layer that adds Gaussian noise, we expand the last hidden layer from 10 units to 192 units, and we use 2 × 2 max-pooling, rather than 3 × 3 .”
- Deep Networks with Stochastic Depth (2016, methodology): “[29] on CIFAR-10 prior to our submission, without resorting to massive data augmentation [6, 24].”
- V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation (2016, methodology): “This strategy serves a similar purpose as pooling layers which, motivated by [20] and other works discouraging the use of maxpooling operations in CNNs, have been replaced in our approach by convolutional ones.”
- Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, methodology): “…compared to following recent successful CNN architectures: AlexNet (Krizhevsky et al., 2012), DSN (Lee et al., 2015), NiN (Lin et al., 2013), Maxout (Goodfellow et al., 2013), All-CNN (Springenberg et al., 2014), Highway Network (Srivastava et al., 2015) and Fractional Max-Pooling (Graham, 2014).”
- Identity Mappings in Deep Residual Networks (2016, neutral): “Better results of [13,14] have been reported using stronger data augmentation and ensembling.”
- Scattering Networks for Hybrid Representation Learning (2018, neutral): “Scat + WRN (ours) 93.1 Highway network [51] 92.4 All-CNN [50] 92.8 WRN 16 - 8 [59] 95.”
- Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks (2015, methodology): “They are similar to the models from Rasmus et al. (2015) who, in turn, derived them from the best models found by Springenberg et al. (2015).”
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (2015, methodology): “Using guided backpropagation as proposed by (Springenberg et al., 2014), we show in Fig.5 that the features learnt by the discriminator activate on typical parts of a bedroom, like beds and windows.”
- Knowledge Neurons in Pretrained Transformers (2021, methodology): “Besides, a set of attribution methods (Shrikumar et al., 2017; Binder et al., 2016; Zeiler and Fergus, 2014; Springenberg et al., 2015) back-propagate the final output to input features.”
- Axiomatic Attribution for Deep Networks (2017, methodology): “These include DeepLift (Shriku-mar et al., 2016), Layer-wise relevance propagation (LRP) (Binder et al., 2016), Deconvolutional networks (De-ConvNets) (Zeiler & Fergus, 2014), and Guided back-propagation (Springenberg et al., 2014).”