Systematic evaluation of CNN advances on the ImageNet

By Dmytro Mishkin, Nikolay Sergievskiy, Jiri Matas. 2016

123 citations recorded by the source corpus.

The paper systematically studies the impact of a range of recent advances in CNN architectures and learning methods on the object categorization (ILSVRC) problem. The evalution tests the influence of the following choices of the architecture: non-linearity (ReLU, ELU, maxout, compatability with batch normalization), pooling variants (stochastic, max, average, mixed), network width, classifier design (convolutional, fully-connected, SPP), image pre-processing, and of learning parameters: learning rate, batch size, cleanliness of the data, etc. The performance gains of the proposed modifications are first tested individually and then in combination. The sum of individual gains is bigger than the observed improvement when all modifications are introduced, but the ”deficit” is small suggesting independence of their benefits. We show that the use of 128x128 pixel images is sufficient to make qualitative conclusions about optimal network structure that hold for the full size Caffe and VGG nets. The results are obtained an order of magnitude faster than with the standard 224 pixel images.

Subfields: Computer Vision and Pattern Recognition, Media Technology, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

In Evaluation & Benchmarks

Citations in the corpus

A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets (2017, methodology): “Mishkin et al. (2016) suggested to use 128x128 pixels ImageNet images to evaluate various deep learning techniques, but their dataset is not available.”
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size (2016, background): “04, and we linearly decrease the learning rate throughout training, as described in (Mishkin et al., 2016).”
PolyNet: A Pursuit of Structural Diversity in Very Deep Networks (2016, background): “In addition, extensive studies [6, 7, 26, 17] have been done to find good design choices, e.”
Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder (2017, result): “The exact choice of these various parameters tends to a ect the results only slightly; we refer an interested reader to a recent survey [Mishkin et al. 2016].”
Deep Fingerprinting: Undermining Website Fingerprinting Defenses with Deep Learning (2018, methodology): “…dataset, we have followed techniques in the deep learning literature [21, 22, 36] to improve the performance of the model, such as using the appropriate number of convolutional layers, mitigation and prevention of overfitting [34] and a suitable activation function for our WF input data [25].”
- LEVEL ACCURACY WITH 50 X FEWER PARAMETERS AND < 0 . 5 MB MODEL SIZE (2016, methodology): “04, and we linearly decrease the learning rate throughout training, as described in (Mishkin et al., 2016).”