Bag of Tricks for Image Classification with Convolutional Neural Networks

By Tong He, Zhi-Li Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, Mu Li. 2018

1,617 citations recorded by the source corpus.

Much of the recent progress made in image classification research can be credited to training procedure refinements, such as changes in data augmentations and optimization methods. In the literature, however, most refinements are either briefly mentioned as implementation details or only visible in source code. In this paper, we will examine a collection of such refinements and empirically evaluate their impact on the final model accuracy through ablation study. We will show that, by combining these refinements together, we are able to improve various CNN models significantly. For example, we raise ResNet-50's top-1 validation accuracy from 75.3% to 79.29% on ImageNet. We will also demonstrate that improvement on image classification accuracy leads to better transfer learning performance in other application domains such as object detection and semantic segmentation.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “We make several modifications to the original version using the ResNetD improvements from He et al. (2019) and the antialiased rect-2 blur pooling from Zhang (2019).”
Fixing the train-test resolution discrepancy (2019, methodology): “1 × B/256, where B is the batch size, as in [15].”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, methodology): “4 Whenever SK is used, we also use the ResNet-D [34] variant of ResNet.”
Training data-efficient image transformers & distillation through attention (2020, methodology): “This performance is remarkable since convnet methods for image classification have benefited from years of tuning and optimization [20, 48].”
Grafit: Learning fine-grained image representations with coarse labels (2020, methodology): “We point out that, thanks to our strong optimization strategy borrowed from recent works [28, 50], this baseline by itself outperforms all published results in several settings, for instance our ResNet-50 baseline without extra training data outperforms on ImageNet a ResNet-50 pretrained on YFCC100M”
Training Generative Adversarial Networks with Limited Data (2020, background): “, leads to increasing invariance to these semantics-preserving distortions — a highly desirable quality in a classifier [17, 8, 9].”
High-Performance Large-Scale Image Recognition Without Normalization (2021, methodology): “We begin with an SE-ResNeXt-D model (Xie et al., 2017; Hu et al., 2018; He et al., 2019) with GELU activations (Hendrycks & Gimpel, 2016), which we found to be a surprisingly strong baseline for Normalizer-Free Networks.”
On Layer Normalization in the Transformer Architecture (2020, background): “Goyal et al. (2017); He et al. (2019); You et al. (2018) showed that a learning rate warm-up stage is preferred when training neural networks with extremely large batch sizes.”
BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models (2020, methodology): “We also tried many other learning rate schedules with an exhaustive hyper-parameter sweep, including linearly decaying [23,35] and cosine decaying [13,22].”
Adversarial AutoAugment (2019, background): “In this section, we directly search the augmentation policy on the full training set and train ResNet-50 (He et al., 2016), ResNet-50-D (He et al., 2018) and ResNet-200 (He et al., 2016) from scratch.”
Neural Prototype Trees for Interpretable Fine-grained Image Recognition (2020, methodology): “part-based methods [55, 60], saliency maps [5, 12, 59] or representer points [52].”
Segmenter: Transformer for Semantic Segmentation (2021, neutral): “Table 3 and Table 4 compares our approach to DeepLabV3+ [9] with ResNet-D backbone [25], one of the best fully-convolutional approaches.”
LambdaNetworks: Modeling Long-Range Interactions Without Attention (2021, methodology): “Similarly to ResNet-RS (Bello et al., 2021), we use the ResNetD (He et al., 2018) and additionally replace the max pooling layer in the stem by a strided 3x3 convolution.”
ResNeSt: Split-Attention Networks (2020, methodology): “During the ﬁnal phase of training, the logits z j tend to be very small for j (cid:54) = c , while z c is being pushed to its optimal value ∞ , and this can induce overﬁtting [26,53].”
Bottleneck Transformers for Visual Recognition (2021, background): “As discussed in the previous sections, SENets are powerful models with strong performance that is better than EfficientNets and can be scaled all the way up to 83.8% top-1 accuracy without any bells and whistles such as ResNet-D, etc. BoTNets initially perform worse than SENets (e.g. T3) but begin t”
Scaling Local Self-Attention for Parameter Efficient Visual Backbones (2021, background): “Utilizing regularizations and architectural modules beyond the core primitive is critical for achieving strong results [16].”
Revisiting ResNets: Improved Training and Scaling Strategies (2021, methodology): “We reiterate that ResNet-RS is a combination of: improved scaling strategies, improved training methodologies, the ResNet-D modiﬁcations (He et al., 2018) and the Squeeze-Excitation module (Hu et al., 2018).”
An Ensemble of Epoch-Wise Empirical Bayes for Few-Shot Learning (2019, background): “monotonically decreasing learning rates, of conventional large-scale network training [21].”