Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network

By Jungkyu Lee, Taeryun Won, Kiho Hong. 2020

58 citations recorded by the source corpus.

Recent studies in image classification have demonstrated a variety of techniques for improving the performance of Convolutional Neural Networks (CNNs). However, attempts to combine existing techniques to create a practical model are still uncommon. In this study, we carry out extensive experiments to validate that carefully assembling these techniques and applying them to basic CNN models (e.g. ResNet and MobileNet) can improve the accuracy and robustness of the models while minimizing the loss of throughput. Our proposed assembled ResNet-50 shows improvements in top-1 accuracy from 76.3\% to 82.78\%, mCE from 76.0\% to 48.9\% and mFR from 57.7\% to 32.3\% on ILSVRC2012 validation set. With these improvements, inference throughput only decreases from 536 to 312. To verify the performance improvement in transfer learning, fine grained classification and image retrieval tasks were tested on several public datasets and showed that the improvement to backbone network performance boosted transfer learning performance significantly. Our approach achieved 1st place in the iFood Competition Fine-Grained Visual Recognition at CVPR 2019, and the source code and trained models are available at this https URL

Subfields: Computer Vision and Pattern Recognition, Industrial and Manufacturing Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Industrial and Manufacturing Engineering

Citations in the corpus

Are we done with ImageNet? (2020, methodology): “We start by assembling a set of 19 models [3, 27, 4, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37] which vary along a number of dimensions, including architecture, training objective and usage of external data (see appendix A).”
LambdaNetworks: Modeling Long-Range Interactions Without Attention (2021, background): “Higher accuracies on ImageNet are commonly obtained by training longer with increased augmentation and regularization (Lee et al., 2020; Tan & Le, 2019).”
ResNeSt: Split-Attention Networks (2020, background): “Despite their state-of-the-art performance, these NAS-derived models are usually not optimized for training eﬃciency or memory usage on general/commercial processing hardware (CPU/GPU) [36].”
ImageNet-21K Pretraining for the Masses (2021, background): “To achieve top results, similar to previous works [29, 53, 50], we added standard knowledge distillation loss into our ImageNet-1K training.”
Bottleneck Transformers for Visual Recognition (2021, methodology): “Lastly, we scale BoTNets, taking inspiration from the training and scaling strategies in [56, 49, 38, 51, 48, 67, 3], after noting that BoTNets do not provide substantial gains in a smaller scale training regime.”
A Simple Way to Make Neural Networks Robust Against Diverse Image Corruptions (2020, background): “Concurrent work to ours demonstrates that having more training data [43], [22] or using stronger backbones [43], [25], [18] can significantly improve model performance on common corruptions.”
Improving robustness against common corruptions by covariate shift adaptation (2020, methodology): “1, we find that as little as eight samples are sufficient to improve over AssembleNet [32], the current state-of-the-art ResNet-50 model on IN-C without adaptation.”
ResNet strikes back: An improved training procedure in timm (2021, methodology): “[28] use ResNet-D [15] with SE attention [21].”
Solving ImageNet: a Unified Scheme for Training any Backbone to Top Results (2022, methodology): “[22] showed marginal improvement for ResNet50 model when using KD.”