MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks

By Zhiqiang Shen, Marios Savvides. 2020

69 citations recorded by the source corpus.

In this paper, we introduce a simple yet effective approach that can boost the vanilla ResNet-50 to 80%+ Top-1 accuracy on ImageNet without any tricks. Generally, our method is based on the recently proposed MEAL, i.e., ensemble knowledge distillation via discriminators. We further simplify it through 1) adopting the similarity loss and discriminator only on the final outputs and 2) using the average of softmax probabilities from all teacher ensembles as the stronger supervision for distillation. One crucial perspective of our method is that the one-hot/hard label should not be used in the distillation process. We show that such a simple framework can achieve state-of-the-art results without involving any commonly-used techniques, such as 1) architecture modification; 2) outside training data beyond ImageNet; 3) autoaug/randaug; 4) cosine learning rate; 5) mixup/cutmix training; 6) label smoothing; etc. On ImageNet, our method obtains 80.67% top-1 accuracy using a single crop-size of 224X224 on the vanilla ResNet-50, outperforming the previous state-of-the-arts by a remarkable margin under the same network structure. Our result can be regarded as a new strong baseline on ResNet-50 using knowledge distillation. To our best knowledge, this is the first work that is able to boost vanilla ResNet-50 to surpass 80% on ImageNet without architecture modification or additional training data. Our code and models are available at: this https URL.

Subfields: Computer Vision and Pattern Recognition, Media Technology, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

In Optimization & Training

Citations in the corpus

Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “Baseline methods We compare the performance of models trained using ImageNet with the following baseline methods: (1) KD [35, 6] (Online distillation): A standard knowledge distillation method with strong teacher models and model-specific augmentations, (2) MEALV2 [54] (Fine-tuning distillation): Di”
S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration (2021, background): “(3) which is equivalent to LKL following MEAL [31, 33].”
Knowledge distillation: A good teacher is patient and consistent (2021, methodology): “2% better than the best ResNet-50 model in the literature, which uses a more complex setup [37].”
SEED: Self-supervised Distillation For Visual Representation (2021, unknown): “For example in image classification task (Mishra & Marr, 2017; Shen & Savvides, 2020; Polino et al., 2018; Cho & Hariharan, 2019), it is normally a cross entropy loss.”
MobileOne: An Improved One millisecond Mobile Backbone (2022, methodology): “Results of competing models are reported from [48].”
BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search (2021, methodology): “To form a common objective for all paths, we can use a scheme analogous to ensemble distillation [59, 60] in supervised learning.”
A Fast Knowledge Distillation Framework for Visual Recognition (2021, background): “MEAL V2 [36] (Fine-tuning Distillation).”
Concept Generalization in Visual Representation Learning (2020, unknown): “28M images and has been used as the standard benchmark for evaluating novel computer vision architectures [23, 50, 53, 58], regularization techniques [49, 59, 69, 74] as well as self- and semi-supervised models [5, 8, 19, 22, 65].”
Dynamic Slimmable Network (2021, unknown): “As pointed out in [58, 59], an ensemble of teacher networks can generate more diverse, more accurate and more general soft labels for distillation training of the student network.”