Knowledge distillation: A good teacher is patient and consistent

By Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, Alexander Kolesnikov. 2021

403 citations recorded by the source corpus.

There is a growing discrepancy in computer vision between large-scale models that achieve state-of-the-art performance and models that are affordable in practical applications. In this paper we address this issue and significantly bridge the gap between these two types of models. Throughout our empirical investigation we do not aim to necessarily propose a new method, but strive to identify a robust and effective recipe for making state-of-the-art large scale models affordable in practice. We demonstrate that, when performed correctly, knowledge distillation can be a powerful tool for reducing the size of large models without compromising their performance. In particular, we uncover that there are certain implicit design choices, which may drastically affect the effectiveness of distillation. Our key contribution is the explicit identification of these design choices, which were not previously articulated in the literature. We back up our findings by a comprehensive empirical study, demonstrate compelling results on a wide range of vision datasets and, in particular, obtain a state-of-the-art ResNet-50 model for ImageNet, which achieves 82.8% top-1 accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, background): “As previously reported (Bagherinezhad et al., 2018; Beyer et al., 2021), we find that it improves accuracy to run distillation with data augmentation.”
Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “Our work has also been inspired by [6] where they proposed imitating the teacher on severe augmentations and train for thousands of epochs.”
On Improving Adversarial Transferability of Vision Transformers (2021, methodology): “We consider convolutional networks including BiT-ResNet50 (BiT50) (Beyer et al., 2021), ResNet152 (Res152) (He et al.”
Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning (2022, background): “Research into how the modality gap phenomenon generalizes to the multi-modal representations obtained by these alternative methods, or even uni-modal settings with teacher and student model [44, 5] would be a promising direction for future work.”
EVA-02: A Visual Representation for Neon Genesis (2023, background): “[9] empirically shows that a robust and effective recipe of knowledge distillation makes state-of-the-art large-scale image classification models affordable in practice.”
Solving ImageNet: a Unified Scheme for Training any Backbone to Top Results (2022, background): “This phenomenon, where in KD the student model benefits from very long training, is called patient teacher [2].”
VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking (2023, methodology): “Using the procedure of [9], we are able to compress VideoMAE V2-g into a mush smaller ViTB.”
Meta Knowledge Distillation (2022, background): “Previous works (Beyer et al., 2021; Cho & Hariharan, 2019) simply conducts a grid search to identify an optimal temperature, and ﬁx it for the entire learning process.”
MobileNetV4 - Universal Models for the Mobile Ecosystem (2024, positive): “9, our method improves top-1 accuracy > 0.8% over the previous SOTA [5].”
Revisiting Neural Scaling Laws in Language and Vision (2022, background): “Recent work has found that scaling up the data size [34], the model size [26, 35], the training schedule [5, 39] or all of them together [8, 40] often lead to improved performance.”
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (2023, methodology): “Knowledge distillation has been successfully used to transfer knowledge from larger, more competent teacher models into smaller student models affordable for practical applications (Buciluˇa et al., 2006; Hinton et al., 2015; Beyer et al., 2022; West et al., 2021; Fu et al., 2023).”
Does Knowledge Distillation Really Work? (2021, background): “[4] find similar qualitative results for Shampoo and Adam, besides faster convergence for Shampoo.”
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (2023, background): “…pseudo-labels is widely used in semi-supervised learning (Laine & Aila, 2016; Tarvainen & Valpola, 2017; Xie et al., 2020), domain adaptation (French et al., 2017; Shu et al., 2018), and knowledge distillation (Hinton et al., 2015; Gou et al., 2021; Stanton et al., 2021; Beyer et al., 2022).”
A Fast Knowledge Distillation Framework for Visual Recognition (2021, methodology): “Knowledge Distillation (KD) [16] has been a widely used technique in various visual domains, such as the supervised recognition [2, 24, 30, 35, 47, 48] and self-supervised representation learning [4, 10, 33].”
Scaling Vision Transformers to 22 Billion Parameters (2023, methodology): “We distill ViT-22B into ViT-B/16 and ViT-L/16 by following the procedure of Beyer et al. (2022b).”
Data Feedback Loops: Model-driven Amplification of Dataset Biases (2022, methodology): “The examples were then labeled by a BigTransfer classifier (Beyer et al., 2022), which has 98.”