Meta Knowledge Distillation

By Jihao Liu, Boxiao Liu, Hongsheng Li, Yu Liu. 2022

35 citations recorded by the source corpus.

Recent studies pointed out that knowledge distillation (KD) suffers from two degradation problems, the teacher-student gap and the incompatibility with strong data augmentations, making it not applicable to training state-of-the-art models, which are trained with advanced augmentations. However, we observe that a key factor, i.e., the temperatures in the softmax functions for generating probabilities of both the teacher and student models, was mostly overlooked in previous methods. With properly tuned temperatures, such degradation problems of KD can be much mitigated. However, instead of relying on a naive grid search, which shows poor transferability, we propose Meta Knowledge Distillation (MKD) to meta-learn the distillation with learnable meta temperature parameters. The meta parameters are adaptively adjusted during training according to the gradients of the learning objective. We validate that MKD is robust to different dataset scales, different teacher/student architectures, and different types of data augmentation. With MKD, we achieve the best performance with popular ViT architectures among compared methods that use only ImageNet-1K as training data, ranging from tiny to large models. With ViT-L, we achieve 86.5% with 600 epochs of training, 0.6% better than MAE that trains for 1,650 epochs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Efficiency & Compression, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohammad Rastegari
2. Sachin Mehta
3. Oncel Tuzel
4. Hadi Pouransari
5. Ali Farhadi
6. Mehrdad Farajtabar
7. Jian Yang
8. Renjie Song

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Mohammad Rastegari
3. Chunhua Shen
4. Ming-Hsuan Yang
5. Xiaogang Wang
6. Jian Yang
7. Oncel Tuzel
8. Dacheng Tao

In Radiology, Nuclear Medicine and Imaging

1. Ali Farhadi
2. Mehrdad Farajtabar
3. Hadi Pouransari
4. Oncel Tuzel
5. Jian Yang
6. Hongpeng Liao
7. Chien-Yao Wang
8. Diganta Misra

In Efficiency & Compression

1. Oncel Tuzel
2. Hadi Pouransari
3. Ali Farhadi
4. Mohammad Rastegari
5. Sachin Mehta
6. Mehrdad Farajtabar
7. Renjie Song
8. Borui Zhao

In Meta & Continual Learning

1. Ali Farhadi
2. Mehrdad Farajtabar
3. Mohammad Rastegari
4. Seyed Iman Mirzadeh
5. Razvan Pascanu

Citations in the corpus

• Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, background): “Recently,[36] suggested that temperature tuning can help in KD from larger teachers.”
• Curriculum Temperature for Knowledge Distillation (2022, background): “The previous work MKD (Liu et al. 2022) shares the same goal as our proposed CTKD, which is to learn an appropriate temperature parameter for the distillation process.”