Improved Knowledge Distillation via Teacher Assistant

By Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, Hassan Ghasemzadeh. 2019

1,428 citations recorded by the source corpus.

Despite the fact that deep neural networks are powerful models and achieve appealing results on many tasks, they are too large to be deployed on edge devices like smartphones or embedded sensor nodes. There have been efforts to compress these networks, and a popular method is knowledge distillation, where a large (teacher) pre-trained network is used to train a smaller (student) network. However, in this paper, we show that the student network performance degrades when the gap between student and teacher is large. Given a fixed student network, one cannot employ an arbitrarily large teacher, or in other words, a teacher can effectively transfer its knowledge to students up to a certain size, not smaller. To alleviate this shortcoming, we introduce multi-step knowledge distillation, which employs an intermediate-sized network (teacher assistant) to bridge the gap between the student and the teacher. Moreover, we study the effect of teacher assistant size and extend the framework to multi-step distillation. Theoretical analysis and extensive experiments on CIFAR-10,100 and ImageNet datasets and on CNN and ResNet architectures substantiate the effectiveness of our proposed approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “Numerous variants of KD have been proposed, including feature distillation [28, 73], iterative distillation [43, 67], and self-distillation [65, 44, 18, 29].”
Designing Interpretable Approximations to Deep Reinforcement Learning with Soft Decision Trees (2020, positive): “Concentrating on edge devices such as smart phones, (Mirzadeh et al. 2019) demonstrate a student of fixed size or complexity will perform poorly if the teacher is too large, and propose a “teaching-assistant” facilitated process involving multiple distillation steps between the original teacher and ”
Knowledge Distillation: A Survey (2020, methodology): “…including the design of knowledge (Hinton et al. 2015; Romero et al. 2015) and the loss functions for matching features or distributions matching (Huang and Wang 2017; Passalis and Tefas 2018; Zagoruyko and Komodakis 2017; Mirzadeh et al. 2020; Li et al. 2020d; Heo et al. 2019b; Asif et al. 2020).”
Orthogonal Gradient Descent for Continual Learning (2019, background): “Knowledge Distillation (Hinton et al., 2015) and its many variants(Romero et al., 2014; Mirzadeh et al., 2019) are useful to retain the previous information.”
Knowledge Distillation from Internal Representations (2019, background): “Mirzadeh et al. (2019) state that, when the gap in between the teacher and the student is large (e.g., shallow vs. deep neural networks), the student struggles to approximate the teacher.”
Depth Anything V2 (2024, neutral): “Moreover, it is indeed non-trivial and risky to directly distill feature representations between two models with a tremendous scale gap [48].”
Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges (2021, positive): “The key intuition behind the success of SC and EE is similar to what has led to the success of techniques such as model pruning [38, 44, 74, 160] and knowledge distillation [46, 61, 98]: most state-of-the-art DNNs are significantly over-parameterized [165, 166].”
Localization Distillation for Object Detection (2022, background): “, Relational KD [48], Self-KD [15], [82], Teacher Assistant KD [46], and Decoupled KD [84], etc.”
Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation (2021, background): “Furthermore, our work is the ﬁrst study on the sequential distillation at the objective level, not at the architecture level such as [ Cho and Hariharan, 2019; Mirzadeh et al. , 2020 ] .”
Decoupled Knowledge Distillation (2022, background): “TAKD [22] introduces an intermediate-sized network named “teacher assistant” to bridge the gap between teachers and students.”
Self-Distillation: Towards Efficient and Compact Neural Networks (2021, methodology): “Recently, researchers ﬁnd that the choice of teacher models has a great impact on the accuracy of student models and the teacher with the highest accuracy is not the best teacher for distillation [32], [33], [34].”
Probabilistic Knowledge Transfer for Lightweight Deep Representation Learning (2020, background): “Finally, also note that a KT approach that employs an intermediate network was also proposed in [37].”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, background): “Various applications with impressive results have been developed for KD, such as mutual learning [190], teacher assistant [191], and self-learning [192].”
An Attentive Survey of Attention Models (2019, background): “Moreover, they observed that introducing a teacher assistant (a concept introduced in Mirzadeh et al. [2020]) also helps the distillation of large pre-trained Transformer models. Similarly, Touvron et al. [2020] employed a teacher-student strategy specific to transformers ensuring that the student l”