Heterogeneous Knowledge Distillation Using Information Flow Modeling
By Nikolaos Passalis, Maria Tzelepi, Anastasios Tefas. 2020
175 citations recorded by the source corpus.
Knowledge Distillation (KD) methods are capable of transferring the knowledge encoded in a large and complex teacher into a smaller and faster student. Early methods were usually limited to transferring the knowledge only between the last layers of the networks, while latter approaches were capable of performing multi-layer KD, further increasing the accuracy of the student. However, despite their improved performance, these methods still suffer from several limitations that restrict both their efficiency and flexibility. First, existing KD methods typically ignore that neural networks undergo through different learning phases during the training process, which often requires different types of supervision for each one. Furthermore, existing multi-layer KD methods are usually unable to effectively handle networks with significantly different architectures (heterogeneous KD). In this paper we propose a novel KD method that works by modeling the information flow through the various layers of the teacher model and then train a student model to mimic this information flow. The proposed method is capable of overcoming the aforementioned limitations by using an appropriate supervision scheme during the different phases of the training process, as well as by designing and training an appropriate auxiliary teacher model that acts as a proxy model capable of “explaining” the way the teacher works to the student. The effectiveness of the proposed method is demonstrated using four image datasets and several different evaluation setups.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
Citations in the corpus
- Knowledge Distillation: A Survey (2020, methodology): “To explore the pairwise hint information, the student model also mimics the mutual information flow from pairs of hint layers of the teacher model (Passalis et al. 2020b).”
- Knowledge Distillation for Incremental Learning in Semantic Segmentation (2019, background): “Recently, novel schemes have been proposed, e.g., to model the information flow through the various layers of the teacher model in order to train a student model to mimic this information flow [40].”
- Collaborative Teacher-Student Learning via Multiple Knowledge Transfer (2021, background): “Unlike both the response-based and the feature-based knowledge from individual instances, the relation-based knowledge from instance relations is modelled for improving student learning [37, 38, 15, 16, 17].”
- Cross-Layer Distillation with Semantic Calibration (2020, methodology): “The transferred knowledge can also be captured by crude pairwise activation similarities (Tung and Mori 2019) or hybrid kernel formulations built on them (Passalis, Tzelepi, and Tefas 2020).”