Cross-Layer Distillation with Semantic Calibration

By Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang, Yan Feng, Chun Chen. 2020

381 citations recorded by the source corpus.

Recently proposed knowledge distillation approaches based on feature-map transfer validate that intermediate layers of a teacher model can serve as effective targets for training a student model to obtain better generalization ability. Existing studies mainly focus on particular representation forms for knowledge transfer between manually specified pairs of teacher-student intermediate layers. However, semantics of intermediate layers may vary in different networks and manual association of layers might lead to negative regularization caused by semantic mismatch between certain teacher-student layer pairs. To address this problem, we propose Semantic Calibration for Cross-layer Knowledge Distillation (SemCKD), which automatically assigns proper target layers of the teacher model for each student layer with an attention mechanism. With a learned attention distribution, each student layer distills knowledge contained in multiple layers rather than a single fixed intermediate layer from the teacher model for appropriate cross-layer supervision in training. Consistent improvements over state-of-the-art approaches are observed in extensive experiments with various network architectures for teacher and student models, demonstrating the effectiveness and flexibility of the proposed attention based soft layer association mechanism for cross-layer distillation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neurology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Neurology

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “…algorithms focus on transferring individual instance knowledge from the teacher to the student, while some recent methods have been proposed to explore the intra-data relationships using graphs (Chen et Speciﬁcally, in Zhang and Peng (2018), each vertex represents a self-supervised teacher.”
CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society (2023, background): “The knowledge transferred from the teacher to the student model can be categorized into three main types: Response-based, Feature-based, and Relation-based knowledge, which have been studied in various works [5, 29, 56, 35, 74, 36, 28, 13, 51, 50].”