ALP-KD: Attention-Based Layer Projection for Knowledge Distillation

By Peyman Passban, Yimeng Wu, Mehdi Rezagholizadeh, Qun Liu. 2020

146 citations recorded by the source corpus.

Knowledge distillation is considered as a training and compression strategy in which two neural networks, namely a teacher and a student, are coupled together during training. The teacher network is supposed to be a trustworthy predictor and the student tries to mimic its predictions. Usually, a student with a lighter architecture is selected so we can achieve compression and yet deliver high-quality results. In such a setting, distillation only happens for final predictions whereas the student could also benefit from teacher’s supervision for internal components. Motivated by this, we studied the problem of distillation for intermediate layers. Since there might not be a one-to-one alignment between student and teacher layers, existing techniques skip some teacher layers and only distill from a subset of them. This shortcoming directly impacts quality, so we instead propose a combinatorial technique which relies on attention. Our model fuses teacher-side information and takes each layer’s significance into consideration, then it performs distillation between combined teacher layers and those of the student. Using our technique, we distilled a 12-layer BERT (Devlin et al. 2019) into 6-, 4-, and 2-layer counterparts and evaluated them on GLUE tasks (Wang et al. 2018). Experimental results show that our combinatorial approach is able to outperform other existing techniques.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, neutral): “ALP-KD (Passban et al. 2021) Attention-based layer projection Hint layer L2(.”
CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society (2023, background): “The knowledge transferred from the teacher to the student model can be categorized into three main types: Response-based, Feature-based, and Relation-based knowledge, which have been studied in various works [5, 29, 56, 35, 74, 36, 28, 13, 51, 50].”