Knowledge Distillation from Internal Representations

By Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Z. Yao, Xing Fan, Edward Guo. 2019

204 citations recorded by the source corpus.

Knowledge distillation is typically conducted by training a small model (the student) to mimic a large and cumbersome model (the teacher). The idea is to compress the knowledge from the teacher by using its output probabilities as soft-labels to optimize the student. However, when the teacher is considerably large, there is no guarantee that the internal knowledge of the teacher will be transferred into the student; even if the student closely matches the soft-labels, its internal representations may be considerably different. This internal mismatch can undermine the generalization capabilities originally intended to be transferred from the teacher to the student. In this paper, we propose to distill the internal representations of a large model such as BERT into a simplified version of it. We formulate two ways to distill such representations and various algorithms to conduct the distillation. We experiment with datasets from the GLUE benchmark and consistently show that adding knowledge distillation from internal representations is a more powerful method than only using soft-label distillation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, methodology): “Aguilar et al. (2019) further introduce the knowledge of self-attention distributions and propose progressive and stacked distillation methods.”
Transfer without Forgetting (2022, methodology): “To enable feature transfer without incurring pretraining forgetting, we maintain a copy of it (the sibling model) and adopt an intermediate feature knowledge distillation [55,2,66,26,41] objective.”
Distill and Replay for Continual Language Learning (2020, methodology): “• Incremental matching: rather than distilling the layers simultaneously, in the second method we progressively match the hidden layers from the bottom to the top, inspired by (Aguilar et al., 2020).”
Knowledge Distillation: A Survey (2020, methodology): “In Aguilar et al. (2020), a simpliﬁed student BERT is proposed by using the internal representations of a large teacher BERT via internal distillation.”
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers (2020, methodology): “It can also be utilized to initialize task-speciﬁc distillation (Sun et al., 2019a; Turc et al., 2019; Aguilar et al., 2019; Mukher-jee and Awadallah, 2020; Xu et al., 2020; Hou et al., 2020; Li et al., 2020), which uses a ﬁne-tuned teacher model to guide the training of the student on speciﬁc…”
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models (2020, background): “…to match that of the teacher (Hinton et al., 2015) as well as using teacher representations (Romero et al., 2015) (refer to (Cheng et al., 2017) Aguilar et al. (2019) train student models leveraging architectural knowledge of the teacher models which adds architectural constraints (e.g.,…”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “IR [2] distills the internal representations of the teacher model to the student model, however, since multiple layers in the teacher are compressed into one layer of the student, there is information loss when matching the features.”
LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understanding (2020, background): “(Aguilar et al. 2020) distilled internal representation knowledge from attention layer and intermediate output layer in a bottom-up way.”
ALP-KD: Attention-Based Layer Projection for Knowledge Distillation (2020, methodology): “Aguilar et al. (2020) followed the same architecture as PKD but they introduced a new training regime, called progressive training.”
Knowledge Transfer via Dense Cross-Layer Mutual-Distillation (2020, background): “Other KD extensions include but are not limited to efficient network design [6,47], style transfer [29], machine translation [22,1] and multi-task learning [25].”
Extreme Compression for Pre-trained Transformers Made Simple and Efficient (2022, methodology): “We make a few observations: First, perhaps a bit surprisingly, the results in Table 5 show that there are no significant improvements from using the more computationally expensive pre-pretraining distillation in comparison to lightweight layer reduction: Skip-BERT 6 (Row 7) achieves the highest aver”
Supplementary Material: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, background): “[1] further introduce the knowledge of self-attention distributions and propose progressive and stacked distillation methods.”
Learning to Reweight with Deep Interactions (2021, methodology): “In knowledge distillation (Romero et al. 2015; Aguilar et al. 2020), a student model mimics the output of the internal layers of the teacher model so as to achieve comparable performances with the teacher model.”