Well-Read Students Learn Better: The Impact of Student Initialization on Knowledge Distillation

By Iulia Turc, Ming-Wei Chang, Kenton C. T. Lee, Kristina Toutanova. 2019

250 citations recorded by the source corpus.

Recent developments in NLP have been accompanied by large, expensive models. Knowledge distillation is the standard method to realize these gains in applications with limited resources: a compact student is trained to recover the outputs of a powerful teacher. While most prior work investigates student architectures and transfer techniques, we focus on an often-neglected aspect---student initialization. We argue that a random starting point hinders students from fully leveraging the teacher expertise, even in the presence of a large transfer set. We observe that applying language model pre-training to students unlocks their generalization potential, surprisingly even for very compact networks. We conduct experiments on 4 NLP tasks and 24 sizes of Transformer-based students; for sentiment classification on the Amazon Book Reviews dataset, pre-training boosts size reduction and TPU speed-up from 3.1x/1.25x to 31x/16x. Extensive ablation studies dissect the interaction between pre-training and distillation, revealing a compound effect even when they are applied on the same unlabeled dataset.

Subfields: Artificial Intelligence, Education, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Education

In Representation & Self-Supervised Learning

Citations in the corpus

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, positive): “2 DOWNSTREAM EVALUATION Following Yang et al. (2019) and Liu et al.”
TinyBERT: Distilling BERT for Natural Language Understanding (2019, methodology): “, 2020), knowledge distillation (Tang et al., 2019; Sanh et al., 2019; Turc et al., 2019; Sun et al., 2020; Liu et al., 2020; Wang et al., 2020), pruning (Cui et al.”
Learning Compact Metrics for MT (2021, neutral): “Distillation has been shown to successfully transfer knowledge from large models to smaller ones (Hinton et al., 2015), but it requires access to a large corpus of un-labelled data (Sanh et al., 2019; Turc et al., 2019), which does not exist for our task.”
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, background): “There have been some works that task-speciﬁcally distill pre-trained large LMs into small models (Tang et al., 2019; Turc et al., 2019b; Sun et al., 2019a; Aguilar et al., 2019).”
Reducing Transformer Depth on Demand with Structured Dropout (2019, methodology): “We succinctly review the Transformer architecture and refer the reader to Vaswani et al. (2017) for additional details.”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “3 100% ×4 BERT24 No WNLI; SQuAD PD (Turc et al., 2019) ×1.”
Knowledge Distillation: A Survey (2020, methodology): “…et al. 2019; Hu et al. 2018; Sun et al. 2019; Nakashole and Flauger 2017; Jiao et al. 2020; Wang et al. 2018d; Zhou et al. 2019a; Sanh et al. 2019; Turc et al. 2019; Arora et al. 2019; Clark et al. 2019; Kim and Rush 2016; Mou et al. 2016; Liu et al. 2019f; Hahn and Choi 2019; Tan et al. 2019;…”
Training with Quantization Noise for Extreme Model Compression (2020, methodology): “Finally, knowledge distillation [6] has been applied to sentence representation [13, 27, 28, 29, 30], to reduce the size of a BERT model [31].”
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices (2020, background): “Turc et al. (2019) propose to pre-train the smaller BERT models to improve task-specific knowledge distillation.”
Pre-trained models for natural language processing: A survey (2020, neutral): “6 PD [183] 6 LKD-CE+LTask+ LMLM ×2.”
Movement Pruning: Adaptive Sparsity by Fine-Tuning (2020, methodology): “We also compare our method against the mini-BERT models, a collection of smaller BERT models with varying hyper-parameters [Turc et al., 2019].”
Analyzing Redundancy in Pretrained Transformer Models (2020, background): “While the work on pretrained model compression (Cao et al., 2020; Shen et al., 2020; Sanh et al., 2019; Turc et al., 2019; Gordon et al., 2020; Guyon and Elisseeff, 2003) indirectly shows that models exhibit redundancy, little has been done to explore the redundancy in the network.”
Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution (2020, methodology): “KD has been implemented in many machine learning tasks, for example image classification [28], object detection [12, 61], video labeling [70], natural language processing [56, 41, 53, 36, 57], and speech recognition [11, 55, 37].”
Structured Pruning of Large Language Models (2019, methodology): “, 2019), and compressing BERT models via distillation (Chia et al., 2019; Jiao et al., 2019; Sanh et al., 2019; Sun et al., 2019; Tsai et al., 2019; Turc et al., 2019).”
DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models (2022, methodology): “In particular, we decouple the training of EMB(wx) and EMB(wy) by replacing EMB(wx) with representations extracted from a pre-trained BERT-tiny model (Turc et al., 2019).”
Know What You Don't Need: Single-Shot Meta-Pruning for Attention Heads (2020, methodology): “Quite a few methods have been proposed to compress pre-trained Transformers during or after fine-tuning to reduce the computational overhead in the inference phase (Tang et al., 2019; Turc et al., 2019; Jiao et al., 2019; McCarley, 2019; Wang et al., 2019b; Fan et al., 2020), while little work attem”