Improved Knowledge Distillation via Teacher Assistant: Bridging the Gap Between Student and Teacher

By Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Hassan Ghasemzadeh. 2019

159 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “In fact, past work has shown that larger and more accurate models are harder to compress (Mirzadeh et al., 2020; Cho & Hariharan, 2019), although this work focuses on distillation rather than pruning/quantization.”
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, methodology): “Finally, we show that introducing a teacher assistant (Mirzadeh et al., 2019) helps the distillation of large pre-trained Transformer based models and the proposed deep self-attention distillation can further boost the performance.”
Knowledge Distillation: A Survey (2020, methodology): “Although offline distillation methods are simple and effective, some issues in offline distillation have attracted increasing attention from the research community (Mirzadeh et al., 2019).”
Dropout as an Implicit Gating Mechanism For Continual Learning (2020, background): “The idea of using knowledge distillation [11, 20, 21] is also found to be a successful regularizer in several works [15, 14].”
Orthogonal Gradient Descent for Continual Learning (2019, background): “, 2015) and its many variants(Romero et al., 2014; Mirzadeh et al., 2019) are useful to retain the previous information.”
Knowledge Distillation from Internal Representations (2019, background): “Mirzadeh et al. (2019) state that, when the gap in between the teacher and the student is large (e.g., shallow vs. deep neural networks), the student struggles to approximate the teacher.”
Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks (2020, unknown): “In this case, the outputs through the SPP can be seen as the outputs of the teacher-assistant (Mirzadeh et al. 2019).”
Structure-Level Knowledge Distillation For Multilingual Sequence Labeling (2020, methodology): “, 2015) is a technique that first trains a strong teacher model and then trains a weak student model through mimicking the output probabilities (Hinton et al., 2015; Lan et al., 2018; Mirzadeh et al., 2019) or hidden states (Romero”
Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation (2020, unknown): “Unlike the tabular settings we consider here, this prior work studies situations where: (a) unlabeled data is plentiful; (b) there are many more training examples than in typical data tables; (c) both teacher and student are neural networks; (d) the task is multiclass classification [5, 10–14].”
Heterogeneous Knowledge Distillation Using Information Flow Modeling (2020, methodology): “Also, note that a hierarchy of auxiliary teachers can be trained in this fashion, as also proposed in [16].”
Revisiting Knowledge Distillation via Label Smoothing Regularization (2020, methodology): “Knowledge Distillation Since [7] proposed knowledge distillation based on prior work [2], KD has been widely adopted or modified [14, 19, 20, 4, 1, 11, 17].”
StyleGAN2 Distillation for Feed-forward Image Manipulation (2020, methodology): “Usage of this additional source of data can be used to improve measures [57] or to reduce size of target model [39].”
Graph-based Knowledge Distillation by Multi-head Attention Network (2019, methodology): “Recently, various methods have been introduced to enhance teacher knowledge[6, 19].”
Residual Knowledge Distillation (2020, methodology): “A very recent work, called TAKD [16], proposed to improve KD method by introducing intermediate teacher assistant model ( TA ).”
ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation (2021, methodology): “[13] suggests that the performance drop is due to the capacity gap between the teacher and the student, and introduces additional student models named teacher assistants with size between teacher and student to alleviate this gap.”