Distillation-Based Training for Multi-Exit Architectures

By Mary Phuong, Christoph H. Lampert. 2019

211 citations recorded by the source corpus.

Multi-exit architectures, in which a stack of processing layers is interleaved with early output layers, allow the processing of a test example to stop early and thus save computation time and/or energy. In this work, we propose a new training procedure for multi-exit architectures based on the principle of knowledge distillation. The method encourages early exits to mimic later, more accurate exits, by matching their probability outputs. Experiments on CIFAR100 and ImageNet show that distillation-based training significantly improves the accuracy of early exits while maintaining state-of-the-art accuracy for late ones. The method is particularly beneficial when training data is limited and also allows a straight-forward extension to semi-supervised learning, i.e. make use also of unlabeled data at training time. Moreover, it takes only a few lines to implement and imposes almost no computational overhead at training time, and none at all at test time.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “Despite a huge number of the knowledge distillation methods and applications, the understanding of knowledge distillation including theoretical explanations and empiri-123 cal evaluations remains insufﬁcient (Lopez-Paz et al. 2016; Phuong and Lampert 2019a; Cho and Hariharan 2019).”
Collaborative Teacher-Student Learning via Multiple Knowledge Transfer (2021, background): “Self-distillation is a novel training scheme for knowledge transfer [23, 24, 27, 11, 10].”
Knowledge Transfer via Dense Cross-Layer Mutual-Distillation (2020, methodology): “In light of this, to the first question, we use a practical principle, adding auxiliary classifiers merely to down-sampling layers of a backbone network [18,58,35,42].”
FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metrics for Automatic Text Generation (2021, background): “, transferring the knowledge of a later exit layer into an earlier one of the same multi-exit network (Phuong and Lampert, 2019).”
Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks (2020, unknown): “P using the same model but with randomly assigned quantization precision for activation at every iteration. Note that the weights in the model are quantized to n W bits. et al. 2019a; Li et al. 2019; Phuong and Lampert 2019; Yu and Huang 2019; Zhang et al. 2019b). The difference between the proposed”
Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges (2021, methodology): “[77] and Phuong and Lampert [108] use a knowledge distillation-based loss such as Equation (3) by treating the final classifier (last exit) as a teacher model and all the early classifiers as student models.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “So is selfdistillation in generations better? [180] find that network architecture heavily determines the success of KD in generations.”
BERxiT: Early Exiting for BERT with Better Fine-Tuning and Extension to Regression (2021, methodology): “FastBERT, on the other hand, uses self-distillation (Zhang et al., 2019; Phuong and Lampert, 2019) for ﬁne-tuning, which works well for small Muppets such as BERT BASE .”
ELF: An Early-Exiting Framework for Long-Tailed Classification (2020, result): “Our inference time exit criterion in Equation (5) exits examples based on prediction confidence which is in line with prior work [9, 34].”