Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students

By Chenglin Yang, Lingxi Xie, Siyuan Qiao, Alan L. Yuille. 2018

81 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Xu Tan
2. Tao Qin
3. Dacheng Tao
4. Yi Ren
5. Tie-Yan Liu
6. Alan L. Yuille
7. Jianping Gou
8. Zhou Zhao

Citations in the corpus

• Knowledge Distillation: A Survey (2020, methodology): “Snapshot distillation (Yang et al. 2019b) is a special variant of self-distillation, in which knowledge in the earlier epochs of the network (teacher) is transferred into its later epochs (student) to support a supervised training process within the same network.”
• Multilingual Neural Machine Translation with Knowledge Distillation (2019, methodology): “Recent works (Furlanello et al., 2018; Yang et al., 2018) even demonstrate that student model can surpass the accuracy of the teacher model, even if the teacher model is of the same capacity as the student model.”
• Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion (2019, background): “[4] first applied knowledge distillation on neural networks, and then a lot of works expand the usage of knowledge distillation to a variety of tasks, such as image classification [5, 24, 25] and natural language processing [6, 7, 8].”
• On the Efficacy of Knowledge Distillation (2019, background): “s that the number is inherited from the original paper. Method Teacher Top-1 Error (%) Scratch - 30.24 Full KD [12] ResNet18 30.57 Full KD [12] ResNet34 30.79 Full KD [12] ResNet50 30.95 Seq. Full KD [23] 3 Gen. 30.12 Seq. Full KD [23] 6 Gen. 29.6 KD+ONE [17] 3 Branches 29.45 0.23 Full KD + AT [26] ”
• Neural Rejuvenation: Improving Deep Network Training by Enhancing Computational Resource Utilization (2018, methodology): “This efficiency issue is addressed by many methods, including weight quantization [10, 51], low-rank approximation [11, 34], knowledge distillation [26, 63] and network pruning [20, 22, 36, 38, 40, 45, 64, 65].”
• Snapshot Distillation: Teacher-Student Optimization in One Generation (2018, methodology): “We do not perform experiments on the CIFAR10 dataset because it does not contain fine-level visual concepts, and thus the benefit brought by T-S optimization is not significant (as observed in [11] and analyzed in [49]).”
• Knowledge Distillation from Bert in Pre-Training and Fine-Tuning for Polyphone Disambiguation (2019, background): “lot of works expand the usage of knowledge distillation to a variety of tasks, such as image classification [23, 24, 25] and natural language processing [26, 27, 28].”
• Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval (2019, background): “Later, knowledge distillation was extended to optimizing deep networks in many generations [9, 40] and [1] pointed out that knowledge distillation could refine ground truth labels.”
• Knowledge Distillation for End-to-End Person Search (2019, unknown): “The teacher supervisory training signal enables training of a student [61], typically smaller.”
• SimCVD: Simple Contrastive Voxel-Wise Representation Distillation for Semi-Supervised Medical Image Segmentation (2021, methodology): “KD has been applied to a variety of tasks, including image classification [36], [55], [56],”