Snapshot Distillation: Teacher-Student Optimization in One Generation

By Chenglin Yang, Lingxi Xie, Chi Su, Alan L. Yuille. 2018

238 citations recorded by the source corpus.

Optimizing a deep neural network is a fundamental task in computer vision, yet direct training methods often suffer from over-fitting. Teacher-student optimization aims at providing complementary cues from a model trained previously, but these approaches are often considerably slow due to the pipeline of training a few generations in sequence, i.e., time complexity is increased by several times. This paper presents snapshot distillation (SD), the first framework which enables teacher-student optimization in one generation. The idea of SD is very simple: instead of borrowing supervision signals from previous generations, we extract such information from earlier epochs in the same generation, meanwhile make sure that the difference between teacher and student is sufficiently large so as to prevent under-fitting. To achieve this goal, we implement SD in a cyclic learning rate policy, in which the last snapshot of each cycle is used as the teacher for all iterations in the next cycle, and the teacher signal is smoothed to provide richer information. In standard image classification benchmarks such as CIFAR100 and ILSVRC2012, SD achieves consistent accuracy gain without heavy computational overheads. We also verify that models pre-trained with SD transfers well to object detection and semantic segmentation in the PascalVOC dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Han-Jia Ye
3. Qi Tian
4. Oncel Tuzel
5. De-chuan Zhan
6. Hadi Pouransari
7. Wei-Lun Chao
8. Lingxi Xie

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Luc Van Gool
3. Bernt Schiele
4. Xiaogang Wang
5. Qi Tian
6. Ming-Hsuan Yang
7. Alan L. Yuille
8. Yi Yang

In Efficiency & Compression

1. Dacheng Tao
2. Oncel Tuzel
3. Hadi Pouransari
4. Lin Wang
5. Kuk-Jin Yoon
6. Qi Tian
7. Jianping Gou
8. Ali Farhadi

Citations in the corpus

• Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “Numerous variants of KD have been proposed, including feature distillation [28, 73], iterative distillation [43, 67], and self-distillation [65, 44, 18, 29].”
• Knowledge Distillation: A Survey (2020, methodology): “ressed multiple teachers in a step-by-step manner, i.e., the student at the t step was used as the teacher of the student at the t+1 step (Furlanello et al., 2018), and similar ideas can be found in (Yang et al., 2019a). To efficiently perform knowledge transfer and explore the power of multiple teach”
• Feature Normalized Knowledge Distillation for Image Classification (2020, background): “Yang et al. [36] extracted information from earlier epochs in the same generation to enable teacher-student optimization in one generation, which could further improve the effi-ciency of training.”
• TranSlider: Transfer Ensemble Learning from Exploitation to Exploration (2020, unknown): “Furthermore, [9, 37] improve the approach with self-distillation.”
• Online Knowledge Distillation via Collaborative Learning (2020, background): “A few recent papers about self-distillation [29, 3, 6, 28] have shown that a converged teacher model supervising a student model of identical architecture could improve the generalization ability over the teacher.”
• Circumventing Outliers of AutoAugment with Knowledge Distillation (2020, background): “Beyond its wide application on model compression (large teacher, small student [33,16]) and model initialization (small teacher, large student [34,3]), researchers later proposed to use it for standard network training, with the teacher and student models sharing the same network architecture [9,50,”
• Progressive Network Grafting for Few-Shot Knowledge Distillation (2020, unknown): “To reduce total training time, online distillation (Yang et al. 2019; Zhang et al. 2019) is proposed to unify the training of student and teacher into one step.”
• Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels (2021, background): “Studies on KD have enriched and diversified the options for the teacher, such as feature map distillation [61, 21, 20], relation-based distillation [37, 51], ensemble distillation [44, 65], or iterative self-distillation [10, 58, 56].”
• Decoupled Knowledge Distillation (2022, unknown): “The following works can be divided into two types, distillation from logits [3,6,22,40,44] and intermediate features [10,11,14,15,23,25,28,33,34,41,43].”
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “[247] propose snapshot distillation method to perform S-T (similar architecture) optimization in one generation based on a cycle learning rate policy (refer to Eqn.”
• Few-Shot Learning With a Strong Teacher (2021, methodology): “We also note that, our method is different from self-distillation [36], [37], [38], [39] — our teacher is a classifier trained with ample data while our student is a few-shot learner.”
• Collaborative Teacher-Student Learning via Multiple Knowledge Transfer (2021, background): “Rather than at the layer level, snapshot distillation [10] transfers knowledge from earlier to later epochs while training a deep model.”
• SimCVD: Simple Contrastive Voxel-Wise Representation Distillation for Semi-Supervised Medical Image Segmentation (2021, background): “As a consequence, it is often hard to accurately recognize object contours; and (3) Generalization ability: considering the limited amount of training data, training deep models is usually deficient due to over-fitting and co-adapting [29], [30].”