Progressive Network Grafting for Few-Shot Knowledge Distillation

By Chengchao Shen, Xinchao Wang, Youtan Yin, Jie Song, Sihui Luo, Mingli Song. 2020

55 citations recorded by the source corpus.

Knowledge distillation has demonstrated encouraging performances in deep model compression. Most existing approaches, however, require massive labeled data to accomplish the knowledge transfer, making the model compression a cumbersome and costly process. In this paper, we investigate the practical few-shot knowledge distillation scenario, where we assume only a few samples without human annotations are available for each category. To this end, we introduce a principled dual-stage distillation scheme tailored for few-shot data. In the first step, we graft the student blocks one by one onto the teacher, and learn the parameters of the grafted block intertwined with those of the other teacher blocks. In the second step, the trained student blocks are progressively connected and then together grafted onto the teacher network, allowing the learned student blocks to adapt themselves to each other and eventually replace the teacher network. Experiments demonstrate that our approach, with only a few unlabeled samples, achieves gratifying results on CIFAR10,CIFAR100, and ILSVRC-2012. On CIFAR10 and CIFAR100, our performances are even on par with those of knowledge distillation schemes that utilize the full datasets. The source code is available at https://github.com/zju-vipa/NetGraft.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Xinchao Wang
3. Jianping Gou
4. S. Maybank
5. Baosheng Yu
6. Mingli Song
7. Yiding Yang
8. Laura Schmid

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Xinchao Wang
3. Mingli Song
4. Jianping Gou
5. S. Maybank
6. Baosheng Yu
7. Yiding Yang
8. Laura Schmid

Citations in the corpus

• Knowledge Distillation: A Survey (2020, background): “Similar to zero-shot learning, a knowledge distillation method with few-shot learning is designed by distilling knowledge from a teacher model into a student neural network (Kimura et al. 2018; Shen et al. 2021).”
• Large Language Models Are Reasoning Teachers (2022, background): “It most closely resembles a form of data-free distillation (Micaelli and Storkey, 2019; Nayak et al., 2019; Shen et al., 2021), where the otherwise inaccessible transfer data is synthetically generated from a large teacher model.”
• Amalgamating Knowledge from Heterogeneous Graph Neural Networks (2021, background): “Following this pioneering teacher-student framework, plenty of algorithms are proposed to fully utilize the knowledge concealed in the pre-trained teachers [30, 28, 48, 7, 55, 26, 33].”