Supplementary Material for Few Sample Knowledge Distillation for Efficient Network Compression

By Tianhong Li, Jianguo Li, Zhuang Liu, Changshui Zhang. 2020

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Dacheng Tao
2. Jianping Gou
3. Lin Wang
4. Kuk-Jin Yoon
5. Baosheng Yu
6. Liang Zheng
7. Liyuan Sun
8. S. Maybank

Citations in the corpus

• Knowledge Distillation: A Survey (2020, methodology): “…including the design of knowledge (Hinton et al. 2015; Romero et al. 2015) and the loss functions for matching features or distributions matching (Huang and Wang 2017; Passalis and Tefas 2018; Zagoruyko and Komodakis 2017; Mirzadeh et al. 2020; Li et al. 2020d; Heo et al. 2019b; Asif et al. 2020).”
• Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation (2019, methodology): “In our implementation, FA takes the form of an 1 × 1 convolutional operation [28, 30, 18].”
• Collaborative Teacher-Student Learning via Multiple Knowledge Transfer (2021, background): “Most of the KD methods employ offline distillation, which is one-way knowledge transfer from a pre-trained large teacher to a small student [20, 22, 13].”
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “[127], the teacher network is first compressed to create student via network pruning [274], and then layer-wise distillation losses are applied to reduce the estimation error on given limited samples.”
• Factorizing Knowledge in Neural Networks (2022, background): “KD is originally designed for model compression [5, 55, 31, 36, 63, 50], but it has been found to be beneficial in other tasks like adversarial defense [46], domain adaptation [15, 43], continual learning [32, 67] and amalgamate the knowledge from multiple teachers [38, 64, 23].”
• Memory-Free Generative Replay For Class-Incremental Learning (2021, methodology): “Knowledge Distillation [13] has been used in many areas such as network compression [20], semantic segmentation [22].”
• Few Shot Network Compression via Cross Distillation (2019, methodology): “…et al. 2016), a data-free approach; 2) Back-propagation (BP) based fine-tuning on L1-norm pruned models; 3) FitNet (Romero et al. 2014) and 4) FSKD (Li et al. 2018), both of which are knowledge distillation meth-ods; 5) ThiNet (Luo, Wu, and Lin 2017) and 6) Channel Pruning (CP) (He, Zhang, and…”
• Progressive Network Grafting for Few-Shot Knowledge Distillation (2020, unknown): “Following (Li et al. 2018) and (Li et al. 2017), we adopt VGG16-half as student model.”