Knowledge Distillation from Few Samples

By Tianhong Li, Jianguo Li, Zhuang Liu, Changshui Zhang. 2018

15 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Dacheng Tao
2. Jianping Gou
3. Ji Lin
4. Yunhe Wang
5. Zhijian Liu
6. Shan You
7. Song Han
8. Kai Han

Citations in the corpus

• Zero-shot Knowledge Transfer via Adversarial Belief Matching (2019, background): “In the few-shot setting, Li et al. (2018) obtain a student by pruning a teacher, and align both networks with 1x1 convolutions using a few samples.”
• Knowledge Distillation: A Survey (2020, background): “, 2015) and the loss functions for matching features or distributions matching (Huang and Wang, 2017; Passalis and Tefas, 2018; Zagoruyko and Komodakis, 2017; Mirzadeh et al., 2019; Li et al., 2018; Heo et al., 2019b; Asif et al., 2019).”
• Few Shot Network Compression via Cross Distillation (2019, methodology): “…et al. 2016), a data-free approach; 2) Back-propagation (BP) based fine-tuning on L1-norm pruned models; 3) FitNet (Romero et al. 2014) and 4) FSKD (Li et al. 2018), both of which are knowledge distillation meth-ods; 5) ThiNet (Luo, Wu, and Lin 2017) and 6) Channel Pruning (CP) (He, Zhang, and…”
• Knowledge Extraction with No Observable Data (2019, background): “[22] added a 1 × 1 convolutional layer at the end of each block of a student network and aligned the teacher and student networks by updating those layers when only a few samples are given.”
• GAN Compression: Efficient Architectures for Interactive Conditional GANs (2020, methodology): “A widely-used method for model compression is knowledge distillation, which matches the distribution of the output layer’s logits [23, 43, 9, 63, 33, 46, 10].”
• Reborn Filters: Pruning Convolutional Neural Networks with Limited Data (2020, methodology): “The use of 1 × 1 convolution can also refer to (Li et al. 2018); however, the difference lies in that (Li et al. 2018) adopts 1 × 1 convolution as a manner for knowledge distillation in a teacher-student paradigm while we use 1 × 1 convolution to develop new compact fil-ters from original filters for…”