FEED: Feature-level Ensemble for Knowledge Distillation

By Seonguk Park, Nojun Kwak. 2019

46 citations recorded by the source corpus.

Knowledge Distillation (KD) aims to transfer knowledge in a teacher-student framework, by providing the predictions of the teacher network to the student network in the training stage to help the student network generalize better. It can use either a teacher with high capacity or {an} ensemble of multiple teachers. However, the latter is not convenient when one wants to use feature-map-based distillation methods. For a solution, this paper proposes a versatile and powerful training algorithm named FEature-level Ensemble for knowledge Distillation (FEED), which aims to transfer the ensemble knowledge using multiple teacher networks. We introduce a couple of training algorithms that transfer ensemble knowledge to the student at the feature map level. Among the feature-map-based distillation methods, using several non-linear transformations in parallel for transferring the knowledge of the multiple teacher{s} helps the student find more generalized solutions. We name this method as parallel FEED, andexperimental results on CIFAR-100 and ImageNet show that our method has clear performance enhancements, without introducing any additional parameters or computations at test time. We also show the experimental results of sequentially feeding teacher's information to the student, hence the name sequential FEED, and discuss the lessons obtained. Additionally, the empirical results on measuring the reconstruction errors at the feature map give hints for the enhancements.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Jianping Gou
3. S. Maybank
4. Baosheng Yu
5. Lin Wang
6. Kuk-Jin Yoon
7. Min Yang
8. Ruifeng Xu

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Jianping Gou
3. S. Maybank
4. Lin Wang
5. Kuk-Jin Yoon
6. Baosheng Yu
7. Rui Yan
8. Min Yang

In Efficiency & Compression

1. Dacheng Tao
2. Lin Wang
3. Kuk-Jin Yoon
4. Jianping Gou
5. Min Yang
6. Ruifeng Xu
7. Chengming Li
8. S. Maybank

Citations in the corpus

• Knowledge Distillation: A Survey (2020, background): “To further explore the advantages of multiple diverse teachers, some multi-teacher knowledge distillation methods have recently been proposed (You et al., 2017; Chen et al., 2019c; Furlanello et al., 2018; Yang et al., 2019a; Zhang et al., 2018b; Sau and Balasubramanian, 2016; Park and Kwak, 2019; P”
• Amalgamating Knowledge from Two Teachers for Task-oriented Dialogue System with Adversarial Training (2020, background): “Instead of considering one single teacher network, several studies trained a student network by incorporating multiple teacher networks in the output layer or the hidden layers (Park and Kwak, 2019; You et al., 2017).”
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “Overall insight: Can we achieve KD when the original data for the teacher or (un)labeled data for training student are not available? One major limitation of most KD methods such as [1, 28, 48, 52] is that they assume the training samples of the original networks (teachers) or of target networks (st”