Feature Fusion for Online Mutual Knowledge Distillation

By Jangho Kim, Minsung Hyun, Inseop Chung, Nojun Kwak. 2019

110 citations recorded by the source corpus.

We propose a learning framework named Feature Fusion Learning (FFL) that efficiently trains a powerful classifier through a fusion module which combines the feature maps generated from parallel neural networks and generates meaningful feature maps. Specifically, we train a number of parallel neural networks as sub-networks, then we combine the feature maps from each sub-network using a fusion module to create a more meaningful feature map. The fused feature map is passed into the fused classifier for overall classification. Unlike existing feature fusion methods, in our framework, an ensemble of sub-network classifiers transfers its knowledge to the fused classifier and then the fused classifier delivers its knowledge back to each subnetwork, mutually teaching one another in an online-knowledge distillation manner. This mutually teaching system not only improves the performance of the fused classifier but also obtains performance gain in each sub-network. Moreover, our model is more beneficial than other alternative methods because different types of network can be used for each sub-network. We have performed a variety of experiments on multiple datasets such as CIFAR-10, CIFAR-100 and ImageNet and proved that our method is more effective than other alternative methods in terms of performances of both sub-networks and the fused classifier, and the aspect of generating meaningful feature maps. The code is available at this link11https://github.com/Jangho-Kim/FFL-pytorch

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Lin Wang
3. Kuk-Jin Yoon
4. Jianping Gou
5. S. Maybank
6. Baosheng Yu
7. Nojun Kwak
8. Jinwon Lee

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Lin Wang
3. Kuk-Jin Yoon
4. Jianping Gou
5. S. Maybank
6. Nojun Kwak
7. Baosheng Yu
8. Jinwon Lee

Citations in the corpus

• Knowledge Distillation: A Survey (2020, methodology): “A variety of online knowledge distillation methods have been proposed, especially in the last few years (Zhang et al. 2018b; Chen et al. 2020a; Xie et al. 2019; Anil et al. 2018; Kim et al. 2019b; Zhou et al. 2018; Walawalkar et al. 2020; Wu and Gong 2021; Zhang et al. 2021a).”
• Collaborative Teacher-Student Learning via Multiple Knowledge Transfer (2021, background): “In contrast, online distillation provides a one-phase end-to-end training scheme via teacher-student collaborative learning on a peer-network architecture instead of a fixed one [25, 33, 36, 32, 28, 12].”
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “Overal insight: Is it possible to enable the student to distill knowledge by itself to achieve plausible performance? The conventional KD approaches [85], [105], [184], [216], [254] still have many setbacks to be tackled although significant performance boost has been achieved.”
• QKD: Quantization-aware Knowledge Distillation (2019, methodology): “Second, online KD methods [54, 19, 51] train both the teacher and the student networks simultaneously without pre-trained teacher models.”