Amalgamating Filtered Knowledge: Learning Task-customized Student from Multi-task Teachers

By Jingwen Ye, Xinchao Wang, Yixin Ji, Kairi Ou, Mingli Song. 2019

33 citations recorded by the source corpus.

Many well-trained Convolutional Neural Network~(CNN) models have now been released online by developers for the sake of effortless reproducing. In this paper, we treat such pre-trained networks as teachers and explore how to learn a target student network for customized tasks, using multiple teachers that handle different tasks. We assume no human-labelled annotations are available, and each teacher model can be either single- or multi-task network, where the former is a degenerated case of the latter. The student model, depending on the customized tasks, learns the related knowledge filtered from the multiple teachers, and eventually masters the complete or a subset of expertise from all teachers. To this end, we adopt a layer-wise training strategy, which entangles the student's network block to be learned with the corresponding teachers. As demonstrated on several benchmarks, the learned student network achieves very promising results, even outperforming the teachers on the customized tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lin Wang
2. Kuk-Jin Yoon
3. Xinchao Wang
4. Mingli Song
5. Jingwen Ye
6. Jie Song
7. Chengchao Shen
8. Yixin Ji

In Computer Vision and Pattern Recognition

1. Lin Wang
2. Kuk-Jin Yoon
3. Xinchao Wang
4. Mingli Song
5. Jingwen Ye
6. Jie Song
7. Chengchao Shen
8. Yixin Ji

Citations in the corpus

• Learning with Recoverable Forgetting (2022, neutral): “To make it further, Ye et al. [58] apply a two-step filter strategy to customize the arbitrary task set on TargetNet.”
• Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation (2019, methodology): “Several knowledge amalgamation methods are also proposed to handle the above task [33, 23, 34].”
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “To tackle these problems, knowledge amalgamation has been initialized by recent works [51], [59], [131], [147], [187], [195], [196], [252], [253], [272] to learn a versatile student model by distilling knowledge from the expertise of all teachers as illustrated in Fig.”
• Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN (2020, methodology): “Other than these works that learn from one single network, knowledge amalgamation (KA) [35] is proposed to effectively reuse multiple pre-trained networks, and make the learned multiknowledge settle well in one single network.”