Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning

By Sihui Luo, Xinchao Wang, Gongfan Fang, Yao Hu, Dapeng Tao, Mingli Song. 2019

59 citations recorded by the source corpus.

An increasing number of well-trained deep networks have been released online by researchers and developers, enabling the community to reuse them in a plug-and-play way without accessing the training annotations. However, due to the large number of network variants, such public-available trained models are often of different architectures, each of which being tailored for a specific task or dataset. In this paper, we study a deep-model reusing task, where we are given as input pre-trained networks of heterogeneous architectures specializing in distinct tasks, as teacher models. We aim to learn a multitalented and light-weight student model that is able to grasp the integrated knowledge from all such heterogeneous-structure teachers, again without accessing any human annotation. To this end, we propose a common feature learning scheme, in which the features of all teachers are transformed into a common space and the student is enforced to imitate them all so as to amalgamate the intact knowledge. We test the proposed approach on a list of benchmarks and demonstrate that the learned student is able to achieve very promising performance, superior to those of the teachers in their specialized tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xinchao Wang
2. Dacheng Tao
3. Lin Wang
4. Kuk-Jin Yoon
5. Mingli Song
6. Jianping Gou
7. S. Maybank
8. Baosheng Yu

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Xinchao Wang
3. Lin Wang
4. Kuk-Jin Yoon
5. Mingli Song
6. Jianping Gou
7. Jingwen Ye
8. S. Maybank

Citations in the corpus

• Knowledge Distillation: A Survey (2020, methodology): “Using multiple teacher models with feature ensembles, knowledge amalgamation is designed in (Shen et al., 2019a; Luo et al., 2019; Shen et al., 2019b).”
• Distilling Knowledge From Graph Convolutional Networks (2020, background): “CFL [25] distills the knowledge by learning a common feature space, wherein the student model mimics the transformed features of the teachers to aggregate knowledge.”
• Factorizing Knowledge in Neural Networks (2022, background): “KD is originally designed for model compression [5,56,32,37,64,51,22], but it has been found to be beneficial in other tasks like adversarial defense [47], domain adaptation [15,44], continual learning [33,68] and amalgamate the knowledge from multiple teachers [39,65,24].”
• Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation (2019, methodology): “Several knowledge amalgamation methods are also proposed to handle the above task [33, 23, 34].”
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “Anil [9] X 7 7 7 7 X X LCE+LEns Chen [28] X X 7 7 7 X X LCE+LEns+LKD Dvornik [51] X 7 7 X 7 X X LCE+LEns+LKD Fukuda [56] X 7 7 7 7 7 7 LCE+LEns Furlanello [57] X 7 7 7 7 7 7 LCE+LEns He [80] 7 X 7 7 X 7 7 L1+LKD Jung [101] X 7 7 7 7 7 7 LCE+LKD Lan [114] X 7 7 7 7 X 7 LCE+LEns Lee [118] X X 7 7 X X ”
• Collaboration by Competition: Self-coordinated Knowledge Amalgamation for Multi-talent Student Learning (2020, background): “Recently, some researches [20, 31, 12, 30] started to investigate how to transfer knowledge from multiple trained models into a single one with unlabeled dataset.”