Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation
By Chengchao Shen, Mengqi Xue, Xinchao Wang, Jie Song, Li Sun, Mingli Song. 2019
65 citations recorded by the source corpus.
A massive number of well-trained deep networks have been released by developers online. These networks may focus on different tasks and in many cases are optimized for different datasets. In this paper, we study how to exploit such heterogeneous pre-trained networks, known as teachers, so as to train a customized student network that tackles a set of selective tasks defined by the user. We assume no human annotations are available, and each teacher may be either single- or multi-task. To this end, we introduce a dual-step strategy that first extracts the task-specific knowledge from the heterogeneous teachers sharing the same sub-task, and then amalgamates the extracted knowledge to build the student network. To facilitate the training, we employ a selective learning scheme where, for each unlabelled sample, the student learns adaptively from only the teacher with the least prediction ambiguity. We evaluate the proposed approach on several datasets and the experimental results demonstrate that the student, learned by such adaptive knowledge amalgamation, achieves performances even better than those of the teachers.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
Citations in the corpus
- Knowledge Distillation: A Survey (2020, methodology): “In particular, these KD-based speech recognition applications have spoken language identification (Shen et al. 2018, 2019c, 2020), audio classification (Gao et al. 2019; Perez et al. 2020), text-independent speaker recognition (Ng et al. 2018), speech enhancement (Watanabe et al. 2017), acoustic…”
- Distilling Knowledge From Graph Convolutional Networks (2020, background): “Knowledge amalgamation [33, 22, 32, 44] aims to learn a student network from multiple teachers from different domains.”
- Collaborative Teacher-Student Learning via Multiple Knowledge Transfer (2021, background): “There are a few works that consider more than one kind of knowledge in knowledge distillation [37, 14] at the same time and explore the efficacy of each kind of knowledge.”
- Collaboration by Competition: Self-coordinated Knowledge Amalgamation for Multi-talent Student Learning (2020, background): “[21] customized the student network by generating component nets as byproducts for attribute learning tasks.”
- Progressive Network Grafting for Few-Shot Knowledge Distillation (2020, unknown): “Following (Hinton, Vinyals, and Dean 2015), researchers pay more attention to the supervision from intermediate representations for better optimization performance (Romero et al. 2014; Wang et al. 2018; Shen et al. 2019a,b; Ye et al. 2020; Luo et al. 2020).”
- Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “Anil [9] X 7 7 7 7 X X LCE+LEns Chen [28] X X 7 7 7 X X LCE+LEns+LKD Dvornik [51] X 7 7 X 7 X X LCE+LEns+LKD Fukuda [56] X 7 7 7 7 7 7 LCE+LEns Furlanello [57] X 7 7 7 7 7 7 LCE+LEns He [80] 7 X 7 7 X 7 7 L1+LKD Jung [101] X 7 7 7 7 7 7 LCE+LKD Lan [114] X 7 7 7 7 X 7 LCE+LEns Lee [118] X X 7 7 X X ”
- Amalgamating Knowledge from Heterogeneous Graph Neural Networks (2021, background): “Also, the works in [32, 34, 22, 45] propose to reuse multiple trained teacher CNNs, working on different tasks, to learn a versatile student model, but built upon a strong assumption that the teacher models share the same CNN architecture.”