Amalgamating Knowledge towards Comprehensive Classification

By Chengchao Shen, Xinchao Wang, Jie Song, Li Sun, Mingli Song. 2018

108 citations recorded by the source corpus.

With the rapid development of deep learning, there have been an unprecedentedly large number of trained deep network models available online. Reusing such trained models can significantly reduce the cost of training the new models from scratch, if not infeasible at all as the annotations used for the training original networks are often unavailable to public. We propose in this paper to study a new model-reusing task, which we term as knowledge amalgamation. Given multiple trained teacher networks, each of which specializes in a different classification problem, the goal of knowledge amalgamation is to learn a lightweight student model capable of handling the comprehensive classification. We assume no other annotations except the outputs from the teacher models are available, and thus focus on extracting and amalgamating knowledge from the multiple teachers. To this end, we propose a pilot two-step strategy to tackle the knowledge amalgamation task, by learning first the compact feature representations from teachers and then the network parameters in a layer-wise manner so as to build the student model. We apply this approach to four public datasets and obtain very encouraging results: even without any human annotation, the obtained student model is competent to handle the comprehensive classification task and in most cases outperforms the teachers in individual sub-tasks.

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

FEED: Feature-level Ensemble for Knowledge Distillation (2019, background): “Shen(Shen et al. 2019) tried to combine knowledge trained from different domains, and You (You et al.”
Knowledge Distillation: A Survey (2020, methodology): “In particular, these KD-based speech recognition applications have spoken language identiﬁcation (Shen et al. 2018, 2019c, 2020), audio classiﬁcation (Gao et al. 2019; Perez et al. 2020), text-independent speaker recognition (Ng et al. 2018), speech enhancement (Watanabe et al. 2017), acoustic…”
Data-Free Learning of Student Networks (2019, background): “[22] conducted feature amalgamation to learn a compact student model by inherting knowledge from multiple teacher networks.”
Distilling Knowledge From Graph Convolutional Networks (2020, background): “Knowledge amalgamation [33, 22, 32, 44] aims to learn a student network from multiple teachers from different domains.”
Amalgamating Knowledge from Heterogeneous Graph Neural Networks (2021, background): “Also, the works in [32, 34, 22, 45] propose to reuse multiple trained teacher CNNs, working on different tasks, to learn a versatile student model, but built upon a strong assumption that the teacher models share the same CNN architecture.”
Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation (2019, methodology): “In our implementation, FA takes the form of an 1 × 1 convolutional operation [28, 30, 18].”
Safe Distillation Box (2021, background): “Shen et al. (2019), on the other hand, train a student classifier by learning from multiple teachers working on different classes.”
Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and More (2019, background): “The work of [32] proposes to train a student classifier that handles the comprehensive classification problem by learning from multiple teachers working on different classes.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “Anil [9] X 7 7 7 7 X X LCE+LEns Chen [28] X X 7 7 7 X X LCE+LEns+LKD Dvornik [51] X 7 7 X 7 X X LCE+LEns+LKD Fukuda [56] X 7 7 7 7 7 7 LCE+LEns Furlanello [57] X 7 7 7 7 7 7 LCE+LEns He [80] 7 X 7 7 X 7 7 L1+LKD Jung [101] X 7 7 7 7 7 7 LCE+LKD Lan [114] X 7 7 7 7 X 7 LCE+LEns Lee [118] X X 7 7 X X ”
Co-Evolutionary Compression for Unpaired Image Translation (2019, background): “matrix/tensor decomposition [5], quantization and binarization [3, 23, 29], knowledge distillation [11, 26, 27] and efficient convolution blocks design [12, 34].”
Highlight Every Step: Knowledge Distillation via Collaborative Teaching (2019, background): “[76] aimed at learning a compact student model capable of handling the”
Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning (2019, methodology): “3, since we focus here on heterogeneous teachers and thus cannot apply Shen’s method.”
Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN (2020, background): “The work of [24], on the other hand, trains a student classifier by learning from multiple teachers with different classes.”
Feature-Level Ensemble Knowledge Distillation for Aggregating Knowledge from Multiple Networks (2020, background): “Shen[23] tried to combine knowledge trained from different domains, and You [30] utilized ensemble of orderings of samples to teach the student network, which is very novel.”
Progressive Network Grafting for Few-Shot Knowledge Distillation (2020, unknown): “Following (Hinton, Vinyals, and Dean 2015), researchers pay more attention to the supervision from intermediate representations for better optimization performance (Romero et al. 2014; Wang et al. 2018; Shen et al. 2019a,b; Ye et al. 2020; Luo et al. 2020).”
Amalgamating Filtered Knowledge: Learning Task-customized Student from Multi-task Teachers (2019, background): “The work of [Shen et al., 2019], on the other hand, trains a student classifier by learning from multiple teachers working on different classes.”
Overcoming Catastrophic Forgetting in Graph Neural Networks (2020, unknown): “This line of work has focused on the memory modules (Lopez-Paz and Ranzato 2017; Re-bufﬁ et al. 2017; Chaudhry et al. 2018), and thus falls within the domain of model reuse (Yang et al. 2020b; Ye et al. 2019; Shen et al. 2019).”
Collaboration by Competition: Self-coordinated Knowledge Amalgamation for Multi-talent Student Learning (2020, background): “Recently, some researches [20, 31, 12, 30] started to investigate how to transfer knowledge from multiple trained models into a single one with unlabeled dataset.”