Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and More

By Jingwen Ye, Yixin Ji, Xinchao Wang, Kairi Ou, Dapeng Tao, Mingli Song. 2019

86 citations recorded by the source corpus.

In this paper, we investigate a novel deep-model reusing task. Our goal is to train a lightweight and versatile student model, without human-labelled annotations, that amalgamates the knowledge and masters the expertise of two pre-trained teacher models working on heterogeneous problems, one on scene parsing and the other on depth estimation. To this end, we propose an innovative training strategy that learns the parameters of the student intertwined with the teachers, achieved by ``projecting'' its amalgamated features onto each teacher's domain and computing the loss. We also introduce two options to generalize the proposed training strategy to handle three or more tasks simultaneously. The proposed scheme yields very encouraging results. As demonstrated on several benchmarks, the trained student model achieves results even superior to those of the teachers in their own expertise domains and on par with the state-of-the-art fully supervised models relying on human-labelled annotations.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Xinchao Wang
3. Lin Wang
4. Kuk-Jin Yoon
5. Mingli Song
6. Jianping Gou
7. Jingwen Ye
8. S. Maybank

Citations in the corpus

• Learning with Recoverable Forgetting (2022, positive): “And in order to handle multi-task problems in one single network, knowledge amalgamation [56] is proposed to train the student network on multiple scene understanding tasks, leading to better performance than the teachers.”
• Knowledge Distillation: A Survey (2020, background): “2019; Liu et al., 2019g; Mullapudi et al., 2019; Siam et al., 2019; Dou et al., 2020), saliency estimation (Li et al., 2019), image retrieval(Liu et al., 2019d), depth estimation(Pilzer et al., 2019; Ye et al., 2019), visual odometry (Saputra et al., 2019) and visual question answering (Mun et al., ”
• Distilling Knowledge From Graph Convolutional Networks (2020, methodology): “For example, a method of knowledge amalgamation is proposed in [43] to train a student model from heterogeneous-task teachers include scene parsing teacher, depth estimation teacher and surface-normal estimation teacher.”
• Curriculum Temperature for Knowledge Distillation (2022, methodology): “Knowledge distillation (Hinton, Vinyals, and Dean 2015), as one of the main network compression techniques, has been widely used in many vision tasks (Liu et al. 2019; Ye et al. 2019; Li et al. 2021b, 2022).”
• Overcoming Catastrophic Forgetting in Graph Neural Networks (2020, unknown): “This line of work has focused on the memory modules (Lopez-Paz and Ranzato 2017; Re-buffi et al. 2017; Chaudhry et al. 2018), and thus falls within the domain of model reuse (Yang et al. 2020b; Ye et al. 2019; Shen et al. 2019).”
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “Anil [9] X 7 7 7 7 X X LCE+LEns Chen [28] X X 7 7 7 X X LCE+LEns+LKD Dvornik [51] X 7 7 X 7 X X LCE+LEns+LKD Fukuda [56] X 7 7 7 7 7 7 LCE+LEns Furlanello [57] X 7 7 7 7 7 7 LCE+LEns He [80] 7 X 7 7 X 7 7 L1+LKD Jung [101] X 7 7 7 7 7 7 LCE+LKD Lan [114] X 7 7 7 7 X 7 LCE+LEns Lee [118] X X 7 7 X X ”
• Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning (2019, background): “The work of Ye et al [Ye et al., 2019], similarly, imposes the same assumption on the network architectures.”
• Amalgamating Knowledge from Heterogeneous Graph Neural Networks (2021, background): “Also, the works in [32, 34, 22, 45] propose to reuse multiple trained teacher CNNs, working on different tasks, to learn a versatile student model, but built upon a strong assumption that the teacher models share the same CNN architecture.”
• Factorizing Knowledge in Neural Networks (2022, background): “KD is originally designed for model compression [5,56,32,37,64,51,22], but it has been found to be beneficial in other tasks like adversarial defense [47], domain adaptation [15,44], continual learning [33,68] and amalgamate the knowledge from multiple teachers [39,65,24].”
• Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation (2019, methodology): “Several knowledge amalgamation methods are also proposed to handle the above task [33, 23, 34].”
• Collaboration by Competition: Self-coordinated Knowledge Amalgamation for Multi-talent Student Learning (2020, background): “Recently, some researches [20, 31, 12, 30] started to investigate how to transfer knowledge from multiple trained models into a single one with unlabeled dataset.”