Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN

By Jingwen Ye, Yixin Ji, Xinchao Wang, Xin Gao, Mingli Song. 2020

59 citations recorded by the source corpus.

Recent advances in deep learning have provided procedures for learning one network to amalgamate multiple streams of knowledge from the pre-trained Convolutional Neural Network (CNN) models, thus reduce the annotation cost. However, almost all existing methods demand massive training data, which may be unavailable due to privacy or transmission issues. In this paper, we propose a data-free knowledge amalgamate strategy to craft a well-behaved multi-task student network from multiple single/multi-task teachers. The main idea is to construct the group-stack generative adversarial networks (GANs) which have two dual generators. First one generator is trained to collect the knowledge by reconstructing the images approximating the original dataset utilized for pre-training the teachers. Then a dual generator is trained by taking the output from the former generator as input. Finally we treat the dual part generator as the target network and regroup it. As demonstrated on several benchmarks of multi-label classification, the proposed method without any training data achieves the surprisingly competitive results, even compared with some full-supervised methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Lin Wang
3. Kuk-Jin Yoon
4. Jianping Gou
5. Xinchao Wang
6. S. Maybank
7. Baosheng Yu
8. Mingli Song

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Xinchao Wang
3. Lin Wang
4. Kuk-Jin Yoon
5. Jianping Gou
6. Mingli Song
7. S. Maybank
8. Baosheng Yu

Citations in the corpus

• Learning with Recoverable Forgetting (2022, neutral): “It has been successfully conducted in deep model compression [61], incremental learning [37], continual learning [25,57] and other tasks other than classification [5,53,22,50,36,60,51,55,23].”
• Knowledge Distillation: A Survey (2020, methodology): “Specifically, in (Chen et al., 2019a; Ye et al., 2020; Micaelli and Storkey, 2019; Yoo et al., 2019; Hu et al., 2020), the transfer data is generated by a GAN.”
• Progressive Network Grafting for Few-Shot Knowledge Distillation (2020, unknown): “Following (Hinton, Vinyals, and Dean 2015), researchers pay more attention to the supervision from intermediate representations for better optimization performance (Romero et al. 2014; Wang et al. 2018; Shen et al. 2019a,b; Ye et al. 2020; Luo et al. 2020).”
• Collaboration by Competition: Self-coordinated Knowledge Amalgamation for Multi-talent Student Learning (2020, background): “Recently, some researches [20, 31, 12, 30] started to investigate how to transfer knowledge from multiple trained models into a single one with unlabeled dataset.”
• Source-Free Domain Adaptation for Semantic Segmentation (2021, methodology): “Most of the data-free KD methods based on generative adversarial networks [3, 49, 30, 11, 47].”
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “Fang [52] 7 7 One Noise/images Teacher + student 7 Ye [251] 7 7 Three Noise Teachers X”