Data-Free Learning of Student Networks

By Hanting Chen, Yunhe Wang, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, Qi Tian. 2019

440 citations recorded by the source corpus.

Learning portable neural networks is very essential for computer vision for the purpose that pre-trained heavy deep models can be well applied on edge devices such as mobile phones and micro sensors. Most existing deep neural network compression and speed-up methods are very effective for training compact deep models, when we can directly access the training dataset. However, training data for the given deep network are often unavailable due to some practice problems (\eg privacy, legal issue, and transmission), and the architecture of the given network are also unknown except some interfaces. To this end, we propose a novel framework for training efficient deep neural networks by exploiting generative adversarial networks (GANs). To be specific, the pre-trained teacher networks are regarded as a fixed discriminator and the generator is utilized for derivating training samples which can obtain the maximum response on the discriminator. Then, an efficient network with smaller model size and computational complexity is trained using the generated data and the teacher network, simultaneously. Efficient student networks learned using the proposed Data-Free Learning (DFL) method achieve 92.22% and 74.47% accuracies without any training data on the CIFAR-10 and CIFAR-100 datasets, respectively. Meanwhile, our student network obtains an 80.56% accuracy on the CelebA benchmark.

Subfields: Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, positive): “During the search process, if we jointly update α and γ , we observe that our model is prone to suffering from the same “architecture collapse” problem observed in (Chen et al., 2019b), i.e., NAS is quickly biased towards some low-latency yet low-performance models.”
Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replay (2022, background): “[4,20] alternatively uses a GAN architecture to synthesize images, where they fix a trained network as a discriminator and optimize a generator to derive images that can be adopted to distill knowledge from the fixed network to a new network.”
Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning (2021, methodology): “The Data-Free Learning method [10] exploits a GAN architecture to synthesize images which match the trained teacher’s statistics while balancing content losses which maximize both temperature scaled linear heads (to drive class-specific content high) and class prediction entropy (to encourage high d”
Knowledge Distillation: A Survey (2020, methodology): “Furthermore, the knowledge transfer from one model to another in knowledge distillation can be extended to other tasks, such as adversarial attacks (Papernot et al. 2016), data augmentation (Lee et al. 2019a; Gordon and Duh 2019), data privacy and security (Wang et al. 2019a).”
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion (2019, methodology): “Chen et al. [7] trained a new generator network for image generation while treating the teacher network as a ﬁxed discriminator.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “An early work in this area [28] exploits Generative Adversarial Networks (GANs) [75] for synthetic data generation.”
Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning (2022, methodology): “DAFL [2] and DFAD [6] use a generator to generate data efficiently, where DAFL optimizes the generator by maximizing the response on prediction and feature level, and DFAD uses an adversarial training scheme to exploit the knowledge in teacher model effectively.”
Memory-Free Generative Replay For Class-Incremental Learning (2021, methodology): “For these requirements, only the first part can be solved by DAFL [5], and we use it as our baseline method.”
GhostNet: More Features From Cheap Operations (2019, background): “Knowledge distillation [19, 12, 3] utilizes larger models to teach smaller ones, which improves the performance of smaller models.”
Large-Scale Generative Data-Free Distillation (2020, methodology): “There are also a few methods that synthesize input data via generative image modeling [5, 11, 41], which create substitute data much more efficiently than optimizing input noise.”
Source-Free Domain Adaptation for Semantic Segmentation (2021, methodology): “More recently, data-free knowledge distillation [29, 3] has drawn surging attention, due to the inevitable data privacy issue.”
Few Shot Network Compression via Cross Distillation (2019, methodology): “2019), feature map statistics (Lopes, Fenu, and Starner 2017; Bhardwaj, Suda, and Marculescu 2019), or an independently trained generative model (Chen et al. 2019) to simulate the distribution of the original training set.”
Federated Mutual Learning (2020, background): “, 2018), or generated by generative adversarial network (GAN)(Chen et al., 2019).”
CARS: Continuous Evolution for Efficient Neural Architecture Search (2019, methodology): “The efficient block design [54, 53, 52], tensor decomposition [24, 49, 50, 42, 43], pruning [25, 26, 12], distillation [4, 48] and quantization [22] are popular techniques to make networks efficient.”
Effectiveness of Arbitrary Transfer Sets for Data-free Knowledge Distillation (2020, methodology): “[15, 4, 1] show that properly optimized generative models can generate samples to be strongly classified by the Teacher models.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “To handle this problem, some representative data-free KD paradigms [111], [112], [113], [114], [115], [116], [117], [118], [119], [120], [121] are newly developed.”