Data-Free Knowledge Distillation for Deep Neural Networks

By Raphael Gontijo Lopes, Stefano Fenu, Thad Starner. 2017

305 citations recorded by the source corpus.

Recent advances in model compression have provided procedures for compressing large neural networks to a fraction of their original size while retaining most if not all of their accuracy. However, all of these approaches rely on access to the original training set, which might not always be possible if the network to be compressed was trained on a very large dataset, or on a dataset whose release poses privacy or safety concerns as may be the case for biometrics tasks. We present a method for data-free knowledge distillation, which is able to compress deep neural networks trained on large-scale datasets to a fraction of their size leveraging only some extra metadata to be provided with a pretrained model release. We also explore different kinds of metadata that can be used with our method, and discuss tradeoffs involved in using each of them.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Dataset Distillation (2018, neutral): “Similar to our approach, data-free knowledge distillation also optimizes synthetic data samples, but with a different objective of matching activation statistics of a teacher model in knowledge distillation (Lopes et al., 2017).”
Zero-shot Knowledge Transfer via Adversarial Belief Matching (2019, background): “Lopes et al. (2017) showed that training images could be reconstructed from a record of the teacher’s training activations, but in practice releasing training activations instead of data is unlikely.”
Zero-Shot Knowledge Distillation in Deep Networks (2019, methodology): “• Using meta data : In (Lopes et al., 2017), activation records are stored at each layer after the training of teacher model and used as meta data to reconstruct training samples and utilize them to train the student model.”
Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning (2021, methodology): “Another early method [37] matches stored layer statistics from a trained “teacher” model while leaving a small memory footprint using frequency-based compression techniques.”
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion (2019, background): “[33] synthesized inputs based on pre-stored auxiliary layer-wise statistics of the teacher network.”
Relational Knowledge Distillation (2019, background): “[17] introduce data-free KD that utilizes metadata of a teacher model.”
Data-Free Learning of Student Networks (2019, methodology): “58% accuracy and “meta data” [16] achieves a higher accuracy of 91.”
Class-Incremental Domain Adaptation (2020, methodology): “methods tackle the domain-shift problem, these require simultaneous access to both source and target domain samples, which makes them unsuitable in cases where the source training data is proprietary [24,32,34] (e.g. in a self-driving car), or simply unavailable during model upgrade [7,23,28]. Moreo”
Few-shot learning of neural networks from scratch by pseudo example optimization (2018, background): “Other previous researches demonstrates that knowledge can be transferred from either intermediate layers of a teacher model [28, 37], from multiple teacher models [38], mismatched unsupervised stimuli [20] or examples reconstructed from meta-data [10].”
The Knowledge Within: Methods for Data-Free Model Compression (2019, methodology): “Previous work explored using data-free knowledge distillation for model compression by collecting metadata related to the statistics of a trained model output [17, 3].”
Data-free Knowledge Distillation for Object Detection (2021, methodology): “Additionally, unlike previous data-free distillation approaches that only work on image classification models [2, 22, 28], we distill deep object detector networks.”
Dream Distillation: A Data-Independent Model Compression Framework (2019, methodology): “Consequently, the industries deploying large deep learning models at the edge must compress them without access to any original, private, or alternate datasets1 (Lopes et al., 2017).”
A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning (2022, background): “[46, 8] release the burden of exemplars by data-free knowledge distillation [33].”
Effectiveness of Arbitrary Transfer Sets for Data-free Knowledge Distillation (2020, background): “[14]) additionally need to store meta-data about the original training dataset (e.”
A Survey on Generative Diffusion Models (2022, background): “1) Knowledge Distillation: Knowledge distillation, a technique for transferring ”knowledge” from larger to simpler models, is becoming increasingly popular [107], [108].”
Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN (2020, methodology): “[19] attempt to regenerate batches of data based on meta data collected at training time describing the activation of the network, rather than relying on the original dataset to guide the optimization.”
Dataless Knowledge Fusion by Merging Weights of Language Models (2022, unknown): “Dataless distillation, although for computer vision architectures and not using Transformer-based approaches, was attempted in (Lopes et al., 2017; Nayak et al., 2019).”