The Knowledge Within: Methods for Data-Free Model Compression

By Matan Haroush, Itay Hubara, Elad Hoffer, Daniel Soudry. 2019

117 citations recorded by the source corpus.

Background: Recently, an extensive amount of research has been focused on compressing and accelerating Deep Neural Networks (DNN). So far, high compression rate algorithms require part of the training dataset for a low precision calibration, or a fine-tuning process. However, this requirement is unacceptable when the data is unavailable or contains sensitive information, as in medical and biometric use-cases. Contributions: We present three methods for generating synthetic samples from trained models. Then, we demonstrate how these samples can be used to calibrate and fine-tune quantized models without using any real data in the process. Our best performing method has a negligible accuracy degradation compared to the original training set. This method, which leverages intrinsic batch normalization layers' statistics of the trained model, can be used to evaluate data similarity. Our approach opens a path towards genuine data-free model compression, alleviating the need for training data during model deployment.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning (2021, methodology): “Importantly: although we optimize Fφ rather than X̂ as done in [18, 62], this method can use the latter with a sacrifice to computational efficiency.”
Knowledge Distillation: A Survey (2020, background): “…have been proposed to overcome problems with unavailable data arising from privacy, legality, security and conﬁdentiality concerns (Chen et al. 2019a; Lopes et al. 2017; Nayak et al. 2019; Micaelli and Storkey 2019; Haroush et al. 2020; Ye et al. 2020; Nayak et al. 2021; Chawla et al. 2021).”
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion (2019, methodology): “work, the proposed paradigm of data-free knowledge distillation via model inversion easily scales to other applications, such as the task of data-free network quantization as independently studied in [19, 7]. Haroush et al. [19] explore The Knowledge Within a trained network for inverted images towa”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “However, Level 2 usually results in higher accuracy, as ﬁnetuning helps the quantized model to recover the accuracy degradation, particularly in ultra-low bit precision settings [84].”
Large-Scale Generative Data-Free Distillation (2020, methodology): “ term to constrain the statistics of the teacher’s intermediate layers as well. Batch normalization [26] layers, a common component of most neural networks, are helpful with providing such statistics [21,55]. The normalization operation is designed to normalize layer activations by re-centering and ”
Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming (2020, background): “Past years have seen the rapid development of efficient deployment techniques (Nagel et al., 2019; Haroush et al., 2019).”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, background): “Haroush [78] X Batch normalization layer 7 Image shape 7 7”
Accurate Post Training Quantization With Small Calibration Sets (2021, background): “Past years have seen the rapid development of efﬁcient deployment techniques (Nagel et al., 2019; Haroush et al., 2019).”
See through Gradients: Image Batch Recovery via GradInversion (2021, unknown): “[5, 17, 42, 48].”
ZeroQ: A Novel Zero Shot Quantization Framework (2020, methodology): “A recent concurrent paper to ours independently proposed to use Batch Normalization statistics to reconstruct input data [12].”