Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion

By Hongxu Yin, Pavlo Molchanov, Zhizhong Li, José Manuel Álvarez, Arun Mallya, Derek Hoiem, Niraj Kumar Jha, Jan Kautz. 2019

706 citations recorded by the source corpus.

We introduce DeepInversion, a new method for synthesizing images from the image distribution used to train a deep neural network. We ``invert'' a trained network (teacher) to synthesize class-conditional input images starting from random noise, without using any additional information about the training dataset. Keeping the teacher fixed, our method optimizes the input while regularizing the distribution of intermediate feature maps using information stored in the batch normalization layers of the teacher. Further, we improve the diversity of synthesized images using Adaptive DeepInversion, which maximizes the Jensen-Shannon divergence between the teacher and student network logits. The resulting synthesized images from networks trained on the CIFAR-10 and ImageNet datasets demonstrate high fidelity and degree of realism, and help enable a new breed of data-free applications - ones that do not require any real images or labeled data. We demonstrate the applicability of our proposed method to three tasks of immense practical importance - (i) data-free network pruning, (ii) data-free knowledge transfer, and (iii) data-free continual learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “An intriguing idea is to replay synthetic data produced by inverting a frozen copy of the old classification model, such as DeepInversion [476], ABD”
FBNetV3: Joint Architecture-Recipe Search using Neural Acquisition Function (2020, methodology): “One method was pruning [9, 5, 46, 2], where either layers or channels were removed according to certain heuristics.”
Generative Negative Text Replay for Continual Vision-Language Pretraining (2022, methodology): “This is inspired by DeepInversion [51], but they do not consider generating negative data points and it is designed for the convolution network with Batch Normalization.”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “Replay-based methods [42, 19, 13, 51, 6, 7, 33, 8, 30, 2, 47, 46, 54] store a subset of each previous task in a separate memory, and replay the tasks when learning a new one, to directly preserve the knowledge on those tasks.”
Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replay (2022, methodology): “Recently, [26,30,29] have integrated the idea of generative reply into addressing class-incremental learning problem using Deep-Inversion.”
Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning (2022, background): “Recently, some works [36, 38, 44] focus on a challenging but practical non-exemplar class-incremental learning problem, where no past data can be stored due to equipment limits or privacy security.”
Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning (2021, methodology): “The closest existing work for the DFCIL problem is DeepInversion [62], which optimizes random noise into images for knowledge distillation using a frozen teacher network.”
Knowledge Distillation: A Survey (2020, background): “Yin et al. (2020) proposed DeepInversion, which uses knowledge distillation to generate synthesized images for data-free knowledge transfer.”
Few-Shot Class-Incremental Learning for Named Entity Recognition (2022, methodology): “Such a setting was previously explored for model compression of image classification (Yin et al., 2020) and text classification (Ma et al.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Model distillation [3, 94, 148, 173, 189, 200, 260, 262, 280] is a method in which a large model with higher accuracy is used as a teacher to help the training of a compact student model.”
I-BERT: Integer-only BERT Quantization (2021, neutral): “These techniques can be broadly categorized into (1) model pruning [18, 31, 35, 38, 40, 67], (2) knowledge distillation [21, 39, 43, 49, 70], (3) efficient neural architecture design [23, 24, 37, 51, 57], (4) hardware and neural architecture co-design [16, 17, 22, 29, 64], and (5) quantization [5, 7”
The Knowledge Within: Methods for Data-Free Model Compression (2019, methodology): “lation. These papers explore interesting ideas such as adaptive sample generation and layer sensitivity heuristics for mixed-precision, while their results further support our ﬁndings. In contrast to [29], which presents a wide range of experiments on data-free applications such as classical KD and ”
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, neutral): “Motivated by recent advances in data-free knowledge distillation [23], [82], SFDA-KTMA [284] exploits the moving average statistics of activations stored in BN layers of the pre-trained source model and imposes the following BN”
Resurrecting Old Classes with New Data for Exemplar-Free Continual Learning (2024, background): “This generation of adversarial samples is computationally cheaper and much faster (only a few iterations) compared to data-inversion methods [51] which inverts embeddings to realistic images.”
Large-Scale Generative Data-Free Distillation (2020, methodology): “h success in practice on large-scale data. It was also observed that while the image are of extremely high 2 quality, using them exclusively for training leads to a signiﬁcant performance degradation [55]. Another promising approach to image synthesis is based on recent work on reversible networks [”
A Unified Continual Learning Framework with General Parameter-Efficient Tuning (2023, positive): “…do not rely on old samples that may raise privacy concerns, they dynamically expand the network or isolate parameters for different tasks [41, 50, 26, 18], regularize the network parameters that are important to learned tasks [28, 20, 52, 1], and replay generative or synthetic data [42, 49, 43, 8].”
Unsolved Problems in ML Safety (2021, background): “[204] Hongxu Yin, Pavlo Molchanov, Zhizhong Li, J.”