Zero-Shot Knowledge Distillation in Deep Networks

By Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, R. Venkatesh Babu, Anirban Chakraborty. 2019

267 citations recorded by the source corpus.

Knowledge distillation deals with the problem of training a smaller model (Student) from a high capacity source model (Teacher) so as to retain most of its performance. Existing approaches use either the training data or meta-data extracted from it in order to train the Student. However, accessing the dataset on which the Teacher has been trained may not always be feasible if the dataset is very large or it poses privacy or safety concerns (e.g., bio-metric or medical data). Hence, in this paper, we propose a novel data-free method to train the Student from the Teacher. Without even using any meta-data, we synthesize the Data Impressions from the complex Teacher model and utilize these as surrogates for the original training data samples to transfer its learning to Student via knowledge distillation. We, therefore, dub our method "Zero-Shot Knowledge Distillation" and demonstrate that our framework results in competitive generalization performance as achieved by distillation using the actual training data samples on multiple benchmark datasets.

Subfields: Artificial Intelligence, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

Citations in the corpus

Zero-shot Knowledge Transfer via Adversarial Belief Matching (2019, methodology): “Concurrent to our work, Nayak et al. (2019) synthesize pseudo data from the weights of the teacher alone and use it to train a student in zero-shot.”
Knowledge Distillation: A Survey (2020, methodology): “To overcome the issue of unavailable data due to the problem of privacy, legality, security and confidentiality concerns, some data-free KD methods have been proposed (Chen et al., 2019b; Micaelli and Storkey, 2019; Lopes et al., 2017; Nayak et al., 2019).”
Class-Incremental Domain Adaptation (2020, methodology): “methods tackle the domain-shift problem, these require simultaneous access to both source and target domain samples, which makes them unsuitable in cases where the source training data is proprietary [24,32,34] (e.g. in a self-driving car), or simply unavailable during model upgrade [7,23,28]. Moreo”
The Knowledge Within: Methods for Data-Free Model Compression (2019, methodology): “, [22] proposed similar data free method dubbed as zero shot distillation, which relies solely on the final layer weights to compute a class similarity matrix.”
Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models (2021, background): “For example, data collected on personal information or medical information will not be open sourced, as mentioned by Nayak et al. (2019).”
A Meta-Learning Framework for Generalized Zero-Shot Learning (2019, methodology): “Because of the wide applicability and more realistic setting the ZSL framework also applied on the different domain like Zero-Shot Task Transfer [34], zero-shot sketch-based image retrieval [42, 23], zero-shot knowledge distillation [31], zero-shot action recognition [56, 13, 30, 28] etc.”
Source-Free Domain Adaptation for Semantic Segmentation (2021, methodology): “In [29, 33], activation records are used to reconstruct training samples for training a compact student model.”
Effectiveness of Arbitrary Transfer Sets for Data-free Knowledge Distillation (2020, methodology): “Table 2 (on left) presents the comparison of distillation performance of the proposed approach against ZSKD [17] and DeGAN [1] on multiple datasets.”
Dataset Condensation with Gradient Matching (2020, methodology): “Recent zero-shot KD methods [25, 26] aim to perform KD from a trained model in the absence of training data by generating synthetic data as the intermediate production to further use.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, background): “Nayak [169] 7 Class similarities 7 Class label+ Number of DIs 7 7”
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, unknown): “Current data-free KD methods can be roughly divided into two categories: adversarial training [77], [78], [79], which focuses on generating worstcase synthetic samples for student learning, and data prior matching [80], [81], [82], where synthetic samples are forced to satisfy priors like class prio”
Large-Scale Generative Data-Free Distillation (2020, methodology): “Follow-up works [43, 55] have developed the approach by using less meta-data or proposing different optimization objectives.”
Dataless Knowledge Fusion by Merging Weights of Language Models (2022, unknown): “Dataless distillation, although for computer vision architectures and not using Transformer-based approaches, was attempted in (Lopes et al., 2017; Nayak et al., 2019).”
Data-free Knowledge Distillation for Object Detection (2021, methodology): “[27] generate a proxy dataset by only modelling the inter-class information from softmax layer.”
Thieves on Sesame Street! Model Extraction of BERT-based APIs (2019, methodology): “Our work is related to prior work on data-efficient distillation, which attempts to distill knowledge from a larger model to a small model with access to limited input data (Li et al., 2018) or in a zeroshot setting (Micaelli & Storkey, 2019; Nayak et al., 2019).”
Progressive Network Grafting for Few-Shot Knowledge Distillation (2020, unknown): “ZSKD (Nayak and Chakraborty 2019) adapts synthetic data impressions from teacher model to replace original training data to achieve knowledge transfer.”