Zero-shot Knowledge Transfer via Adversarial Belief Matching

By Paul Micaelli, Amos J. Storkey. 2019

257 citations recorded by the source corpus.

Performing knowledge transfer from a large teacher network to a smaller student is a popular task in modern deep learning applications. However, due to growing dataset sizes and stricter privacy regulations, it is increasingly common not to have access to the data that was used to train the teacher. We propose a novel method which trains a student to match the predictions of its teacher without using any data or metadata. We achieve this by training an adversarial generator to search for images on which the student poorly matches the teacher, and then using them to train the student. Our resulting student closely approximates its teacher for simple datasets like SVHN, and on CIFAR10 we improve on the state-of-the-art for few-shot distillation (with 100 images per class), despite using no data. Finally, we also propose a metric to quantify the degree of belief matching between teacher and student in the vicinity of decision boundaries, and observe a significantly higher match between our zero-shot student and the teacher, than between a student distilled with real data and the teacher. Code available at: this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replay (2022, methodology): “Note that we adopt mean square error (MSE) as objective in both phases other than KL divergence adopted in [20], because logit matching has better generalization capacity [16].”
Knowledge Distillation: A Survey (2020, methodology): “Furthermore, Micaelli and Storkey (2019) utilized an adversarial generator to generate hard examples for knowledge transfer.”
Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks (2019, methodology): “the model [10] which exploits the increased confidence of the model on the training data to reconstruct images used for training; [23] proposes a method for performing zero-shot knowledge distillation by adversarially generating a set of exciting images to train a student network.”
High-Fidelity Extraction of Neural Network Models (2019, background): “For other adversaries, progress in generative modeling is likely to offer ways to remove this assumption [29].”
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, unknown): “Current data-free KD methods can be roughly divided into two categories: adversarial training [77], [78], [79], which focuses on generating worstcase synthetic samples for student learning, and data prior matching [80], [81], [82], where synthetic samples are forced to satisfy priors like class prio”
See through Gradients: Image Batch Recovery via GradInversion (2021, background): “Multiple works have also looked into training GANs given only a pretrained model [6, 34], but result in images that lack details or perceptual similarities to original data.”
High Accuracy and High Fidelity Extraction of Neural Networks (2019, background): “This is a natural assumption for a theft-motivated adversary who wishes to steal the oracle for local use—the adversary has data they want to learn the labels of without querying the model! For other adversaries, progress in generative modeling is likely to offer ways to remove this assumption [29].”
Progressive Network Grafting for Few-Shot Knowledge Distillation (2020, unknown): “ZSKT (Micaelli and Storkey 2019) and DFAD (Fang et al. 2019) introduce an adversarial strategy to synthesize training samples for knowledge distillation.”
Self-Distillation as Instance-Specific Label Smoothing (2020, background): “Recently, data-free distillation, a novel scenario in which the original data for the teacher is unavailable to students, has also been extensively studied [6, 7, 25, 40].”
Thieves on Sesame Street! Model Extraction of BERT-based APIs (2019, methodology): “…work (Tramèr et al., 2016) reports successful extraction on SVMs and 1-layer networks using i.i.d noise, but no prior work has scaled this idea to deeper neural networks for which a single class tends to dominate model predictions on most noise inputs (Micaelli & Storkey, 2019; Pal et al., 2019).”
Effectiveness of Arbitrary Transfer Sets for Data-free Knowledge Distillation (2020, background): “[17, 15]) since it is common now-a-days that many popular pre-trained models are released without providing access to the training data (e.”
Adversarial Self-Supervised Data Free Distillation for Text Classification (2020, background): “Exploratory researches (Micaelli and Storkey, 2019; Fang et al., 2019) also show that GANs can synthesize harder and more diversified images by exploiting disagreements between teachers and students.”
A Survey of Machine Unlearning (2022, unknown): “Hence, inspired by [107], Chundawat et al.”
Large-Scale Generative Data-Free Distillation (2020, methodology): “There are also a few methods that synthesize input data via generative image modeling [5, 11, 41], which create substitute data much more efﬁciently than optimizing input noise.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “Considering the limitation of metadata and similarity-based distillation methods, some works [31], [52], [157], [251], [255], [257] propose novel data-free KD methods via adversarial learning [65], [222], [224].”
Principles and Practice of Explainable Machine Learning (2020, background): “The idea of transferring knowledge from a complex model (the teacher ) to a simpler one (the student ) been explored in other works, for example [37, 12, 58].”
Data-free Knowledge Distillation for Object Detection (2021, background): “[5] and Micaelli and Storkey [25] reformulate the classification network as a discriminator and train an external generator network to synthesize images that maximize the discriminator’s response.”
Dataset Inference: Ownership Resolution in Machine Learning (2021, methodology): “The adversary has complete access to the victim model, and uses data-free knowledge transfer (Micaelli & Storkey, 2019; Fang et al., 2019) to train a student model.”