Adversarially Robust Distillation

By Micah Goldblum, Liam H. Fowl, Soheil Feizi, Tom Goldstein. 2019

270 citations recorded by the source corpus.

Knowledge distillation is effective for producing small, high-performance neural networks for classification, but these small networks are vulnerable to adversarial attacks. This paper studies how adversarial robustness transfers from teacher to student during knowledge distillation. We find that a large amount of robustness may be inherited by the student even when distilled on only clean images. Second, we introduce Adversarially Robust Distillation (ARD) for distilling robustness onto student networks. In addition to producing small models with high test accuracy like conventional distillation, ARD also passes the superior robustness of large networks onto the student. In our experiments, we find that ARD student models decisively outperform adversarially trained networks of identical architecture in terms of robust accuracy, surpassing state-of-the-art methods on standard robustness benchmarks. Finally, we adapt recent fast adversarial training methods to ARD for accelerated robust distillation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety Research

Citations in the corpus

Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, unknown): “To this end, recent works [3], [19], [31], [41], [61], [63], [84], [89], [126], [132], [135], [138], [140], [183], [197], [198], [227], [228], [229], [243], [244], [262] have tried to explore adversarial learning to improve the performance for KD.”
Defense without Forgetting: Continual Adversarial Defense with Anisotropic & Isotropic Pseudo Replay (2024, background): “Recently, adversarial training (AT) [16, 18, 27] and defensive distillation [15, 46, 46, 55] have become mainstream defense due to their essential robustness, and the latest research primarily focus on exploring their potential.”
Cross-Entropy Loss Functions: Theoretical Analysis and Applications (2023, methodology): “…et al., 2020), weight averaging/perturbation (Gowal et al., 2020; Wu et al., 2020; Tsai et al., 2021; Yu et al., 2018; Prabhu et al., 2019) and other techniques (Zhang & Wang, 2019; Qin et al., 2019; Goldblum et al., 2020; Song et al., 2019; Pang et al., 2020b; Lee et al., 2020; Qian et al., 2021).”
Queried Unlabeled Data Improves and Robustifies Class-Incremental Learning (2022, unknown): “Indeed, recent studies have identiﬁed such challenges, even in the much simpler case of transferring robustness from one source to another target domain (Hendrycks et al., 2019; Shafahi et al., 2020; Goldblum et al., 2020; Chan et al., 2019).”
Adversarially Robust Few-Shot Learning: A Meta-Learning Approach (2019, background): “A number of adversarial training variants have emerged which improve performance or achieve various tasks from data augmentation to domain generalization to model compression [28, 10, 23].”
Does Knowledge Distillation Really Work? (2021, background): “Knowledge distillation can improve model efficiency [30, 34], unsupervised domain adaptation [29], improved object detection [9], model transparency [37], and adversarial robustness [14, 33].”