Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks
By Nicolas Papernot, Patrick Drew McDaniel, Xi Wu, Somesh Jha, Ananthram Swami. IEEE Symposium on Security and Privacy, 2015
3,315 citations recorded by the source corpus.
Deep learning algorithms have been shown to perform extremely well on many classical machine learning problems. However, recent studies have shown that deep learning, like other machine learning techniques, is vulnerable to adversarial samples: inputs crafted to force a deep neural network (DNN) to provide adversary-selected outputs. Such attacks can seriously undermine the security of the system supported by the DNN, sometimes with devastating consequences. For example, autonomous vehicles can be crashed, illicit or illegal content can bypass content filters, or biometric authentication systems can be manipulated to allow improper access. In this work, we introduce a defensive mechanism called defensive distillation to reduce the effectiveness of adversarial samples on DNNs. We analytically investigate the generalizability and robustness properties granted by the use of defensive distillation when training DNNs. We also empirically study the effectiveness of our defense mechanisms on two DNNs placed in adversarial settings. The study shows that defensive distillation can reduce effectiveness of sample creation from 95% to less than 0.5% on a studied DNN. Such dramatic gains can be explained by the fact that distillation leads gradients used in adversarial sample creation to be reduced by a factor of 1030. We also find that distillation increases the average minimum number of features that need to be modified to create adversarial samples by about 800% on one of the DNNs we tested.
Subfields: Artificial Intelligence, Hardware and Architecture, Molecular Biology, Safety, Robustness & Interpretability
View Paper: https://arxiv.org/abs/1511.04508
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Yoshua Bengio
- Mohit Bansal
- Quoc V. Le
- Jianfeng Gao
- Bernt Schiele
- Sergey Levine
- Dacheng Tao
In Hardware and Architecture
- Nicholas Carlini
- Matthias Hein
- Bo Li
- J. Kolter
- Cho-Jui Hsieh
- Dawn Song
- Nicolas Papernot
- Ian J. Goodfellow
In Molecular Biology
- Nicholas Carlini
- Jun Zhu
- Nicolas Papernot
- Michael I. Jordan
- Cho-Jui Hsieh
- Patrick Drew McDaniel
- Tianyu Pang
- Dan Hendrycks
In Safety, Robustness & Interpretability
- Nicholas Carlini
- Matthias Hein
- Nicolas Papernot
- Cho-Jui Hsieh
- Huan Zhang
- Florian Tramèr
- Aleksander Madry
- Bo Li
Citations in the corpus
- Benchmarking Neural Network Robustness to Common Corruptions and Perturbations (2019, background): “In the literature, new defenses (Lu et al., 2017; Papernot et al., 2017; Metzen et al., 2017; Hendrycks & Gimpel, 2017a) often quickly succumb to new attacks (Evtimov et al.”
- Towards the Science of Security and Privacy in Machine Learning (2016, methodology): “In experiments with the fast gradient sign method [85] and the Jacobian attack [84], larger perturbations are required to achieve misclassification of adversarial examples by the distilled model.”
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “lutional networks. We consider the middle two architectures in Figure 1. Conv2 is a variation of lenet-5 [30] with two convolutional layers and three fully-connected layers. Conv4, which derives from [43] and [6], has four convolutional layers and three fully-connected layers. We investigate three q”
- Born Again Neural Networks (2018, methodology): “(Papernot et al., 2016) incorporate KD into an adversarial training scheme.”
- Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (2019, methodology): “, [10,25,26,38,41,48,64]) were explored to mitigate them.”
- Certified Robustness to Adversarial Examples with Differential Privacy (2018, methodology): “Model distillation, proposed as a robust defense in [44], was subsequently broken in [7].”
- Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning (2017, background): “[13] proposed an attack framework specifically aimed to assess the security of deep networks, and a defense mechanism (referred to as distillation) based on masking the gradient of deep networks to make gradient-based attacks against them ineffective [5].”
- Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks (2019, background): “A number of defenses have been proposed [43], [44], [45], [46], [47], yet many have shown to be less effective against an adaptive adversary [48], [49], [50], [51].”
- Wasserstein Adversarial Examples via Projected Sinkhorn Iterations (2019, background): “To name a few, distillation was proposed as a defense but was defeated (Papernot et al., 2016; Carlini & Wagner, 2017), realistic transformations seen by vehicles were thought to be safe until more robust adversarial examples were created (Lu et al., 2017; Athalye et al., 2018b), and many defenses…”
- Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks (2018, background): “A variety of defenses against adversarial inputs have been proposed [37,13] and broken [20,9,5]; research into defenses that provide strong guarantees of robustness is ongoing.”
- Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples (2018, background): “To the best of our knowledge, all defenses against adversarial examples published in peer-reviewed venues to date (Papernot et al., 2016; Hendrik Metzen et al., 2017; Hendrycks & Gimpel, 2017; Meng & Chen, 2017; Zantedeschi et al., 2017) are vulnerable to powerful optimization-based attacks (Carlini”
- Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models (2017, background): “…attacks is to mask the gradients, for example by adding non-differentiable elements either implicitly through means like defensive distillation (Papernot et al., 2016) or saturated non-linearities (Nayebi & Ganguli, 2017), or explicitly through means like non-differentiable classifiers (Lu et…”
- A Rotation and a Translation Suffice: Fooling CNNs with Simple Transformations (2017, background): “[20] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami.”
- ZOO: Zeroth Order Optimization Based Black-box Attacks to Deep Neural Networks without Training Substitute Models (2017, methodology): “mples, one natural defense mechanism is to hide the gradient information while training a DNN, known as gradient masking. A typical example of gradient masking is the defense distillation proposed in [38]. The authors proposed to retrain a DNN using distillation [15] based on the original confidence”
- Ensemble Adversarial Training: Attacks and Defenses (2017, background): “Two proposed defenses against adversarial examples (Defensive Distillation [16] and Saturated Networks [11]) hinder first-order attacks (in part) by rendering gradient computations numerically unstable [4, 3].”
- Adversarial Machine Learning at Scale (2016, positive): “Another existing approach is to use defensive distillation to train the network (Papernot et al., 2015).”
- Towards Evaluating the Robustness of Neural Networks (2016, methodology): “We evaluate the quality of the adversarial examples found on the MNIST and CIFAR datasets.”