The Limitations of Deep Learning in Adversarial Settings
By Nicolas Papernot, Patrick Drew McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, Ananthram Swami. 2015
4,317 citations recorded by the source corpus.
Deep learning takes advantage of large datasets and computationally efficient training algorithms to outperform other approaches at various machine learning tasks. However, imperfections in the training phase of deep neural networks make them vulnerable to adversarial samples: inputs crafted by adversaries with the intent of causing deep neural networks to misclassify. In this work, we formalize the space of adversaries against deep neural networks (DNNs) and introduce a novel class of algorithms to craft adversarial samples based on a precise understanding of the mapping between inputs and outputs of DNNs. In an application to computer vision, we show that our algorithms can reliably produce samples correctly classified by human subjects but misclassified in specific targets by a DNN with a 97% adversarial success rate while only modifying on average 4.02% of the input features per sample. We then evaluate the vulnerability of different sample classes to adversarial perturbations by defining a hardness measure. Finally, we describe preliminary work outlining defenses against adversarial samples by defining a predictive measure of distance between a benign input and a target classification.
Subfields: Artificial Intelligence, Safety Research, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Nicholas Carlini
- Bernt Schiele
- Yoshua Bengio
- Mohit Bansal
- Quoc V. Le
- Sergey Levine
- Jianfeng Gao
In Safety Research
- Nicolas Papernot
- Patrick Drew McDaniel
- Nicholas Carlini
- Ian J. Goodfellow
- Xiaoyu Cao
- Reza Shokri
- Michael Backes
- Jamie Hayes
In Safety, Robustness & Interpretability
- Nicholas Carlini
- Matthias Hein
- Nicolas Papernot
- Florian Tramèr
- Bo Li
- Cho-Jui Hsieh
- Bernt Schiele
- Aleksander Madry
Citations in the corpus
- Towards the Science of Security and Privacy in Machine Learning (2016, methodology): “For the purpose of exposition of the following Sections, we expand upon previous approaches at articulating a threat model for ML [25], [26].”
- Towards Deep Learning Models Resistant to Adversarial Attacks (2017, background): “Due to the growing body of work on adversarial examples [8, 6, 23, 17, 3, 25, 7, 12], we focus only on the most related papers here.”
- Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (2019, methodology): “Our threat model is inspired by multiple existing studies [30, 47, 48, 56] on adversarial machine learning.”
- Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning (2017, background): “[66] show that an adversarially crafted input can be fed to deep learning models and make them prone to error, i.”
- Certified Robustness to Adversarial Examples with Differential Privacy (2018, background): “Attacks have used various norms to quantify an attacker’s ability to perturb an image: the 0-norm [43], 1-norm [28], [7], 2-norm [53], [7], [2], and ∞-norm [19], [59], [7], [2], [35].”
- Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks (2015, methodology): “3: Adversarial crafting framework: Existing algorithms for adversarial sample crafting [7], [9] are a succession of two steps: (1) direction sensitivity estimation and (2) perturbation selection.”
- Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning (2017, background): “3In [14], the authors defined targeted and indiscriminate attacks (at test time) depending on whether the attacker aims to cause specific or generic errors.”
- Explaining Explanations: An Overview of Interpretability of Machine Learning (2018, background): “Extending this observation, a number of techniques have been shown for changing a network’s classification of any image to any target class by making imperceptible alterations to the pixels [3], [4], [5].”
- Deep Variational Information Bottleneck (2017, background): “…attacks fall into three broad categories: optimization-based attacks (Szegedy et al., 2014; Carlini & Wagner, 2016; Moosavi-Dezfooli et al., 2016; Papernot et al., 2015; Robinson & Graham, 2015; Sabour et al., 2016), which directly run an optimizer such as L-BFGS or ADAM (Kingma & Ba, 2015) on…”
- Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, background): “We establish theoretically that if a model is vulnerable to adversarial examples, such as the ones presented in [28, 29, 30, 31, 32], then, under mild conditions, backdoor attacks are unavoidable.”
- Privacy Risks of Securing Machine Learning Models against Adversarial Examples (2019, background): “d into two categories: evasion attacks and poisoning attacks [22]. Evasion attacks, also known as adversarial examples, perturb inputs at the test time to induce wrong predictions by the target model [5, 8, 15, 38, 56]. In contrast, poisoning attacks target the training process by maliciously modify”
- Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks (2019, background): “These can be applied to DNNs during inference [38], [39], [40], [41], [42].”
- Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models (2017, methodology): “This includes black-box variants of JSMA (Narodytska & Kasiviswanathan, 2016) and of the Carlini & Wagner attack (Chen et al., 2017) as well as generator networks that predict adversarials (Hayes & Danezis, 2017).”
- ZOO: Zeroth Order Optimization Based Black-box Attacks to Deep Neural Networks without Training Substitute Models (2017, methodology): “Each iteration, JSMA recomputes the saliency map and uses the derivative of the DNN with respect to the input image as an indicator of modification for adversarial attacks.”
- Ensemble Adversarial Training: Attacks and Defenses (2017, background): “Machine learning (ML) models are often vulnerable to adversarial examples, maliciously perturbed inputs designed to mislead a model at test time [2, 19, 5, 14].”
- Contamination Attacks and Mitigation in Multi-Party Machine Learning (2018, background): “Our attacks exploit misaligned goals between parties in multi-party machine learning as opposed to exploiting vulnerabilities within the model itself, such as with adversarial examples [7, 13, 24, 34, 30].”
- Model-Reuse Attacks on Deep Learning Systems (2018, background): “One line of work focuses on developing new evasion attacks against deep neural networks (DNNs) [14, 25, 29, 44].”