On Detecting Adversarial Perturbations

By Jan Hendrik Metzen, Tim Genewein, Volker Fischer, Bastian Bischoff. 2017

1,037 citations recorded by the source corpus.

Machine learning and deep learning in particular has advanced tremendously on perceptual tasks in recent years. However, it remains vulnerable against adversarial perturbations of the input that have been crafted specifically to fool the system while being quasi-imperceptible to a human. In this work, we propose to augment deep neural networks with a small "detector" subnetwork which is trained on the binary classification task of distinguishing genuine data from data containing adversarial perturbations. Our method is orthogonal to prior work on addressing adversarial perturbations, which has mostly focused on making the classification network itself more robust. We show empirically that adversarial perturbations can be detected surprisingly well even though they are quasi-imperceptible to humans. Moreover, while the detectors have been trained to detect only a specific adversary, they generalize to similar and weaker adversaries. In addition, we propose an adversarial attack that fools both the classifier and the detector and a novel training procedure for the detector that counteracts this attack.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations (2019, background): “In the literature, new defenses (Lu et al., 2017; Papernot et al., 2017; Metzen et al., 2017; Hendrycks & Gimpel, 2017a) often quickly succumb to new attacks (Evtimov et al., 2017; Carlini & Wagner, 2017; 2016), with some exceptions for perturbations on small images (Schott et al., 2018; Madry et…”
Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (2019, neutral): “, [10,25,26,38,41,48,64]) were explored to mitigate them.”
Certified Robustness to Adversarial Examples with Differential Privacy (2018, methodology): “These defenses include model distialltion [44], automated detection of adversarial examples [22], [41], [40], application of different input transformations [27], [10], randomization [21], [11], and application of generative models [49], [25], [60].”
SentiNet: Detecting Localized Universal Attacks Against Deep Learning Systems (2018, methodology): “Metzen et al. [17] use a similar approach by training a modiﬁed target classiﬁcation network to detect adversarial perturbations.”
ZOO: Zeroth Order Optimization Based Black-box Attacks to Deep Neural Networks without Training Substitute Models (2017, background): “readers can refer to recent works in [10, 13, 18, 28, 42, 43] and references therein for details.”
Mitigating Evasion Attacks to Deep Neural Networks via Region-based Classification (2017, background): “We note that the second limitation also applies to other approaches [4, 13] that aim to detect adversarial examples.”
Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks (2017, methodology): “Multiple recent studies [25, 11, 9] focused on detecting adversarial examples.”
On the (Statistical) Detection of Adversarial Examples (2017, background): “This work was supported by the German Federal Ministry of Education and Research (BMBF) through funding for the Center for IT-Security, Privacy and Accountability (CISPA) (FKZ: 16KIS0753).”
Mitigating adversarial effects through randomization (2017, background): “Recent works (Papernot et al., 2016b; Kurakin et al., 2017; Tram`er et al., 2017; Metzen et al., 2017; Feinman et al., 2017; Meng & Chen, 2017) are making progress on this line of research.”
Provable defenses against adversarial examples via the convex outer adversarial polytope (2017, background): “Even detecting the presence of adversarial examples is challenging (Metzen et al., 2017; Carlini & Wagner, 2017a), and attacks are not limited to synthetic examples, having been demonstrated repeatedly on real-world objects (Sharif et al.”
PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples (2017, background): “Most recent work on detecting adversarial examples focuses on adding an outlier class detection module to the classiﬁer, such as Grosse et al. (2017), Gong et al. (2017) and Metzen et al. (2017).”
NO Need to Worry about Adversarial Examples in Object Detection in Autonomous Vehicles (2017, background): “In the past few years, researchers tried to explain why neural nets are susceptible to such examples despite their impressive success on random test datasets [6, 5, 2], suggested new methods to generate adversarial examples and measure NN robustness against them [4, 21, 16, 9, 17, 19, 13], and propo”
MagNet: A Two-Pronged Defense against Adversarial Examples (2017, methodology): “An representative work of this idea is [20].”
Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods (2017, methodology): “We study ten detection schemes proposed in seven papers over the last year [4, 11, 12, 15, 18, 19, 24], and compare their efficacy with the other defenses in a consistent manner.”
DeepXplore: Automated Whitebox Testing of Deep Learning Systems (2017, background): “This observation has inspired several researchers to try to improve the robustness and reliability of DNNs [9, 13, 17, 30, 32, 46, 51, 53, 62, 84, 89, 90].”
Efficient Defenses Against Adversarial Attacks (2017, methodology): “To this end, a certain number of directions have been explored, such as performing statistical tests [13], using an additional model for detection [11, 26] or applying dropout [36] at test time [9].”
SafetyNet: Detecting and Rejecting Adversarial Examples Robustly (2017, methodology): “[17]); (d) transfer attacks work because an example that generates unfamiliar patterns in one network tends to generate unfamiliar patterns in other networks too; (e) transfer attacks could be defended as well (section 3).”
Adversarial Examples for Semantic Segmentation and Object Detection (2017, background): “and showed that this brings robustness to adversarial attack. This is imporved by [34], which proposed an ensemble adversarial training method to increase the network robustness to black-box attacks. [21] trained a detector on the inner layer of the classiﬁer to detect adversarial examples. There ar”