Towards the first adversarially robust neural network model on MNIST

By Lukas Schott, Jonas Rauber, Matthias Bethge, Wieland Brendel. 2018

391 citations recorded by the source corpus.

Despite much effort, deep neural networks remain highly susceptible to tiny input perturbations and even for MNIST, one of the most common toy datasets in computer vision, no neural network model exists for which adversarial perturbations are large and make semantic sense to humans. We show that even the widely recognized and by far most successful defense by Madry et al. (1) overfits on the L-infinity metric (it's highly susceptible to L2 and L0 perturbations), (2) classifies unrecognizable images with high certainty, (3) performs not much better than simple input binarization and (4) features adversarial perturbations that make little sense to humans. These results suggest that MNIST is far from being solved in terms of adversarial robustness. We present a novel robust classification model that performs analysis by synthesis using learned class-conditional data distributions. We derive bounds on the robustness and go to great length to empirically evaluate our model using maximally effective adversarial attacks by (a) applying decision-based, score-based, gradient-based and transfer-based attacks for several different Lp norms, (b) by designing a new attack that exploits the structure of our defended model and (c) by devising a novel decision-based attack that seeks to minimize the number of perturbed pixels (L0). The results suggest that our approach yields state-of-the-art robustness on MNIST against L0, L2 and L-infinity perturbations and we demonstrate that most adversarial examples are strongly perturbed towards the perceptual boundary between the original and the adversarial class.

Subfields: Artificial Intelligence, Molecular Biology, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Shortcut learning in deep neural networks (2020, background): “Adversarially robust models are somewhat more aligned with humans and show promising generalisation abilities [129, 130].”
Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One (2019, methodology): “To analyze this property, we follow Schott et al. (2018).”
Towards Deep Learning Models Resistant to Adversarial Attacks (2017, background): “Indeed, subsequent work [20, 25] has found that PGD is in fact overestimating the `2-robustness of this model.”
Toward Causal Representation Learning (2021, background): “11 Recent work supports this view: it was shown that a possible defense against adversarial attacks is to solve the anticausal classiﬁcation problem by modeling the causal generative direction, a method that, in vision, is referred to as analysis by synthesis [222].”
Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbations (2020, methodology): “Note that the Binary-ABS defense of (Schott et al., 2019) boasts 60% (empirical) robust accuracy on `1-attacks with " = 0.4 (see (Schott et al., 2019)).”
Adversarial Training and Robustness for Multiple Perturbations (2019, methodology): “Extending prior observations [25, 31, 23], we show that models trained against an l∞-adversary learn representations that mask gradients for attacks in other lp-norms.”
Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness (2019, methodology): “In fact, MNIST is the only dataset for which robustness to adversarial examples is considered even remotely close to being solved (Schott et al., 2019) and researchers working on (provable) robustness to adversarial examples have moved on to other, larger vision datasets such as CIFAR-10 (Madry et…”
Towards Causal Representation Learning (2021, background): “10 Recent work supports this view: it was shown that a possible defense against adversarial attacks is to solve the anticausal classification problem by modeling the causal generative direction, a method which in vision is referred to as analysis by synthesis [222].”
Causality for Machine Learning (2019, background): “Recent work supports this view: it was shown that a possible defense against adversarial attacks is to solve the anticausal classification problem by modeling the causal generative direction, a method which in vision is referred to as analysis by synthesis (Schott et al., 2019).”
Towards neural networks that provably know when they don't know (2019, background): “Worst-case aspects of OOD detection have previously been studied in Nguyen et al. (2015); Schott et al. (2018); Hein et al. (2019); Sehwag et al. (2019), but no robustness guarantees have yet been proposed for this setting.”
Excessive Invariance Causes Adversarial Vulnerability (2018, neutral): “While classifying MNIST digits is considered a toy example, it has been shown to be challenging from the perspective of (cid:15) -robustness (Schott et al., 2018).”
Invertible Residual Networks (2018, background): “• Adversarial robustness with likelihood-based generative models (Schott et al., 2019; Jacobsen et al., 2019)”
On Evaluating Adversarial Robustness (2019, background): “Also, remember that there is no universally strongest attack: each attack makes specific assumptions about the model and so an attack that is best on one model might perform much worse on another (Schott et al., 2019).”
Engineering a Less Artificial Intelligence. (2019, background): “However, a recent paper (Schott et al., 2019) showed that the defending network does not learn more causal, human-like features but instead just exploits the binary nature of the dataset (MNIST, a collection of handwritten digits) and is thus unlikely to generalize to all natural images.”
Batch Normalization is a Cause of Adversarial Vulnerability (2019, background): “This has led to a variety of methods that aim to improve robustness, but doing so effectively remains a challenge (Athalye et al., 2018; Schott et al., 2019; Hendrycks & Dietterich, 2019; Jacobsen et al., 2019).”
Prototype-based Neural Network Layers: Incorporating Vector Quantization (2018, methodology): “Surprisingly, if we follow the results in [9], one of the most robust methods on MNIST is a simple knearest-neighbor classifier, which is a predecessor of Learning Vector Quantization.”
Recent Advances in Adversarial Training for Adversarial Robustness (2021, background): “…various defense strategies, Adversarial Training (AT) [Goodfellow et al. , 2015; Madry et al. , 2018] currently proves to be the most effective against adversarial attacks [Pang et al. , 2020a; Maini et al. , 2020; Schott et al. , 2019], receiving considerable attention from the research community.”