Adversarial Machine Learning at Scale
By Alexey Kurakin, Ian J. Goodfellow, Samy Bengio. 2016
3,411 citations recorded by the source corpus.
Adversarial examples are malicious inputs designed to fool machine learning models. They often transfer from one model to another, allowing attackers to mount black box attacks without knowledge of the target model's parameters. Adversarial training is the process of explicitly training a model on adversarial examples, in order to make it more robust to attack or to reduce its test error on clean inputs. So far, adversarial training has primarily been applied to small problems. In this research, we apply adversarial training to ImageNet. Our contributions include: (1) recommendations for how to succesfully scale adversarial training to large models and datasets, (2) the observation that adversarial training confers robustness to single-step attack methods, (3) the finding that multi-step attack methods are somewhat less transferable than single-step attack methods, so single-step attacks are the best for mounting black-box attacks, and (4) resolution of a "label leaking" effect that causes adversarially trained models to perform better on adversarial examples than on clean examples, because the adversarial example construction process uses the true label and the model can learn to exploit regularities in the construction process.
Subfields: Artificial Intelligence, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Percy Liang
- Quoc V. Le
- Mohit Bansal
- Sergey Levine
- Jianfeng Gao
- Ruslan Salakhutdinov
- Bernt Schiele
In Safety, Robustness & Interpretability
- Nicholas Carlini
- Matthias Hein
- Huan Zhang
- Cho-Jui Hsieh
- Aleksander Madry
- Florian Tramèr
- Tianyu Pang
- Jun Zhu
Citations in the corpus
- Re-imagining Algorithmic Fairness in India and Beyond (2021, background): “perturbation testing [156], data augmentation [220], adversarial testing [117], and adherence to terminology guidelines for oppressed groups, such as SIGACCESS.”
- Natural Adversarial Examples (2019, background): “Increasing network capacity has been shown to improve performance on p adversarial examples [37], common corruptions [26], and now also improves performance for adversarially filtered images.”
- Benchmarking Neural Network Robustness to Common Corruptions and Perturbations (2019, background): “These deceptive distortions can occasionally fool black-box classifiers (Kurakin et al., 2017).”
- Towards Deep Learning Models Resistant to Adversarial Attacks (2017, background): “Due to the growing body of work on adversarial examples (Gu & Rigazio, 2014; Fawzi et al., 2015; Torkamani, 2016; Papernot et al., 2016; Carlini & Wagner, 2016a; Tramèr et al., 2017b; Goodfellow et al., 2014; Kurakin et al., 2016), we focus only on the most related papers here.”
- Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (2016, background): “[33], for improving predictive uncertainty in the semi-supervised setting as well.”
- Adversarial Training and Robustness for Multiple Perturbations (2019, methodology): “We show that Projected Gradient Descent [22, 25] is inefficient in the `1-case, and design a new attack, Sparse `1 Descent (SLIDE), that is both efficient and competitive with strong optimization attacks [8], We experiment with MNIST and CIFAR10.”
- Adversarial Training for Free! (2019, background): “It is believed that increased network capacity leads to greater robustness from adversarial training [Madry et al., 2017, Kurakin et al., 2016b].”
- Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks (2019, background): “These can be applied to DNNs during inference [38], [39], [40], [41], [42].”
- Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples (2018, methodology): “As benchmarking against iterative optimization-based attacks (e.g., Kurakin et al. (2016a); Madry et al. (2018); Carlini & Wagner (2017c)) has become standard practice in evaluating defenses, new defenses have arisen that appear to be robust against these powerful optimization-based attacks.”
- Attacking the Madry Defense Model with L1-based Adversarial Examples (2017, methodology): “Iterative fast gradient methods (I-FGM) were proposed in (Kurakin et al., 2016), which iteratively use FGM with a finer distortion, followed by an (cid:15) -ball clipping.”
- ZOO: Zeroth Order Optimization Based Black-box Attacks to Deep Neural Networks without Training Substitute Models (2017, background): “baseline algorithm for improved adversarial example generation [21, 22], and it can be viewed as an attack framework based on first-order projected gradient descent [27].”
- Ensemble Adversarial Training: Attacks and Defenses (2017, methodology): “While training models to be robust to all white-box attacks appears out-of-reach using current techniques [8], our work gives promising new directions and results towards securing ML systems against black-box adversaries.”
- Mitigating Evasion Attacks to Deep Neural Networks via Region-based Classification (2017, background): “Several variants [14, 18, 22, 34, 37] of adversarial training were also proposed.”
- Certified Adversarial Robustness via Randomized Smoothing (2019, background): “Empirical defenses The most successful empirical defense to date is adversarial training (Kurakin et al., 2017; Madry et al., 2018), in which a neural network is trained to optimize the worst-case loss over balls around the training data.”
- Formal Security Analysis of Neural Networks using Symbolic Intervals (2018, methodology): “Existing methods to test DNNs against corner cases focus on finding adversarial examples [7, 11, 27, 28, 33, 35, 37, 38, 44] without providing formal guarantees about the non-existence of adversarial inputs even within very small input ranges.”
- Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach (2018, methodology): “We apply the state-of-the-art white-box attack methods, iterative fast gradient sign method (I-FGSM) (Goodfellow et al., 2015; Kurakin et al., 2016b) and Carlini and Wagner’s attack (CW) (Carlini & Wagner, 2017b), to find adversarial examples for 11 networks, including 4 networks trained on CIFAR, 4…”
- Towards Robust Neural Networks via Random Self-ensemble (2017, methodology): “We do not compare the results on MNIST since it is a much easier dataset and existing defense methods such as [16, 27, 12, 11] can effectively increase image distortion under adversarial attacks.”
- Mitigating adversarial effects through randomization (2017, methodology): “Recently, adversarial training (Kurakin et al., 2017; Tramèr et al., 2017) was developed to defend against single-step attacks.”