Feature Denoising for Improving Adversarial Robustness

By Cihang Xie, Yuxin Wu, L. V. D. van der Maaten, Alan L. Yuille, Kaiming He. 2018

1,015 citations recorded by the source corpus.

Adversarial attacks to image classification systems present challenges to convolutional networks and opportunities for understanding them. This study suggests that adversarial perturbations on images lead to noise in the features constructed by these networks. Motivated by this observation, we develop new network architectures that increase adversarial robustness by performing feature denoising. Specifically, our networks contain blocks that denoise the features using non-local means or other filters; the entire networks are trained end-to-end. When combined with adversarial training, our feature denoising networks substantially improve the state-of-the-art in adversarial robustness in both white-box and black-box attack settings. On ImageNet, under 10-iteration PGD white-box attacks where prior art has 27.9% accuracy, our method achieves 55.7%; even under extreme 2000-iteration PGD white-box attacks, our method secures 42.6% accuracy. Our method was ranked first in Competition on Adversarial Attacks and Defenses (CAAD) 2018 --- it achieved 50.6% classification accuracy on a secret, ImageNet-like test dataset against 48 unknown attackers, surpassing the runner-up approach by ~10%. Code is available at https://github.com/facebookresearch/ImageNet-Adversarial-Training.

Subfields: Artificial Intelligence, Molecular Biology, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Safety, Robustness & Interpretability

Citations in the corpus

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, background): “Additionally, CNNs augmented with non-local means [10] are shown to be more robust to adversarial attacks [89].”
Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, methodology): “This category includes 86 models with an explicit robustness intervention such as adversarially robust models [13, 27, 72, 74, 101], models with special data augmentation [20, 28, 34, 41, 100, 108, 113], and models with architecture modiﬁcations [115].”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “, trained models are able to resist adversarial attacks on clean images at the expense of performance [42, 73, 84].”
Natural Adversarial Examples (2019, background): “Even with both model scale and self-attention, classiﬁer accuracies are still below that of adversarially trained classiﬁers on typical (cid:96) ∞ adversarial examples [44], so that natural adversarial examples are in some sense currently more challenging.”
Do Image Classifiers Generalize Across Time? (2019, methodology): “We evaluate the model from [31], which currently performs best against l∞-attacks on ImageNet.”
Ensemble Adversarial Training: Attacks and Defenses (2017, neutral): “We further note that adversarial training with multi-step attacks has now been scaled to Ima-geNet (Xie et al., 2019a), resulting in models with plausible and non-negligible robustness to white-box attacks , thereby superseding the results obtained with Ensemble Adversarial Training.”
Using Pre-Training Can Improve Model Robustness and Uncertainty (2019, methodology): “Instead, we choose to adversarially pre-train a Downsampled ImageNet model against an untargeted adversary, contra Kurakin et al. (2017); Kannan et al. (2018); Xie et al. (2018).”
Dynamic Graph CNN for Learning on Point Clouds (2018, background): “…among neighboring edge features. a connection to existing work, Non-local Neural Networks [Wang et al. 2018a] explored similar ideas in the video recognition field, and follow-up work by Xie et al. [2018] proposed using non-local blocks to denoise feature maps to defend against adversarial attacks.”
Theoretically Principled Trade-off between Robustness and Accuracy (2019, methodology): “, feature denoising technique (Xie et al., 2018) and network architecture design (Cisse et al.”
Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty (2019, methodology): “In all experiments, we use 40-2 Wide Residual Networks Zagoruyko and Komodakis [2016].”
Bottleneck Transformers for Visual Recognition (2021, background): “Prior work [65] has shown that adding Non-Local blocks to ResNets and training them using canonical settings does not provide substantial gains.”
Adversarial Examples Improve Image Recognition (2019, background): “However, such trained models usually cannot generalize well to clean images as shown in [23, 45].”
FreeLB: Enhanced Adversarial Training for Natural Language Understanding (2019, background): “While adversarial training boosts the robustness, it is widely accepted by computer vision researchers that it is at odds with generalization, with classification accuracy on non-corrupted images dropping as much as 10% on CIFAR-10, and 15% on Imagenet (Madry et al., 2018; Xie et al., 2019).”
Transfer of Adversarial Robustness Between Perturbation Types (2019, background): “tion, considerable work has focused on robustness to “L∞bounded” perturbations (perturbations with bounded perpixel magnitude) (Goodfellow et al., 2014; Madry et al., 2017; Xie et al., 2018).”
A Simple Way to Make Neural Networks Robust Against Diverse Image Corruptions (2020, result): “This result contradicts our ﬁndings on full ImageNet as we see a slight decline in accuracy on those two classes for the adversarially trained model from [42] and severe drops in accuracy on other corruptions.”
Understanding Adversarial Attacks on Deep Learning Based Medical Image Analysis Systems (2019, background): “0/255 for targeted attacks to generally succeed (Madry et al., 2018b; Xie et al., 2019; Wang et al., 2019a).”