Smooth Adversarial Training

By Cihang Xie, Mingxing Tan, Boqing Gong, Alan L. Yuille, Quoc V. Le. 2020

165 citations recorded by the source corpus.

It is commonly believed that networks cannot be both accurate and robust, that gaining robustness means losing accuracy. It is also generally believed that, unless making networks larger, network architectural elements would otherwise matter little in improving adversarial robustness. Here we present evidence to challenge these common beliefs by a careful study about adversarial training. Our key observation is that the widely-used ReLU activation function significantly weakens adversarial training due to its non-smooth nature. Hence we propose smooth adversarial training (SAT), in which we replace ReLU with its smooth approximations to strengthen adversarial training. The purpose of smooth activation functions in SAT is to allow it to find harder adversarial examples and compute better gradient updates during adversarial training. Compared to standard adversarial training, SAT improves adversarial robustness for "free", i.e., no drop in accuracy and no increase in computational cost. For example, without introducing additional computations, SAT significantly enhances ResNet-50's robustness from 33.0% to 42.3%, while also improving accuracy by 0.9% on ImageNet. SAT also works well with larger networks: it helps EfficientNet-L1 to achieve 82.2% accuracy and 58.6% robustness on ImageNet, outperforming the previous state-of-the-art defense by 9.5% for accuracy and 11.6% for robustness.

Subfields: Artificial Intelligence, Molecular Biology, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jun Zhu
2. Alan L. Yuille
3. Yoshua Bengio
4. Percy Liang
5. Tianyu Pang
6. Sergey Levine
7. Quoc V. Le
8. Hang Su

In Molecular Biology

1. Sven Gowal
2. Dan Hendrycks
3. Tianyu Pang
4. Xiao Yang
5. Jun Zhu
6. Yinpeng Dong
7. Hang Su
8. Timothy A. Mann

In Safety, Robustness & Interpretability

1. Tianyu Pang
2. Jun Zhu
3. Yinpeng Dong
4. Hang Su
5. Sven Gowal
6. Nicholas Carlini
7. Timothy A. Mann
8. Pushmeet Kohli

Citations in the corpus

• Towards Robust Vision Transformer (2021, background): “, stronger data augmentation [16, 18, 33], carefully designed [36,44] or searched [9,13] network architecture, improved training strategy [22, 26, 47], quantization [24] and pruning [49] of the weights, better pooling [41, 53] or activation functions [46], etc.”
• Unsolved Problems in ML Safety (2021, background): “Others could create architectures that are more adversarially robust [203].”
• Bag of Tricks for Adversarial Training (2020, methodology): “Most of the previous AT methods apply ReLU as the non-linear activation function in their models, while Xie et al. (2020) empirically demonstrate that smooth activation functions can better improve model robustness on ImageNet.”
• Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples (2020, background): “Other “smooth” activation functions (Xie et al., 2020b) do not necessarily correlate positively with robustness in our experiments.”
• Are Transformers More Robust Than CNNs? (2021, background): “However, when increasing the perturbation radius to 4/255, a more challenging but standard case studied in previous works [34, 46, 47], both models will be circumvented completely , i.e ., 0% robustness on defending against AutoAttack.”
• Recent Advances in Adversarial Training for Adversarial Robustness (2021, methodology): “Some replace the fundamental components of models for better performances, like hypersphere embedding [Pang et al. , 2020b], and smoothed ReLU function [Xie et al. , 2020].”
• Deep Repulsive Prototypes for Adversarial Robustness (2021, background): “Capacity is known to help adversarial robustness (Madry et al. 2017; Xie et al. 2020).”
• Regional Homogeneity: Towards Learning Transferable Universal Adversarial Perturbations Against Defenses (2019, background): “To mitigate this, large efforts have been devoted to adversarial defense [7, 34, 47, 72], via adversarial training [42, 48, 69, 76, 68, 77, 74], randomized transformation [14, 26, 45, 75] etc.”
• MedViT: A Robust Vision Transformer for Generalized Medical Image Classification (2023, background): “36], strong data augmentation [37, 38], searched network architecture [39, 40], improved training strategy [41–43], pruning [44] of the weights, and quantization [45], activation functions [46] or better pooling [47, 48].”
• An Effective Anti-Aliasing Approach for Residual Networks (2020, methodology): “Finally, we evaluate the impact of anti-aliasing relative to other architectural changes proposed for OOD problems, such as smooth activation functions [20].”