Bag of Tricks for Adversarial Training
By Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, Jun Zhu. International Conference on Learning Representations, 2020
306 citations recorded by the source corpus.
Adversarial training (AT) is one of the most effective strategies for promoting model robustness. However, recent benchmarks show that most of the proposed improvements on AT are less effective than simply early stopping the training procedure. This counter-intuitive fact motivates us to investigate the implementation details of tens of AT methods. Surprisingly, we find that the basic training settings (e.g., weight decay, learning rate schedule, etc.) used in these methods are highly inconsistent, which could largely affect the model performance as shown in our experiments. For example, a slightly different value of weight decay can reduce the model robust accuracy by more than 7%, which is probable to override the potential promotion induced by the proposed methods. In this work, we provide comprehensive evaluations on the effects of basic training tricks and hyperparameter settings for adversarially trained models. We provide a reasonable baseline setting and re-implement previous defenses to achieve new state-of-the-art results.
Subfields: Artificial Intelligence, Emergency Medicine
View Paper: https://www.semanticscholar.org/paper/f1209119572225d3bf3ec0e4f277fa4fcd187b8c
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Emergency Medicine
Citations in the corpus
- Cross-Entropy Loss Functions: Theoretical Analysis and Applications (2023, background): “ADVCOMP-SUM with the smallest model WRN-28-10 surpasses (Pang et al., 2020; Rice et al., 2020; Qin et al., 2019).”
- Boosting Adversarial Training with Hypersphere Embedding (2020, methodology): “We train WRN models via PGD-AT+HE, with weight decay of 5× 10−4 [47].”
- Better Diffusion Models Further Improve Adversarial Training (2023, methodology): “Pang et al. (2021) investigate a wide range of batch sizes, from 64 to 512, and find that 128 is optimal for CIFAR-10.”
- Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples (2020, background): “Since Madry et al. (2017), various modifications to their algorithm have been proposed (Xie et al., 2018; Pang et al., 2020b; Huang et al., 2020; Qin et al., 2019; Zoran et al., 2019; Andriushchenko & Flammarion, 2020).”
- Recent Advances in Adversarial Training for Adversarial Robustness (2021, methodology): “Some replace the fundamental components of models for better performances, like hypersphere embedding [Pang et al. , 2020b], and smoothed ReLU function [Xie et al. , 2020].”
- Defense without Forgetting: Continual Adversarial Defense with Anisotropic & Isotropic Pseudo Replay (2024, unknown): “In adversarial training, Jia et al. [18] proposed a learnable strategy, Mustafa et al. [32] enhanced adversarial robustness by perturbing feature representations, and Pang et al. [33] fully utilized tricks to maximize the potential of AT.”
- Relating Adversarially Robust Generalization to Flat Minima (2021, background): “In fact, “standard” AT is shown to perform surprisingly well on recent benchmarks [11, 10] when tuning hyper-parameters properly [42, 18].”
- Smooth Adversarial Training (2020, background): “As suggested in [45, 70], adversarial training on CIFAR-10 is highly sensitive to different parameter settings, therefore we leave the exploring of the “optimal” adversarial training setting with ELU and SILU on CIFAR-10 as a future work.”