Adversarial Training Versus Weight Decay

By Angus Galloway, Thomas Tanay, Graham W. Taylor. 2018

23 citations recorded by the source corpus.

Performance-critical machine learning models should be robust to input perturbations not seen during training. Adversarial training is a method for improving a model's robustness to some perturbations by including them in the training process, but this tends to exacerbate other vulnerabilities of the model. The adversarial training framework has the effect of translating the data with respect to the cost function, while weight decay has a scaling effect. Although weight decay could be considered a crude regularization technique, it appears superior to adversarial training as it remains stable over a broader range of regimes and reduces all generalization errors. Equipped with these abstractions, we provide key baseline results and methodology for characterizing robustness. The two approaches can be combined to yield one small model that demonstrates good robustness to several white-box attacks associated with different metrics.

Subfields: Artificial Intelligence, Optimization & Training, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Nicholas Carlini
2. Ludwig Schmidt
3. Dacheng Tao
4. Quoc V. Le
5. Percy Liang
6. B. Recht
7. Vaishaal Shankar
8. Achal Dave

In Optimization & Training

1. Ludwig Schmidt
2. Nicholas Carlini
3. B. Recht
4. Vaishaal Shankar
5. Tongliang Liu
6. Dacheng Tao
7. Graham W. Taylor
8. Medhat A. Moussa

In Safety, Robustness & Interpretability

1. Nicholas Carlini
2. Ludwig Schmidt
3. Graham W. Taylor
4. Thomas Tanay
5. Angus Galloway
6. Vaishaal Shankar
7. B. Recht
8. Achal Dave

Citations in the corpus

• Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, background): “Prior work often evaluates on synthetic distribution shifts to measure robustness [9, 32, 38].”
• Why ResNet Works? Residuals Generalize (2019, background): “Other technical advances and empirical analysis include [56]–[59].”
• Batch Normalization is a Cause of Adversarial Vulnerability (2019, background): “We believe this gap could be made larger by training a baseline with greater (cid:96) 2 regularization (Galloway et al., 2018).”
• On Adversarial Robustness of Small vs Large Batch Training (2019, background): “While Galloway et al. (2018) observe that weight decay itself can give a robust network which generalized better than robustness achieved by adversarial training.”