Invariance-inducing regularization using worst-case transformations suffices to boost accuracy and spatial robustness

By Fanny Yang, Zuowen Wang, Christina Heinze-Deml. 2019

46 citations recorded by the source corpus.

This work provides theoretical and empirical evidence that invariance-inducing regularizers can increase predictive accuracy for worst-case spatial transformations (spatial robustness). Evaluated on these adversarially transformed examples, we demonstrate that adding regularization on top of standard or adversarial training reduces the relative error by 20% for CIFAR10 without increasing the computational cost. This outperforms handcrafted networks that were explicitly designed to be spatial-equivariant. Furthermore, we observe for SVHN, known to have inherent variance in orientation, that robust training also improves standard accuracy on the test set. We prove that this no-trade-off phenomenon holds for adversarial examples from transformation groups in the infinite data limit.

Subfields: Artificial Intelligence, Materials Chemistry, Optimization & Training, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Percy Liang
2. Tatsunori B. Hashimoto
3. Pang Wei Koh
4. Aditi Raghunathan
5. Mohit Bansal
6. Shiori Sagawa
7. Luke S. Zettlemoyer
8. Noah A. Smith

In Materials Chemistry

1. Percy Liang

In Optimization & Training

1. Percy Liang
2. Tatsunori B. Hashimoto
3. Pang Wei Koh
4. Shiori Sagawa
5. Fanny Yang
6. John C. Duchi
7. Aditi Raghunathan
8. Sang Michael Xie

In Safety, Robustness & Interpretability

1. Percy Liang
2. Pang Wei Koh
3. Aditi Raghunathan
4. Shiori Sagawa
5. Tatsunori B. Hashimoto
6. Fanny Yang
7. John C. Duchi
8. Sang Michael Xie

Citations in the corpus

• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization (2019, background): “…prediction models, mostly from the causal inference literature, similarly aim to achieve high performance on a range of test distributions (Peters et al., 2016; B¨uhlmann & Meinshausen, 2016; Heinze-Deml & Meinshausen, 2017; Rothenh¨ausler et al., 2018; Yang et al., 2019; Arjovsky et al., 2019).”
• Understanding and Mitigating the Tradeoff Between Robustness and Accuracy (2020, methodology): “We use two common forms of robust training for spatial perturbations, where we approximately maximize over T (x) with either adversarial (worst-of-10) or random augmentations (Yang et al., 2019; Engstrom et al., 2019).”