Adversarial Robustness with Semi-Infinite Constrained Learning

By Alexander Robey, Luiz F. O. Chamon, George Pappas, Hamed Hassani, Alejandro Ribeiro. 2021

50 citations recorded by the source corpus.

Despite strong performance in numerous applications, the fragility of deep learning to input perturbations has raised serious questions about its use in safety-critical domains. While adversarial training can mitigate this issue in practice, state-of-the-art methods are increasingly application-dependent, heuristic in nature, and suffer from fundamental trade-offs between nominal performance and robustness. Moreover, the problem of finding worst-case perturbations is non-convex and underparameterized, both of which engender a non-favorable optimization landscape. Thus, there is a gap between the theory and practice of adversarial training, particularly with respect to when and why adversarial training works. In this paper, we take a constrained learning approach to address these questions and to provide a theoretical foundation for robust learning. In particular, we leverage semi-infinite optimization and non-convex duality theory to show that adversarial training is equivalent to a statistical problem over perturbation distributions, which we characterize completely. Notably, we show that a myriad of previous robust training techniques can be recovered for particular, sub-optimal choices of these distributions. Using these insights, we then propose a hybrid Langevin Monte Carlo approach of which several common algorithms (e.g., PGD) are special cases. Finally, we show that our approach can mitigate the trade-off between nominal and robust performance, yielding state-of-the-art results on MNIST and CIFAR-10. Our code is available at: https://github.com/arobey1/advbench.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

The Ideal Continual Learner: An Agent That Never Forgets (2023, background): “Constrained learning naturally arises when learning under certain requirements, e.g., robustness (Zhang et al., 2019; Robey et al., 2021), safety (Paternain et al., 2019), or fairness (Cotter et al., 2019).”
Cross-Entropy Loss Functions: Theoretical Analysis and Applications (2023, background): “…2021; Bubeck et al., 2021), consistency guarantees and optimal adversarial classifiers (Bao et al., 2020; Awasthi et al., 2021a;c;b; 2022b;a; Meunier et al., 2022; Li & Telgarsky, 2023; Frank & Niles-Weed, 2023) and optimization (Awasthi et al., 2019; Robey et al., 2021; Awasthi et al., 2023a).”