Precise Tradeoffs in Adversarial Training for Linear Regression

By Adel Javanmard, M. Soltanolkotabi, Hamed Hassani. 2020

124 citations recorded by the source corpus.

Despite breakthrough performance, modern learning models are known to be highly vulnerable to small adversarial perturbations in their inputs. While a wide variety of recent \emph{adversarial training} methods have been effective at improving robustness to perturbed inputs (robust accuracy), often this benefit is accompanied by a decrease in accuracy on benign inputs (standard accuracy), leading to a tradeoff between often competing objectives. Complicating matters further, recent empirical evidence suggest that a variety of other factors (size and quality of training data, model size, etc.) affect this tradeoff in somewhat surprising ways. In this paper we provide a precise and comprehensive understanding of the role of adversarial training in the context of linear regression with Gaussian features. In particular, we characterize the fundamental tradeoff between the accuracies achievable by any algorithm regardless of computational power or size of the training data. Furthermore, we precisely characterize the standard/robust accuracy and the corresponding tradeoff achieved by a contemporary mini-max adversarial training approach in a high-dimensional regime where the number of data points and the parameters of the model grow in proportion to each other. Our theory for adversarial training algorithms also facilitates the rigorous study of how a variety of factors (size and quality of training data, model overparametrization etc.) affect the tradeoff between these two competing accuracies.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Feature Purification: How Adversarial Training Performs Robust Deep Learning (2020, background): “We emphasize that prior theoretical works [35, 51, 84, 112] mainly study adversarial examples in the context of linear models (such as linear regression, linear regression over prescribed feature mappings, or the neural tangent kernels).”
Adversarial Robustness with Semi-Infinite Constrained Learning (2021, background): “[51] Adel Javanmard, Mahdi Soltanolkotabi, and Hamed Hassani.”
SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks (2023, unknown): “In adversarial examples research, several results on both the empirical and theoretical side point to the fact that higher robustness often comes at the cost of degraded nominal performance [40–42].”