Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbations

By Florian Tramèr, Jens Behrmann, Nicholas Carlini, Nicolas Papernot, Jörn-Henrik Jacobsen. 2020

104 citations recorded by the source corpus.

Adversarial examples are malicious inputs crafted to induce misclassification. Commonly studied sensitivity-based adversarial examples introduce semantically-small changes to an input that result in a different model prediction. This paper studies a complementary failure mode, invariance-based adversarial examples, that introduce minimal semantic changes that modify an input's true label yet preserve the model's prediction. We demonstrate fundamental tradeoffs between these two types of adversarial examples. We show that defenses against sensitivity-based attacks actively harm a model's accuracy on invariance-based attacks, and that new approaches are needed to resist both attack types. In particular, we break state-of-the-art adversarially-trained and certifiably-robust models by generating small perturbations that the models are (provably) robust to, yet that change an input's class according to human labelers. Finally, we formally show that the existence of excessively invariant classifiers arises from the presence of overly-robust predictive features in standard datasets.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Adversarial Robustness with Semi-Infinite Constrained Learning (2021, background): “For common choices of the hypothesis class H, including DNNs, classifiers obtained by solving (P-NOM) are known to be sensitive to small, normbounded input perturbations [61].”
Partial success in closing the gap between human and machine vision (2021, background): “(5)This might be related to [68], who studied a potentially related tradeoff between selectivity and invariance.”
Improving Robustness using Generated Data (2021, background): “We note that while robustness and invariance to input perturbations is crucial to the deployment of machine learning models in various applications, it can also have broader negative impacts to society such as hindering privacy [64] or increasing bias [69].”
Blind Backdoors in Deep Learning Models (2020, background): “Further, certified defenses are not robust against attacks that use a different metric than the defense [89] and can break a model [88] because some small changes—e.”