Are Labels Required for Improving Adversarial Robustness?

By Jonathan Uesato, Jean-Baptiste Alayrac, Po-Sen Huang, Robert Stanforth, Alhussein Fawzi, Pushmeet Kohli. 2019

361 citations recorded by the source corpus.

Recent work has uncovered the interesting (and somewhat surprising) finding that training models to be invariant to adversarial perturbations requires substantially larger datasets than those required for standard classification. This result is a key hurdle in the deployment of robust machine learning models in many real world applications where labeled data is expensive. Our main insight is that unlabeled data can be a competitive alternative to labeled data for training adversarially robust models. Theoretically, we show that in a simple statistical setting, the sample complexity for learning an adversarially robust model from unlabeled data matches the fully supervised case up to constant factors. On standard datasets like CIFAR-10, a simple Unsupervised Adversarial Training (UAT) approach using unlabeled data improves robust accuracy by 21.7% over using 4K supervised examples alone, and captures over 95% of the improvement from the same number of labeled examples. Finally, we report an improvement of 4% over the previous state-of-the-art on CIFAR-10 against the strongest known attack by using additional unlabeled data from the uncurated 80 Million Tiny Images dataset. This demonstrates that our finding extends as well to the more realistic case where unlabeled data is also uncurated, therefore opening a new avenue for improving adversarial training.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Safety, Robustness & Interpretability

Citations in the corpus

Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, background): “[94] Uesato, J.”
Do Image Classifiers Generalize Across Time? (2019, background): “etoahuman. Adversarialattacksinthe‘ p modelarepowerfulanddiﬃculttodefend against. Forinstance,thestateoftheartdefensesstillachievemediocreclassiﬁcationaccuracieson adversarialinputs(below65%onCIFAR-10[1,6,22,31,34]). Motivated by the artiﬁcial nature of ‘ p attacks, recent work has proposed more rea”
Adversarial Examples Improve Image Recognition (2019, background): “Other works try to explain this tradeoff phenomenon from the perspective of the increased sample complexity of adversary [37, 25, 28], the limited amount of training data [1, 27, 34, 44, 48], or network overparameterization [31].”
Unlabeled Data Improves Adversarial Robustness (2019, methodology): “[41] also study semisupervised adversarial learning theoretically in the Gaussian model of [35] and empirically via experiments on CIFAR-10, Tiny Images, and SVHN.”
Overfitting in adversarially robust deep learning (2020, methodology): “Semi-supervised learning methods augment the dataset with unlabeled data, and have been shown to improve generalization when used in the adversarially robust setting (Carmon et al., 2019; Zhai et al., 2019; Alayrac et al., 2019).”
Metric Learning for Adversarial Robustness (2019, methodology): “In addition, unlabeled data [39] and model ensemble [37, 25] have been shown to improve the robustness of the model.”
Adversarial Training Can Hurt Generalization (2019, background): “Furthermore, we show that we can eliminate the tradeoff in our constructed problem using the recently-proposed robust self-training [15, 1, 8, 18] on additional unlabeled data.”
Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning (2020, methodology): “Both works [31, 4] use unlabeled data to form an unsupervised auxiliary loss (e.”
Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks (2020, methodology): “91% for the model of (Alayrac et al., 2019), so that it is better than the value reported in the original paper, and of 0.”
Better Diffusion Models Further Improve Adversarial Training (2023, methodology): “When the training set is dramatically expanded, using a large amount of external data (Carmon et al., 2019; Alayrac et al., 2019) or synthetic data (Gowal et al., 2021), significant improvements in both clean and robust accuracy are observed.”
Boosting Adversarial Training with Hypersphere Embedding (2020, background): “…(Shafahi et al., 2019; Zhang et al., 2019a; Wong et al., 2020) and exploiting extra labeled and unlabeled training data (Hendrycks et al., 2019; Alayrac et al., 2019; Carmon et al., 2019), which are effective in the cases with limited computational resources or additional data accessibility.”