Unlabeled Data Improves Adversarial Robustness

By Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, John C. Duchi. 2019

814 citations recorded by the source corpus.

We demonstrate, theoretically and empirically, that adversarial robustness can significantly benefit from semisupervised learning. Theoretically, we revisit the simple Gaussian model of Schmidt et al. that shows a sample complexity gap between standard and robust classification. We prove that unlabeled data bridges this gap: a simple semisupervised learning procedure (self-training) achieves high robust accuracy using the same number of labels required for achieving high standard accuracy. Empirically, we augment CIFAR-10 with 500K unlabeled images sourced from 80 Million Tiny Images and use robust self-training to outperform state-of-the-art robust accuracies by over 5 points in (i) $\ell_\infty$ robustness against several strong attacks via adversarial training and (ii) certified $\ell_2$ and $\ell_\infty$ robustness via randomized smoothing. On SVHN, adding the dataset's own extra training set with the labels removed provides gains of 4 to 10 points, within 1 point of the gain from using the extra labels.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Self-Training With Noisy Student Improves ImageNet Classification (2019, positive): “Our finding is consistent with similar arguments that using unlabeled data can improve adversarial robustness [8, 64, 46, 80].”
Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, background): “Self-training has recently been shown to help adversarial robustness as well [10, 61, 94].”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “Adversarial Training Many efforts have been devoted to improving AT from different angles: (i) use triplet-wise metric learning [8, 7] and optimal transport [20] to leverage inter-sample interactions; (ii) exploit extra unlabeled training data [12, 1]; and (iii) accelerate the training procedure [11”
Unsolved Problems in ML Safety (2021, background): “Researchers could enhance data for adversarial robustness by simulating more data [208], augmenting data [151], repurposing existing real data [31, 80], and extracting more information from available data [82].”
Differentially Private Learning Needs Better Features (or Much More Data) (2020, methodology): “To obtain a larger dataset comparable to CIFAR-10, we use 500K pseudo-labeled Tiny Images5 (Torralba et al., 2008) collected by Carmon et al. (2019).6 We then train private models on subsets of size 10,000 ≤ N ≤ 550,000 from this dataset.”
Adversarial Examples Improve Image Recognition (2019, background): “Other works try to explain this tradeoff phenomenon from the perspective of the increased sample complexity of adversary [37, 25, 28], the limited amount of training data [1, 27, 34, 44, 48], or network overparameterization [31].”
Are Labels Required for Improving Adversarial Robustness? (2019, background): “The observation that adversarial robustness can be optimized without labels was made independently and concurrently by [10, 33, 53].”
Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers (2019, methodology): “Specifically, we pursue two ideas: 1) pre-training similar to [17], and 2) semi-supervised learning as in [5].”
An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models (2020, background): “Similarly, recent work has shown that semi-supervised learning with generic auxiliary data improves model robustness to adversarial examples (Schmidt et al., 2018; Carmon et al., 2019).”
Rethinking the Value of Labels for Improving Class-Imbalanced Learning (2020, methodology): “[8] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang.”
Improving Robustness using Generated Data (2021, methodology): “We highlight the works by Carmon et al. [11], Najaﬁ et al. [52], Uesato et al. [73], Zhai et al. [81] who simultaneously proposed the use of additional unlabeled external data.”
Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples (2020, methodology): “While they analyze some properties that we also analyze in this manuscript (such as training batch size, label smoothing, weight decay, activation functions), they also complement our analyses with experiments on early stopping and perturbation radius warm-up, optimizers, model architectures beyond ”
Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift (2022, methodology): “We ﬁnd that ID-calibrated ensembles outperform prior approaches based on self-training [Carmon et al., 2019, Uesato et al., 2019, Xie et al., 2021], despite not using any additional unlabeled data.”
Better Diffusion Models Further Improve Adversarial Training (2023, methodology): “When the training set is dramatically expanded, using a large amount of external data (Carmon et al., 2019; Alayrac et al., 2019) or synthetic data (Gowal et al., 2021), significant improvements in both clean and robust accuracy are observed.”
Cross-Entropy Loss Functions: Theoretical Analysis and Applications (2023, background): “, 2022), unlabeled data (Carmon et al., 2019; Alayrac et al., 2019; Zhai et al., 2019), data augmentation (Rebuffi et al.”