Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty

By Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, Dawn Song. 2019

1,072 citations recorded by the source corpus.

Self-supervision provides effective representations for downstream tasks without requiring labels. However, existing approaches lag behind fully supervised training and are often not thought beneficial beyond obviating the need for annotations. We find that self-supervision can benefit robustness in a variety of ways, including robustness to adversarial examples, label corruption, and common input corruptions. Additionally, self-supervision greatly benefits out-of-distribution detection on difficult, near-distribution outliers, so much so that it exceeds the performance of fully supervised methods. These results demonstrate the promise of self-supervision for improving robustness and uncertainty estimation and establish these tasks as new axes of evaluation for future self-supervised learning research.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Pretrained Transformers Improve Out-of-Distribution Robustness (2020, methodology): “Similar to computer vision (Orhan, 2019; Xie et al., 2020; Hendrycks et al., 2019a), pretraining on larger and more diverse datasets can improve robustness.”
Pretraining Language Models with Human Preferences (2023, background): “Moreover, reducing the diversity of training data can negatively impact alignment with human preferences by decreasing robustness (Hendrycks et al., 2019; 2020) and amplifying existing social biases (Xu et al.”
On Robustness and Transferability of Convolutional Neural Networks (2020, background): “Auxiliary self-supervision [7, 71] can improve robustness to label noise and common corruptions [28].”
Natural Adversarial Examples (2019, background): “For out-of-distribution (OOD) detection [27, 39, 28, 29] models learn a distribution, such as the ImageNet-1K distribution, and are tasked with producing quality anomaly scores that distinguish between usual test set examples and examples from held-out anomalous distributions.”
Unsolved Problems in ML Safety (2021, background): “Researchers could enhance data for adversarial robustness by simulating more data [208], augmenting data [151], repurposing existing real data [31, 80], and extracting more information from available data [82].”
S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learning (2023, methodology): “…can notably improve the performance of downstream tasks like few-shot learning [13], semi-supervised learning [43], to improve the model robustness [14], class imbalance [40], etc. Recently, Lee et al. [22] used SSL to improve the performance for supervised classification, by augmenting the…”
A Theoretical Study on Solving Continual Learning (2022, background): “[81] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song.”
Test-Time Training for Out-of-Distribution Generalization (2019, methodology): “Separate from our method, it is interesting to note that joint training consistently improves over the baseline, as discovered by Hendrycks et al. (2019b).”
PANDA: Adapting Pretrained Features for Anomaly Detection and Segmentation (2020, background): “We compare our PANDA-OE to the OE baseline in [15] on CIFAR10, as the code or results for other classes were unavailable.”
Continual Learning Based on OOD Detection and Task Masking (2022, methodology): “The OOD detection algorithm in CLOM is inspired by several recent advances in self-supervised learning, data augmentation [21], contrastive learning [11, 48], and their applications to OOD detection [18, 23, 65].”
Self-supervised Label Augmentation via Input Transformations (2019, methodology): “, 2020), improving robustness (Hendrycks et al., 2019), and training generative adversarial networks (Chen et al.”
Neurosymbolic Programming (2021, background): “, 2020), which is closely related to the concepts of self-supervision (Hendrycks et al., 2019; Chen et al., 2020a) and data augmentation (Taylor and Nitschke, 2017).”
i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning (2021, methodology): “SSL has also been used as an auxiliary task to improve the performance on the main task, such as generative model learning (Chen et al., 2019), semi-supervised learning (Zhai et al., 2019), and improving robustness and uncertainty (Hendrycks et al., 2019).”
Self-Supervised Learning Aided Class-Incremental Lifelong Learning (2020, background): “However, while combined with fullysupervised learning, SSL is generally used for data augmentation, and has negative effect on accuracy, though benefits on generality, robustness and uncertainty [14].”
Intriguing properties of adversarial training (2019, background): “…the model robustness (Xie et al., 2019b; Zhang et al., 2019b; Hendrycks et al., 2019a; Qin et al., 2019; Zhang & Wang, 2019; Carmon et al., 2019; Hendrycks et al., 2019b; Alayrac et al., 2019; Zhai et al., 2019) or accelerate the adversarial training process (Shafahi et al., 2019; Zhang et al.,…”
CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances (2020, methodology): “We compare our method on two in-distribution datasets: CIFAR-10 [33] and ImageNet-30 [26].”
Scaling Out-of-Distribution Detection for Real-World Settings (2022, methodology): “However, as Hendrycks et al. (2019b) observe, many of these works tune hyperparameters on a particular type of anomaly that is also seen at test time, so their evaluation setting is more lenient.”