Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning

By Tianlong Chen, Sijia Liu, Shiyu Chang, Yu Cheng, Lisa Amini, Zhangyang Wang. 2020

295 citations recorded by the source corpus.

Pretrained models from self-supervision are prevalently used in fine-tuning downstream tasks faster or for better accuracy. However, gaining robustness from pretraining is left unexplored. We introduce adversarial training into self-supervision, to provide general-purpose robust pretrained models for the first time. We find these robust pretrained models can benefit the subsequent fine-tuning in two ways: i) boosting final model robustness; ii) saving the computation cost, if proceeding towards adversarial fine-tuning. We conduct extensive experiments to demonstrate that the proposed framework achieves large performance margins (eg, 3.83% on robust accuracy and 1.3% on standard accuracy, on the CIFAR-10 dataset), compared with the conventional end-to-end adversarial training baseline. Moreover, we find that different self-supervised pretrained models have diverse adversarial vulnerability. It inspires us to ensemble several pretraining tasks, which boosts robustness more. Our ensemble strategy contributes to a further improvement of 3.59% on robust accuracy, while maintaining a slightly higher standard accuracy on CIFAR-10. Our codes are available at https://github.com/TAMU-VITA/Adv-SS-Pretraining.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Safety, Robustness & Interpretability

Citations in the corpus

Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “Due to growing dominance of large-scale pre-training, very recent work has started to explore adversarial training in the pre-training stage [19, 10, 37].”
Better Diffusion Models Further Improve Adversarial Training (2023, methodology): “…made employing strategies inspired by other areas, including metric learning (Mao et al., 2019; Pang et al., 2020a;b), self-supervised learning (Chen et al., 2020a;b; Naseer et al., 2020; Wang & Liu, 2022), ensemble learning (Tramèr et al., 2018; Pang et al., 2019), fairness (Ma et al., 2022;…”
RGB-D salient object detection: A survey (2020, background): “Furthermore, several studies [191, 192] have suggested that models pretrained using self-supervision can effectively be used to achieve better performance.”
Adversarial Self-Supervised Contrastive Learning (2020, methodology): “Existing works [40, 18] have shown that finetuning the supervised or self-supervised pretrained networks with adversarial training improves robustness.”
Graph Contrastive Learning with Augmentations (2020, background): “Self-supervision with handcrafted pretext tasks [29, 30, 31, 32] relies on heuristics to design, and thus could limit the generality of the learned representations.”
Queried Unlabeled Data Improves and Robustifies Class-Incremental Learning (2022, unknown): “The power of unlabeled data has recently drawn explosive interests, in multiple contexts such as semi-supervised learning (Chen et al., 2020d), self-training (Xie et al., 2020), and self-supervised learning (Chen et al., 2020e;c).”
Advances in adversarial attacks and defenses in computer vision: A survey (2021, unknown): “[341] proposed self-supervised adversarial training, whereas adversarial training is independently analyzed for self-supervision by incorporating it in pretraining in [342].”
Geometry-aware Instance-reweighted Adversarial Training (2020, background): “Furthermore, there is still a large room to apply adversarial training techniques into other domains such as pre-training (Hendrycks et al., 2019; Chen et al., 2020; Jiang et al., 2020; Salman et al., 2020), noisy labels (Zhu et al., 2021) and so on.”
A Closer Look at Accuracy vs. Robustness (2020, background): “Outside of neural networks, prior works suggest that lack of data may be responsible for low robustness [1, 5, 8, 9, 28, 48, 49, 59, 66].”
Recent advances and clinical applications of deep learning in medical image analysis (2021, methodology): “The pre-trained models from self-supervision are followed by supervised fine-tuning to achieve faster and better performance in subsequent classification tasks (Chen et al., 2020c).”