Self-Supervised Pretraining Improves Self-Supervised Pretraining

By Colorado Reed, Xiangyu Yue, Aniruddha Nrusimha, Sayna Ebrahimi, Vivek Vijaykumar, Richard Mao, Bo Li, Shanghang Zhang, Devin Guillory, Sean L. Metzger and 2 more. 2021

137 citations recorded by the source corpus.

While self-supervised pretraining has proven beneficial for many computer vision tasks, it requires expensive and lengthy computation, large amounts of data, and is sensitive to data augmentation. Prior work demonstrates that models pretrained on datasets dissimilar to their target data, such as chest X-ray models trained on ImageNet, underperform models trained from scratch. Users that lack the resources to pretrain must use existing models with lower performance. This paper explores Hierarchical PreTraining (HPT), which decreases convergence time and improves accuracy by initializing the pretraining process with an existing pretrained model. Through experimentation on 16 diverse vision datasets, we show HPT converges up to 80× faster, improves accuracy across tasks, and improves the robustness of the self-supervised pretraining process to changes in the image augmentation policy or amount of pretraining data. Taken together, HPT provides a simple framework for obtaining better pretrained representations with less computational resources.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jiashi Feng
2. Yifan Zhang
3. Lanqing Hong
4. Xinchao Wang
5. Zhenguo Li
6. Dapeng Hu
7. Shipeng Yan
8. Mat'ias Mendieta

In Computer Vision and Pattern Recognition

1. Jiashi Feng
2. Xinchao Wang
3. Zhenguo Li
4. Yifan Zhang
5. Lanqing Hong
6. Dapeng Hu
7. Dacheng Tao
8. Shipeng Yan

Citations in the corpus

• Continual Barlow Twins: Continual Self-Supervised Learning for Remote Sensing Semantic Segmentation (2022, neutral): “similar intuitions reached, for example, in [34].”
• How Well Does Self-Supervised Pre-Training Perform with Streaming Data? (2021, background): “Some recent works (Gururangan et al., 2020; Reed et al., 2021) propose to pre-train with the downstream dataset for a better transfer.”
• Recent advances and clinical applications of deep learning in medical image analysis (2021, background): “For example, Reed et al. (2021) proposed a hierarchical pre-training strategy to make the self-supervised pre-training process converge up to 80× faster with an improved accuracy across different tasks.”
• Towards Geospatial Foundation Models via Continual Pretraining (2023, background): “[36] explores a hierarchical pretraining approach for task adaptation.”