How Well Does Self-Supervised Pre-Training Perform with Streaming Data?

By Dapeng Hu, Shipeng Yan, Qizhengqiu Lu, Lanqing Hong, Hailin Hu, Yifan Zhang, Zhenguo Li, Xinchao Wang, Jiashi Feng. 2021

39 citations recorded by the source corpus.

Prior works on self-supervised pre-training focus on the joint training scenario, where massive unlabeled data are assumed to be given as input all at once, and only then is a learner trained. Unfortunately, such a problem setting is often impractical if not infeasible since many real-world tasks rely on sequential learning, e.g., data are decentralized or collected in a streaming fashion. In this paper, we conduct the first thorough and dedicated investigation on self-supervised pre-training with streaming data, aiming to shed light on the model behavior under this overlooked setup. Specifically, we pre-train over 500 models on four categories of pre-training streaming data from ImageNet and DomainNet and evaluate them on three types of downstream tasks and 12 different downstream datasets. Our studies show that, somehow beyond our expectation, with simple data replay or parameter regularization, sequential self-supervised pre-training turns out to be an efficient alternative for joint pre-training, as the performances of the former are mostly on par with those of the latter. Moreover, catastrophic forgetting, a common issue in sequential supervised learning, is much alleviated in sequential self-supervised learning (SSL), which is well justified through our comprehensive empirical analysis on representations and the sharpness of minima in the loss landscape. Our findings, therefore, suggest that, in practice, for SSL, the cumbersome joint training can be replaced mainly by sequential learning, which in turn enables a much broader spectrum of potential application scenarios.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jun Zhu
2. Hang Su
3. Liyuan Wang
4. Xingxing Zhang
5. Mohamed Elhoseiny
6. Shipeng Yan
7. Xuming He
8. Bernard Ghanem

In Computer Vision and Pattern Recognition

1. Jun Zhu
2. Luc Van Gool
3. Philip H. S. Torr
4. Dacheng Tao
5. Bernt Schiele
6. Hang Su
7. Ming-Hsuan Yang
8. Trevor Darrell

In Radiology, Nuclear Medicine and Imaging

1. Mohamed Elhoseiny
2. Liyuan Wang
3. Philip H. S. Torr
4. Jun Zhu
5. Hang Su
6. Shipeng Yan
7. Bernard Ghanem
8. Xuming He

In Representation & Self-Supervised Learning

1. Liyuan Wang
2. Mohamed Elhoseiny
3. Jun Zhu
4. Xingxing Zhang
5. Hang Su
6. Shipeng Yan
7. Xuming He
8. Jiangwei Xie

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “In particular,sinceself-supervisedandpre-trainedrepresentations arenaturallymorerobusttocatastrophicforgetting[28],[87], [89],[93],creating,improvingandexploitingsuchrepresenta-tionaladvantageshasbecomeapromisingdirection.”
• General Incremental Learning with Domain-aware Categorical Representations (2022, background): “This paradigm is important in many real world applications [14,22] in which offline learning regularly outperforms the online version.”
• Continual Learning on a Diet: Learning from Sparsely Labeled Streams Under Constrained Computation (2024, result): “We also compare against the joint training, where pre-training and fine-tuning are performed on the whole dataset, following (Hu et al., 2021) to show the upper bound.”