How Well Self-Supervised Pre-Training Performs with Streaming Data?

12 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model (2023, background): “On the other hand, since the large amount of pre-training data are typically unlabeled and may also arrive incrementally, it seems more reasonable to use self-supervised pre-training than supervised pre-training [5, 43], also regarding that (upstream) continual learning in a self-supervised manner i”
Generative Negative Text Replay for Continual Vision-Language Pretraining (2022, methodology): “For the instance incremental split, it models a real-world scenario in which data chunks are continuously collected in the same environment [22].”
CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One (2022, positive): “SimSiam [8] Unsupervised Continual Learning (UCL): has the unique property of being naturally robust to catastrophic forgetting when fine-tuning on incremental unlabeled data [16,25].”
Probing Representation Forgetting in Supervised and Unsupervised Continual Learning (2022, methodology): “These methods have been recently evaluated in a limited continual learning setting [19] where a sequence was trained on non-iid unsupervised streaming data and then applied in transfer learning settings on multiple datasets.”