Divide and Contrast: Self-supervised Learning from Uncurated Data

By Yonglong Tian, Olivier J. H'enaff, Aäron van den Oord. 2021

113 citations recorded by the source corpus.

Self-supervised learning holds promise in leveraging large amounts of unlabeled data, however much of its progress has thus far been limited to highly curated pre-training data such as ImageNet. We explore the effects of contrastive learning from larger, less-curated image datasets such as YFCC, and find there is indeed a large difference in the resulting representation quality. We hypothesize that this curation gap is due to a shift in the distribution of image classes—which is more diverse and heavy-tailed—resulting in less relevant negative samples to learn from. We test this hypothesis with a new approach, Divide and Contrast (DnC), which alternates between contrastive learning and clustering-based hard negative mining. When pretrained on less curated datasets, DnC greatly improves the performance of self-supervised learning on downstream tasks, while remaining competitive with the current state-of-the-art on curated datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

SLIP: Self-supervision meets Language-Image Pre-training (2021, background): “Finally, while progress on ImageNet has been encouraging, there has not been strong evidence that current self-supervised methods scale well to larger uncurated datasets [54].”
Self-Supervised Models are Continual Learners (2021, positive): “This is probably due to a combination of factors: (i) Clipart is the last task, therefore it probably benefits in forward transfer and (ii) a similar effect to the one found in [26], where dividing data in subgroups tends to enable the learning of better representations.”
DINOv2: Learning Robust Visual Features without Supervision (2023, background): “A growing body of work has focused on the scaling abilities of self-supervised learning in terms of data and model size (Caron et al., 2019; Goyal et al., 2019; Tian et al., 2021; Goyal et al., 2022a).”
Efficient Self-supervised Vision Transformers for Representation Learning (2021, background): “The impact of pre-training dataset size/quality is explored for ConvNets in SSL [23, 68].”
SODA10M: Towards Large-Scale Object Detection Benchmark for Autonomous Driving (2021, unknown): “Traditional self-supervised algorithms [11, 42, 16] are usually pre-trained on ImageNet, while recent works [6, 51] have shown the consistency between upstream and downstream data distribution has a positive impact on the ﬁnal performance.”
Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence? (2023, background): “Several works have showed that scaling model and dataset size improves performance on vision tasks like image classification (Zhai et al., 2022; Tian et al., 2021; Goyal et al., 2021).”
Point Transformer V3: Simpler, Faster, Stronger (2023, background): “Deep learning models have experienced rapid advancements in various areas, such as 2D vision [24, 38, 78, 86] and natural language processing (NLP) [1, 37, 56, 79], with their progress often attributed to the effective utilization of scale, encompassing factors such as the size of datasets, the…”
Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation (2022, background): “Another issue is that its scalability to model capacity and data size is rarely studied, or is only performed in a linear evaluation setup [35].”