Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision

By Priya Goyal, Quentin Duval, Isaac Seessel, Mathilde Caron, Ishan Misra, Levent Sagun, Armand Joulin, Piotr Bojanowski. 2022

125 citations recorded by the source corpus.

Discriminative self-supervised learning allows training models on any random group of internet images, and possibly recover salient information that helps differentiate between the images. Applied to ImageNet, this leads to object centric features that perform on par with supervised features on most object-centric downstream tasks. In this work, we question if using this ability, we can learn any salient and more representative information present in diverse unbounded set of images from across the globe. To do so, we train models on billions of random images without any data pre-processing or prior assumptions about what we want the model to learn. We scale our model size to dense 10 billion parameters to avoid underfitting on a large data size. We extensively study and validate our model performance on over 50 benchmarks including fairness, robustness to distribution shift, geographical diversity, fine grained recognition, image copy detection and many image classification datasets. The resulting model, not only captures well semantic information, it also captures information about artistic style and learns salient information such as geolocations and multilingual word embeddings based on visual content only. More importantly, we discover that such model is more robust, more fair, less harmful and less biased than supervised models or models trained on object centric datasets such as ImageNet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

The Challenges of Continuous Self-Supervised Learning (2022, background): “As a result, recent attempts of scaling up augmentation invariance [9, 24, 25] have seen some performance gains.”
DINOv2: Learning Robust Visual Features without Supervision (2023, methodology): “We compare our model to SEERv2 in Table 13.”
R3M: A Universal Visual Representation for Robot Manipulation (2022, background): “The best representations in these fields did not arise out of task-specific and carefully curated datasets, but rather the use of abundant in-the-wild data [4, 11, 12, 13].”
A Cookbook of Self-Supervised Learning (2023, unknown): “There’s also evidence SSL models can learn representations that are more robust to adversarial examples, label corruption, and input perturbations—and are more fair—compared to their supervised counterparts [Hendrycks et al., 2019, Goyal et al., 2022].”