Billion-scale semi-supervised learning for image classification

By I. Z. Yalniz, Herv'e J'egou, Kan Chen, Manohar Paluri, Dhruv Kumar Mahajan. 2019

493 citations recorded by the source corpus.

This paper presents a study of semi-supervised learning with large convolutional networks. We propose a pipeline, based on a teacher/student paradigm, that leverages a large collection of unlabelled images (up to 1 billion). Our main goal is to improve the performance for a given target architecture, like ResNet-50 or ResNext. We provide an extensive analysis of the success factors of our approach, which leads us to formulate some recommendations to produce high-accuracy models for image classification with semi-supervised learning. As a result, our approach brings important gains to standard architectures for image, video and fine-grained classification. For instance, by leveraging one billion unlabelled images, our learned vanilla ResNet-50 achieves 81.2% top-1 accuracy on the ImageNet benchmark.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Scaling & Learning Theory

Citations in the corpus

Fixing the train-test resolution discrepancy (2019, neutral): “8 ResNet-50 Billion-scale [38] X 224 224 25.”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, methodology): “Inspired by the recent successes of learning from unlabeled data [19, 20, 1, 11, 24, 12], the proposed semi-supervised learning framework leverages unlabeled data in both task-agnostic and task-specific”
VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “5B Instagram images, using hashtags as labels [45, 46].”
Emerging Properties in Self-Supervised Vision Transformers (2021, background): “This propagation can either be done with hard assignments of labels [41, 78, 79] or with a soft assignment [76].”
Grafit: Learning fine-grained image representations with coarse labels (2020, background): “…strategy borrowed from recent works [28, 50], this baseline by itself outperforms all published results in several settings, for instance our ResNet-50 baseline without extra training data outperforms on ImageNet a ResNet-50 pretrained on YFCC100M [66] (see Table 12 in Appendix B for a comparison).”
Self-training Improves Pre-training for Natural Language Understanding (2020, background): “Self-training has been successfully applied to a variety of tasks, including image recognition (Yalniz et al., 2019; Xie et al., 2020; Zoph et al., 2020), automatic speech recognition (Syn-naeve et al., 2019; Kahn et al., 2020; Park et al., 2020), sequence generation (He et al., 2019), and parsing…”
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, neutral): “In computer vision and natural language processing, the best performing models are often pre-trained on a large dataset before being fine-tuned on data from the target task [29, 100, 80, 38, 61, 51, 99, 50, 12].”
RegionCLIP: Region-based Language-Image Pretraining (2021, background): “The learned features can be transferred to recognition tasks [16], and the classifier can be used to label images for semi-supervised learning [36, 55, 57].”
Rethinking Pre-training and Self-training (2020, background): “The recent successes of self-training [11–14] raise the question to what degree does self-training work better than pre-training.”
Meta Pseudo Labels (2020, methodology): “The method of Pseudo Labels, also known as self-training, is a simple Semi-Supervised Learning (SSL) approach that has been successfully applied to improve the state-of-the-art of many tasks, such as: image classification [79, 77], object detection, semantic segmentation [89], machine translation [2”
Support-set bottlenecks for video-text representation learning (2020, background): “Large-scale training data has enabled the more eﬀective pretraining of image (Sun et al., 2017; Yalniz et al., 2019), video (Ghadiyaram et al., 2019; Thomee et al., 2016) and textual representations (Raﬀel et al., 2019).”
MiniVLM: A Smaller and Faster Vision-Language Model (2020, methodology): “in image classifications [62, 22] and object detection [39].”
CVPR 2020 Continual Learning in Computer Vision Competition: Approaches, Results, Current Challenges and Future Directions (2020, methodology): “Among the different architectures and training procedures considered for the feature extractor, ResNet-50 (He et al., 2016) trained by Facebook AI using the Semi-Weakly Super-vised Learning procedure (Yalniz et al., 2019) was selected.”
Compressing Visual-linguistic Model via Knowledge Distillation (2021, background): “For example, [25, 6] transfer the learned knowledge by mimicking the mass function of the output probability across classes, or by minimizing the divergence of intermediate features [78, 36, 28, 77, 73].”