Big Self-Supervised Models are Strong Semi-Supervised Learners

By Ting-Li Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, Geoffrey E. Hinton. 2020

2,576 citations recorded by the source corpus.

One paradigm for learning from few labeled examples while making best use of a large amount of unlabeled data is unsupervised pretraining followed by supervised fine-tuning. Although this paradigm uses unlabeled data in a task-agnostic way, in contrast to most previous approaches to semi-supervised learning for computer vision, we show that it is surprisingly effective for semi-supervised learning on ImageNet. A key ingredient of our approach is the use of a big (deep and wide) network during pretraining and fine-tuning. We find that, the fewer the labels, the more this approach (task-agnostic use of unlabeled data) benefits from a bigger network. After fine-tuning, the big network can be further improved and distilled into a much smaller one with little loss in classification accuracy by using the unlabeled examples for a second time, but in a task-specific way. The proposed semi-supervised learning algorithm can be summarized in three steps: unsupervised pretraining of a big ResNet model using SimCLRv2 (a modification of SimCLR), supervised fine-tuning on a few labeled examples, and distillation with unlabeled examples for refining and transferring the task-specific knowledge. This procedure achieves 73.9\% ImageNet top-1 accuracy with just 1\% of the labels ($\le$13 labeled images per class) using ResNet-50, a $10\times$ improvement in label efficiency over the previous state-of-the-art. With 10\% of labels, ResNet-50 trained with our method achieves 77.5\% top-1 accuracy, outperforming standard supervised training with all of the labels.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “SimCLRv2 The SimCLRv2 (Chen et al., 2020d) project released pre-trained and fine-tuned models in various settings.”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “Among discriminative methods, contrastive methods [9, 10, 32, 33, 34, 11, 35, 36] currently achieve state-of-the-art performance in self-supervised learning [37, 8, 38, 12].”
Emerging Properties in Self-Supervised Vision Transformers (2021, neutral): “4× faster run time than previous state of the art [12].”
An Empirical Study of Training Self-Supervised Vision Transformers (2021, methodology): “8 we compare with the state-of-the-art big ResNets reported by SimCLR v2 [10] and BYOL [17].”
Self-supervised Pretraining of Visual Features in the Wild (2021, methodology): “They are RegNet architectures [40] trained with the SwAV self-supervised method [7] We compare with the original models trained in Caron et al. [7] as well as the pretraining on curated data from SimCLRv2 [9] and ViT [14].”
Vector-quantized Image Modeling with Improved VQGAN (2021, methodology): “Similar to findings reported in iGPT (Chen et al., 2020a), the representations from the middle transformer blocks has better linear-probe accuracy (more study can be found in Appendix).”
SLIP: Self-supervision meets Language-Image Pre-training (2021, methodology): “In this work we primarily use an adaptation of SimCLR [9, 10], a representative example of these methods, as the self-supervised objective in SLIP.”
Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, methodology): “, CLIP [71]) have been trained on joint image(s) and corresponding text captions via variants of a masked language modeling objective [72, 73, 74, 75], a contrastive loss [76, 77, 71] or other supervised objectives[78, 79].”
Training Neural Networks with Fixed Sparse Masks (2021, background): “Since modern neural networks often have millions or billions of parameters [9, 7, 40], this communication can become excessively expensive.”
SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network (2021, neutral): “This has been observed in language modelling [14, 51, 26], computer vision [24, 51, 52], and speech recognition [53, 5].”
Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition (2020, background): “This idea has been explored recently for image tasks in [3, 4].”
ZeroPrompt: Scaling Prompt-Based Pretraining to 1, 000 Tasks Improves Zero-Shot Generalization (2022, background): “For fair evaluation of zero-shot generalization, we investigate and collect diverse public Chinese NLP datasets with different task types.”
Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “A promising development in self-supervised learning is the contrastive loss, with representative works like CPC (van den Oord et al., 2018), SimCLR (Chen et al., 2020a,b) and MoCo (He et al., 2020; Chen et al., 2020c).”
Pix2seq: A Language Modeling Framework for Object Detection (2021, methodology): “Following (Howard, 2013; Chen et al., 2020a;b), we also use color distortion with a strength of 0.5.”
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision (2021, methodology): “…on the popular multi-modal tasks with state-of-the-art (SOTA) VLP methods including LXMERT (Tan & Bansal, 2019), VL-T5 (Cho et al., 2021), UNITER (Chen et al., 2020b), OSCAR (Li et al., 2020), Villa (Gan et al., 2020), SOHO (Huang et al., 2021), UNIMO (Li et al., 2021), and VinVL (Zhang et al.,…”
Multimodal Chain-of-Thought Reasoning in Language Models (2023, neutral): “5 models (Chen et al., 2020).”