Rethinking Pre-training and Self-training

By Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, Quoc V. Le. 2020

743 citations recorded by the source corpus.

Pre-training is a dominant paradigm in computer vision. For example, supervised ImageNet pre-training is commonly used to initialize the backbones of object detection and segmentation models. He et al., however, show a surprising result that ImageNet pre-training has limited impact on COCO object detection. Here we investigate self-training as another method to utilize additional data on the same setup and contrast it against ImageNet pre-training. Our study reveals the generality and flexibility of self-training with three additional insights: 1) stronger data augmentation and more labeled data further diminish the value of pre-training, 2) unlike pre-training, self-training is always helpful when using stronger data augmentation, in both low-data and high-data regimes, and 3) in the case that pre-training is helpful, self-training improves upon pre-training. For example, on the COCO object detection dataset, pre-training benefits when we use one fifth of the labeled data, and hurts accuracy when we use all labeled data. Self-training, on the other hand, shows positive improvements from +1.3 to +3.4AP across all dataset sizes. In other words, self-training works well exactly on the same setup that pre-training does not work (using ImageNet to help COCO). On the PASCAL segmentation dataset, which is a much smaller dataset than COCO, though pre-training does help significantly, self-training improves upon the pre-trained model. On COCO object detection, we achieve 54.3AP, an improvement of +1.5AP over the strongest SpineNet model. On PASCAL segmentation, we achieve 90.5 mIOU, an improvement of +1.5% mIOU over the previous state-of-the-art result by DeepLabv3+.

Subfields: Applied Psychology, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Self-training Improves Pre-training for Natural Language Understanding (2020, background): “Self-training has been successfully applied to a variety of tasks, including image recognition (Yalniz et al., 2019; Xie et al., 2020; Zoph et al., 2020), automatic speech recognition (Synnaeve et al.”
Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition (2020, background): “This idea has been explored recently for image tasks in [3, 4].”
Florence: A New Foundation Model for Computer Vision (2021, methodology): “In addition, we generate pseudo bounding boxes on ImageNet-22K dataset (Deng et al., 2009) by following (Zoph et al., 2020), which further enlarges our data.”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, methodology): “However, models trained by these methods so far show limited transferability to downstream tasks (Zoph et al., 2020).”
RegionCLIP: Region-based Language-Image Pretraining (2021, background): “Recently, semi-supervised learning was explored [48, 56, 66], where pretrained detectors are used to create pseudo labels of image regions.”
Grounded Language-Image Pre-training (2021, background): “Why can the student model possibly outperform the teacher model? While discussions remain active in the selftraining literature [68], in the context of visual grounding, we posit that the teacher model is utilizing the language context and language generalization ability to accurately ground concept”
Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation (2020, methodology): “We then show that the Copy-Paste augmentation strategy provides additional gains with self-training [44, 73] wherein we extract instances from ground-truth data and paste them onto unlabeled data annotated with pseudo-labels.”
Towards Label-Efficient Incremental Learning: A Survey (2023, methodology): “The pseudo labels are used for further self-training [Sahito et al., 2022; Zoph et al., 2020].”
Pre-Trained Models: Past, Present and Future (2021, methodology): “For those conventional PTMs based on supervised pre-training, we refer to the papers of He et al. (2019) and Zoph et al. (2020).”
Meta Pseudo Labels (2020, methodology): “The method of Pseudo Labels, also known as self-training, is a simple Semi-Supervised Learning (SSL) approach that has been successfully applied to improve the state-of-the-art of many tasks, such as: image classification [79, 77], object detection, semantic segmentation [89], machine translation [2”
Self-Supervised Learning: Generative or Contrastive (2020, methodology): “Some following work [84], [153] investigates this problem and attributes it to the gap between the instance discrimination and object detection.”
End-to-End Semi-Supervised Object Detection with Soft Teacher (2021, methodology): “In [20, 36], the predictions of different data augmentation are ensembled to form the pseudo labels for unlabeled images.”
Revisiting ResNets: Improved Training and Scaling Strategies (2021, background): “Furthermore, Zoph et al. (2020a) presents evidence that the addition of data augmentation shrinks the L2 norm of the weights, which renders some of the effects of weight decay redundant.”
Image Segmentation Using Deep Learning: A Survey (2020, methodology): “Network (UperNet) [136], segmentation by re-training and self-training [137], densely connected neural architecture search [138], hierarchical multiscale attention [139], Efficient RGB-D Semantic Segmentation (ESA-Net) [140], Iterative Pyramid Contexts [141], and Learning Dynamic Routing for Semanti”
Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (2024, background): “This belongs to the research area of semi-supervised learning [31, 55, 89], which is popular with various applications [70, 74].”
Self-Supervised Representation Learning: Introduction, advances, and challenges (2021, neutral): “A preliminary result in computer vision suggests not [99].”
Knowledge distillation: A good teacher is patient and consistent (2021, background): “Recent state-of-the-art models for image classification [6, 22, 39, 41, 48], object detection [7, 26] or semantic segmentation [52] push model size to the limits allowed by modern hardware.”