Rethinking ImageNet Pre-Training

By Kaiming He, Ross B. Girshick, Piotr Doll'ar. 2018

1,207 citations recorded by the source corpus.

We report competitive results on object detection and instance segmentation on the COCO dataset using standard models trained \textbf{from random initialization}. The results are \textbf{no worse} than their ImageNet pre-training counterparts even when using the hyper-parameters of the baseline system (Mask R-CNN) that were optimized for fine-tuning pre-trained models, with the sole exception of increasing the number of training iterations so the randomly initialized models may converge. Training from random initialization is surprisingly robust; our results hold even when: (i) using only 10\% of the training data, (ii) for deeper and wider models, and (iii) for multiple tasks and metrics. Experiments show that ImageNet pre-training speeds up convergence early in training, but does not necessarily provide regularization or improve final target task accuracy. To push the envelope we demonstrate {50.9}~AP on COCO object detection without using any external data---a result on par with the top COCO 2017 competition results that used ImageNet pre-training. These observations challenge the conventional wisdom of ImageNet pre-training for dependent tasks and we expect these discoveries will encourage people to rethink the current de facto paradigm of `pre-training and fine-tuning' in computer vision.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “There is some early empirical work along these lines in the field of computer vision [Huh et al., 2016; Kornblith et al., 2018; He et al., 2018].”
End-to-End Object Detection with Transformers (2020, positive): “s we attempt to make a Faster R-CNN baseline stronger. To align it with DETR, we add generalized IoU [38] to the box loss, the same random crop augmentation and long training known to improve results [13]. Results are presented in Table1. In the top section we show Faster R-CNN results from Detectro”
A Simple Framework for Contrastive Learning of Visual Representations (2020, neutral): “Other approaches include shufﬂing data examples (He et al., 2019), or replacing BN with layer norm (Hénaff et al., 2019).”
Momentum Contrast for Unsupervised Visual Representation Learning (2019, background): “In green are the gaps of at least + 0.5 point. ity of features, so our experiments are on controlled schedules, e.g ., the 1 ( ∼ 12 epochs) or 2 schedules [22] for COCO, in contrast to 6 ∼ 9 in [31].”
To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks (2019, background): “Recent results suggest is less useful for more distant target tasks such as semantic segmentation (He et al., 2018).”
Big Transfer (BiT): General Visual Representation Learning (2019, background): “wnstream labeled examples. Transfer learning can also shift computational requirements upstream. Although the pre-training phase is expensive, downstream tasks often require fewer training iterations [37,12]. In this paper, we revisit this simple paradigm: pre-train on a large supervised source doma”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, background): “Prior work (He et al., 2018) shows that with sufﬁcient data and training time, from-scratch training is competitive for detection.”
Learning Representations by Maximizing Mutual Information Across Views (2019, positive): “Though, the necessity of pre-trained networks for many vision tasks has been convincingly questioned in recent work [He et al., 2018].”
VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “This model is trained from scratch on COCO, following [102].”
Benchmarking Detection Transfer Learning with Vision Transformers (2021, methodology): “These upgrades, developed primarily in [39, 18, 13], allow the model to be trained effectively from random initialization, thus enabling a meaningful from-scratch baseline.”
Cascade R-CNN: High Quality Object Detection and Instance Segmentation (2019, background): “[23] showed it is possible to achieve state-of-the-art object detectors without ImageNet pretraining, with a help of the Cascade R-CNN.”
VL-BERT: Pre-training of Generic Visual-Linguistic Representations (2019, background): “Recently there are research works on directly training CNNs from scratch on massive-scale target datasets, without ImageNet pre-training (He et al., 2018).”
Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity (2022, methodology): “We note that different from closedworld [21], ImageNet supervised pre-training is a strong initialization for open-world (see supplementary).”
K-Net: Towards Unified Image Segmentation (2021, background): “The short edge of images is randomly sampled from [640, 800] [21].”
Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation (2020, background): “32 and a step learning rate decay [25].”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, methodology): “egmentation [8,9]. A task is rst dened, then data is collected and annotated to learn a generalisable model for that task. Increasingly, datasets have been used for novel tasks, through pre-training [10,11], self-supervision [12,13] or additional annotations [14,15]. However, task adaptation demonst”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, positive): “In the left panel of Table 6, we show performance after training a linear classiﬁer on top of frozen representations on different datasets while on the right panel we evaluate the features by ﬁnetuning a ResNet-50 on object detection with Faster R-CNN [49] and DETR [6].”
Billion-scale semi-supervised learning for image classification (2019, background): “This approach is typically required for tasks for which no enough labels are available [29], such a ﬁne-grain classiﬁcation or detection (although a recent work [15] shows that learning from scratch is possible in the latter case).”