Exploring the Limits of Weakly Supervised Pretraining

By Dhruv Kumar Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin R. Bharambe, L. V. D. van der Maaten. 2018

1,475 citations recorded by the source corpus.

State-of-the-art visual perception models for a wide range of tasks rely on supervised pretraining. ImageNet classification is the de facto pretraining task for these models. Yet, ImageNet is now nearly ten years old and is by modern standards "small". Even so, relatively little is known about the behavior of pretraining with datasets that are multiple orders of magnitude larger. The reasons are obvious: such datasets are difficult to collect and annotate. In this paper, we present a unique study of transfer learning with large convolutional networks trained to predict hashtags on billions of social media images. Our experiments demonstrate that training for large-scale hashtag prediction leads to excellent results. We show improvements on several image classification and object detection tasks, and report the highest ImageNet-1k single-crop, top-1 accuracy to date: 85.4% (97.6% top-5). We also perform extensive experiments that provide novel empirical data on the relationship between large-scale pretraining and transfer learning performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Universal Language Model Fine-tuning for Text Classification (2018, background): “…a small performance drop (Huh et al., 2016), while recent work shows that an alignment be-tween source and target task label sets is important (Mahajan et al., 2018)—focusing on predicting a subset of words such as the most frequent ones might retain most of the performance while speeding up…”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “…research argues that general methods that can leverage additional computation ultimately win out against methods that rely on human expertise [Sutton, 2019; Hestness et al., 2017; Shazeer et al., 2017; Jozefowicz et al., 2016; Mahajan et al., 2018; Shazeer et al., 2018, 2017; Huang et al., 2018b].”
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “Therefore, in large-scale image recognition, classic ResNet-like architectures are still state of the art (Mahajan et al., 2018; Xie et al., 2020; Kolesnikov et al., 2020).”
Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “Mahajan et al. (2018) showed that predicting ImageNet related hashtags on Instagram images is an effective pre-training task.”
Masked Autoencoders Are Scalable Vision Learners (2021, positive): “On Places, our MAE outperforms the previous best results [19, 40], which were obtained via pre-training on billions of images.”
Momentum Contrast for Unsupervised Visual Representation Learning (2019, background): “Following [63, 2], we take two random “views” of the same image under random data augmentation to form a positive pair.”
Do Better ImageNet Models Transfer Better? (2018, background): “Other work has investigated transfer from extremely large image datasets to ImageNet, demonstrating that transfer learning can be useful even when the target dataset is large [65, 47].”
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, methodology): “We omit ensemble and multi-crop models (Hu et al., 2018), or models pretrained on 3.5B Instagram images (Mahajan et al., 2018).”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “Scaling neural networks brings dramatic quality gains over a wide array of machine learning problems such as computer vision, language understanding and neural machine translation (Devlin et al., 2018; Mahajan et al., 2018; Arivazhagan et al., 2019; Huang et al., 2019; Brown et al., 2020b).”
Are we done with ImageNet? (2020, methodology): “We start by assembling a set of 19 models [3, 27, 4, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37] which vary along a number of dimensions, including architecture, training objective and usage of external data (see appendix A).”
Big Transfer (BiT): General Visual Representation Learning (2019, background): “A similar eﬀect is observed when training on Instagram hash-tags [36] and in language modelling [25].”
Self-Training With Noisy Student Improves ImageNet Classification (2019, methodology): “This result is also a new state-of-the-art and 1% better than the previous best method that used an order of magnitude more weakly labeled data [44, 71].”