Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
By Chen Sun, Abhinav Shrivastava, Saurabh Singh, Abhinav Kumar Gupta. 2017
2,747 citations recorded by the source corpus.
The success of deep learning in vision can be attributed to: (a) models with high capacity; (b) increased computational power; and (c) availability of large-scale labeled data. Since 2012, there have been significant advances in representation capabilities of the models and computational capabilities of GPUs. But the size of the biggest dataset has surprisingly remained constant. What will happen if we increase the dataset size by 10 × or 100 × ? This paper takes a step towards clearing the clouds of mystery surrounding the relationship between ‘enormous data’ and visual deep learning. By exploiting the JFT-300M dataset which has more than 375M noisy labels for 300M images, we investigate how the performance of current vision tasks would change if this data was used for representation learning. Our paper delivers some surprising (and some expected) findings. First, we find that the performance on vision tasks increases logarithmically based on volume of training data size. Second, we show that representation learning (or pre-training) still holds a lot of promise. One can improve performance on many vision tasks by just training a better base model. Finally, as expected, we present new state-of-the-art results for different vision tasks including image classification, object detection, semantic segmentation and human pose estimation. Our sincere hope is that this inspires vision community to not undervalue the data and develop collective efforts in building larger datasets.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Quoc V. Le
- Jianfeng Gao
- Yoshua Bengio
- Percy Liang
- Sergey Levine
- Luke S. Zettlemoyer
- Trevor Darrell
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Alan L. Yuille
In Scaling & Learning Theory
- Jason D. Lee
- Neil Houlsby
- Simon Shaolei Du
- Behnam Neyshabur
- Julien Mairal
- Xiaohua Zhai
- Quoc V. Le
- Basil Mustafa
In Evaluation & Benchmarks
- Mohit Bansal
- Jitendra Malik
- Yejin Choi
- Jianfeng Gao
- Kaiming He
- Trevor Darrell
- Abhinav Kumar Gupta
- Ali Farhadi
Citations in the corpus
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, positive): “…model scalability, we use the ILSVRC-2012 ImageNet dataset with 1k classes and 1.3M images (we refer to it as ImageNet in what follows), its superset ImageNet-21k with 21k classes and 14M images (Deng et al., 2009), and JFT (Sun et al., 2017) with 18k classes and 303M high-resolution images.”
- A Constructive Prediction of the Generalization Error Across Scales (2019, background): “Data scaling: It has long been recognized that more data improves performance, and various studies report such trends in both computer vision (e.g., Zhu et al., 2012; Sun et al., 2017) and language processing tasks (e.g., Banko & Brill, 2001; Talmor & Berant, 2019).”
- Deep Learning Scaling is Predictable, Empirically (2017, background): “Sun et al. (2017) show image classification accuracy improves with training data size, but curiously, they conclude that accuracy "increases logarithmically based on volume of training data size".”
- Rethinking ImageNet Pre-Training (2018, background): “If the gain of large-scale classification-level pre-training becomes exponentially diminishing [44, 30], it would be more effective to collect data in the target domain.”
- Do Better ImageNet Models Transfer Better? (2018, background): “Other work has investigated transfer from extremely large image datasets to ImageNet, demonstrating that transfer learning can be useful even when the target dataset is large [65, 47].”
- One Epoch Is All You Need (2019, background): “Some works have analyzed the relationship between the performance and the dataset size (Sun et al., 2017; Hestness et al., 2017).”
- Beyond human-level accuracy: computational challenges in deep learning (2019, background): “Google Research recently showed the importance of training image classi ers with 300× more images than prior datasets [36].”
- Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, background): “(2020) and pre-train ViT on the JFT dataset (Sun et al., 2017) with 18k classes and 303M images, for 7 epochs.”
- On Robustness and Transferability of Convolutional Neural Networks (2020, methodology): “For models trained on JFT we observe a similar behavior except that training for 1.12M steps often leads to a higher accuracy than training for 457k steps even when only 1M or 5M data points are used (c.f. Appendix B).”
- Big Transfer (BiT): General Visual Representation Learning (2019, methodology): “The largest, BiT-L is trained on the JFT-300M dataset [51], which contains 300 M noisily labelled”
- VideoBERT: A Joint Model for Video and Language Representation Learning (2019, background): “Deep learning can benefit a lot from labeled data [24], but this is hard to acquire at scale.”
- Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, methodology): “We examine two pipeline specifications: one that trains a ResNet-50 model (He et al., 2016) on ImageNet, and another that pretrains a ResNet-101x3 Big Transfer (BiT) model (Kolesnikov et al., 2019) on the JFT-300M dataset (Sun et al., 2017) then fine-tunes it on ImageNet.”
- Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, neutral): “23 % [82].”
- Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, background): “The effectiveness of big models have been demonstrated on supervised learning [64–67], fine-tuning supervised models on a few examples [68], and unsupervised learning on language [9, 69, 10, 70].”
- VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “Recent works [42–44] also use JFT-300M [42] dataset, curated by automatic labeling of images from web signals using Google’s internal tooling.”
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “In vision, [28, 49] explore the use of image metadata such as Instagram hashtags to pretrain image classifiers.”
- Self-supervised Pretraining of Visual Features in the Wild (2021, methodology): “Benefiting from the advances in distributed training [18], several works have shown the advantages of pretraining on large curated image datasets with weak-supervised learning [27, 35], semisupervised learning [54] or supervised training on hundreds of millions of filtered images [29, 47].”
- Zero-Shot Text-to-Image Generation (2021, methodology): “To scale up to 12-billion parameters, we created a dataset of a similar scale to JFT-300M (Sun et al., 2017) by collecting 250 million text-images pairs from the internet.”