Revisiting Unreasonable Effectiveness of Data in Deep Learning Era

By Chen Sun, Abhinav Shrivastava, Saurabh Singh, Abhinav Kumar Gupta. 2017

2,747 citations recorded by the source corpus.

The success of deep learning in vision can be attributed to: (a) models with high capacity; (b) increased computational power; and (c) availability of large-scale labeled data. Since 2012, there have been significant advances in representation capabilities of the models and computational capabilities of GPUs. But the size of the biggest dataset has surprisingly remained constant. What will happen if we increase the dataset size by 10 × or 100 × ? This paper takes a step towards clearing the clouds of mystery surrounding the relationship between ‘enormous data’ and visual deep learning. By exploiting the JFT-300M dataset which has more than 375M noisy labels for 300M images, we investigate how the performance of current vision tasks would change if this data was used for representation learning. Our paper delivers some surprising (and some expected) findings. First, we find that the performance on vision tasks increases logarithmically based on volume of training data size. Second, we show that representation learning (or pre-training) still holds a lot of promise. One can improve performance on many vision tasks by just training a better base model. Finally, as expected, we present new state-of-the-art results for different vision tasks including image classification, object detection, semantic segmentation and human pose estimation. Our sincere hope is that this inspires vision community to not undervalue the data and develop collective efforts in building larger datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Quoc V. Le
  3. Jianfeng Gao
  4. Yoshua Bengio
  5. Percy Liang
  6. Sergey Levine
  7. Luke S. Zettlemoyer
  8. Trevor Darrell

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Ming-Hsuan Yang
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Alan L. Yuille

In Scaling & Learning Theory

  1. Jason D. Lee
  2. Neil Houlsby
  3. Simon Shaolei Du
  4. Behnam Neyshabur
  5. Julien Mairal
  6. Xiaohua Zhai
  7. Quoc V. Le
  8. Basil Mustafa

In Evaluation & Benchmarks

  1. Mohit Bansal
  2. Jitendra Malik
  3. Yejin Choi
  4. Jianfeng Gao
  5. Kaiming He
  6. Trevor Darrell
  7. Abhinav Kumar Gupta
  8. Ali Farhadi

Citations in the corpus