The Unreasonable Effectiveness of Data

By Alon Y. Halevy, Peter Norvig, Fernando C Pereira. 2009

1,711 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Management Science and Operations Research, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Evaluation & Benchmarks

Citations in the corpus

Shortcut learning in deep neural networks (2020, background): “t opportunities are by no means disappearing when the size of a dataset is simply scaled up by some orders of magnitude (in the hope that this is sufﬁcient to sample the diverse world that we live in [34]). Systematic biases are still present even in “Big Data” with large volume and variety, and con”
A Survey of Machine Learning for Big Code and Naturalness (2017, background): “Do we believe in the unreasonable effectiveness of data [83]? Yes, but we do not have sufficient data.”
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era (2017, background): “t the discussion from models to data. Our paper is inspired from several papers which have time and again paid closer look to impact and properties of data rather than models. In 2009, Pereira et al. [30] presented a survey paper to look into impact of data in ﬁelds such as natural language processi”
Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks (2015, background): “It is well known that working with large amounts of real data (“big data”) tends to lead researchers to simpler models as “simple models and a lot of data trump more elaborate models based on less data” (Halevy et al., 2009).”
Green AI (2019, background): “Similar trends exist with increasing the quantity of training data [40, 13] and the number of experiments [3].”
Data and its (dis)contents: A survey of dataset development and use in machine learning research (2020, background): “ance on crowdworkers, was seen as a boon to machine learning.(1,3,6,7) The enormous scale of such datasets has been mythologized as beneficial to the perceived generality of trained systems,(7) but they continue to be impacted by the limitations and biases that impact all datasets.”
Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure (2020, background): “ML fairness practitioners have identified collection and curation of representative data as the most common way to mitigate model biases [71], echoing the broader tendency within ML communities to valorize larger datasets as necessarily better—exemplified by [62] and subsequently in [85, 136].”
Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2016, background): “These advances have made long-lasting impacts on the way we sense, communicate, and make decisions [4], a trend that is only expected to grow in the foresee-”
The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition (2015, background): “Our work is inspired by [19], whose stance is that large datasets are an unreasonably effective way to learn highly accurate models, in a generic and scalable fashion.”
Knowledge Infused Learning (K-IL): Towards Deep Incorporation of Knowledge in Deep Learning (2019, background): “The manipulation of attentional focus is effective in solving real world problems involving massive amounts of data (Halevy, Norvig, and Pereira 2009; Sun et al. 2017).”
Natural Language Processing for EHR-Based Computational Phenotyping (2018, background): “that inclusion of more data resources can improve the model performances [174].”
Representation Matters: Offline Pretraining for Sequential Decision Making (2021, background): “Indeed, the last decade has witnessed ever more impressive models learned from ever larger static datasets (Halevy et al., 2009; Krizhevsky et al., 2012; Brown et al., 2020; Dosovitskiy et al., 2020).”
The Effectiveness of Data Augmentation in Image Classification using Deep Learning (2017, background): “For example, text-to-speech and text-based models have improved signiﬁcantly due to the release of a trillion-word corpus by Google [8].”
Quantity doesn’t buy quality syntax with neural language models (2019, neutral): “Can we hope to address these deﬁcits by training larger and larger networks on larger and larger corpora, relying on the “unreasonable effective-ness” of massive datasets (Halevy et al., 2009) and computational power?”
Embracing data abundance: BookTest Dataset for Reading Comprehension (2016, background): “…question-answering datasets; 2. it is widely accepted that more data can significantly improve performance of most models (Banko and Brill, 2001; Halevy et al., 2009); 3. training a model takes only about two hours on the CBT or about two days on the Daily Mail dataset, the largest of the three…”
Use What You Have: Video retrieval using representations from collaborative experts (2019, neutral): “sets, reminiscent of the “big-data” hypothesis [20].”
Low-N protein engineering with data-efficient deep learning (2020, background): “To meet the enormous data requirement of supervised deep learning — typically greater than 10 6 labeled data points 26,27 — current machine learning-guided protein design approaches must gather high-throughput experimental data 28–31 or abandon deep learning altogether 18,20,21,32–37 .”
Robustness properties of Facebook's ResNeXt WSL models (2019, background): “On the one hand, these models achieve a remarkable degree of robustness against common image corruptions and perturbations and even a limited degree of adversarial robustness despite not having been explicitly trained for adversarial robustness, demonstrating yet another example of the “unreasonable”