The Deluge of Spurious Correlations in Big Data
By Cristian S. Calude, Giuseppe Longo. Foundations of Science, 2016
339 citations recorded by the source corpus.
Very large databases are a major opportunity for science and data analytics is a remarkable new field of investigation in computer science. The effectiveness of these tools is used to support a “philosophy” against the scientific method as developed throughout history. According to this view, computer-discovered correlations should replace understanding and guide prediction and action. Consequently, there will be no need to give scientific meaning to phenomena, by proposing, say, causal relations, since regularities in very large databases are enough: “with enough data, the numbers speak for themselves”. The “end of science” is proclaimed. Using classical results from ergodic theory, Ramsey theory and algorithmic information theory, we show that this “philosophy” is wrong. For example, we prove that very large databases have to contain arbitrary correlations. These correlations appear only due to the size, not the nature, of data. They can be found in “randomly” generated, large enough databases, which—as we will prove—implies that most correlations are spurious. Too much information tends to behave like very little information. The scientific method can be enriched by computer mining in immense databases, but not replaced by it.
Subfields: Computational Theory and Mathematics, Discrete Mathematics and Combinatorics
View Paper: https://www.semanticscholar.org/paper/9e7be12082f58cbf7ebdb84a8cbdc897a4e41683
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Theory and Mathematics
Citations in the corpus
- Machine Learning With Big Data: Challenges and Approaches (2017, background): “Recently, Calude and Longo [44] have discussed the impact and incidence of spurious correlations in Big Data.”
- Situating methods in the magic of Big Data and AI (2018, background): “One problem, common to all statistical practices, is the production of spurious correlations, or results that suggest connections between data that have no real connection (Calude & Longo, 2016).”