The natural selection of bad science

By Paul E. Smaldino, R. Mcelreath. Royal Society Open Science, 2016

721 citations recorded by the source corpus.

Poor research design and data analysis encourage false-positive findings. Such poor methods persist despite perennial calls for improvement, suggesting that they result from something more than just misunderstanding. The persistence of poor methods results partly from incentives that favour them, leading to the natural selection of bad science. This dynamic requires no conscious strategizing—no deliberate cheating nor loafing—by scientists, only that publication is a principal factor for career advancement. Some normative methods of analysis have almost certainly been selected to further publication instead of discovery. In order to improve the culture of science, a shift must be made away from correcting misunderstandings and towards rewarding understanding. We support this argument with empirical evidence and computational modelling. We first present a 60-year meta-analysis of statistical power in the behavioural sciences and show that power has not improved despite repeated demonstrations of the necessity of increasing power. To demonstrate the logical consequences of structural incentives, we then present a dynamic model of scientific communities in which competing laboratories investigate novel or previously published hypotheses using culturally transmitted research methods. As in the real world, successful labs produce more ‘progeny,’ such that their methods are more often copied and their students are more likely to start labs of their own. Selection for high output leads to poorer methods and increasingly high false discovery rates. We additionally show that replication slows but does not stop the process of methodological deterioration. Improving the quality of research requires change at the institutional level.

Subfields: Ecological Modeling, Safety Research, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/835a5484292f32a3c02f507cbd8fb1f5d9f4aacf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Ecological Modeling

In Safety Research

In Statistics and Probability

Citations in the corpus

The Future of Sensitivity Analysis: An essential discipline for systems modeling and policy support (2020, neutral): “Current non-reproducible science is ecologically fit to the existing science governance arrangements (Smaldino and McElreath, 2016), including its ‘publish or perish ’ culture (Banobi et al., 2011), and is resistent to reform (Chalmers and Glasziou, 2009; Edwards and Roy, 2017).”
Power Analysis and Effect Size in Mixed Effects Models: A Tutorial (2018, background): “Unless these are based on a proper power analysis, it is to be expected that many of these “reasons” will be fallacies aimed at decreasing the work load rather than cherishing the quality of the research done (Smaldino & McElreath, 2016).”
Statistical Rituals: The Replication Delusion and How We Got There (2018, methodology): “A meta-analysis of 44 reviews published in the social and behavioral sciences, beginning with Cohen’s 1962 study, found that power had not increased over half a century (Smaldino & McElreath, 2016).”
Quality Uncertainty Erodes Trust in Science (2017, background): “If researchers can achieve the same result by doing shoddy science (which is cheaper, faster, and easier than rigorous science; Bakker, van Dijk, & Wicherts, 2012), there is little external incentive for them to do things the right way (Smaldino & McElreath, 2016; Tullett, 2015).”
The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities (2018, neutral): “In a final ironic twist, similar evolutionary arguments (applied to a higher level of biological organization) suggest that current incentive systems in science similarly produce surprising (and undesirable) byproducts [128].”
Abandon Statistical Significance (2017, background): “Consequently, low replication rates are to be expected given existing scientiﬁc practices (Ioannidis, 2005; Smaldino and McElreath, 2016), and calls for reform, which are not new (see, for example, Meehl (1978)), have become insistent.”
The science of science: from the perspective of complex systems (2017, background): “focused on the false-positive findings and confirmed via a case study that they persist in scientific publications [341].”