Why Most Published Research Findings Are False
By John P. A. Ioannidis. 2005
9,605 citations recorded by the source corpus.
Summary There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.
Subfields: Economics and Econometrics, Statistics and Probability, Social Sciences & Humanities
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Economics and Econometrics
- Fotios Petropoulos
- Daniël Lakens
- Rob J Hyndman
- John P. A. Ioannidis
- Stanley E. Lazic
- Zoltan Dienes
- Douglas B. Kell
- Paul E. Smaldino
In Statistics and Probability
- Bryan G. Cook
- Daniël Lakens
- John P. A. Ioannidis
- Jonathan E. Taylor
- Andrew Gelman
- Vitaly Feldman
- Robert Tibshirani
- Andrea Saltelli
In Social Sciences & Humanities
- Paul E. Smaldino
- Daniël Lakens
- Jana Winter
- Charles Lambdin
- Daniele Fanelli
- Uri Simonsohn
- Joseph P. Simmons
- Leif D. Nelson
Citations in the corpus
- What Do NLP Researchers Believe? Results of the NLP Community Metasurvey (2022, background): “On the other hand, it could simply mean that many reported findings are of little importance or are not robust, which would arguably not make NLP unique among sciences (Ioannidis, 2005).”
- Show Your Work: Improved Reporting of Experimental Results (2019, background): “In NLP, the use of standardized test sets and public leaderboards (which limit test evaluations) has helped to mitigate the so-called “replication crisis” happening in fields such as psychology and medicine (Ioannidis, 2005; Gelman and Loken, 2014).”
- Critical data studies: An introduction (2016, background): “…extracted from data subjects (Hauge et al., 2016; Metcalf and Crawford, 2016), hijacked to serve agendas that benefit research and industry (Ioannidis, 2005, 2016), and compromised by the interests of not only powerful business organizations but also hackers and rogue agents (Coleman,…”
- Preserving Statistical Validity in Adaptive Data Analysis (2014, background): “Throughout the scientific community there is a growing recognition that claims of statistical significance in published research are frequently invalid [29, 28, 35, 8].”
- The ethics of algorithms: Mapping the debate (2016, neutral): “The search for causal links is difficult, as correlations established in large, proprietary datasets are frequently not reproducible or falsifiable (cf. Ioannidis, 2005; Lazer et al., 2014).”
- The Ladder: A Reliable Leaderboard for Machine Learning Competitions (2015, background): “Finally, in the context of false discovery in the empirical sciences (Ioannidis, 2005; Gelman & Loken, 2014), one could imagine using the the Ladder mechanism as a way of keeping track of scientific progress on important public data sets.”
- GWAS of 126,559 Individuals Identifies Genetic Variants Associated with Educational Attainment (2013, background): “…the likelihood that effect sizes of individual polymorphisms are small, that studies have therefore been underpowered, and that multiple hypotheses and methods of analysis have been explored; these conditions will result in an unacceptably high proportion of false findings (Ioannidis 2005).””
- A Simple Method for Removing Bias From a Popular Measure of Standardized Effect Size: Adjusted Partial Eta Squared (2019, background): “Another strength is that η p 2 is a convenient starting point for a priori power analysis (see, e.g., Cohen, 1988), which is of high interest to all empirical scientists, especially those working in fields that may be suffering from a replication crisis (see, e.g., Ioannidis, 2005).”
- Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature (2016, methodology): “…examined p value reporting errors in the cognitive neuroscience literature as recent studies found that 12.5% to 20% of psychology papers have major p value reporting errors which affect judgments about statistical significance (Bakker and Wicherts, 2011; Nuijten et al. 2015; Veldkamp et al. 2014).”
- Is There Really Granger Causality between Energy Use and Output? (2014, background): “This has led a prominent meta-analyst to claim that: “most published research findings are false” (Ioannidis, 2005).”
- Issues in information theory-based statistical inference—a commentary from a frequentist’s perspective (2010, background): “For recent contributions to the issue in other research areas see, for example, Ioannidis (2005), Young et al. (2009) or Vul et al. (2010).”
- Implementing Reproducible Research (2014, background): “The self-correcting nature of science has been questioned repeatedly in the decade since Ioannidis (2005), “Why Most Published Research Findings Are False”.”
- On doing relevant and rigorous experiments: Review and recommendations (2018, background): “…& Joo, 2013), are heavily unstable with respect to the analytical strategy chosen (Button et al., 2013), and have low statistical power (Ioannidis, 2005); at the research field level, small sample sizes tend to inflate the effect size of the treatment, given that only significant…”
- Estimating the Reproducibility of Experimental Philosophy (2018, background): “…analogous initiative for cancer biology (Nosek & Errington, 2017), meta-scientist John Ioannidis’s modeling efforts in biomedicine and beyond (e.g., Ioannidis, 2005) and a 2015 estimate produced by the Board of Governors of the Federal Reserve System for research in economics (Chang & Li, 2015).”