Research: increasing value, reducing waste 2

By John P. A. Ioannidis, Sander Greenland, M. Hlatky, M. Khoury, M. Macleod, David Moher, K. F. Schulz, Robert Tibshirani. The Lancet, 2014

1,340 citations recorded by the source corpus.

Correctable weaknesses in the design, conduct, and analysis of biomedical and public health research studies can produce misleading results and waste valuable resources. Small effects can be difficult to distinguish from bias introduced by study design and analyses. An absence of detailed written protocols and poor documentation of research is common. Information obtained might not be useful or important, and statistical precision or power is often too low or used in a misleading way. Insufficient consideration might be given to both previous and continuing studies. Arbitrary choice of analyses and an overemphasis on random extremes might affect the reported findings. Several problems relate to the research workforce, including failure to involve experienced statisticians and methodologists, failure to train clinical researchers and laboratory scientists in research methods and design, and the involvement of stakeholders with conflicts of interest. Inadequate emphasis is placed on recording of research decisions and on reproducibility of research. Finally, reward systems incentivise quantity more than quality, and novelty more than reliability. We propose potential solutions for these problems, including improvements in protocols and documentation, consideration of evidence from studies in progress, standardisation of research efforts, optimisation and training of an experienced and non-conflicted scientific workforce, and reconsideration of scientific reward systems.

Subfields: Economics and Econometrics, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/29c6fcbe81f3832244e751a943c5a213d81d3e48

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Statistics and Probability

Citations in the corpus

Analysis of drug combinations: current methodological landscape (2015, neutral): “As methods can be complex and continue to evolve, statisticians and methodologists should be involved in all stages of drug combination research and development, which in practice is too often sporadically the case leading to flawed designs and analysis (Ioannidis et al. 2014).”
Surrogate Science (2015, background): “Scientists at Amgen were unable to replicate 47 out of 53 major oncological findings for potential drug targets, nor were researchers at Bayer able to replicate 43 out of 67 oncological and cardiovascular findings (see Ioannidis et al., 2014).”
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research (2015, neutral): “More pessimistically, investigators may just see this as another checklist they have to satisfy that has no real effect on improving their research.(6) Again, a monitoring component would be key to include alongside a change in process that attempts to recognize quality.”
COSMIN Risk of Bias checklist for systematic reviews of Patient-Reported Outcome Measures (2017, methodology): “Research performed with Patient-Reported Outcome Measures (PROMs) of poor or unknown quality constitutes a waste of resources and is unethical [1].”
Experimental design and analysis and their reporting: new guidance for publication in BJP (2015, background): “…research suggests that fewer than 30% of studies in the bio-medical preclinical literature report randomization and fewer than 5% report blinded analysis (Ioannidis et al ., 2014), a level that has been described, correctly in our view, as ‘piti-fully infrequent’ (Macleod et al ., 2014).”
The Human Behaviour-Change Project: harnessing the power of artificial intelligence and machine learning for evidence synthesis and interpretation (2017, background): “The situation has been improved by the publication of a number of guidelines [6], but intervention evaluations still vary widely in quality and format, and are reported inconsistently and incompletely using terminology with limited standardisation [7].”
External validation of multivariable prediction models: a systematic review of methodological conduct and reporting (2014, background): “Replicating findings obtained during the original development of the prediction model in different data but from the same underlying target population is key [19-21].”
A new framework to enhance the interpretation of external validation studies of clinical prediction models. (2015, neutral): “[44].”
The Economics of Reproducibility in Preclinical Research (2015, background): “However, it is reasonable to state that cumulative errors in the following broad categories—as well as underlying biases that could contribute to each problem area [14] or even result in entire studies never being published or reported [15]—are the primary causes of irreproducibility [16]: (1) study”