Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research: A Multidisciplinary View

By Wei Luo, Dinh Phung, Truyen Tran, Sunil Gupta, Santu Rana, Chandan Kumar Karmakar, Alistair Shilton, J. Yearwood, N. Dimitrova, T. Ho and 2 more. 2016

874 citations recorded by the source corpus.

Background As more and more researchers are turning to big data for new opportunities of biomedical discoveries, machine learning models, as the backbone of big data analysis, are mentioned more often in biomedical journals. However, owing to the inherent complexity of machine learning methods, they are prone to misuse. Because of the flexibility in specifying machine learning models, the results are often insufficiently reported in research articles, hindering reliable assessment of model validity and consistent interpretation of model outputs. Objective To attain a set of guidelines on the use of machine learning predictive models within clinical settings to make sure the models are correctly applied and sufficiently reported so that true discoveries can be distinguished from random coincidence. Methods A multidisciplinary panel of machine learning experts, clinicians, and traditional statisticians were interviewed, using an iterative process in accordance with the Delphi method. Results The process produced a set of guidelines that consists of (1) a list of reporting items to be included in a research article and (2) a set of practical sequential steps for developing predictive models. Conclusions A set of guidelines was generated to enable correct application of machine learning models and consistent reporting of model specifications and results in biomedical research. We believe that such guidelines will accelerate the adoption of big data analysis, particularly with machine learning methods, in the biomedical research community.

Subfields: Artificial Intelligence, Health Informatics, Sociology and Political Science

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Sociology and Political Science

Citations in the corpus

The myth of generalisability in clinical research and machine learning in health care (2020, background): “8 This request is only natural: such journals often have diverse readerships, and research articles with widespread relevance to many readers are more likely to be read and circulated, increasing the visibility of the journal and publishers.”
Coarse race data conceals disparities in clinical risk score performance (2023, methodology): “We split the dataset at the patient level, not the visit level, as is standard to prevent data leakage (Luo et al., 2016; Tampu et al., 2022); thus, no patient appears in both the train and test set.”
A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. (2019, background): “Some researchers claim that ML will not outperform LR when only a limited set of prespeciﬁed predictors is considered, and that the advantage of ML lies in better handling a huge amount of predictors [3,7,12,15,16,104].”
Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022, methodology): “The literature stresses that detailed description of any preprocessing steps applied to the raw data should be provided, including software used to perform the processing steps 3,6,7,9,13 – 15, 22, 50, 61, 62, Data coding standards .”
Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A Guide for Authors and Reviewers. (2020, background): “When the direct calculation of confidence intervals is not possible, report non-parametric estimates from bootstrap samples (11).”
Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers (2019, methodology): “This was to ensure that the validation data were not just a timesplit subsample of the original large dataset, as that results in a type of internal validation (25).”
Detecting Narcissistic Personality Disorder Traits on Forums: Proof-of-Concept Study (2026, unknown): “We adhered to the Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research [51], documenting each stage of our workflow, from data sourcing and outcome definition to feature engineering, hyperparameter tuning, internal validation, and performance reporting.”