A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models.
By Evangelia Christodoulou, Jie Ma, Gary S. Collins, Ewout W. Steyerberg, J. Verbakel, B. Van calster. 2019
1,645 citations recorded by the source corpus.
OBJECTIVES The objective of this study was to compare performance of logistic regression (LR) with machine learning (ML) for clinical prediction modeling in the literature. STUDY DESIGN AND SETTING We conducted a Medline literature search (1/2016 to 8/2017) and extracted comparisons between LR and ML models for binary outcomes. RESULTS We included 71 of 927 studies. The median sample size was 1,250 (range 72-3,994,872), with 19 predictors considered (range 5-563) and eight events per predictor (range 0.3-6,697). The most common ML methods were classification trees, random forests, artificial neural networks, and support vector machines. In 48 (68%) studies, we observed potential bias in the validation procedures. Sixty-four (90%) studies used the area under the receiver operating characteristic curve (AUC) to assess discrimination. Calibration was not addressed in 56 (79%) studies. We identified 282 comparisons between an LR and ML model (AUC range, 0.52-0.99). For 145 comparisons at low risk of bias, the difference in logit(AUC) between LR and ML was 0.00 (95% confidence interval, -0.18 to 0.18). For 137 comparisons at high risk of bias, logit(AUC) was 0.34 (0.20-0.47) higher for ML. CONCLUSION We found no evidence of superior performance of ML over LR. Improvements in methodology and reporting are needed for studies that compare modeling algorithms.
Subfields: Artificial Intelligence, Health Informatics, Evaluation & Benchmarks, Medical & Health AI
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Lesia Semenova
- Chudi Zhong
- Zhi Chen
- Haiyang Huang
- Cynthia Rudin
- Momin M. Malik
- Gary S. Collins
- Tammy Jiang
In Health Informatics
- Zhi Chen
- Lesia Semenova
- Chudi Zhong
- Haiyang Huang
- Momin M. Malik
- Gary S. Collins
- Karel G. M. Moons
- Cynthia Rudin
In Evaluation & Benchmarks
In Medical & Health AI
- Lesia Semenova
- Momin M. Malik
- Chudi Zhong
- Zhi Chen
- Haiyang Huang
- John P. A. Ioannidis
- Gary S. Collins
- Cynthia Rudin
Citations in the corpus
- Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, methodology): “This means it is often difficult even to beat logistic regression, assuming one is willing to perform minor preprocessing such as creating dummy variables (e.g., see Christodoulou et al., 2019).”
- A Hierarchy of Limitations in Machine Learning (2020, result): “36Confusion around this is shown in a recent exchange of Christodoulou et al. (2019); Bian et al. (2019); Van Calster et al. (2019).”
- Early detection of cancer (2022, positive): “In some cases, AI and MLmethodology might simply not have advantages over statistical methods such as logistic regression (94)—the right tools should be used for the intended purpose.”
- Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness (2020, background): “40 We would urge researchers to keep this in mind when carrying out such comparisons.”
- Artificial Intelligence Applications in Dermatology: Where Do We Stand? (neutral): “…the review does summarize various theoretical reasons why machine learning may be superior to longitudinal regression in certain instances such as processing data with a strong signal to noise ratio (e.g., handwriting) or with a significant number of predictor subcategories (e.g., images) (86).”
- Supervised machine learning: A brief primer (2020, background): “Further, there is the potential for negligible incremental performance compared to traditional approaches and thus machine learning may not be preferable or efficient for certain data structures (Christodoulou et al., 2019).”