Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints

By T. van der Ploeg, Peter C. Austin, Ewout W. Steyerberg. BMC Medical Research Methodology, 2014

674 citations recorded by the source corpus.

BackgroundModern modelling techniques may potentially provide more accurate predictions of binary outcomes than classical techniques. We aimed to study the predictive performance of different modelling techniques in relation to the effective sample size (“data hungriness”).MethodsWe performed simulation studies based on three clinical cohorts: 1282 patients with head and neck cancer (with 46.9% 5 year survival), 1731 patients with traumatic brain injury (22.3% 6 month mortality) and 3181 patients with minor head injury (7.6% with CT scan abnormalities). We compared three relatively modern modelling techniques: support vector machines (SVM), neural nets (NN), and random forests (RF) and two classical techniques: logistic regression (LR) and classification and regression trees (CART). We created three large artificial databases with 20 fold, 10 fold and 6 fold replication of subjects, where we generated dichotomous outcomes according to different underlying models. We applied each modelling technique to increasingly larger development parts (100 repetitions). The area under the ROC-curve (AUC) indicated the performance of each model in the development part and in an independent validation part. Data hungriness was defined by plateauing of AUC and small optimism (difference between the mean apparent AUC and the mean validated AUC <0.01).ResultsWe found that a stable AUC was reached by LR at approximately 20 to 50 events per variable, followed by CART, SVM, NN and RF models. Optimism decreased with increasing sample sizes and the same ranking of techniques. The RF, SVM and NN models showed instability and a high optimism even with >200 events per variable.ConclusionsModern modelling techniques such as SVM, NN and RF may need over 10 times as many events per variable to achieve a stable AUC and a small optimism than classical modelling techniques such as LR. This implies that such modern techniques should only be used in medical prediction problems if very large data sets are available.

Subfields: Artificial Intelligence, Neurology

View Paper: https://www.semanticscholar.org/paper/145fbe557741acce5c43afb4193dc2bef6723acb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Neurology

Citations in the corpus

Multicenter Comparison of Machine Learning Methods and Conventional Regression for Predicting Clinical Deterioration on the Wards (2016, background): “This enhanced flexibility can lead to improved accuracy but also increases the chance of overfitting, and studies suggest that large amounts of data are needed to estimate a stable random forest model (17).”
Comparison of Variable Selection Methods for Clinical Predictive Modeling (2018, methodology): “However, there has been limited research comparing the performance of classic and modern modeling methods in clinical datasets [5].”
Review and evaluation of penalised regression methods for risk prediction in low‐dimensional data with few events (2015, methodology): “Also, techniques from machine learning, such as random forests, neural networks and support vector machines, have been used recently for prediction with medical data and have the potential to outperform standard regression methods in some cases, although recent studies showed that those methods requ”
A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. (2019, background): “In contradiction with the aforementioned claim, recent research suggests that ML requires more data than LR [17].”
Moving beyond regression techniques in cardiovascular risk prediction: applying machine learning to address analytic challenges (2016, neutral): “24”
Minimum sample size for developing a multivariable prediction model: PART II ‐ binary and time‐to‐event outcomes (2018, methodology): “When R(2)Royston_app is reported it can be used to obtain R 2 ÓQuigley_app by rearranging Equation (19) as”