The effect of machine learning regression algorithms and sample size on individualized behavioral prediction with functional connectivity features

By Zaixu Cui, Gaolang Gong. NeuroImage, 2018

337 citations recorded by the source corpus.

Abstract Individualized behavioral/cognitive prediction using machine learning (ML) regression approaches is becoming increasingly applied. The specific ML regression algorithm and sample size are two key factors that non‐trivially influence prediction accuracies. However, the effects of the ML regression algorithm and sample size on individualized behavioral/cognitive prediction performance have not been comprehensively assessed. To address this issue, the present study included six commonly used ML regression algorithms: ordinary least squares (OLS) regression, least absolute shrinkage and selection operator (LASSO) regression, ridge regression, elastic‐net regression, linear support vector regression (LSVR), and relevance vector regression (RVR), to perform specific behavioral/cognitive predictions based on different sample sizes. Specifically, the publicly available resting‐state functional MRI (rs‐fMRI) dataset from the Human Connectome Project (HCP) was used, and whole‐brain resting‐state functional connectivity (rsFC) or rsFC strength (rsFCS) were extracted as prediction features. Twenty‐five sample sizes (ranged from 20 to 700) were studied by sub‐sampling from the entire HCP cohort. The analyses showed that rsFC‐based LASSO regression performed remarkably worse than the other algorithms, and rsFCS‐based OLS regression performed markedly worse than the other algorithms. Regardless of the algorithm and feature type, both the prediction accuracy and its stability exponentially increased with increasing sample size. The specific patterns of the observed algorithm and sample size effects were well replicated in the prediction using re‐testing fMRI data, data processed by different imaging preprocessing schemes, and different behavioral/cognitive scores, thus indicating excellent robustness/generalization of the effects. The current findings provide critical insight into how the selected ML regression algorithm and sample size influence individualized predictions of behavior/cognition and offer important guidance for choosing the ML regression algorithm or sample size in relevant investigations. HighlightsIndividualized prediction is influenced by regression algorithm and sample size.LASSO regression performed worse than other algorithms using rsFC feature.OLS regression performed worse than other algorithms using rsFCS feature.Prediction accuracy and its stability exponentially increased with sample size.The observed algorithm and sample size effects are robust and generalizable.

Subfields: Cognitive Neuroscience, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/37a4f26df5e2c7a51729f1e8fc8f2ad88dfd35da

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

1. Dustin Scheinost
2. X. Shen
3. S. Noble
4. Monica D. Rosenberg
5. R. Todd Constable

In Radiology, Nuclear Medicine and Imaging

1. R. Todd Constable
2. S. Noble
3. X. Shen
4. Dustin Scheinost

Citations in the corpus

• Evaluation of a decided sample size in machine learning applications (2023, background): “Taken together, a small training sample size may exaggerate the accuracy of ML due to overfitting or random effects, whereas large-scale studies require more financial resources and consume more time [7, 12, 13].”
• Ten simple rules for predictive modeling of individual differences in neuroimaging (2019, background): “Increasing the sample size for training and testing will help to create more generalizable models and less variable prediction results (Cui and Gong, 2018; Schnack and Kahn, 2016; Varoquaux et al., 2017).”