Assessing and tuning brain decoders: Cross‐validation, caveats, and guidelines

By Gaël Varoquaux, Pradeep Reddy Raamana, Denis-Alexander Engemann, Andrés Hoyos Idrobo, Yannick Schwartz, Bertrand Thirion. NeuroImage, 2016

748 citations recorded by the source corpus.

ABSTRACT Decoding, i.e. prediction from brain images or signals, calls for empirical evaluation of its predictive power. Such evaluation is achieved via cross‐validation, a method also used to tune decoders' hyper‐parameters. This paper is a review on cross‐validation procedures for decoding in neuroimaging. It includes a didactic overview of the relevant theoretical considerations. Practical aspects are highlighted with an extensive empirical study of the common decoders in within‐ and across‐subject predictions, on multiple datasets –anatomical and functional MRI and MEG– and simulations. Theory and experiments outline that the popular “leave‐one‐out” strategy leads to unstable and biased estimates, and a repeated random splits method should be preferred. Experiments outline the large error bars of cross‐validation in neuroimaging settings: typical confidence intervals of 10%. Nested cross‐validation can tune decoders' parameters while avoiding circularity bias. However we find that it can be favorable to use sane defaults, in particular for non‐sparse decoders. Graphical abstract Figure. No Caption available. HighlightsWe give a primer on cross‐validation to measure decoders predictive power.We assess on many datasets its practical use for decoding selection and tuning.Cross‐validation displays large confidence intervals, in particular leave one out.Default parameters on standard decoders can outperform parameter tuning.

Subfields: Cognitive Neuroscience

View Paper: https://www.semanticscholar.org/paper/d9915e99afd3136e6f2429d4e65dca8d89f0faa9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

Citations in the corpus

Benchmarking functional connectome‐based predictive models for resting‐state fMRI (2019, positive): “We perform cross-validation (CV) by randomly shuf ﬂ ing and splitting each dataset over 100 folds, forming two sets of subjects: 75% for training the classi ﬁ er and learning brain atlases with data-driven models and the remaining 25% for testing on unseen data (Varoquaux et al., 2017).”
Spatial Topography of Individual-Specific Cortical Networks Predicts Human Cognition, Personality and Emotion (2017, background): “Because a single 20-fold cross-validation might be sensitive to the particular split of the data into folds (Varoquaux et al., 2017), the above 20-fold cross-validation was repeated 100 times.”
Ten simple rules for predictive modeling of individual differences in neuroimaging (2019, background): “More generally, best practices appropriate to the stage of the discovery (i.e., confirmatory or exploratory) alleviate many multiple testing concerns (Wagenmakers et al., 2012).”
Cross‐validation failure: Small sample sizes lead to large error bars (2017, methodology): “Technically, this test is done via cross-validation: the available data is split in two, a first part, the train set used to fit themodel, and a second part, the test set used to test themodel (Pereira et al., 2009; Varoquaux et al., 2017).”
Machine learning algorithm validation with a limited sample size (2019, background): “Practically a distribution of performance measures could be produced by, for example, multiple times differently/randomly splitting the data to CV folds [27].”
Intracranial EEG signals disentangle multi-areal neural dynamics of vicarious pain perception (2024, methodology): “The useof aseparate sub-dataset,insteadofthefulldataset, to select relevant features has been suggested as a way to avoid potential problems of information leakage 104 – 106 .”