Circular analysis in systems neuroscience – the dangers of double dipping

By Nikolaus Kriegeskorte, Kyle Simmons, P. Bellgowan, Chris L. Baker. Nature Neuroscience, 2009

2,754 citations recorded by the source corpus.

A neuroscientific experiment typically generates a large amount of data, of which only a small fraction is analyzed in detail and presented in a publication. However, selection among noisy measurements can render circular an otherwise appropriate analysis and invalidate results. Here we argue that systems neuroscience needs to adjust some widespread practices to avoid the circularity that can arise from selection. In particular, 'double dipping', the use of the same dataset for selection and selective analysis, will give distorted descriptive statistics and invalid statistical inference whenever the results statistics are not inherently independent of the selection criteria under the null hypothesis. To demonstrate the problem, we apply widely used analyses to noise data known to not contain the experimental effects in question. Spurious effects can appear in the context of both univariate activation analysis and multivariate pattern-information analysis. We suggest a policy for avoiding circularity.

Subfields: Cognitive Neuroscience

View Paper: https://www.semanticscholar.org/paper/13eb77434b41b977164d59fc5f4c2907aba1f4f3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

Citations in the corpus

An algorithmic method for functionally defining regions of interest in the ventral visual pathway (2012, methodology): “Because we wanted to later examine the response pro ﬁ les of the resulting fROIs in an independent subset of the data (see Kriegeskorte et al., 2009; Vul and Kanwisher, 2010)), we excluded the ﬁ rst functional run from these contrasts, and de ﬁ ned fROIs using the remaining three runs.”
BCILAB: a platform for brain–computer interface development (2013, methodology): “In particular, it is difﬁcult to accidentally employ unsound evaluation procedures such as ‘ testing on the training data ’ or other forms of inadvertent ‘ double-dipping ’ [26] or to make other often encountered methodological errors such as inadvertent non-causal ﬁltering.”
A review on neuroimaging‐based classification studies and associated feature extraction methods for Alzheimer's disease and its prodromal stages (2017, neutral): “The term double-dipping, or circular analysis, refers to the use of test subjects in any part of the training process, such as selection of features and training of classification algorithm, and may lead to over-fitted classification (Kriegeskorte et al., 2009).”
The Decoding Toolbox (TDT): a versatile software package for multivariate analyses of functional imaging data (2015, background): “Just as problematic—and not uncommon even for experts—is the unintentional wrong use of decoding methods, for example circular analyses (Kriegeskorte et al., 2009).”
Behavioral Interpretations of Intrinsic Connectivity Networks (2011, background): “In addition, they may be useful as masks for seeding specific a priori cortical regions or networks of interest in prospective neuroimaging studies or as a technique for circumventing the inherent problems associated with double dipping (Kriegeskorte, Simmons, Bellgowan, & Baker, 2009 ).”
The thing that should not be: predictive coding and the uncanny valley in perceiving human and humanoid robot actions (2011, methodology): “In a balanced factorial design with equiprobable conditions (as was used here), this process does not bias the chances of finding an interaction (Kriegeskorte et al., 2009).”
Intrinsic and task-evoked network architectures of the human brain. (2014, positive): “…because it sampled from every major brain system, was estimated using independent data (reducing potential circularity or biases from overfitting in our analyses [Kriegeskorte et al., 2009]), and came with an independently identified node community partition (again, to reduce potential biases).”
Best practice guidance for linear mixed-effects models in psychological science (2019, methodology): “Thus, this paper reports on the use of LMMs in the context of ongoing concerns regarding statistical best practices across the cognitive and neurosciences (e.g., Carp, 2012a, 2012b; Chabris et al., 2012; Cumming, 2013a, 2013b; Ioannidis, 2005; Kriegeskorte et al., 2009; Lieberman & Cunningham, 2009;”
Neuroimaging of the Periaqueductal Gray: State of the Field (2011, background): “There is little reason to believe that the PAG literature is free from the over-interpretation (Logothetis, 2008), publication bias (Ioannidis, 2011; Jennings and Van Horn, 2011), double dipping (Kriegeskorte et al., 2009) and other methodological shortcomings observed in other neuroimaging fields (”
A Reproducible Evaluation of ANTs Similarity Metric Performance in Brain Image Registration (2010, methodology): “Results are artificially inflated when the same data are used in both testing and training (Vul et al., 2009; Kriegeskorte et al., 2009).”
Dysfunction of the prefrontal cortex in addiction: neuroimaging findings and clinical implications (2011, methodology): “A way to circumvent the limitations of post-hoc analyses is to perform both whole-brain analyses and use a priori defined anatomical ROI”
Whole genome association study of brain-wide imaging phenotypes for identifying quantitative trait loci in MCI and AD: A study of the ADNI cohort (2010, result): “Given the recent interest in the non-independent analysis issue (e.g., (Kriegeskorte et al., 2009)), independent datasets for replication will be important for future studies to confirm the findings.”
The problem of pseudoreplication in neuroscientific studies: is it affecting your analysis? (2010, background): “There have been some recent papers critical of common statistical practices in the neuroimaging field, where lack of independence is also a central issue [51].”
Early developmental emergence of human amygdala–prefrontal connectivity after maternal deprivation (2013, methodology): “To avoid nonindependence in statistical analyses (68), we ﬁ rst identi ﬁ ed a group × emotional run interaction in mPFC in AFNI and then extracted beta weights for the identi ﬁ ed cluster for the fear run.”
A Convergent Functional Architecture of the Insula Emerges Across Imaging Modalities (2012, neutral): “, 2011) the use of pre-defined ROIs allows researchers to test hypotheses concerning differential localization or timing of stimulus or task-evoked responses without “double-dipping” (Kriegeskorte et al., 2009; Vul et al., 2009).”