Bayesian Model Selection for Group Studies

By Klaas E. Stephan, W. D. Penny, Jean Daunizeau, Rosalyn J. Moran, Karl J. Friston. NeuroImage, 2009

1,478 citations recorded by the source corpus.

Bayesian model selection (BMS) is a powerful method for determining the most likely among a set of competing hypotheses about the mechanisms that generated observed data. BMS has recently found widespread application in neuroimaging, particularly in the context of dynamic causal modelling (DCM). However, so far, combining BMS results from several subjects has relied on simple (fixed effects) metrics, e.g. the group Bayes factor (GBF), that do not account for group heterogeneity or outliers. In this paper, we compare the GBF with two random effects methods for BMS at the between-subject or group level. These methods provide inference on model-space using a classical and Bayesian perspective respectively. First, a classical (frequentist) approach uses the log model evidence as a subject-specific summary statistic. This enables one to use analysis of variance to test for differences in log-evidences over models, relative to inter-subject differences. We then consider the same problem in Bayesian terms and describe a novel hierarchical model, which is optimised to furnish a probability density on the models themselves. This new variational Bayes method rests on treating the model as a random variable and estimating the parameters of a Dirichlet distribution which describes the probabilities for all models considered. These probabilities then define a multinomial distribution over model space, allowing one to compute how likely it is that a specific model generated the data of a randomly chosen subject as well as the exceedance probability of one model being more likely than any other model. Using empirical and synthetic data, we show that optimising a conditional density of the model probabilities, given the log-evidences for each model over subjects, is more informative and appropriate than both the GBF and frequentist tests of the log-evidences. In particular, we found that the hierarchical Bayesian approach is considerably more robust than either of the other approaches in the presence of outliers. We expect that this new random effects method will prove useful for a wide range of group studies, not only in the context of DCM, but also for other modelling endeavours, e.g. comparing different source reconstruction methods for EEG/MEG or selecting among competing computational models of learning and decision-making.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/3b7eda93a15e908a22d90ca3ee0cbcbbdb8fafb8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Statistics and Probability

Citations in the corpus

Understanding DCM: Ten simple rules for the clinician (2013, neutral): “This is not the same as “structural connectivity” measured through “tractography” or diffusion tensor imaging (DTI) white matter assessments (for primer, see Mori & Zhang 2006) - although structural and functional/effective connectivity are not independent as such (Stephan et al. 2009).”
Observing the Observer (II): Deciding When to Decide (2010, methodology): “However, note that this difference in model complexity does not distort or bias our model comparisons since the free energy approximation to the model evidence accounts for such differences in complexity [20].”
Hierarchical Active Inference: A Theory of Motivated Control (2018, background): “In short, appealing to active inference allows one to treat the costs and benefits of hierarchical imperatives in terms of Bayesian model selection in statistics, in which more complex models are penalized but may also enjoy a bonus if they confer greater accuracy over extended periods of applicatio”
Attention Reverses the Effect of Prediction in Silencing Sensory Signals (2011, neutral): “For both hemispheres, the most appropriate model, as determined by BMS (Stephan et al. 2009) and family level inference (Penny et al. 2010), contained modulatory influences of prediction on the selfconnection of V1, while the model in which prediction did not modulate this connection (model B) perfo”
EEG and MEG Data Analysis in SPM8 (2011, methodology): “SPM8 is integrated with the FieldTrip toolbox , making it possible for users to combine a variety of standard analysis methods with new schemes implemented in SPM and build custom analysis tools using powerful graphical user interface (GUI) and batching tools.”
Bayesian model selection for group studies - Revisited (2014, positive): “In this section, we first revisit the approach to random effects BMS proposed in Stephan et al. (2009), recasting it as an extension of Polya's urn model.”
How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis (2012, neutral): “We also report exceedance probabilities reﬂecting the likelihood that each model is the best of all candidate models given the distribution of AIC values across all subjects (Stephan et al. , 2009).”
The feedback-related negativity (FRN) revisited: New insights into the localization, meaning and network organization (2014, methodology): “Using a random-effect Bayesian model selection approach (Stephan et al., 2009), the model with the anticorrelated valuation showed a clearly better fit than the standard Rescorla–Wagner model (exceedance probability p N .999).”
VBA: A Probabilistic Treatment of Nonlinear Models for Neurobiological and Behavioural Data (2014, background): “[16] also introduced the notion of exceedance probability, which measures how likely it is that any given model is more frequent than all other models in the comparison set.”
Ten Simple Rules for Dynamic Causal Modelling (2009, neutral): “Incorporating anatomical information in this way can improve the evidence of DCMs (Stephan et al., 2009b).”
Cortical Hierarchies Perform Bayesian Causal Inference in Multisensory Perception (2015, methodology): “We performed Bayesian model selection [42] at the group level as implemented in SPM8 [43] to obtain the exceedance probability for the candidate models (i.”
Trial-by-trial data analysis using computational models (2011, background): “Note that one alternative sometimes observed in the literature (Stephan et al., 2009) is summary statistics reported and tested for individual subject Bayes factors (e.g., “across subjects, model M 1 is favored over M 2 by an average log Bayes factor of 4.1, which is signiﬁcantly different from…”
Information Selection in Noisy Environments with Large Action Spaces (2014, methodology): “For each model, we computed the (participantspecific) Bayesian Information Criterion approximation to the marginal likelihood, and then submitted the models to the Bayesian model selection algorithm of Stephan et al. (2009), which estimates the group-level exceedance probability for each model (the ”
Comparing Dynamic Causal Models using AIC, BIC and Free Energy (2012, background): “, 2010) and to look for consistent differences over a group of subjects (Stephan et al., 2009).”
Neural computations underlying arbitration between model-based and model-free learning (2013, methodology): “To formally test which version of the reliability computation provides the best account of responses in inferior lateral prefrontal cortex, we ran a Bayesian model selection(Stephan et al., 2009) on three models.”
Bayesian mixed-effects inference on classification performance in hierarchical data sets (2012, neutral): “In other words, the model evidence encodes how well a model strikes the balance between explaining the data and remaining simple (Pitt and Myung, 2002; Beal, 2003; Stephan et al., 2009).”
Generalization of value in reinforcement learning by humans (2012, methodology): “Such a test examines the null hypothesis that any improvement in model fit is due to chance, correcting for the inclusion of additional free parameters (e.g. Stephan et al., 2009).”
Modulating cortical connectivity in stroke patients by rTMS assessed with fMRI and dynamic causal modeling (2010, neutral): “Recent advances for group analyses in DCM overcome this limitation by using a random effects approach based on the conditional density of the models per se (Stephan et al., 2009).”
Mapping the self in the brain's default mode network (2016, neutral): “We computed posterior probabilities and protected exceedance probabilities at the group level (Stephan et al., 2009; Rigoux et al., 2014).”