On the interpretation of weight vectors of linear models in multivariate neuroimaging
By Stefan Haufe, Frank C. Meinecke, Kai Görgen, Sven Dähne, John-Dylan Haynes, Benjamin Blankertz, Felix Biessmann. 2014
1,365 citations recorded by the source corpus.
The increase in spatiotemporal resolution of neuroimaging devices is accompanied by a trend towards more powerful multivariate analysis methods. Often it is desired to interpret the outcome of these methods with respect to the cognitive processes under study. Here we discuss which methods allow for such interpretations, and provide guidelines for choosing an appropriate analysis for a given experimental goal: For a surgeon who needs to decide where to remove brain tissue it is most important to determine the origin of cognitive functions and associated neural processes. In contrast, when communicating with paralyzed or comatose patients via brain-computer interfaces, it is most important to accurately extract the neural processes specific to a certain mental state. These equally important but complementary objectives require different analysis methods. Determining the origin of neural processes in time or space from the parameters of a data-driven model requires what we call a forward model of the data; such a model explains how the measured data was generated from the neural sources. Examples are general linear models (GLMs). Methods for the extraction of neural information from data can be considered as backward models, as they attempt to reverse the data generating process. Examples are multivariate classifiers. Here we demonstrate that the parameters of forward models are neurophysiologically interpretable in the sense that significant nonzero weights are only observed at channels the activity of which is related to the brain process under study. In contrast, the interpretation of backward model parameters can lead to wrong conclusions regarding the spatial or temporal origin of the neural signals of interest, since significant nonzero weights may also be observed at channels the activity of which is statistically independent of the brain process under study. As a remedy for the linear case, we propose a procedure for transforming backward models into forward models. This procedure enables the neurophysiological interpretation of the parameters of linear backward models. We hope that this work raises awareness for an often encountered problem and provides a theoretical basis for conducting better interpretable multivariate neuroimaging analyses.
Subfields: Artificial Intelligence, Cognitive Neuroscience, Geometry and Topology, Safety, Robustness & Interpretability, Neuroscience
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- K. Müller
- Wojciech Samek
- Grgoire Montavon
- Yoshua Bengio
- Been Kim
- Sebastian Lapuschkin
- Mohit Bansal
- Christopher J. Anders
In Cognitive Neuroscience
- Cuntai Guan
- Dustin Scheinost
- X. Shen
- K. Müller
- Grgoire Montavon
- Martin N. Hebart
- Erico Tjoa
- Sebastian Lapuschkin
In Geometry and Topology
- Aristeidis Sotiras
- Grgoire Montavon
- K. Müller
- John-Dylan Haynes
- Max Welling
- Wojciech Samek
- Bernhard Schölkopf
- Christos Davatzikos
In Safety, Robustness & Interpretability
- K. Müller
- Wojciech Samek
- Grgoire Montavon
- Been Kim
- Sebastian Lapuschkin
- Christopher J. Anders
- Pieter-Jan Kindermans
- Alexander Binder
In Neuroscience
- Grgoire Montavon
- Sebastian Lapuschkin
- Wojciech Samek
- Bernhard Schölkopf
- K. Müller
- Alexander Binder
- John-Dylan Haynes
- Max Welling
Citations in the corpus
- A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, background): “In the medical field (see later section), [62] [80] suggest linear combination of variables as the means to interpretability (for example clinical variables, metabolites signals for MRS etc).”
- A Survey of Methods for Explaining Black Box Models (2018, methodology): “In particular, the f described in Reference [39] enables the neurophysiological interpretation of the parameters of nonlinear models.”
- Methods for interpreting and understanding deep neural networks (2017, neutral): “For classical linear explanation of neural activation patterns in cognitive brain science experiments or Brain Computer Interfacing, see [13, 30, 12, 22].”
- Learning how to explain neural networks: PatternNet and PatternAttribution (2017, methodology): “Figure 2: For linear models, i.e., a simple neural network, the weight vector does not explain the signal it detects Haufe et al. (2014).”
- Explaining nonlinear classification decisions with deep Taylor decomposition (2015, background): “Linear models readily provide explanations in terms of input variables [20, 21, 22], however, due to their lim-”
- Causal and anti-causal learning in pattern recognition for neuroimaging (2015, background): “While in linear decoding models potential misinterpretations can be avoided by converting them into encoding models [3], this is a substantially more difficult”
- Visualizing Deep Neural Network Decisions: Prediction Difference Analysis (2017, background): “These are independent of the input image, and, as argued by Gaonkar & Davatzikos (2013) and Haufe et al. (2014), interpreting these weights can be misleading in general.”
- Investigating the influence of noise and distractors on the interpretation of neural networks (2016, critical): “Finally, we propose a new rule based on what is commonly applied in Neuroimaging [2].”
- Exploring EEG Features in Cross-Subject Emotion Recognition (2018, methodology): “Haufe et al. (2014) indicated that the interpretation of the parameters in backward methods (multivariate classifiers) may lead to the wrong conclusions in neuroimaging data modeling.”
- Analysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Survey (2019, methodology): “Based on an analysis of linear models, see also [104], [65] they propose a generalization that yielded the two neuron-wise explanation techniques PatternNet (for signal visualization) and PatternAttribution (for decomposition methods) by taking the data distribution into account.”
- Predicting Age Using Neuroimaging: Innovative Brain Ageing Biomarkers. (2017, neutral): “However, there are several important reasons why interpreting the ‘weight maps’ derived from machine learning is complicated and does not offer a straightforward interpretation in the context of brain ageing [51].”
- EEG Based Emotion Recognition: A Tutorial and Review (2022, background): “[118] pointed out that the research on the parameters of backward methods (multivariate classifiers) is not recommended in brain imaging data analysis.”
- On The Stability of Interpretable Models (2018, background): “• On the subset of instances (instance selection): instance generation/perturbation in black-box explanation can be purely random [3], [4], or adopt refined approaches, e.”
- Real-time Neuroimaging and Cognitive Monitoring Using Wearable Dry EEG (2015, neutral): “While caution should be exercised in overinterpreting such weights [70], the ProxConn regression approach may likewise yield insight into source-level networks predictive of cognitive and”
- Closed-loop stimulation of temporal cortex rescues functional networks and improves memory (2018, positive): “To determine what information the model used to classify encoding trials we computed a Forward Model for each patient using the classi fi er weights and neural encoding activity 34,38 .”
- PatternNet and PatternLRP - Improving the interpretability of neural networks (2017, methodology): “Using the terminology of [16], the direction as along which the signal varies is called the pattern.”