Encoding and decoding in fMRI

By Thomas Naselaris, Kendrick Norris Kay, Shinji Nishimoto, Jack L. Gallant. 2010

996 citations recorded by the source corpus.

Over the past decade fMRI researchers have developed increasingly sensitive techniques for analyzing the information represented in BOLD activity. The most popular of these techniques is linear classification, a simple technique for decoding information about experimental stimuli or tasks from patterns of activity across an array of voxels. A more recent development is the voxel-based encoding model, which describes the information about the stimulus or task that is represented in the activity of single voxels. Encoding and decoding are complementary operations: encoding uses stimuli to predict activity while decoding uses activity to predict information about stimuli. However, in practice these two operations are often confused, and their respective strengths and weaknesses have not been made clear. Here we use the concept of a linearizing feature space to clarify the relationship between encoding and decoding. We show that encoding and decoding operations can both be used to investigate some of the most common questions about how information is represented in the brain. However, focusing on encoding models offers two important advantages over decoding. First, an encoding model can in principle provide a complete functional description of a region of interest, while a decoding model can provide only a partial description. Second, while it is straightforward to derive an optimal decoding model from an encoding model it is much more difficult to derive an encoding model from a decoding model. We propose a systematic modeling approach that begins by estimating an encoding model for every voxel in a scan and ends by using the estimated encoding models to perform decoding.

Subfields: Cognitive Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

Citations in the corpus

Causal and anti-causal learning in pattern recognition for neuroimaging (2015, background): “It has been argued that only encoding models can provide a complete functional description of a region of interest [1].”
On the interpretation of weight vectors of linear models in multivariate neuroimaging (2014, methodology): “Doing so, we build on prior work contained in Parra et al. (2005), Hyvärinen et al. (2009), Blankertz et al. (2011), Naselaris et al. (2011) and Bießmann et al. (2012b).”
Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision (2016, methodology): “Such models were de ﬁ ned separately for each voxel, namely voxel-wise encoding models (Naselaris et al. 2011), through which the voxel response was predicted from a linear combination of the feature representations of the input movie.”
Neural Coding of Cognitive Control: The Representational Similarity Analysis Approach. (2020, methodology): “These methods can be decomposed (noncomprehensively) into two variants: classification-based decoding, which we refer to here as ‘classification’, and RSA (Box 1; see also encoding methods [50]).”
Decoding neural representational spaces using multivariate pattern analysis. (2014, background): “…locations of pattern u t g Several excellent reviews have focused on MVP classiﬁcation (Norman et al. 2006, Haynes & Rees 2006, O’Toole et al. 2007, Pereira et al. 2009, Tong & Pratte 2012), RSA (Kriegeskorte & Kievet 2013), or stimulus-model-based encoding and decoding (Naselaris et al. 2011).”
Machine learning for neuroimaging with scikit-learn (2014, background): “The alternative that consists in predicting the imaging data given external variables, such as stimuli descriptors, is called encoding (Naselaris et al., 2011).”
Representational models: A common framework for understanding encoding, pattern-component, and representational-similarity analysis (2017, methodology): “In encoding analysis [10, 11], representational models are defined in terms of the underlying 147 features (Fig.”
Similarity judgments and cortical visual responses reflect different properties of object and scene categories in naturalistic images (2018, methodology): “An alternative approach to using multi-voxel patterns 673 is to model feature-selectivity at the individual voxel level (Naselaris et al., 2011).”
Relating Brain Structures To Open-Ended Descriptions Of Cognition (2017, methodology): “Neurosynth can be used in encoding settings, mapping the probability of activations in the brain associated with a given cognitive term (Naselaris, Kay, Nishimoto, & Gallant, 2011), or in decoding settings, mapping brain locations in which activity implies a given behavior (POLDRACK, 2006).”
Decoding Dynamic Brain Patterns from Evoked Responses: A Tutorial on Multivariate Pattern Analysis Applied to Time Series Neuroimaging Data (2016, methodology): “The term “ decoding ” refers to the prediction of a model from the data ( “ encoding ” approaches do the reverse, predicting the data from the model, reviewed in Naselaris, Kay, Nishimoto, & Gallant, 2011; see also, e.g., Ding & Simon, 2012, for an example of encoding models for MEG).”
Shared computational principles for language processing in humans and deep language models (2022, methodology): “The decoding analysis inverts this procedure to find a mapping from neural responses, across multiple electrodes and time points, to the embedding space 47 .”
A Task-Optimized Neural Network Replicates Human Auditory Behavior, Predicts Brain Responses, and Reveals a Cortical Processing Hierarchy. (2018, positive): “This general procedure has become standard for evaluating encoding models of brain responses (G€uçl€u and van Gerven, 2015; Klindt et al., 2017; Naselaris et al., 2011; Santoro et al., 2014; Yamins et al., 2014), the rationale being that the linear transformation discovered by regression aligns…”
Analyzing Neuroimaging Data Through Recurrent Deep Learning Models (2018, neutral): “, 1994) represents a univariate brain encoding model (Naselaris et al., 2011; Kriegeskorte and Douglas, 2018).”
A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence (2021, methodology): “Adopting the framework of encoding models 46 , we took NSD betas from visual areas V1–hV4, divided these data into a training set (used for parameter tuning) and a validation set (used to assess prediction performance) and evaluated how accurately different computational models predict brain…”