Representational Similarity Analysis – Connecting the Branches of Systems Neuroscience
By Nikolaus Kriegeskorte, Marieke Mur, Peter A. Bandettini. 2008
3,437 citations recorded by the source corpus.
A fundamental challenge for systems neuroscience is to quantitatively relate its three major branches of research: brain-activity measurement, behavioral measurement, and computational modeling. Using measured brain-activity patterns to evaluate computational network models is complicated by the need to define the correspondency between the units of the model and the channels of the brain-activity data, e.g., single-cell recordings or voxels from functional magnetic resonance imaging (fMRI). Similar correspondency problems complicate relating activity patterns between different modalities of brain-activity measurement (e.g., fMRI and invasive or scalp electrophysiology), and between subjects and species. In order to bridge these divides, we suggest abstracting from the activity patterns themselves and computing representational dissimilarity matrices (RDMs), which characterize the information carried by a given representation in a brain or model. Building on a rich psychological and mathematical literature on similarity analysis, we propose a new experimental and data-analytical framework called representational similarity analysis (RSA), in which multi-channel measures of neural activity are quantitatively related to each other and to computational theory and behavior by comparing RDMs. We demonstrate RSA by relating representations of visual objects as measured with fMRI in early visual cortex and the fusiform face area to computational models spanning a wide range of complexities. The RDMs are simultaneously related via second-level application of multidimensional scaling and tested using randomization and bootstrap techniques. We discuss the broad potential of RSA, including novel approaches to experimental design, and argue that these ideas, which have deep roots in psychology and neuroscience, will allow the integrated quantitative analysis of data from all three branches, thus contributing to a more unified systems neuroscience.
Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Neuroscience
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Cognitive Neuroscience
- Nikolaus Kriegeskorte
- James V. Haxby
- John-Dylan Haynes
- Jörn Diedrichsen
- Josh Tenenbaum
- Xiao-Jing Wang
- Pamela K. Douglas
- Guangyu Robert Yang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Jitendra Malik
- Xiaogang Wang
- Andrew Zisserman
- Ming-Hsuan Yang
- Andrea Vedaldi
In Neuroscience
- Nikolaus Kriegeskorte
- Grace W. Lindsay
- Yonatan Belinkov
- Josh Tenenbaum
- James L. McClelland
- Dharshan Kumaran
- Demis Hassabis
- Guangyu Robert Yang
Citations in the corpus
- Probing Classifiers: Promises, Shortcomings, and Advances (2021, methodology): “Alternatively, some work has used representational similarity analysis (Kriegeskorte, Mur, and Bandettini 2008) to measure similarity between word or sentence representations and syntactic properties, both local properties like determining a verb’s subject (Lepori and McCoy 2020) and more…”
- Similarity of Neural Network Representations Revisited (2019, methodology): “In neuroscience, such matrices representing the similarities between examples are called representational similarity matrices (Kriegeskorte et al., 2008a).”
- Comparing continual task learning in minds and machines (2018, methodology): “RSA.”
- Biologically inspired protection of deep networks from adversarial attacks (2017, background): “Beyond examining the weights and the activations, we also examine the global structure of internal representations by constructing, for each network and layer, the representational dissimilarity matrix (RDM) of its activities (Kriegeskorte et al., 2008).”
- Unsupervised Model Selection for Variational Disentangled Representation Learning (2019, methodology): “Finally, Representation Similarity Matrix (RSM) (Kriegeskorte et al., 2008) is a commonly used method in Neuroscience for comparing the representations of different systems to the same set of stimuli.”
- Toward an Integration of Deep Learning and Neuroscience (2016, methodology): “Various techniques exist to carry out such comparisons in fRMI studies, including population receptive field estimation (Dumoulin and Wandell, 2008; Güçlü and van Gerven, 2015) and representational dissimilarity matrices (Kriegeskorte et al., 2008; Khaligh-Razavi and Kriegeskorte, 2014).”
- The Birth of Bias: A case study on the evolution of gender bias in an English language model (2022, positive): “Finally, we measure qualitative changes in the topological organisation of the semantic space of the occupation and gendered words 6 , by measuring Representational Similarity (Kriegeskorte et al., 2008) between the original model and the debiased models.”
- Training Neural Networks with Local Error Signals (2019, background): “The category clusters surprisingly match be-tween humans and monkeys when exposed to the same real-world object images (Kriegeskorte et al., 2008b).”
- Reliability of dissimilarity measures for multi-voxel pattern analysis (2016, positive): “In MVPA, activation patterns are analyzed using either classification (Cox & Savoy, 2003; Haxby et al., 2001) or representational similarity analysis (RSA, Kriegeskorte et al., 2008).”
- The Decoding Toolbox (TDT): a versatile software package for multivariate analyses of functional imaging data (2015, positive): “In addition, representational similarity analysis which exploits the correlation similarity structure of voxels and different distance metrics can be used to illustrate the representational distance of different conditions (Kriegeskorte et al., 2008; Nili et al., 2014).”
- Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence (2016, neutral): “2) 27,31 .”
- Comparison of multivariate classifiers and response normalizations for pattern-information fMRI (2010, neutral): “In the literature, some pattern-information studies used beta estimates to define the response patterns (e.g. Haxby 2001; De Martino et al., 2008) and other studies used t-values (e.g. Martínez- Ramón et al., 2006; Kriegeskorte et al., 2008a).”
- Representation Similarity Analysis for Efficient Task Taxonomy & Transfer Learning (2019, methodology): “In [18], RSA is used to compute similarities between brain responses in different regions of visual cortex with categorical models and computational vision models.”
- Stepmothers are mean and academics are pretentious: What do pretrained language models learn about you? (2021, positive): “Comparison across models We systematically compare the emotion profiles elicited by the social groups across different models by adapting the Representational Similarity Analysis (RSA) from Kriegeskorte et al. (2008).”
- Does language help generalization in vision models? (2021, methodology): “To better understand the similarities and differences between the feature spaces learned by the various models, we now compare them using Representational Similarity Analysis (RSA) [12].”