Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation

By Seyed-Mahdi Khaligh-Razavi, Nikolaus Kriegeskorte. 2014

1,311 citations recorded by the source corpus.

Inferior temporal (IT) cortex in human and nonhuman primates serves visual object recognition. Computational object-vision models, although continually improving, do not yet reach human performance. It is unclear to what extent the internal representations of computational models can explain the IT representation. Here we investigate a wide range of computational model representations (37 in total), testing their categorization performance and their ability to account for the IT representational geometry. The models include well-known neuroscientific object-recognition models (e.g. HMAX, VisNet) along with several models from computer vision (e.g. SIFT, GIST, self-similarity features, and a deep convolutional neural network). We compared the representational dissimilarity matrices (RDMs) of the model representations with the RDMs obtained from human IT (measured with fMRI) and monkey IT (measured with cell recording) for the same set of stimuli (not used in training the models). Better performing models were more similar to IT in that they showed greater clustering of representational patterns by category. In addition, better performing models also more strongly resembled IT in terms of their within-category representational dissimilarities. Representational geometries were significantly correlated between IT and many of the models. However, the categorical clustering observed in IT was largely unexplained by the unsupervised models. The deep convolutional network, which was trained by supervision with over a million category-labeled images, reached the highest categorization performance and also best explained IT, although it did not fully explain the IT data. Combining the features of this model with appropriate weights and adding linear combinations that maximize the margin between animate and inanimate objects and between faces and other objects yielded a representation that fully explained our IT data. Overall, our results suggest that explaining IT requires computational features trained through supervised learning to emphasize the behaviorally important categorical divisions prominently reflected in IT.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

Citations in the corpus

A Neural Algorithm of Artistic Style (2015, neutral): “Ltotal(~ p,~a, ~x) = αLcontent(~ p, ~x) + βLstyle(~a, ~x) (7)”
Similarity of Neural Network Representations Revisited (2019, background): “…brain areas (Haxby et al., 2001; Freiwald & Tsao, 2010), individuals (Connolly et al., 2012), species (Kriegeskorte et al., 2008b), and behaviors (Elsayed et al., 2016), as well as between brains and neural networks (Yamins et al., 2014; Khaligh-Razavi & Kriegeskorte, 2014; Sussillo et al., 2015).”
Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream (2014, methodology): “We manually assigned each image in the test set to one of two categories (animate vs inanimate), as this appears to be the strongest categorical division in inferior temporal cortex (Khaligh-Razavi and Kriegeskorte, 2014).”
Texture synthesis and the controlled generation of natural stimuli using convolutional neural networks (2015, background): “Together with the increasing amount of evidence for the similarity of the representations in convolutional networks and those in the ventral visual pathway [21, 9], these properties make them compelling candidate models for studying visual information processing in the brain.”
Toward an Integration of Deep Learning and Neuroscience (2016, methodology): “Various techniques exist to carry out such comparisons in fRMI studies, including population receptive ﬁeld estimation (Dumoulin and Wandell, 2008; Güçlü and van Gerven, 2015) and representational dissimilarity matrices (Kriegeskorte et al., 2008; Khaligh-Razavi and Kriegeskorte, 2014).”
The role of textural statistics vs. outer contours in deep CNN and neural responses to objects (2018, background): “…cortex; visual features; CNNs Introduction At present, deep convolutional neural network activations are the best predictors of object responses in object selective cortex (Khaligh-Razavi & Kriegeskorte, 2014; Güclü & van Gerven, 2015; Long, Yu, & Konkle, 2017; Yamins et al., 2014).”
Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence (2016, neutral): “2) 27,31 .”
What Learning Systems do Intelligent Agents Need? Complementary Learning Systems Theory Updated. (2016, neutral): “…during passive viewing of pictures reveals coding of ﬁ ne-grained categorical structure (e.g., of a set of animate and inanimate objects) – that is well ﬁ t by deep convolutional neural networks which have algorithmic parallels with feedforward processing in the ventral visual stream [17,40].”
Deep Neural Networks as Scientific Models. (2019, background): “Consider the benchmarking of different models with respect to how well they predict brain activity [12,15,39].”
Representational models: A common framework for understanding encoding, pattern-component, and representational-similarity analysis (2017, background): “G = ωECi E [18, 54-56], with the weights being free second-level 1013”
Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future (2020, methodology): “This method was used in 2014 [18] to demonstrate that the later layers of an AlexNet network trained on ImageNet match multiple higher areas of the human visual system, along with monkey IT, better than previously-used models.”
Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision (2016, background): “In line with previous studies (Khaligh-Razavi and Kriegeskorte 2014; Yamins et al. 2014; Güçlü and van Gerven 2015a, b; Cichy et al. 2016; Kubilius et al. 2016; Eickenberg et al. 2016; Horikawa and Kamitani 2017), we explored the correspondence between individual layers in the CNN and individual…”
Towards deep learning with segregated dendrites (2016, background): “In fact, deep learning models can match cortical representations even more so than some models that explicitly attempt to mimic the real brain (Khaligh-Razavi and Kriegeskorte, 2014).”