Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence

By Radoslaw Martin Cichy, Aditya Khosla, Dimitrios Pantazis, Antonio Torralba, Aude Oliva. 2016

751 citations recorded by the source corpus.

The complex multi-stage architecture of cortical visual pathways provides the neural basis for efficient visual object recognition in humans. However, the stage-wise computations therein remain poorly understood. Here, we compared temporal (magnetoencephalography) and spatial (functional MRI) visual brain representations with representations in an artificial deep neural network (DNN) tuned to the statistics of real-world visual recognition. We showed that the DNN captured the stages of human visual processing in both time and space from early visual areas towards the dorsal and ventral streams. Further investigation of crucial DNN parameters revealed that while model architecture was important, training on real-world categorization was necessary to enforce spatio-temporal hierarchical relationships with the brain. Together our results provide an algorithmically informed view on the spatio-temporal dynamics of visual object recognition in the human visual brain.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

Citations in the corpus

Global-and-local attention networks for visual recognition (2018, background): “We expect that a more systematic analysis – beyond identifying what features are selected but also when they are selected [45, 46] could help further our understanding of the different attention mechanisms responsible for the selection of diagnostic image features.”
Neural population control via deep image synthesis (2018, background): “In particular, visually-evoked internal neural representations of these specific ANNs 17 are remarkably similar to the visually-evoked neural representations in mid-level (area V4) and 18 high-level (area IT) cortical stages of the ventral stream (2, 3), a finding that has been extended 19 to neural”
Deep Neural Networks as Scientific Models. (2019, background): “Proponents argue that DNNs do explain either in the same way as other models, in a qualitatively different way [13,15,36,49], or that DNNs afford rich post hoc explanations.”
The Origins and Prevalence of Texture Bias in Convolutional Neural Networks (2019, background): “ImageNet-trained CNNs have emerged as the model of choice in neuroscience for modeling electrophysiological and neuroimaging data from primate visual cortex [101, 52, 15, 71, 7].”
Cognitive computational neuroscience (2018, background): “As we move gradually along the ventral visual stream, higher layers of the neural networks gradually come to provide a better basis for explaining the representations.(122,123,124) Higher layers of deep convolutional neural networks also resemble the IT cortical representation in that both enable th”
A parametric texture model based on deep convolutional features closely matches texture appearance for humans (2017, neutral): “…the stimulus input better than existing models built explicitly for that purpose ( Cadieu et al., 2014; Cichy, Khosla, Pantazis, & Oliva, 2016; Cichy, Khosla, Pantazis, Torralba, & Oliva, 2016; Guclu & van Gerven, 2015; Hong et al., 2016; Khaligh-Razavi & Kriegeskorte , 2014; Yamins, Hong,…”
What shapes feature representations? Exploring datasets, architectures, and training (2020, background): “rained ones after ﬁnetuning on a new task [12, 23]. Furthermore, features in untrained models can explain substantial amounts of neural data, although trained models generally perform slightly better [44, 18, 39, 6, 5]. Overall, this prior work observes that untrained feature representations capture”
The neural architecture of language: Integrative modeling converges on predictive processing (2020, background): “This result is consistent with findings in models of early (5, 7, 95) and high-level visual processing (95) and speech perception (122), as well as recent results in NLP (96–98), but it raises important questions of interpretation in the context of human language.”
A Task-Optimized Neural Network Replicates Human Auditory Behavior, Predicts Brain Responses, and Reveals a Cortical Processing Hierarchy. (2018, neutral): “…fruitful in the visual system, where the presence of hierarchy is well established—different network layers best predict responses at different stages of the visual cortical hierarchy (Cichy et al., 2016; G€uçl€u and van Gerven, 2015; Khaligh-Razavi and Kriegeskorte, 2014; Yamins et al., 2014).”
The Rhetoric and Reality of Anthropomorphism in Artificial Intelligence (2019, methodology): “DNNs have been used to model information processing at various stages of the visual cortex of human and nonhuman primates (Cichy et al. 2016; Kriegeskorte 2015; Yamins and DiCarlo 2016), achieving state of the art predictive performance while simultaneously suggesting novel subcortical functions.”