Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition

By Charles F. Cadieu, Ha Hong, Daniel L. K. Yamins, Nicolas Pinto, Diego Ardila, E. Solomon, Najib J. Majaj, James J. DiCarlo. 2014

837 citations recorded by the source corpus.

The primate visual system achieves remarkable visual object recognition performance even in brief presentations, and under changes to object exemplar, geometric transformations, and background variation (a.k.a. core visual object recognition). This remarkable performance is mediated by the representation formed in inferior temporal (IT) cortex. In parallel, recent advances in machine learning have led to ever higher performing models of object recognition using artificial deep neural networks (DNNs). It remains unclear, however, whether the representational performance of DNNs rivals that of the brain. To accurately produce such a comparison, a major difficulty has been a unifying metric that accounts for experimental limitations, such as the amount of noise, the number of neural recording sites, and the number of trials, and computational limitations, such as the complexity of the decoding classifier and the number of classifier training examples. In this work, we perform a direct comparison that corrects for these experimental limitations and computational considerations. As part of our methodology, we propose an extension of “kernel analysis” that measures the generalization accuracy as a function of representational complexity. Our evaluations show that, unlike previous bio-inspired models, the latest DNNs rival the representational performance of IT cortex on this visual object recognition task. Furthermore, we show that models that perform well on measures of representational performance also perform well on measures of representational similarity to IT, and on measures of predicting individual IT multi-unit responses. Whether these DNNs rely on computational mechanisms similar to the primate visual system is yet to be determined, but, unlike all previous bio-inspired models, that possibility cannot be ruled out merely on representational performance grounds.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Finding Generalizable Evidence by Learning to Convince Q&A Models (2019, background): “Convolutional networks trained on ImageNet share similarities with the human visual cortex (Cadieu et al., 2014).”
ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness (2018, methodology): “This is problematic, since CNNs are being used as computational models for human object recognition (e.g. Cadieu et al., 2014; Yamins et al., 2014).”
Generalisation in humans and deep neural networks (2018, background): “Since DNNs and humans achieve similar accuracy, a number of studies have started investigating similarities and differences between DNNs and human vision [9–24].”
Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner (2016, neutral): “Related work exist in the area of computational neuroscience, which attempts to use machine learning architectures (deep learning) and bring it to bear to the analysis of neural representations for visual inputs (Cadieu et al., 2014; Isik, Tacchetti, & Poggio, 2016; Leibo, Liao, Anselmi, & Poggio, 2”
Texture Synthesis Using Convolutional Neural Networks (2015, background): “Together with the increasing amount of evidence for the similarity of the representations in convolutional networks and those in the ventral visual pathway [29, 2, 14], these properties make them compelling candidate models for studying visual information processing in the brain.”
Classification and Geometry of General Perceptual Manifolds (2017, background): “[10] Charles F Cadieu, Ha Hong, Daniel LK Yamins, Nicolas Pinto, Diego Ardila, Ethan A Solomon, Najib J Majaj, and James J DiCarlo.”
Deep learning for visual understanding (2017, neutral): “However, we cannot conclude that the representational performance of a CNN rivals that of the brain [205].”
Illusory Motion Reproduced by Deep Neural Networks Trained for Prediction (2018, background): “…of object recognition, stimulus representations developed by the DNNs have been shown to account for neural signals in primates inferior temporal cortex and in fMRI recording data from the human ventral stream (Cadieu et al., 2014; Khaligh-Razavi and Kriegeskorte, 2014; Güçlü and van Gerven, 2015).”
Angular Visual Hardness (2019, background): “Image degradation: Because CNNs and humans achieve similar accuracy on a wide range of tasks on benchmark datasets, a number of works have investigated similarities and differences between CNNs and human vision [3, 5, 9, 11, 12, 29, 46, 51, 67].”
Deep neural networks: a new framework for modelling biological vision and brain information processing (2015, background): “Indeed, the recent successes at predicting brain representations of novel natural images are largely driven by task training (Yamins et al. 2014, Khaligh-Razavi & Kriegeskorte 2014, Cadieu et al. 2014).”
Large-Scale, High-Resolution Comparison of the Core Visual Object Recognition Behavior of Humans, Monkeys, and State-of-the-Art Deep Artificial Neural Networks (2018, background): “…IC models display internal feature representations similar to neuronal representations along the primate ventral visual stream (Yamins et al., 2013; Cadieu et al., 2014; Khaligh-Razavi and Kriegeskorte, 2014; Yamins et al., 2014), and they exhibit behavioral patterns similar to the behavioral…”