Large-Scale, High-Resolution Comparison of the Core Visual Object Recognition Behavior of Humans, Monkeys, and State-of-the-Art Deep Artificial Neural Networks

By Rishi Rajalingham, Elias B. Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, James J. DiCarlo. 2018

355 citations recorded by the source corpus.

Primates, including humans, can typically recognize objects in visual images at a glance despite naturally occurring identity-preserving image transformations (e.g., changes in viewpoint). A primary neuroscience goal is to uncover neuron-level mechanistic models that quantitatively explain this behavior by predicting primate performance for each and every image. Here, we applied this stringent behavioral prediction test to the leading mechanistic models of primate vision (specifically, deep, convolutional, artificial neural networks; ANNs) by directly comparing their behavioral signatures against those of humans and rhesus macaque monkeys. Using high-throughput data collection systems for human and monkey psychophysics, we collected more than one million behavioral trials from 1472 anonymous humans and five male macaque monkeys for 2400 images over 276 binary object discrimination tasks. Consistent with previous work, we observed that state-of-the-art deep, feedforward convolutional ANNs trained for visual categorization (termed DCNNIC models) accurately predicted primate patterns of object-level confusion. However, when we examined behavioral performance for individual images within each object discrimination task, we found that all tested DCNNIC models were significantly nonpredictive of primate performance and that this prediction failure was not accounted for by simple image attributes nor rescued by simple model modifications. These results show that current DCNNIC models cannot account for the image-level behavioral patterns of primates and that new ANN models are needed to more precisely capture the neural mechanisms underlying primate object vision. To this end, large-scale, high-resolution primate behavioral benchmarks such as those obtained here could serve as direct guides for discovering such models. SIGNIFICANCE STATEMENT Recently, specific feedforward deep convolutional artificial neural networks (ANNs) models have dramatically advanced our quantitative understanding of the neural mechanisms underlying primate core object recognition. In this work, we tested the limits of those ANNs by systematically comparing the behavioral responses of these models with the behavioral responses of humans and monkeys at the resolution of individual images. Using these high-resolution metrics, we found that all tested ANN models significantly diverged from primate behavior. Going forward, these high-resolution, large-scale primate behavioral benchmarks could serve as direct guides for discovering better ANN models of the primate visual system.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

Citations in the corpus

BrainGNN: Interpretable Brain Graph Neural Network for fMRI Analysis (2020, neutral): “understand the human brain network [13,36,49].”
Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense (2020, background): “Despite the fact that these approaches achieved good performances on scene categorization in terms of recognition accuracy in publicly available datasets, a recent large-scale neuroscience study [21] has shown that current DNNs cannot account for the image-level behavior patterns of primates (both h”
The neural architecture of language: Integrative modeling converges on predictive processing (2020, background): “…findings that across hundreds of model variants, DCNNs that perform better on object recognition tasks also better capture human recognition behavior and neural responses in IT cortex of both human and non-human primates (Rajalingham et al., 2018; Schrimpf et al., 2018, 2020; Yamins et al., 2014).”
Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs (2019, methodology): “We fit a logistic regression on 80% of data at each 10 ms and tested on 10% at which time point the normalized d′ scores per image [33] surpassed a threshold values defined by the monkey’s behavioral output for that image, which we refer to as "object solution times", or OSTs [see 14, for details], ”
Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like? (2018, methodology): “We here use the metric of the image-by-image patterns of difficulty, broken down by the object choice alternatives (termed I2n), because recent work (Rajalingham et al., 2018) suggests that it has the most power to distinguish among alternative ANNs (assuming that sufficient amounts of behavioral da”
Teacher Guided Architecture Search (2018, background): “Nevertheless, while these networks have largely progressed our understanding of neural computations in many parts of the primate brain, there are still significant differences between the two [16, 34].”
Neural population control via deep image synthesis (2018, background): “This shows that even today’s ANNmodels—which are clearly not yet perfect (1, 6)—already give us new ability to find manifolds of more optimal stimuli for each neural site at a much finer degree of granularity and to discover such stimuli unconstrained by human intuition and the limits of human langu”
Physion: Evaluating Physical Prediction from Vision in Humans and Machines (2021, neutral): “In other words, models that performed better on the prediction task also made errors that were more like those made by humans, strongly analogous to the situation with core visual object recognition [48].”
Neural network models and deep learning - a primer for biologists (2019, background): “…Yamins et al., 2014; Güçlü and van Gerven, 2015; Cichy et al., 2016; Eickenberg et al., 2017; Wen et al., 2017; Nayebi et al., 2018), and yet multiple functional incompatibilities have already been reported (Szegedy et al., 2013; Geirhos et al., 2017; Jo and Bengio, 2017; Rajalingham et al., 2018).”
Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future (2020, background): “lar matches to human and monkey object classification, though the models are not predictive down to the image level (Rajalingham et al., 2018).”
If deep learning is the answer, what is the question? (2020, background): “In one study, networks were prone to confuse object classes that humans and even monkeys could safely tell apart, such as dogs and guitars, and patterns of confusion among individual images were shared by humans and macaques, but not deep networks 11 ( Fig.”
Partial success in closing the gap between human and machine vision (2021, methodology): “[32, 75])? This is quantified via error consistency (defined in Section 2); which additionally allows us to compare models with each other, asking e.”
Training Spiking Neural Networks Using Lessons From Deep Learning (2021, background): “This includes links between convolutional neural networks and measured activity from the visual cortex [215], [216], [217] and auditory processing regions [218].”
Selective Replay Enhances Learning in Online Continual Analogical Reasoning (2021, background): “Multiple primate species are also capable of image classification at levels that rival humans [22, 63, 64].”