Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like?

By Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Kailyn Schmidt and 2 more. 2018

725 citations recorded by the source corpus.

The internal representations of early deep artificial neural networks (ANNs) were found to be remarkably similar to the internal neural representations measured experimentally in the primate brain. Here we ask, as deep ANNs have continued to evolve, are they becoming more or less brain-like? ANNs that are most functionally similar to the brain will contain mechanisms that are most like those used by the brain. We therefore developed Brain-Score – a composite of multiple neural and behavioral benchmarks that score any ANN on how similar it is to the brain’s mechanisms for core object recognition – and we deployed it to evaluate a wide range of state-of-the-art deep ANNs. Using this scoring system, we here report that: (1) DenseNet-169, CORnet-S and ResNet-101 are the most brain-like ANNs. There remains considerable variability in neural and behavioral responses that is not predicted by any ANN, suggesting that no ANN model has yet captured all the relevant mechanisms. (3) Extending prior work, we found that gains in ANN ImageNet performance led to gains on Brain-Score. However, correlation weakened at ≥ 70% top-1 ImageNet performance, suggesting that additional guidance from neuroscience is needed to make further advances in capturing brain mechanisms. (4) We uncovered smaller (i.e. less complex) ANNs that are more brain-like than many of the best-performing ImageNet models, which suggests the opportunity to simplify ANNs to better understand the ventral stream. The scoring system used here is far from complete. However, we propose that evaluating and tracking model-benchmark correspondences through a Brain-Score that is regularly updated with new brain data is an exciting opportunity: experimental benchmarks can be used to guide machine network evolution, and machine networks are mechanistic hypotheses of the brain’s network and thus drive next experiments. To facilitate both of these, we release Brain-Score.org: a platform that hosts the neural and behavioral benchmarks, where ANNs for visual processing can be submitted to receive a Brain-Score and their rank relative to other models, and where new experimental data can be naturally incorporated.

Subfields: Cognitive Neuroscience, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

Orthogonal representations for robust context-dependent task performance in brains and neural networks (2022, neutral): “…reported improve- 10 Neuron 110, 1–13, April 6, 2022 ments in model fit of trained over untrained networks tend to be relatively modest, as if the visual cortex mainly recapitulates the input data through random high-dimensional projections (G€uçl€u and van Gerven, 2015; Schrimpf et al., 2018).”
Teacher Guided Architecture Search (2018, background): “While our ability to simultaneously record from large populations of neurons is fast growing [40], these measurements have already been shown to have remarkable similarities to internal activations of CNNs [43, 39].”
The neural architecture of language: Integrative modeling converges on predictive processing (2020, methodology): “…transformed our algorithmic understanding of the primate ventral visual stream (Bao et al., 2020; Cadena et al., 2019; Cichy et al., 2016; Kietzmann et al., 2019; Kubilius et al., 2019; Schrimpf et al., 2018, 2020; Yamins et al., 2014), and holds great promise for other aspects of brain function.”
Individual differences among deep neural network models (2020, background): “To date, task-optimized deep neural networks constitute the best model class for predicting activity across multiple regions of the primate visual cortex (Cadieu et al., 2014; Guclu and van Gerven, 2015; Khaligh-Razavi and Kriegeskorte, 2014; Schrimpf et al., 2018; Yamins et al., 2014).”
Neural population control via deep image synthesis (2018, methodology): “This shows that even today’s ANN models – which 301 are clearly not yet perfect (1,6) – already give us new ability to find manifolds of more optimal 302 stimuli for each neural site at a much finer degree of granularity and to discover such stimuli 303 unconstrained by human intuition and difficult”
Deep Neural Networks as Scientific Models. (2019, neutral): “Consider the benchmarking of different models with respect to how well they predict brain activity [12,15,39].”
Unsupervised neural network models of the ventral visual stream (2020, methodology): “The resulting image-by-category confusion matrix was compared to data from humans performing the same alternative forced-choice task, where each trial started with a 500-ms fixation point, presented the image for 100 ms, and required the subject to choose from the true and another distractor categor”
Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future (2020, background): “But it can be beat by somewhat deeper architectures such as VGG models and ResNets [52,53].”
Neuromorphic computing at scale (2025, neutral): “There are also tools to evaluate the performance of neuromorphic systems and models for neuroscience and psychological modelling 95,96 .”
The Notorious Difficulty of Comparing Human and Machine Perception (2020, neutral): “…and/or monkey behavioral representations (Eberhardt, Cader, & Serre, 2016; Golan, Raju, & Kriegeskorte, 2019; Peterson, Abbott, & Griﬃths, 2016; Schrimpf et al., 2018; Yamins et al., 2014), human fMRI representations (Han et al., 2019; Khaligh-Razavi & Kriegeskorte, 2014) or monkey cell…”
The Platonic Representation Hypothesis (2024, methodology): “Since it is not trivial to determine where the alignment might occur, we draw inspiration from BrainScore(Schrimpf et al., 2018) and compute pairwise alignment scores, then take the maximum.”
If deep learning is the answer, what is the question? (2020, background): “However, neural networks that perform substantially worse at image classification explain just 5% less 55 .”
The Origins and Prevalence of Texture Bias in Convolutional Neural Networks (2019, methodology): “CORNet-S, the model with the highest BrainScore [81], had a shape bias of 20.3%, and shape and texture accuracies of 13.3% and 51.9%.”
Electrophysiological decoding captures the temporal trajectory of face categorization in infants (2024, unknown): “AlexNet was chosen as its layers are supposed to follow roughly a brain-like processing hierarchy (Schrimpf et al., 2020; Nonaka et al., 2021).”