Deep convolutional models improve predictions of macaque V1 responses to natural images

By Santiago A. Cadena, George H. Denfield, Edgar Y. Walker, Leon A. Gatys, Andreas S. Tolias, Matthias Bethge, Alexander S. Ecker. bioRxiv, 2017

402 citations recorded by the source corpus.

Despite great efforts over several decades, our best models of primary visual cortex (V1) still predict spiking activity quite poorly when probed with natural stimuli, highlighting our limited understanding of the nonlinear computations in V1. Recently, two approaches based on deep learning have been successfully applied to neural data: On the one hand, transfer learning from networks trained on object recognition worked remarkably well for predicting neural responses in higher areas of the primate ventral stream, but has not yet been used to model spiking activity in early stages such as V1. On the other hand, data-driven models have been used to predict neural responses in the early visual system (retina and V1) of mice, but not primates. Here, we test the ability of both approaches to predict spiking activity in response to natural images in V1 of awake monkeys. Even though V1 is rather at an early to intermediate stage of the visual system, we found that the transfer learning approach performed similarly well to the data-driven approach and both outperformed classical linear-nonlinear and wavelet-based feature representations that build on existing theories of V1. Notably, transfer learning using a pre-trained feature space required substantially less experimental time to achieve the same performance. In conclusion, multi-layer convolutional neural networks (CNNs) set the new state of the art for predicting neural responses to natural images in primate V1 and deep features learned for object recognition are better explanations for V1 computation than all previous filter bank theories. This finding strengthens the necessity of V1 models that are multiple nonlinearities away from the image domain and it supports the idea of explaining early visual cortex based on high-level functional goals. Author summary Predicting the responses of sensory neurons to arbitrary natural stimuli is of major importance for understanding their function. Arguably the most studied cortical area is primary visual cortex (V1), where many models have been developed to explain its function. However, the most successful models built on neurophysiologists’ intuitions still fail to account for spiking responses to natural images. Here, we model spiking activity in primary visual cortex (V1) of monkeys using deep convolutional neural networks (CNNs), which have been successful in computer vision. We both trained CNNs directly to fit the data, and used CNNs trained to solve a high-level task (object categorization). With these approaches, we are able to outperform previous models and improve the state of the art in predicting the responses of early visual neurons to natural images. Our results have two important implications. First, since V1 is the result of several nonlinear stages, it should be modeled as such. Second, functional models of entire visual pathways, of which V1 is an early stage, do not only account for higher areas of such pathways, but also provide useful representations for V1 predictions.

Subfields: Cognitive Neuroscience, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/e44b3703c9411fb43bad9399f504539804135613

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

Large-Scale, High-Resolution Comparison of the Core Visual Object Recognition Behavior of Humans, Monkeys, and State-of-the-Art Deep Artificial Neural Networks (2018, background): “…activity at multiple stages of the primate ventral visual stream (Yamins et al., 2013; Cadieu et al., 2014; Khaligh-Razavi and Kriegeskorte, 2014; Yamins et al., 2014; Güçlü and van Gerven, 2015; Cichy et al., 2016; Hong et al., 2016; Seibert et al., 2016; Cadena et al., 2017; Wen et al., 2017).”
Correspondence of Deep Neural Networks and the Brain for Visual Textures (2018, background): “Although CNNs ([3], [4], [1],[5]) are only crudely matched to the hierarchical structure of the brain, such models have been intriguingly able to predict some aspects of cortical visual processing [6], [7], [8], [9], [10], [11], [12], [13], [14].”
Teacher Guided Architecture Search (2018, background): “These models are able to predict the neural responses in parts of the primate visual [43, 12, 11] and auditory cortices [23], explain patterns of object similarity judgements [35, 22], shape sensitivity in primates [26], and even to control the neural activity in a mid-level visual cortical area [6]”
A Task-Optimized Neural Network Replicates Human Auditory Behavior, Predicts Brain Responses, and Reveals a Cortical Processing Hierarchy. (2018, neutral): “One difference between our results and analogous work in the visual system is that primary visual cortical responses have been found to be best predicted by early layers of a deep network (Cadena et al., 2017; G€ uçl€ u and van Gerven, 2015; Khaligh-Razavi andKriegeskorte, 2014), whereaswe found pri”
Deep Neural Networks as Scientific Models. (2019, background): “In this spirit, researchers have shown that DNNs predict human perceptual similarity judgements [10,11] and outperform any other model in accounting for neural activity in primate sensory cortices [12 – 18] (Figure 1).”
Towards the neural population doctrine. (2019, result): “In [60], the authors address this question in the context of V1 neurons, and find that both these modeling approaches perform similarly, while outperforming other known models.”
A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence (2021, result): “Noticeably, the simple Gabor model accounts for substantial variance in the responses; nonetheless, the more complex CNN-based models provide additional predictive power, consistent with previous observations 48 .”
Engineering a Less Artificial Intelligence. (2019, methodology): “Some of the studies simultaneously predict thousands of neurons with less than 100 hidden feature dimensions (Sinz et al., 2018; Ecker et al., 2019; Cadena et al., 2019; Klindt et al., 2017).”
Integrative Benchmarking to Advance Neurally Mechanistic Models of Human Intelligence. (2020, background): “A successful strategy has been to simply ask labs if they would be willing to share data (Majaj et al., 2015; Rajalingham et al., 2018; Cadena et al., 2019; David et al., 2004; Freeman et al., 2013; Kar et al., 2019), and modelers interested in evaluating their models (i.e., the ‘‘second person’’…”
Diverse feature visualizations reveal invariances in early layers of deep neural networks (2018, neutral): “As a inal practical use case, we applied our method to a three-layer CNN that has been trained to predict neural responses in V1 when monkeys are shown natural images (data from [4]; see also their Fig.”