Neural population control via deep image synthesis

By Pouya Bashivan, Kohitij Kar, James J. DiCarlo. Science, 2018

464 citations recorded by the source corpus.

Predicting behavior of visual neurons To what extent are predictive deep learning models of neural responses useful for generating experimental hypotheses? Bashivan et al. took an artificial neural network built to model the behavior of the target visual system and used it to construct images predicted to either broadly activate large populations of neurons or selectively activate one population while keeping the others unchanged. They then analyzed the effectiveness of these images in producing the desired effects in the macaque visual cortex. The manipulations showed very strong effects and achieved considerable and highly selective influence over the neuronal populations. Using novel and non-naturalistic images, the neural network was shown to reproduce the overall behavior of the animals' neural responses. Science, this issue p. eaav9436 A deep artificial neural network can model primate vision. INTRODUCTION The pattern of light that strikes the eyes is processed and re-represented via patterns of neural activity in a “deep” series of six interconnected cortical brain areas called the ventral visual stream. Visual neuroscience research has revealed that these patterns of neural activity underlie our ability to recognize objects and their relationships in the world. Recent advances have enabled neuroscientists to build ever more precise models of this complex visual processing. Currently, the best such models are particular deep artificial neural network (ANN) models in which each brain area has a corresponding model layer and each brain neuron has a corresponding model neuron. Such models are quite good at predicting the responses of brain neurons, but their contribution to an understanding of primate visual processing remains controversial. RATIONALE These ANN models have at least two potential limitations. First, because they aim to be high-fidelity computerized copies of the brain, the total set of computations performed by these models is difficult for humans to comprehend in detail. In that sense, each model seems like a “black box,” and it is unclear what form of understanding has been achieved. Second, the generalization ability of these models has been questioned because they have only been tested on visual stimuli that are similar to those used to “teach” the models. Our goal was to assess both of these potential limitations through nonhuman primate neurophysiology experiments in a mid-level visual brain area. We sought to answer two questions: (i) Despite these ANN models’ opacity to simple “understanding,” is the knowledge embedded in them already useful for a potential application (i.e., neural activity control)? (ii) Do these models accurately predict brain responses to novel images? RESULTS We conducted several closed-loop neurophysiology experiments: After matching model neurons to each of the recorded brain neural sites, we used the model to synthesize entirely novel “controller” images based on the model’s implicit knowledge of how the ventral visual stream works. We then presented those images to each subject to test the model’s ability to control the subject’s neurons. In one test, we asked the model to try to control each brain neuron so strongly as to activate it beyond its typically observed maximal activation level. We found that the model-generated synthetic stimuli successfully drove 68% of neural sites beyond their naturally observed activation levels (chance level is 1%). In an even more stringent test, the model revealed that it is capable of selectively controlling an entire neural subpopulation, activating a particular neuron while simultaneously inactivating the other recorded neurons (76% success rate; chance is 1%). Next, we used these non-natural synthetic controller images to ask whether the model’s ability to predict the brain responses would hold up for these highly novel images. We found that the model was indeed quite accurate, predicting 54% of the image-evoked patterns of brain response (chance level is 0%), but it is clearly not yet perfect. CONCLUSION Even though the nonlinear computations of deep ANN models of visual processing are difficult to accurately summarize in a few words, they nonetheless provide a shareable way to embed collective knowledge of visual processing, and they can be refined by new knowledge. Our results demonstrate that the currently embedded knowledge already has potential application value (neural control) and that these models can partially generalize outside the world in which they “grew up.” Our results also show that these models are not yet perfect and that more accurate ANN models would produce even more precise neural control. Such noninvasive neural control is not only a potentially powerful tool in the hands of neuroscientists but also could lead to a new class of therapeutic applications. Collection of images synthesized by a deep neural network model to control the activity of neural populations in primate cortical area V4. We used a deep artificial neural network to control the activity pattern of a population of neurons in cortical area V4 of macaque monkeys by synthesizing visual stimuli that, when applied to the subject’s retinae, successfully induced the experimenter-desired neural response patterns. Particular deep artificial neural networks (ANNs) are today’s most accurate models of the primate brain’s ventral visual stream. Using an ANN-driven image synthesis method, we found that luminous power patterns (i.e., images) can be applied to primate retinae to predictably push the spiking activity of targeted V4 neural sites beyond naturally occurring levels. This method, although not yet perfect, achieves unprecedented independent control of the activity state of entire populations of V4 neural sites, even those with overlapping receptive fields. These results show how the knowledge embedded in today’s ANN models might be used to noninvasively set desired internal brain states at neuron-level resolution, and suggest that more accurate ANN models would produce even more accurate control.

Subfields: Biophysics, Cognitive Neuroscience

View Paper: https://www.semanticscholar.org/paper/aa0ad8dd35f7b385feba07cedb30d7e3a1935381

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Cognitive Neuroscience

Citations in the corpus

Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future (2020, methodology): “Starting from the above-mentioned procedure for predicting neural activity with CNNs, stimuli that were intended to maximally drive a neuron’s firing rate were produced [47].”
Unsupervised neural network models of the ventral visual stream (2020, methodology): “Additionally, we examined optimal stimuli driving neurons in intermediate and higher model layers using techniques similar to those used in recent model-driven electrophysiology experiments (55).”
Artificial Neural Networks for Neuroscientists: A Primer (2020, methodology): “For example, compared to purely feedforward networks, recurrently-connected deep networks are better at predicting responses of higher visual area neurons to behaviorally challenging images [Kar et al., 2019].”
Distributional reinforcement learning in the brain (2020, background): “Arti ﬁ cial neural networks speci ﬁ cally adapted for performance on machine learning tasks may reveal novel combinations of neural codes and related computations, as has been widely documented in the primate visual system [86,87].”
If deep learning is the answer, what is the question? (2020, methodology): “One approach, illustrated schematically here, uses ‘closed loop’ experimental designs to test the predictive power of deep networks 57,58,149 .”
Integrative Benchmarking to Advance Neurally Mechanistic Models of Human Intelligence. (2020, background): “model-synthesized patterns of light applied to retinae (Bashivan et al., 2019; Ponce et al., 2019).”
The neural architecture of language: Integrative modeling converges on predictive processing (2020, background): “…on the sentence stimuli that elicit the strongest responses in the language network to generate hypotheses about the critical response-driving feature/feature spaces, and perhaps to discover new organizing principles of the language system (cf. vision; (Bashivan et al., 2019; Ponce et al., 2019)).”
Deep Neural Networks as Scientific Models. (2019, background): “Recent research has demonstrated the feasibility of using DNNs for systematically manipulating brain activity in visual regions V1 [37] and V4 [38].”
Quo vadis artificial intelligence? (2022, background): “This is more due to ethical concerns than technical feasibility, as experiments with animals have demonstrated the effectiveness of artificial stimuli to control visual and motion perceptions [97].”
The Origins and Prevalence of Texture Bias in Convolutional Neural Networks (2019, background): “ImageNet-trained CNNs have emerged as the model of choice in neuroscience for modeling electrophysiological and neuroimaging data from primate visual cortex [101, 52, 15, 71, 7].”
Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs (2019, background): “Having strong models of the brain opened up unexpected possibilities of brain-response control by showing optimal stimuli to primates [2], pointing the way to a potentially exciting future of noninvasive brain-machine interfaces.”
Computation Through Neural Population Dynamics (2020, background): “…research on using optogenetics to spatiotemporally pattern neural activity in visual cortex has demonstrated that experimental control over population activity is indeed possible (e.g., Bashivan et al. 2019, Carrillo-Reid et al. 2019, Chettih & Harvey 2019, Marshel et al. 2019, Yang et al. 2018).”