Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision

By Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayu Cao, Zhongming Liu. Cerebral Cortex, 2016

328 citations recorded by the source corpus.

Convolutional neural network (CNN) driven by image recognition has been shown to be able to explain cortical responses to static pictures at ventral-stream areas. Here, we further showed that such CNN could reliably predict and decode functional magnetic resonance imaging data from humans watching natural movies, despite its lack of any mechanism to account for temporal dynamics or feedback processing. Using separate data, encoding and decoding models were developed and evaluated for describing the bi-directional relationships between the CNN and the brain. Through the encoding models, the CNN-predicted areas covered not only the ventral stream, but also the dorsal stream, albeit to a lesser degree; single-voxel response was visualized as the specific pixel pattern that drove the response, revealing the distinct representation of individual cortical location; cortical activation was synthesized from natural images with high-throughput to map category representation, contrast, and selectivity. Through the decoding models, fMRI signals were directly decoded to estimate the feature representations in both visual and semantic spaces, for direct visual reconstruction and semantic categorization, respectively. These results corroborate, generalize, and extend previous findings, and highlight the value of using deep learning, as an all-in-one model of the visual cortex, to understand and decode natural vision.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Signal Processing

View Paper: https://www.semanticscholar.org/paper/c1b778f8a57c94a8933882aa28046bc38790b150

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Signal Processing

Citations in the corpus

Deep Nets: What have They Ever Done for Vision? (2018, background): “non-invasive measurements, and there are a growing number of examples (Cichy et al., 2016; Wen et al., 2017).”
Deep image reconstruction from human brain activity (2017, methodology): “…activity into images or movies, the methods were essentially limited to the image reconstruction with low-level image bases (Miyawaki et al., 2008; Wen et al., 2016) or to the matching to exemplar images or movies (Naselaris et al., 2009; Nishi-moto et al., 2011), failing to combine visual…”
Large-Scale, High-Resolution Comparison of the Core Visual Object Recognition Behavior of Humans, Monkeys, and State-of-the-Art Deep Artificial Neural Networks (2018, background): “…activity at multiple stages of the primate ventral visual stream (Yamins et al., 2013; Cadieu et al., 2014; Khaligh-Razavi and Kriegeskorte, 2014; Yamins et al., 2014; Güçlü and van Gerven, 2015; Cichy et al., 2016; Hong et al., 2016; Seibert et al., 2016; Cadena et al., 2017; Wen et al., 2017).”
Neural network models and deep learning - a primer for biologists (2019, background): “(Cadieu et al., 2014; Khaligh-Razavi and Kriegeskorte, 2014; Yamins et al., 2014; Güçlü and van Gerven, 2015; Cichy et al., 2016; Eickenberg et al., 2017; Wen et al., 2017; Nayebi et al., 2018), and yet multiple functional incompatibilities have already been reported (Szegedy et al.”
High-resolution image reconstruction with latent diffusion models from human brain activity (2023, methodology): “…and design of deep neural network models, have allowed direct comparisons be-tween latent representations in biological brains and architectural characteristics of artiﬁcial networks, providing important insights into how these systems operate [3, 8– 10, 13, 18, 19, 21, 42, 43, 54, 55].”