Reconstructing visual experiences from brain activity evoked by natural movies

By Shinji Nishimoto, An T. Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, Jack L. Gallant. Current Biology, 2011

1,018 citations recorded by the source corpus.

Summary Quantitative modeling of human brain activity can provide crucial insights about cortical representations [1, 2], and can form the basis for brain decoding devices [3–5]. Recent functional magnetic resonance imaging (fMRI) studies have modeled brain activity elicited by static visual patterns, and have shown that it is possible to reconstruct these images from brain activity measurements [6–8]. However, blood oxygen level dependent (BOLD) signals measured using fMRI are very slow [9], so it has been difficult to model brain activity elicited by dynamic stimuli such as natural movies. Here we present a new motion-energy [10, 11] encoding model that largely overcome this limitation. Our motion-energy model describes fast visual information and slow hemodynamics by separate components. We recorded BOLD signals in occipito-temporal visual cortex of human subjects who passively watched natural movies, and fit the encoding model separately to individual voxels. Visualization of the fit models reveals how early visual areas represent moving stimuli. To demonstrate the power of our approach we also constructed a Bayesian decoder [8], by combining estimated encoding models with a sampled natural movie prior. The decoder provides remarkable reconstructions of natural movies, capturing the spatio-temporal structure of the viewed movie. These results demonstrate that dynamic brain activity measured under naturalistic conditions can be decoded using current fMRI technology.

Subfields: Cognitive Neuroscience

View Paper: https://www.semanticscholar.org/paper/1c63c8bc5654a13575640b5c2a1b9725f02e028a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

Citations in the corpus

Sparse Modeling for Image and Vision Processing (2014, methodology): “Models based on such filters have been subsequently widely used, and are still present in state-of-the-art predictive models of the neuronal activity in V1 [Kay et al., 2008, Nishimoto et al., 2011].”
Natural speech reveals the semantic maps that tile human cerebral cortex (2016, neutral): “This suggests that the contents of thought, or internal speech, might be decoded using these voxel-wise models 17 .”
Semantic reconstruction of continuous language from non-invasive brain recordings (2022, methodology): “Bayesian decoders have previously been used to decode perceived images and videos 18,19 .”
Inverting Convolutional Networks with Convolutional Networks (2015, background): “Since it does not require gradients of the feature representation, it could be applied, for example, to reconstruct visual input based on recordings from a real brain, somewhat similar to [7].”
HOGgles: Visualizing Object Detection Features (2013, methodology): “We also point out that a similar nearest neighbor method is used in brain research to visualize what a person might be seeing [16].”
Seeing it all: Convolutional network layers map the function of the human visual system (2017, methodology): “While spatiotemporal features do obtain an increase in performance 532 for low-level features even for BOLD measure [19], the spatiotemporally separated setting 533 is nevertheless an acceptable approximation, which improves with layer abstraction level.”
Deconstructing multivariate decoding for the study of brain function (2017, methodology): “In addition, multivariate decoding for prediction has been used for read-out of information from visual cortex during perception (Kay et al., 2008; Miyawaki et al., 2008; Naselaris et al., 2009; Nishimoto et al., 2011; Thirion et al., 2006) and during sleep (Horikawa et al.”
Cross‐validation failure: Small sample sizes lead to large error bars (2017, background): “…the neural support of cognition (Haynes and Rees, 2006), information mapping (Kriegeskorte et al., 2006), prediction of individual diﬀerences –behavioral or clinical– (Smith et al., 2015), rich encoding models (Nishimoto et al., 2011), principled reverse inferences (Poldrack et al., 2009), etc .”
A Primer on Pattern-Based Approaches to fMRI: Principles, Pitfalls, and Perspectives. (2015, background): “Encoding models also can incorporate nonlinear processing stages (Nishimoto et al., 2011).”
Realist Magic: Objects, Ontology, Causality (2013, background): “This is not simply a realm of mere appetite, as Kant suggests, because that would reproduce a difference between humans and nonhumans (animals, for instance) that is untenable and problematic.(41) Moreover, in appetite I roam like a hungry wolf over the carcass of things—it seems as if powerful obje”
Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision (2016, methodology): “Our decoding method does not require any prior set of candidate images, setting itself apart from the encoding-model-based decoding (Nishimoto et al. 2011).”
Inverting Visual Representations with Convolutional Networks (2015, methodology): “The method of [19] requires gradients of the feature representation, therefore it could not be directly applied to non-differentiable representations such as LBP, or recordings from a real brain [20].”
The present and future of deep learning in radiology. (2019, background): “T E This effectively translates to the functionality and processing of human brain's neural responses to the different visual stimuli [26, 27].”
Decoding neural representational spaces using multivariate pattern analysis. (2014, methodology): “This innovation affords predictions of / 16 F o patterns of brain response to novel stimuli based on their features Nishimoto et al. 2011, Horikawa et al. 2013 s on 02 / ( representational spaces.”