Bayesian reconstruction of natural images from human brain activity

By Thomas Naselaris, Ryan J. Prenger, Kendrick Norris Kay, Michael Oliver, Jack L. Gallant. Neuron, 2009

582 citations recorded by the source corpus.

Summary Recent studies have used fMRI signals from early visual areas to reconstruct simple geometric patterns. Here, we demonstrate a new Bayesian decoder that uses fMRI signals from early and anterior visual areas to reconstruct complex natural images. Our decoder combines three elements: a structural encoding model that characterizes responses in early visual areas; a semantic encoding model that characterizes responses in anterior visual areas; and prior information about the structure and semantic content of natural images. By combining all these elements, the decoder produces reconstructions that accurately reflect both the spatial structure and semantic category of the objects contained in the observed natural image. Our results show that prior information has a substantial effect on the quality of natural image reconstructions. We also demonstrate that much of the variance in the responses of anterior visual areas to complex natural images is explained by the semantic category of the image alone.

Subfields: Biophysics, Cognitive Neuroscience, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ae9990bf7f761a7a1c89d2fbe4384fc3cf587ec6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

Citations in the corpus

Processing of Natural Sounds in Human Auditory Cortex: Tonotopy, Spectral Tuning, and Relation to Voice Sensitivity (2012, methodology): “To calculate the voxels’ frequency profile and topographic maps, we used an fMRI “encoding” procedure (Kay et al., 2008b; Naselaris et al., 2009, 2011).”
Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream (2014, methodology): “The stimulus-response pairs were taken from the vim-1 data set [10] that was originally published in [11, 12].”
Reconstructing visual experiences from brain activity evoked by natural movies (2011, unknown): “An earlier study showed that the static model tested here recovered much more information from BOLD signals than had been obtained with any previous model [8, 19].”
High-resolution image reconstruction with latent diffusion models from human brain activity (2023, background): “Reconstructing visual images from brain activity, such as that measured by functional Magnetic Resonance Imaging (fMRI), is an intriguing but challenging problem, because the underlying representations in the brain are largely unknown, and the sample size typically associated with brain data is rela”
Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision (2016, methodology): “2006) and encoding-modelbased decoding (Kay et al. 2008; Naselaris et al. 2009; Nishimoto et al. 2011).”
A Primer on Pattern-Based Approaches to fMRI: Principles, Pitfalls, and Perspectives. (2015, methodology): “…also makes it possible to extend encoding models and incorporate prior knowledge of stimulus probabilities, which 266 Neuron 87, July 15, 2015 ª2015 Elsevier Inc. can substantially improve reconstruction (Thirion et al., 2006; Naselaris et al., 2009; Schoenmakers et al., 2013; Figure 7E).”
Encoding and decoding in fMRI (2010, methodology): “On the encoding side, voxel-based encoding models (Dumoulin and Wandell, 2008; Gourtzelidis et al., 2005; Jerde et al., 2008; Kay et al., 2008; Mitchell et al., 2008; Naselaris et al., 2009; Schönwiesner and Zatorre, 2009; Thirion et al., 2006) have recently emerged as a promising computational tech”