High-resolution image reconstruction with latent diffusion models from human brain activity

By Yu Takagi, Shinji Nishimoto. bioRxiv, 2023

397 citations recorded by the source corpus.

Reconstructing visual experiences from human brain activity offers a unique way to understand how the brain represents the world, and to interpret the connection between computer vision models and our visual system. While deep generative models have recently been employed for this task, reconstructing realistic images with high semantic fidelity is still a challenging problem. Here, we propose a new method based on a diffusion model (DM) to reconstruct images from human brain activity obtained via functional magnetic resonance imaging (fMRI). More specifically, we rely on a latent diffusion model (LDM) termed Stable Diffusion. This model reduces the computational cost of DMs, while preserving their high generative performance. We also characterize the inner mechanisms of the LDM by studying how its different components (such as the latent vector of image Z, conditioning inputs C, and different elements of the denoising U-Net) relate to distinct brain functions. We show that our proposed method can reconstruct high-resolution images with high fidelity in straight-forward fashion, without the need for any additional training and fine-tuning of complex deep-learning models. We also provide a quantitative interpretation of different LDM components from a neuroscientific perspective. Overall, our study proposes a promising method for reconstructing images from human brain activity, and provides a new framework for understanding DMs. Please check out our webpage at https://sites.google.com/view/stablediffusion-with-brain/.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/2824f18b3aeaae51b3fbe0d629c9b8a728da86d7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

Robin Rombach

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis (2023, background): “Remarkably, the scope of applications has been extraordinarily extensive, encompassing fields as diverse as music generation [9] and reconstructing images from fMRI brain scans [49].”
DreamLLM-3D: Affective Dream Reliving using Large Language Model and 3D Generative AI (2025, unknown): “Some directly look at how AI can help decode fMRI signals and reconstruct visual images [Miyawaki et al., 2008, Kay et al., 2008, Takagi and Nishimoto, 2023], and further decode visual imagery during sleep [Horikawa et al., 2013] and dreamed objects [Horikawa and Kamitani, 2017].”