Generic decoding of seen and imagined objects using hierarchical visual features

By Tomoyasu Horikawa, Yukiyasu Kamitani. Nature Communications, 2015

569 citations recorded by the source corpus.

Object recognition is a key function in both human and machine vision. While brain decoding of seen and imagined objects has been achieved, the prediction is limited to training examples. We present a decoding approach for arbitrary objects using the machine vision principle that an object category is represented by a set of features rendered invariant through hierarchical processing. We show that visual features, including those derived from a deep convolutional neural network, can be predicted from fMRI patterns, and that greater accuracy is achieved for low-/high-level features with lower-/higher-level visual areas, respectively. Predicted features are used to identify seen/imagined object categories (extending beyond decoder training) from a set of computed features for numerous object images. Furthermore, decoding of imagined objects reveals progressive recruitment of higher-to-lower visual representations. Our results demonstrate a homology between human and machine vision and its utility for brain-based information retrieval. Machine learning algorithms can decode objects that people see or imagine from their brain activity. Here the authors present a predictive decoder combined with deep neural network representations that generalizes beyond the training set and correctly identifies novel objects that it has never been trained on.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/025dbb99945ca3c12365d3e635813045722c9e6c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep image reconstruction from human brain activity (2017, methodology): “We trained the decoders that predict the DNN features of viewed images from fMRI activity patterns, following the procedures of Horikawa and Kamitani (2017).”
Deep Learning for Spatio-Temporal Data Mining: A Survey (2019, methodology): “[44], [46], [48], [64], [104], [123] modeled the fMRI data as spatial maps, and then used them as the input of the classification models.”
Invariant object recognition is a personalized selection of invariant features in humans, not simply explained by hierarchical feed-forward vision models (2017, result): “This is interesting as recent studies have shown a high level of brain-plausibility for the hierarchical model (AlexNet 20 ) used in the current paper 2,4,6,21,43,55 .”
Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision (2016, methodology): “In line with previous studies (Khaligh-Razavi and Kriegeskorte 2014; Yamins et al. 2014; Güçlü and van Gerven 2015a, b; Cichy et al. 2016; Kubilius et al. 2016; Eickenberg et al. 2016; Horikawa and Kamitani 2017), we explored the correspondence between individual layers in the CNN and individual cor”