Object representations in the human brain reflect the co-occurrence statistics of vision and language

By Michael F. Bonner, Russell A. Epstein. 2020

90 citations recorded by the source corpus.

A central regularity of visual perception is the co-occurrence of objects in the natural environment. Here we use machine learning and fMRI to test the hypothesis that object co-occurrence statistics are encoded in the human visual system and elicited by the perception of individual objects. We identified low-dimensional representations that capture the latent statistical structure of object co-occurrence in real-world scenes, and we mapped these statistical representations onto voxel-wise fMRI responses during object viewing. We found that cortical responses to single objects were predicted by the statistical ensembles in which they typically occur, and that this link between objects and their visual contexts was made most strongly in parahippocampal cortex, overlapping with the anterior portion of scene-selective parahippocampal place area. In contrast, a language-based statistical model of the co-occurrence of object names in written text predicted responses in neighboring regions of object-selective visual cortex. Together, these findings show that the sensory coding of objects in the human brain reflects the latent statistics of object context in visual and linguistic experience. When people view an object, they can often guess the setting from which it was drawn and the other objects that might be found in that setting. Here the authors identify regions of the human visual system that represent this information about which objects tend to appear together in the world.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

  1. Sepp Hochreiter

In Computer Vision and Pattern Recognition

  1. Günter Klambauer
  2. Hubert Ramsauer
  3. Angela Bitto-Nemling
  4. Fei Tang
  5. Viet-Hung Tran
  6. Michael Kopp
  7. Johannes Lehner
  8. Andreas Furst

In Representation & Self-Supervised Learning

  1. Günter Klambauer
  2. Viet-Hung Tran
  3. Sepp Hochreiter
  4. J. Urgen Schmidhuber

Citations in the corpus