Toward a universal decoder of linguistic meaning from brain activation

By Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J. Gershman, Nancy Kanwisher, Matthew Botvinick, E. Fedorenko. Nature Communications, 2018

379 citations recorded by the source corpus.

Prior work decoding linguistic meaning from imaging data has been largely limited to concrete nouns, using similar stimuli for training and testing, from a relatively small number of semantic categories. Here we present a new approach for building a brain decoding system in which words and sentences are represented as vectors in a semantic space constructed from massive text corpora. By efficiently sampling this space to select training stimuli shown to subjects, we maximize the ability to generalize to new meanings from limited imaging data. To validate this approach, we train the system on imaging data of individual concepts, and show it can decode semantic vector representations from imaging data of sentences about a wide variety of both concrete and abstract topics from two separate datasets. These decoded representations are sufficiently detailed to distinguish even semantically similar sentences, and to capture the similarity structure of meaning relationships between sentences. Previous work decoding linguistic meaning from imaging data has generally been limited to a small number of semantic categories. Here, authors show that a decoder trained on neuroimaging data of single concepts sampling the semantic space can robustly decode meanings of semantically diverse new sentences with topics not encountered during training.

Subfields: Artificial Intelligence, Cognitive Neuroscience

View Paper: https://www.semanticscholar.org/paper/8e5acc4b6fd3126c41f74dae3644b7daa3f374ce

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

Citations in the corpus

Semantic reconstruction of continuous language from non-invasive brain recordings (2022, background): “Although previous studies have shown that the BOLD signal contains rich semantic information 5,11 , our results show that this information is captured at the granularity of individual words and phrases.”
The neural architecture of language: Integrative modeling converges on predictive processing (2020, methodology): “We used the data from Pereira et al.’s (2018) Experiments 2 (n=9) and 3 (n=6) (10 unique participants).”
Blackbox Meets Blackbox: Representational Similarity & Stability Analysis of Neural Language Models and Brains (2019, methodology): “models, or using NLP models to build predictive models of the human brain, or both (Murphy et al., 2012; Wehbe et al., 2014a; Ruan et al., 2016; Søgaard, 2016; Xu et al., 2016; Fyshe et al., 2014; Bingel et al., 2016; Bulat et al., 2017; Abnar et al., 2018; Pereira et al., 2018; Huth et al., 2016).”