The neural architecture of language: Integrative modeling converges on predictive processing

By Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Josh Tenenbaum, E. Fedorenko. 2020

663 citations recorded by the source corpus.

Significance Language is a quintessentially human ability. Research has long probed the functional architecture of language in the mind and brain using diverse neuroimaging, behavioral, and computational modeling approaches. However, adequate neurally-mechanistic accounts of how meaning might be extracted from language are sorely lacking. Here, we report a first step toward addressing this gap by connecting recent artificial neural networks from machine learning to human recordings during language processing. We find that the most powerful models predict neural and behavioral responses across different datasets up to noise levels. Models that perform better at predicting the next word in a sequence also better predict brain measurements—providing computationally explicit evidence that predictive processing fundamentally shapes the language comprehension mechanisms in the brain. The neuroscience of perception has recently been revolutionized with an integrative modeling approach in which computation, brain function, and behavior are linked across many datasets and many computational models. By revealing trends across models, this approach yields novel insights into cognitive and neural mechanisms in the target domain. We here present a systematic study taking this approach to higher-level cognition: human language processing, our species’ signature cognitive skill. We find that the most powerful “transformer” models predict nearly 100% of explainable variance in neural responses to sentences and generalize across different datasets and imaging modalities (functional MRI and electrocorticography). Models’ neural fits (“brain score”) and fits to behavioral responses are both strongly correlated with model accuracy on the next-word prediction task (but not other language tasks). Model architecture appears to substantially contribute to neural fit. These results provide computationally explicit evidence that predictive processing fundamentally shapes the language comprehension mechanisms in the human brain.

Subfields: Cognitive Neuroscience, Electrical and Electronic Engineering, Social Psychology, Neuroscience, Cognitive Science & Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Tijmen Blankevoort

In Social Psychology

In Neuroscience

In Cognitive Science & Psychology

Citations in the corpus

VeRA: Vector-based Random Matrix Adaptation (2023, background): “Other works (Lu et al., 2022; Schrimpf et al., 2021; Frankle et al., 2021) have shown that frozen, randomly initialized models, with small sections finetuned, can perform surprisingly well.”
High-resolution image reconstruction with latent diffusion models from human brain activity (2023, methodology): “…and design of deep neural network models, have allowed direct comparisons be-tween latent representations in biological brains and architectural characteristics of artiﬁcial networks, providing important insights into how these systems operate [3, 8– 10, 13, 18, 19, 21, 42, 43, 54, 55].”
Over-reliance on English hinders cognitive science. (2022, positive): “, reading times [82]) and, vice versa, those with the most brain-like representations perform the best at this task [83], all of which reinforces the idea that the similarities between Language Models and humans might be rooted in processes and representations shared by both.”
Cortical encoding of probabilistic temporal predictions during speech perception (2026, unknown): “S2), and may also relate to the observation that a model's predictive accuracy does not systematically translate into its neural predictivity (63), but see (10).”