Shared computational principles for language processing in humans and deep language models

By Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, A. Price, Bobbi Aubrey, Samuel A. Nastase, Amir Feder, Dotan Emanuel, Alon Cohen and 22 more. 2022

506 citations recorded by the source corpus.

Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models generate appropriate linguistic responses in a given context. In the current study, nine participants listened to a 30-min podcast while their brain responses were recorded using electrocorticography (ECoG). We provide empirical evidence that the human brain and autoregressive DLMs share three fundamental computational principles as they process the same natural narrative: (1) both are engaged in continuous next-word prediction before word onset; (2) both match their pre-onset predictions to the incoming word to calculate post-onset surprise; (3) both rely on contextual embeddings to represent words in natural contexts. Together, our findings suggest that autoregressive DLMs provide a new and biologically feasible computational framework for studying the neural basis of language. Deep language models have revolutionized natural language processing. The paper discovers three computational principles shared between deep language models and the human brain, which can transform our understanding of the neural basis of language.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Social Psychology, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Social Psychology

In Large Language Models

In Natural Language Processing

Citations in the corpus

Can language models learn from explanations in context? (2022, background): “, 1992), and given accumulating evidence that LMs predict a great deal of human neural language processing (Schrimpf et al., 2021; Goldstein et al., 2022), it is natural to ask whether there are cognitive implications of our work.”
Understanding Natural Language Understanding Systems. A Critical Analysis (2023, background): “In fact, Goldstein et al. (2022) claim that autoregressive language models like GPT share with the brain important computational principles, since the brain is also constantly involved in next-word prediction as it processes natural language and represents words with contextual embeddings that…”
High-resolution image reconstruction with latent diffusion models from human brain activity (2023, methodology): “…and design of deep neural network models, have allowed direct comparisons be-tween latent representations in biological brains and architectural characteristics of artiﬁcial networks, providing important insights into how these systems operate [3, 8– 10, 13, 18, 19, 21, 42, 43, 54, 55].”
Evaluating large language models in theory of mind tasks (2023, background): “Studying AI can also advance psychological science ( 80 – 82 ).”
How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection (2023, background): “From the linguistic or NLP perspectives, we are also interested in where are the remaining gaps between ChatGPT and humans and what are their implicit linguistic differences [14, 18].”
Cortical encoding of probabilistic temporal predictions during speech perception (2026, unknown): “In this sense, temporal prediction appears to be a distinct mechanism from content prediction, which is time-locked to the unit’s onset and traditionally modeled via surprisal (4, 8, 9).”
ROSE: A Neurocomputational Architecture for Syntax (2023, unknown): “Recent research has shown that the human brain and autoregressive deep language models both engage in continuous next-word prediction before word onset (Goldstein et al., 2022), but this literature has focused almost exclusively on single-element, lexical statistics and prediction, which many lingui”