Brains and algorithms partially converge in natural language processing

By Charlotte Caucheteux, Jean-Remi King. Communications Biology, 2022

353 citations recorded by the source corpus.

Deep learning algorithms trained to predict masked words from large amount of text have recently been shown to generate activations similar to those of the human brain. However, what drives this similarity remains currently unknown. Here, we systematically compare a variety of deep language models to identify the computational principles that lead them to generate brain-like representations of sentences. Specifically, we analyze the brain responses to 400 isolated sentences in a large cohort of 102 subjects, each recorded for two hours with functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG). We then test where and when each of these algorithms maps onto the brain responses. Finally, we estimate how the architecture, training, and performance of these models independently account for the generation of brain-like representations. Our analyses reveal two main findings. First, the similarity between the algorithms and the brain primarily depends on their ability to predict words from context. Second, this similarity reveals the rise and maintenance of perceptual, lexical, and compositional representations within each cortical region. Overall, this study shows that modern language algorithms partially converge towards brain-like solutions, and thus delineates a promising path to unravel the foundations of natural language processing. Charlotte Caucheteux and Jean-Rémi King examine the ability of transformer neural networks trained on word prediction tasks to fit representations in the human brain measured with fMRI and MEG. Their results provide further insight into the workings of transformer language models and their relevance to brain responses.

Subfields: Cognitive Neuroscience, Cultural Studies, Social Psychology

View Paper: https://www.semanticscholar.org/paper/83a491b6dfab0a9f30ce66d7dad1d7409e4d6e4d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Cultural Studies

In Social Psychology

Citations in the corpus

Spikformer: When Spiking Neural Network Meets Transformer (2022, background): “Self-attention, the key part of Transformer, selectively focuses on information of interest, and is also an important feature of the human biological system (Whittington et al., 2022; Caucheteux & King, 2022).”
Semantic reconstruction of continuous language from non-invasive brain recordings (2022, methodology): “Previous studies have shown that middle layers of language models extract the best semantic features for predicting brain responses to natural language 8,14,15,17 .”
High-resolution image reconstruction with latent diffusion models from human brain activity (2023, methodology): “…and design of deep neural network models, have allowed direct comparisons be-tween latent representations in biological brains and architectural characteristics of artiﬁcial networks, providing important insights into how these systems operate [3, 8– 10, 13, 18, 19, 21, 42, 43, 54, 55].”
PolyRNN: A time-resolved model of polyphonic musical expectations aligned with human brain responses (2025, unknown): “…growing trend in cognitive neurosciences to adopt RNN and Transformer models, which offer a more accurate representation of the neural encoding of perceptual statistical properties (Donhauser & Baillet, 2020; Saxe et al., 2021; Caucheteux & King, 2022; Kanwisher et al., 2023; Arana et al., 2024).”