Mechanisms for handling nested dependencies in neural-network language models and humans.

By Yair Lakretz, Dieuwke Hupkes, A. Vergallito, Marco Marelli, Marco Baroni, Stanislas Dehaene. 2020

75 citations recorded by the source corpus.

Recursive processing in sentence comprehension is considered a hallmark of human linguistic abilities. However, its underlying neural mechanisms remain largely unknown. We studied whether a modern artificial neural network trained with "deep learning" methods mimics a central aspect of human sentence processing, namely the storing of grammatical number and gender information in working memory and its use in long-distance agreement (e.g., capturing the correct number agreement between subject and verb when they are separated by other phrases). Although the network, a recurrent architecture with Long Short-Term Memory units, was solely trained to predict the next word in a large corpus, analysis showed the emergence of a very sparse set of specialized units that successfully handled local and long-distance syntactic agreement for grammatical number. However, the simulations also showed that this mechanism does not support full recursion and fails with some long-range embedded dependencies. We tested the model's predictions in a behavioral experiment where humans detected violations in number agreement in sentences with systematic variations in the singular/plural status of multiple nouns, with or without embedding. Human and model error patterns were remarkably similar, showing that the model echoes various effects observed in human data. However, a key difference was that, with embedded long-range dependencies, humans remained above chance level, while the model's systematic errors brought it below chance. Overall, our study shows that exploring the ways in which modern artificial neural networks process sentences leads to precise and testable hypotheses about human linguistic performance.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Natural Language Processing, Computational Linguistics & Semantics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Natural Language Processing

In Computational Linguistics & Semantics

Citations in the corpus

Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, background): “…have studied whether models prefer grammatical completions in subjectverb agreement contexts (Marvin and Linzen, 2018; van Schijndel et al., 2019; Goldberg, 2019; Mueller et al., 2020; Lakretz et al., 2021; Futrell et al., 2019), as well as in filler-gap dependencies (Wilcox et al., 2018, 2019).”