Linguistic generalization and compositionality in modern artificial neural networks

By Marco Baroni. 2019

162 citations recorded by the source corpus.

In the last decade, deep artificial neural networks have achieved astounding performance in many natural language-processing tasks. Given the high productivity of language, these models must possess effective generalization abilities. It is widely assumed that humans handle linguistic productivity by means of algebraic compositional rules: are deep networks similarly compositional? After reviewing the main innovations characterizing current deep language-processing networks, I discuss a set of studies suggesting that deep networks are capable of subtle grammar-dependent generalizations, but also that they do not rely on systematic compositional rules. I argue that the intriguing behaviour of these devices (still awaiting a full understanding) should be of interest to linguists and cognitive scientists, as it offers a new perspective on possible computational strategies to deal with linguistic productivity beyond rule-based compositionality, and it might lead to new insights into the less systematic generalization patterns that also appear in natural language. This article is part of the theme issue ‘Towards mechanistic models of meaning composition’.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Cultural Studies, Natural Language Processing, Computational Linguistics & Semantics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Cultural Studies

In Natural Language Processing

In Computational Linguistics & Semantics

Citations in the corpus

Explainability in Deep Reinforcement Learning (2020, background): “Imitation learning is a way of enabling algorithms to learn from human demonstrations, such as teaching robots to learn assembly skills [35, 98].”
Compositional Generalization by Learning Analytical Expressions (2020, background): “It relates itself to the unboundedness of languages, which means languages license a theoretically infinite set of possible sentences [4].”
Understanding Natural Language Understanding Systems. A Critical Analysis (2023, background): “Therefore, the fact that neural models strive to generalize compositionally and systematically does not entail they are unable to capture important aspects of natural language, since it also departs from these same properties (Baroni 2019).”
Is the Red Square Big? MALeViC: Modeling Adjectives Leveraging Visual Contexts (2019, result): “This is in line with recent evidence showing that deep learning models do not rely on systematic compositional rules (Baroni, 2019; Bahdanau et al., 2019).”
Value-Agnostic Conversational Semantic Parsing (2021, background): “However, there is an additional potential beneﬁt from parameter sharing due to the compositional structure of the embeddings (see e.g., Baroni, 2020).”