The emergence of number and syntax units in LSTM language models
By Yair Lakretz, Germán Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, Marco Baroni. 2019
190 citations recorded by the source corpus.
Recent work has shown that LSTMs trained on a generic language modeling objective capture syntax-sensitive generalizations such as long-distance number agreement. We have however no mechanistic understanding of how they accomplish this remarkable feat. Some have conjectured it depends on heuristics that do not truly take hierarchical structure into account. We present here a detailed study of the inner mechanics of number tracking in LSTMs at the single neuron level. We discover that long-distance number information is largely managed by two “number units”. Importantly, the behaviour of these units is partially controlled by other units independently shown to track syntactic structure. We conclude that LSTMs are, to some extent, implementing genuinely syntactic processing mechanisms, paving the way to a more general understanding of grammatical encoding in LSTMs.
Subfields: Artificial Intelligence, Natural Language Processing, Computational Linguistics & Semantics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yonatan Belinkov
- Mohit Bansal
- Yoav Goldberg
- Percy Liang
- Luke S. Zettlemoyer
- Douwe Kiela
- Samuel R. Bowman
- Christopher D. Manning
In Natural Language Processing
- Yonatan Belinkov
- Yoav Goldberg
- Mohit Bansal
- Douwe Kiela
- Samuel R. Bowman
- James R. Glass
- Tal Linzen
- Percy Liang
In Computational Linguistics & Semantics
- Yonatan Belinkov
- Yoav Goldberg
- Tal Linzen
- Dieuwke Hupkes
- Willem H. Zuidema
- Ivan Titov
- Douwe Kiela
- James R. Glass
Citations in the corpus
- Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, background): “Recent work has performed causal analyses of syntactic agreement units in LSTM (Hochreiter and Schmidhuber, 1997)-based LMs (Lakretz et al., 2019; Lu et al., 2020) or causal analyses of LSTM hidden representations’ impact on syntactic agreement (Giulianelli et al., 2018), but the agreement…”
- Editing Factual Knowledge in Language Models (2021, background): “certain behaviour has been studied in the context of research in interpreting neural networks (Lakretz et al., 2019; Vig et al., 2020; Elazar et al., 2021; Csordás et al., 2021).”
- Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “11 Other studies that perform interventions to interpret NLP models without involving probing classifiers (e.g., Bau et al. 2019; Lakretz et al. 2019; Vig et al. 2020) are left out of the present scope.”
- Open Sesame: Getting inside BERT’s Linguistic Knowledge (2019, background): “A flurry of recent work (Linzen et al., 2016; Gulordava et al., 2018; Marvin and Linzen, 2018; Lakretz et al., 2019) has explored how recurrent neural language models perform in cases that require sensitivity to hierarchical syntactic structure, and study how they do so, particularly in the domain…”
- One Size Does Not Fit All: Comparing NMT Representations of Different Granularities (2019, result): “Other studies carried a more fine-grained neuron-level analysis for NMT and LM (Dalvi et al., 2019; Bau et al., 2019; Lakretz et al., 2019).”
- Measuring Causal Effects of Data Statistics on Language Model's 'Factual' Predictions (2022, methodology): “…of a given human-interpretable concept of a pretrained model, and relating the intervention to the change in the model’s behavior (Giulianelli et al., 2018; Lakretz et al., 2019; Bau et al., 2019; Elazar et al., 2021b; Feder et al., 2021b; Ravfogel et al., 2021; Antverg and Be-linkov, 2022).”
- Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little (2021, background): “pers provide compelling evidence that BERT contains a surprising amount of syntax, semantics, and world knowledge (Giulianelli et al., 2018; Rogers et al., 2020; Lakretz et al., 2019; Jumelet et al., 2019, 2021).”
- Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignment (2019, methodology): “This confirms our previous finding from the decomposition matrix, which showed a relatively high contribution of plural subjects to plural verbs, as well as the conclusion of Lakretz et al. (2019) that the model is in fact keeping track of syntactic structure.”
- Similarity Analysis of Contextual Word Representation Models (2020, background): “While neural language representations are thought to be densely distributed, several recent studies have pointed out the importance of individual neurons (Qian et al., 2016; Shi et al., 2016; Radford et al., 2017; Lakretz et al., 2019; Bau et al., 2019; Dalvi et al., 2019; Baan et al., 2019).”
- Probing Classifiers: Promises, Shortcomings, and Alternatives (2021, background): “Similarly, Lakretz et al. (2019) ablate neurons in language models by setting certain dimensions of f l ( x ) to zero.”
- Brains and algorithms partially converge in natural language processing (2022, background): “Recent neuroimaging studies suggest that they might — at least partially 8 – 12 .”
- On the Linguistic Representational Power of Neural Machine Translation Models (2019, background): “Recent studies carried a more fine-grained neuron-level analysis for NMT and LM (Dalvi et al. 2019a; Bau et al. 2019; Lakretz et al. 2019).”
- Linguistic generalization and compositionality in modern artificial neural networks (2019, background): “Lakretz and colleagues [55] conducted extensive ablation and connectivity studies of the Gulordava network.”
- Influence Paths for Characterizing Subject-Verb Number Agreement in LSTM Language Models (2020, methodology): “…syntactic rules in artificial languages (Gers and Schmidhuber, 2001), it is unclear whether they are as capable in natural languages (Linzen et al., 2016; Lakretz et al., 2019) in the context of rules such as subject-verb number agreement, especially when not supervised for the particular feature.”