Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models

By Matthew Finlayson, Aaron Mueller, Stuart M. Shieber, Sebastian Gehrmann, Tal Linzen, Yonatan Belinkov. 2021

151 citations recorded by the source corpus.

Targeted syntactic evaluations have demonstrated the ability of language models to perform subject-verb agreement given difficult contexts. To elucidate the mechanisms by which the models accomplish this behavior, this study applies causal mediation analysis to pre-trained neural language models. We investigate the magnitude of models’ preferences for grammatical inflections, as well as whether neurons process subject-verb agreement similarly across sentences with different syntactic structures. We uncover similarities and differences across architectures and model sizes—notably, that larger models do not necessarily learn stronger preferences. We also observe two distinct mechanisms for producing subject-verb agreement depending on the syntactic structure of the input sentence. Finally, we find that language models rely on similar sets of neurons when given sentences with similar syntactic structure.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yonatan Belinkov
2. Hinrich Schütze
3. Yoav Goldberg
4. Mohit Bansal
5. Roi Reichart
6. Abhilasha Ravichander
7. Yanai Elazar
8. Timo Schick

In Large Language Models

1. Hinrich Schütze
2. Yoav Goldberg
3. Nora Kassner
4. Abhilasha Ravichander
5. Yanai Elazar
6. Jacob Eisenstein
7. Roi Reichart
8. Diyi Yang

In Natural Language Processing

1. Yonatan Belinkov
2. Hinrich Schütze
3. Yoav Goldberg
4. Yanai Elazar
5. Abhilasha Ravichander
6. Mohit Bansal
7. Roi Reichart
8. Noah A. Smith

In Causal Inference & Discovery

1. Yonatan Belinkov
2. Katherine A. Keith
3. Zach Wood-Doughty
4. Jacob Eisenstein
5. Hinrich Schütze
6. Roi Reichart
7. Yoav Goldberg
8. Shauli Ravfogel

Citations in the corpus

• The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets (2023, unknown): “Other authors have trained probes to classify truthfulness from LLM activations, using both logistic regression (Azaria & Mitchell, 2023; Li et al., 2023b) and unsupervised techniques (Burns et al., 2023).”
• Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (2024, unknown): “Following prior work (Vig et al., 2020; Finlayson et al., 2021), we quantify the importance of a on a pair of inputs ( x clean , x patch ) via its indirect effect (IE; Pearl, 2001) on m : (2) Here, a patch is the value that a takes in the computation of m ( x patch ) , and m ( x clean | do ( a = a…”
• Measuring Causal Effects of Data Statistics on Language Model's 'Factual' Predictions (2022, methodology): “Finlayson et al. (2021) followed a similar framework to investigate mechanisms of subject–verb agreement in language models.”
• Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small (2022, unknown): “In this way our use case is similar to Finlayson et al. (2021).”
• Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond (2021, background): “Finally, a causal perspective on attention-based explanations is to view internal nodes as mediators of the causal effect from the input to the output (Vig et al., 2020; Finlayson et al., 2021).”