Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information
By Mario Giulianelli, J. Harding, Florian Mohnert, Dieuwke Hupkes, Willem H. Zuidema. 2018
202 citations recorded by the source corpus.
How do neural language models keep track of number agreement between subject and verb? We show that ‘diagnostic classifiers’, trained to predict number from the internal states of a language model, provide a detailed understanding of how, when, and where this information is represented. Moreover, they give us insight into when and where number information is corrupted in cases where the language model ends up making agreement errors. To demonstrate the causal role played by the representations we find, we then use agreement information to influence the course of the LSTM during the processing of difficult sentences. Results from such an intervention reveal a large increase in the language model’s accuracy. Together, these results show that diagnostic classifiers give us an unrivalled detailed look into the representation of linguistic information in neural models, and demonstrate that this knowledge can be used to improve their performance.
Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability, Computational Linguistics & Semantics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yonatan Belinkov
- Mohit Bansal
- Percy Liang
- Yoav Goldberg
- Luke S. Zettlemoyer
- Samuel R. Bowman
- Yejin Choi
- Christopher D. Manning
In Natural Language Processing
- Yonatan Belinkov
- Samuel R. Bowman
- Mohit Bansal
- Yoav Goldberg
- Tal Linzen
- Douwe Kiela
- Ellie Pavlick
- Percy Liang
In Safety, Robustness & Interpretability
- Yonatan Belinkov
- Adina Williams
- Ellie Pavlick
- Samuel R. Bowman
- Percy Liang
- Douwe Kiela
- Sebastian Gehrmann
- Isabelle Augenstein
In Computational Linguistics & Semantics
- Yonatan Belinkov
- Yoav Goldberg
- Tal Linzen
- James R. Glass
- Dieuwke Hupkes
- Willem H. Zuidema
- Roger Levy
- Ivan Titov
Citations in the corpus
- Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, background): “, 2020) or causal analyses of LSTM hidden representations’ impact on syntactic agreement (Giulianelli et al., 2018), but the agreement mechanisms of Transformer-based LMs have not been as extensively investigated.”
- Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “Thus, contrary to Giulianelli et al. (2018), they conclude that probing classifiers do not always identify features that are actually used by the model.”
- Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “In related work, Giulianelli et al. (2018) demonstrated that changing hidden representations in an LSTM based on the output of diagnostic classifiers can decrease the error on a subject-verb agreement classification task.”
- Analysis Methods in Neural Language Processing: A Survey (2018, background): “(2018) and Glockner et al. (2018) extracted examples from SNLI (Bowman et al.”
- What Does BERT Look at? An Analysis of BERT’s Attention (2019, background): “Another line of work investigates the internal vector representations of the model (Adi et al., 2017; Giulianelli et al., 2018; Zhang and Bow-man, 2018), often using probing classifiers.”
- Designing and Interpreting Probes with Control Tasks (2019, methodology): “Probes have also been used to test hypotheses about the mechanisms by which models perform tasks (Hupkes et al., 2018; Giulianelli et al., 2018).”
- Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly (2019, background): “A wide range of literature analyzes linguistic knowledge stored in pretrained embeddings (Jumelet and Hupkes, 2018; Gulordava et al., 2018; Giulianelli et al., 2018; McCoy et al., 2019; Dasgupta et al., 2018; Marvin and Linzen, 2018; Warstadt and Bowman, 2019; Kann et al., 2019).”
- Open Sesame: Getting inside BERT’s Linguistic Knowledge (2019, background): “This is similar to the concept of diagnostic classification by Hupkes et al. (2018); Giulianelli et al. (2018). the cat will sleep (BERT) ↓ ↓ ↓ ↓ e1 e2 e3 e4 (Classifier) ↓ ↓ ↓ ↓ ŷ1 ŷ2 ŷ3 ŷ4 Each input sentence is tokenized and processed by BERT, and the resulting embeddings {ei} are…”
- Probing What Different NLP Tasks Teach Machines about Function Word Comprehension (2019, background): “Gulordava et al. (2018), Giulianelli et al. (2018), Rønning et al. (2018), and Jumelet and Hupkes (2018) make a focused contribution towards a particular linguistic phenomenon (agreement, ellipsis, negative polarity).”
- The emergence of number and syntax units in LSTM language models (2019, methodology): “To verify this, we tested whether subject number can be decoded from the whole pattern of activities in the network (excluding the two LR-number units) and whether this decoding is stable across time (see Giulianelli et al., 2018, for similar observations and related methods).”
- Measuring Causal Effects of Data Statistics on Language Model's 'Factual' Predictions (2022, methodology): “…of the encoding of a given human-interpretable concept of a pretrained model, and relating the intervention to the change in the model’s behavior (Giulianelli et al., 2018; Lakretz et al., 2019; Bau et al., 2019; Elazar et al., 2021b; Feder et al., 2021b; Ravfogel et al., 2021; Antverg and…”
- Emergent Linear Representations in World Models of Self-Supervised Sequence Models (2023, background): “Intervention is commonly used to study causality (Giulianelli et al., 2018; Tucker et al., 2021), but have often reached mixed conclusions (Belinkov, 2022).”
- Negated LAMA: Birds cannot fly (2019, background): “There is a wide range of literature analyzing linguistic knowledge stored in pretrained embed-dings (Jumelet and Hupkes, 2018; Gulordava et al., 2018; Giulianelli et al., 2018; McCoy et al., 2019; Dasgupta et al., 2018; Martin and Linzen, 2018; Warstadt and Bowman, 2019; Kann et al., 2019).”