Information-Theoretic Probing for Linguistic Structure
By Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, Ryan Cotterell. 2020
280 citations recorded by the source corpus.
The success of neural networks on a diverse set of NLP tasks has led researchers to question how much these networks actually “know” about natural language. Probes are a natural way of assessing this. When probing, a researcher chooses a linguistic task and trains a supervised model to predict annotations in that linguistic task from the network’s learned representations. If the probe does well, the researcher may conclude that the representations encode knowledge related to the task. A commonly held belief is that using simpler models as probes is better; the logic is that simpler models will identify linguistic structure, but not learn the task itself. We propose an information-theoretic operationalization of probing as estimating mutual information that contradicts this received wisdom: one should always select the highest performing probe one can, even if it is more complex, since it will result in a tighter estimate, and thus reveal more of the linguistic information inherent in the representation. The experimental portion of our paper focuses on empirically estimating the mutual information between a linguistic property and BERT, comparing these estimates to several baselines. We evaluate on a set of ten typologically diverse languages often underrepresented in NLP research—plus English—totalling eleven languages. Our implementation is available in https://github.com/rycolab/info-theoretic-probing.
Subfields: Artificial Intelligence, Natural Language Processing, Information Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yonatan Belinkov
- Mohit Bansal
- Percy Liang
- Samuel R. Bowman
- Luke S. Zettlemoyer
- Christopher D. Manning
- Yejin Choi
- Yoav Goldberg
In Natural Language Processing
- Yonatan Belinkov
- Mohit Bansal
- Samuel R. Bowman
- Douwe Kiela
- Percy Liang
- Yoav Goldberg
- Christopher D. Manning
- Yejin Choi
In Information Theory
- Yonatan Belinkov
- Jacob Andreas
- Kawin Ethayarajh
- Tiago Pimentel
- Roger Levy
- Christopher D. Manning
- Joelle Pineau
- Percy Liang
Citations in the corpus
- Factual Probing Is [MASK]: Learning vs. Learning to Recall (2021, methodology): “This is analogous to a control function Pimentel et al. (2020), a function that removes information from a linguistic representation.”
- Probing Classifiers: Promises, Shortcomings, and Advances (2021, methodology): “Taking an information-theoretic perspective on probing, Pimentel et al. (2020b) proposed to use control functions instead of control tasks in order to compare probes.”
- Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals (2020, background): “Indeed, there is an ongoing debate on the assertion that certain information is ‘‘encoded’’ in the representation (Voita and Titov, 2020; Pimentel et al., 2020).”
- When Do You Need Billions of Words of Pretraining Data? (2020, background): “has found problems with many probing methods: Classifier probes can learn too much from training data (Hewitt and Liang, 2019) and can fail to distinguish features that are extractable from features that are actually used when generalizing on downstream tasks (Voita and Titov, 2020; Pimentel et al.,”
- What Context Features Can Transformer Language Models Use? (2021, positive): “Our approach also shares similar motivations to information-theoretic work on probing (Voita and Titov, 2020; Pimentel et al., 2020), which uses related tools to interpret linguistic structure in LM representations rather than characterizing their effect on LM predictions.”
- Emergent Structures and Training Dynamics in Large Language Models (2022, methodology): “Many of these approaches highlight the need to for-208 malize the "effort" required in encoding a linguistic property, often via some form of a control function (Pimentel et al., 2020b).”
- Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little (2021, methodology): “To probe the array of pre-trained models for syntactic, semantic and other linguistic properties, we investigate dependency parsing using Pareto probing (Pimentel et al., 2020a) as well as the probing tasks from Conneau et al. (2018) available in the SentEval toolkit (Conneau and Kiela, 2018).”
- Probing Classifiers: Promises, Shortcomings, and Alternatives (2021, background): “Taking an information-theoretic perspective on probing, Pimentel et al. (2020b) proposed to use control functions instead of control tasks in order to compare probes.”
- Analyzing Individual Neurons in Pre-trained Language Models (2020, unknown): “Pimentel et al. (2020) discussed the complexity of the probes and argued for using highest performing probes for tighter estimates.”
- Do Language Embeddings capture Scales? (2020, methodology): “Probes We use linear probes in all cases following many previous probing work (Shi et al., 2016; Ettinger et al., 2016; Pimentel et al., 2020) since we want to use a simple probe to find easily accessible information in a representation.”
- Finding Neurons in a Haystack: Case Studies with Sparse Probing (2023, background): “…between class means for each neuron (Antverg & Belinkov, 2021), the mutual information between each neuron and the labels (Ross, 2014; Voita & Titov, 2020; Pimentel et al., 2020), or the absolute magnitude of the coefficients of a dense probe trained with l 1 regularization (Dalvi et al., 2019).”
- Probing Pretrained Language Models for Lexical Semantics (2020, background): “…their mas- 1The interplay between the complexity of a probe and its accuracy, as well as its effect on the overall procedure, remain controversial (Pimentel et al., 2020; Voita and Titov, 2020). sively multilingual counterparts; 2) providing context around the target word in input; 3) including…”
- What if This Modified That? Syntactic Interventions with Counterfactual Embeddings (2021, methodology): “On the other hand, Pimentel et al. (2020) argue from an information-theoretical perspective that more expressive probes are always preferable.”
- Do Language Embeddings capture Scales? (2020, methodology): “Probes We use linear probes in all cases following many previous probing work (Shi et al., 2016; Ettinger et al., 2016; Pimentel et al., 2020) since we want to use a simple probe to find easily accessible information in a representation.”
- Conditional probing: measuring usable information beyond a baseline (2021, methodology): “While Pimentel et al. (2020b) conclude from the data processing inequality that probing is not meaningful, we conclude that estimating mutual information is not the goal of probing.”
- Emergent Linear Representations in World Models of Self-Supervised Sequence Models (2023, background): “Figure 12: Cosine similarity scores between linear probes across layers. as inadequate (Pimentel et al., 2020b; Saphra and Lopez, 2019).”
- Analyzing Redundancy in Pretrained Transformer Models (2020, background): “Recently, Voita and Titov (2020); Pimentel et al. (2020) proposed probing methods based on information theoretic measures.”