Designing and Interpreting Probes with Control Tasks

By John Hewitt, Percy Liang. 2019

737 citations recorded by the source corpus.

Probes, supervised models trained to predict properties (like parts-of-speech) from representations (like ELMo), have achieved high accuracy on a range of linguistic tasks. But does this mean that the representations encode linguistic structure or just that the probe has learned the linguistic task? In this paper, we propose control tasks, which associate word types with random outputs, to complement linguistic tasks. By construction, these tasks can only be learned by the probe itself. So a good probe, (one that reflects the representation), should be selective, achieving high linguistic task accuracy and low control task accuracy. The selectivity of a probe puts linguistic task accuracy in context with the probe’s capacity to memorize from word types. We construct control tasks for English part-of-speech tagging and dependency edge prediction, and show that popular probes on ELMo representations are not selective. We also find that dropout, commonly used to control probe complexity, is ineffective for improving selectivity of MLPs, but that other forms of regularization are effective. Finally, we find that while probes on the first layer of ELMo yield slightly better part-of-speech tagging accuracy than the second, probes on the second layer are substantially more selective, which raises the question of which layer better represents parts-of-speech.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “41. github.com/bigscience-workshop/massive-probing-framework Logistic regression is chosen due to its higher selectivity as opposed to non-linear probing classifiers (Hewitt and Liang, 2019).”
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021, background): “Still, to make this methodology rigorous, interpreting and controlling the strength of the probes will be an important future direction (Hewitt and Liang, 2019).”
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers (2021, methodology): “We focus on zero-shot evaluation as it enables us to examine the representations without confounding our ﬁndings with the side-effects of ﬁne-tuning (Yogatama et al., 2019) or probing classi-ﬁers ( e.g. , Zhang and Bowman, 2018; Hewitt and Liang, 2019).”
Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs (2021, background): “Subsequent work has explored how much knowledge is stored in LMs (Heinzerling and Inui, 2021), approaches to querying models for knowledge (Hewitt and Liang, 2019; Jiang et al., 2020; Voita and Titov, 2020; West et al., 2021), and meth-”
Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, background): “Probing is more difficult to interpret than behavioral approaches because the addition of a trained classifier introduces confounds (Hewitt and Liang, 2019): most notably, whether the probe maps from”
Factual Probing Is [MASK]: Learning vs. Learning to Recall (2021, methodology): “These include random baselines (Hewitt and Liang, 2019) and information-theoretic measurements (Voita and Titov, 2020).”
Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “They found that MDL provides more information about how a probe g works, for instance by revealing differences in com-5 plexity of probes when performing control tasks from D P,Rand , as in Hewitt and Liang (2019).”
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals (2020, background): “These results strengthen recent works that question the usefulness of probing as an analysis tool (Hewitt and Liang, 2019; Ravichander et al., 2020), but measure it from the usefulness of properties on the main task.”
Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “In addition, it is challenging to differentiate the information learned by the probing classiﬁer from that learned by the underlying model (Hewitt and Liang, 2019).”
Revisiting Few-sample BERT Fine-tuning (2020, methodology): “BERT Representation Transferability BERT pre-trained representations have been widely studied using probing methods showing that the pre-trained features from intermediate layers are more transferable [15, 16, 25, 40, 41] or applicable [55] to new tasks than features from later layers, which change ”
Eliciting Knowledge from Language Models Using Automatically Generated Prompts (2020, background): “However, probing classifiers require additional learned parameters and are thus susceptible to false positives; high probing accuracy is not a sufficient condition to conclude that an LM contains a certain piece of knowledge (Hewitt and Liang, 2019; Voita and Titov, 2020).”
Information-Theoretic Probing with Minimum Description Length (2020, methodology): “Accuracy is shown for the standard probe as in Hewitt and Liang (2019); for the variational probe, scores are similar (see Table 3).”
Implicit Representations of Meaning in Neural Language Models (2021, neutral): “A phenomenon is judged to be encoded by a model if the probe’s accuracy cannot be explained by its accuracy when trained on control tasks (Hewitt and Liang, 2019) or baseline models (Pimentel et al., 2020).”
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions (2020, background): “, 2018), probing approaches that test internal representations of models for certain tasks and properties (Liu et al., 2019b; Hewitt and Liang, 2019), and generative approaches that make the model jointly extract or”
FNet: Mixing Tokens with Fourier Transforms (2021, background): “Although potentially limited in their effectiveness (Hewitt and Liang, 2019), these probes of attention layers generally back the intuition that, by allowing higher order units to form out of compositions of the input, Transformer models can flexibly capture”
When Do You Need Billions of Words of Pretraining Data? (2020, background): “Hewitt and Liang (2019) and Voita and Titov (2020) caution that the performance achieved in the classiﬁer probing setting reﬂects a combined effort of the representations and the probe, so a probing classiﬁer’s performance does not precisely reveal the quality of the representations.”
Information-Theoretic Probing for Linguistic Structure (2020, methodology): “Actually, Hewitt and Liang likely had an intuition about this in mind when they wrote “[a] sufficiently expressive probe with enough training data could learn any task on top of it” (Hewitt and Liang, 2019).”
A Bayesian Framework for Information-Theoretic Probing (2021, background): “Most work in this field claims that, when performing this analysis, we should prefer simple models as probes (Alain and Bengio, 2016; Hewitt and Liang, 2019; Voita and Titov, 2020).”
Pareto Probing: Trading-Off Accuracy and Complexity (2020, background): “Most saliently, Hewitt and Liang (2019) attempts to operationalize complexity in terms of control tasks, which constrain a probe’s capacity for memorization.”