Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?

By Abhilasha Ravichander, Yonatan Belinkov, Eduard H. Hovy. 2020

169 citations recorded by the source corpus.

Although neural models have achieved impressive results on several NLP benchmarks, little is understood about the mechanisms they use to perform language tasks. Thus, much recent attention has been devoted to analyzing the sentence representations learned by neural encoders, through the lens of ‘probing’ tasks. However, to what extent was the information encoded in sentence representations, as discovered through a probe, actually used by the model to perform its task? In this work, we examine this probing paradigm through a case study in Natural Language Inference, showing that models can learn to encode linguistic properties even if they are not needed for the task on which the model was trained. We further identify that pretrained word embeddings play a considerable role in encoding these properties rather than the training task itself, highlighting the importance of careful controls when designing probing experiments. Finally, through a set of controlled synthetic tasks, we demonstrate models can encode these properties considerably above chance-level, even when distributed in the data as random noise, calling into question the interpretation of absolute claims on probing tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “Ravichander, Belinkov, and Hovy (2021) point out that datasets are imperfect proxies for tasks.”
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals (2020, background): “This is consistent with the recent findings of Ravichander et al. (2020).”
Discovering the Compositional Structure of Vector Representations with Role Learning Networks (2019, background): “classifier to extract that information does not mean that the model is using it (Voita and Titov, 2020; Ravichander et al., 2020; Vanmassenhove et al., 2017).”
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors (2021, background): “…work on introspecting and understanding BERT (see Rogers et al. (2020) for an overview), e.g., via probing (Tenney et al., 2019), analyzes by Hewitt and Liang (2019); Eger et al. (2020); Ravichander et al. (2021) indicate that probing results (based on supervision) are not always trustworthy.”
Unsupervised Distillation of Syntactic Information from Contextualized Word Representations (2020, background): “This approach can introduce confounding between extracting information and learning it by the probe (Hewitt and Liang, 2019; Ravichander et al., 2020; Maudslay et al., 2020; Elazar et al., 2020).”
Finding Neurons in a Haystack: Case Studies with Sparse Probing (2023, methodology): “However, sparse probing also inherits many of the weaknesses of the general probing paradigm (Hewitt & Liang, 2019; Donnelly & Roegiest, 2019; Ravichander et al., 2020; Maudslay & Cotterell, 2021; Elazar et al., 2021).”
Probing Classifiers: Promises, Shortcomings, and Alternatives (2021, background): “Ravichander, Belinkov, and Hovy (2021) point out that datasets are imperfect proxies for tasks.”
Language Models Represent Space and Time (2023, background): “High predictive performance on out-of-sample data indicates that the base model has temporal and spatial information linearly decodable in its representations, although this does not imply that the model actually uses these representations (Ravichander et al., 2020).”
Causal Abstractions of Neural Networks (2021, background): “Probes are fundamentally unable to directly measure causal properties of neural representations, and Ravichander et al. [21], Elazar et al. [9], and Geiger et al. [11] have argued that probes are limited in their ability to provide even indirect evidence of causal properties.”
DirectProbe: Studying Representations without Classifiers (2021, background): “Controlled test sets (¸Senel et al., 2018; Jastrzebski et al., 2017) are designed and errors are analyzed to reverse-engineer what information can be encoded by the model (e.g., Marvin and Linzen, 2018; Ravichander et al., 2020; Wu et al., 2020).”