Probing Classifiers: Promises, Shortcomings, and Advances

By Yonatan Belinkov. 2021

901 citations recorded by the source corpus.

Probing classifiers have emerged as one of the prominent methodologies for interpreting and analyzing deep neural network models of natural language processing. The basic idea is simple—a classifier is trained to predict some linguistic property from a model’s representations—and has been used to examine a wide variety of models and properties. However, recent studies have demonstrated various methodological limitations of this approach. This squib critically reviews the probing classifiers framework, highlighting their promises, shortcomings, and advances.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “, 2022), although it comes with certain shortcomings (Belinkov, 2022).”
Locating and Editing Factual Associations in GPT (2022, background): “However, such approaches suffer from various limitations, notably being dissociated from the network’s behavior (Belinkov, 2021).”
Universal and Independent: Multilingual Probing Framework for Exhaustive Model Interpretation and Evaluation (2022, methodology): “The proposed framework allows for different probing methods to be used similarly, including the widely criticized ones (Belinkov, 2022).”
Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 (2024, neutral): “However, unlike previous interpretability techniques like steering vectors (Turner et al., 2024; Li et al., 2023) or probing (Belinkov, 2022), sparse autoencoders can be highly expensive and difficult to train, limiting the ambition of interpretability research.”
The Linear Representation Hypothesis and the Geometry of Large Language Models (2023, background): “…work on linear representations for interpreting (probing) (e.g., Alain & Ben-gio, 2017; Kim et al., 2018; nostalgebraist, 2020; Rogers et al., 2021; Belinkov, 2022; Li et al., 2022; Geva et al., 2022; Nanda et al., 2023) and controlling (steering) (e.g., Wang et al., 2023; Turner et al., 2023;…”
Mechanistic Interpretability for AI Safety - A Review (2024, methodology): “As highlighted by Belinkov (2021), this technique faces a notable challenge: the probe’s performance may often reflect its own learning capacities more than the actual characteristics of the model’s representations.”
Large Language Models (2023, methodology): “If this works, one can go on to modify the internal representation in a minimal way which changes the probe prediction, and check if this leads to the corresponding effects on the output (see [12] and references there).”
Locating and Editing Factual Knowledge in GPT (2022, background): “Approaches that seek to identify correlations between network representations and external information, such as probing classiﬁers, are often dissociated from the network’s behavior (Belinkov, 2021).”
Explainability for Large Language Models: A Survey (2023, methodology): “Although high probing performance is often attributed to the quality and interpretability of representations [Belinkov 2022], this assumption remains largely unproven and difficult to validate.”
Language Models Represent Space and Time (2023, methodology): “Probing To find evidence of spatial and temporal representations in LLMs, we use the standard technique of probing Alain & Bengio (2016); Belinkov (2022), which fits a simple model on the network activations to predict some target label associated with labeled input data.”
Eliciting Latent Predictions from Transformers with the Tuned Lens (2023, background): “This is especially important for probing techniques, since it’s known that probes can learn to rely on spurious features unrelated to the model’s performance (Hewitt and Liang, 2019; Belinkov, 2022).”
Is neural language acquisition similar to natural? A chronological probing study (2022, background): “A huge body of probing studies rely on linear models (e.g., external classiﬁers (Belinkov, 2016)) that try to establish the relationship between internal representations from the language model and the desired linguistic phenomena.”
Post-hoc Interpretability for Neural NLP: A Survey (2021, background): “If this concern is valid, it would mean that the probing methods do not provide functionally-grounded explanations [13].”
LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (2024, methodology): “This is akin to the linear probing setup commonly used in the language model analysis literature (Belinkov, 2022).”
Emergent Linear Representations in World Models of Self-Supervised Sequence Models (2023, background): “Probing has produced both excitement and skepticism amongst researchers (Belinkov, 2022).”
Inference-Time Intervention: Eliciting Truthful Answers from a Language Model (2023, methodology): “Where in the network is factuality represented? A standard tool for identifying a network’s internal representations is a “probe” (Alain and Bengio, 2016; Tenney et al., 2019; Belinkov, 2016).”
Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (2024, neutral): “On the other hand, prior methods for analyzing behaviors in terms of fine-grained units (Kim et al., 2018; Belinkov, 2022; Geiger et al., 2023; Zou et al., 2023) attempt to fit model internals to researcher-specified mechanistic hypotheses using researcher-curated data.”
Finding Neurons in a Haystack: Case Studies with Sparse Probing (2023, methodology): “Probing is a technique to localize where specific information resides in a model by training a simple classifier (a probe) to predict a labeled feature of the input using the internal activations of the model [3].”