Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals

By Yanai Elazar, Shauli Ravfogel, Alon Jacovi, Yoav Goldberg. 2020

281 citations recorded by the source corpus.

Abstract A growing body of work makes use of probing in order to investigate the working of neural models, often considered black boxes. Recently, an ongoing debate emerged surrounding the limitations of the probing paradigm. In this work, we point out the inability to infer behavioral conclusions from probing results, and offer an alternative method that focuses on how the information is being used, rather than on what information is encoded. Our method, Amnesic Probing, follows the intuition that the utility of a property for a given task can be assessed by measuring the influence of a causal intervention that removes it from the representation. Equipped with this new analysis tool, we can ask questions that were not possible before, for example, is part-of-speech information important for word prediction? We perform a series of analyses on BERT to answer these types of questions. Our findings demonstrate that conventional probing performance is not correlated to task importance, and we call for increased scrutiny of claims that draw behavioral or causal conclusions from probing results.1

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Causal Inference & Discovery

Citations in the corpus

Locating and Editing Factual Associations in GPT (2022, background): “Some research has been done for masked models (Petroni et al., 2019; Jiang et al., 2020; Elazar et al., 2021a; Geva et al., 2021; Dai et al., 2022; De Cao et al., 2021), but GPT has architectural differences such as unidirectional attention and generation capabilities that provide an opportunity…”
Editing Factual Knowledge in Language Models (2021, background): “certain behaviour has been studied in the context of research in interpreting neural networks (Lakretz et al., 2019; Vig et al., 2020; Elazar et al., 2021; Csordás et al., 2021).”
Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “Figure 1 Components constituting the probing classifiers framework. stable form by several groups who studied sentence embeddings (Ettinger, Elgohary, and Resnik 2016; Adi et al. 2017; Conneau et al. 2018) and recurrent/recursive neural networks (Belinkov et al. 2017a; Hupkes, Veldhoen, and Zuidema ”
Measuring and Improving Consistency in Pretrained Language Models (2021, background): “…in analyzing how well PLMs (Rogers et al., 2020) perform on linguistic tasks (Goldberg, 2019; Hewitt and Man-ning, 2019; Tenney et al., 2019; Elazar et al., 2021), commonsense (Forbes et al., 2019; Da and Kasai, 2019; Zhang et al., 2020) and reasoning (Talmor et al., 2020; Kassner et al.,…”
Measuring Causal Effects of Data Statistics on Language Model's 'Factual' Predictions (2022, methodology): “…of a given human-interpretable concept of a pretrained model, and relating the intervention to the change in the model’s behavior (Giulianelli et al., 2018; Lakretz et al., 2019; Bau et al., 2019; Elazar et al., 2021b; Feder et al., 2021b; Ravfogel et al., 2021; Antverg and Be-linkov, 2022).”
Locating and Editing Factual Knowledge in GPT (2022, methodology): “Research has been done for masked models (Petroni et al., 2019; Jiang et al., 2020; Elazar et al., 2021a; Geva et al., 2021; Dai et al., 2021; De Cao et al., 2021), but GPT’s architectural differences (e.g., unidirectional attention, generation capabilities) provide an opportunity for new insights.”
What’s in Your Head? Emergent Behaviour in Multi-Task Transformer Models (2021, methodology): “Moreover, we perform interventions (Woodward, 2005; Elazar et al., 2021), where we modify the representation based on the output of the extractive head, and show this leads to predictable changes in the behaviour of the generative head.”
Data Efficient Masked Language Modeling for Vision and Language (2021, unknown): “We follow the approach of amnesic probing (Elazar et al., 2021).”
Mechanistic Interpretability for AI Safety - A Review (2024, background): “These techniques, including causal tracing (Meng et al., 2022a), interchange intervention (Geiger et al., 2021b), causal mediation analysis (Vig et al., 2020), and causal ablation (Wang et al., 2023), share the common goal of modifying a neural model’s internal state by replacing specific…”
Probing Classifiers: Promises, Shortcomings, and Alternatives (2021, background): “Similarly, Elazar et al. (2021) remove certain properties z (such as parts of speech, syntactic dependencies) from representations in f by repeatedly training (linear) probing classiﬁers g and projecting them out of the representation.”
Eliciting Latent Predictions from Transformers with the Tuned Lens (2023, background): “Amnesic probing (Elazar et al., 2021) provides one way to do this—it seeks a direction whose erasure maximally degrades a model’s accuracy.”
What if This Modified That? Syntactic Interventions with Counterfactual Embeddings (2021, background): “Our work is most closely related to that of Elazar et al. (2020), who, as in this work, used probes to generate counterfactual embeddings within a network.”
Post-hoc Concept Bottleneck Models (2022, background): “Recent works evaluated the causal validity of explanations (Feder et al., 2021; Elazar et al., 2021; Goyal et al., 2019), e.”
Finding Neurons in a Haystack: Case Studies with Sparse Probing (2023, background): “While probing in general has a number of limitations [14, 28, 15, 29, 30], our work seeks to address, at least partially, the shortcomings most associated with probing individual neurons [13].”
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2023, background): “This relationship has been explored in machine learning and NLP for commonsense reasoning (Kıcıman et al., 2023), interpretability (Elazar et al., 2021; Geiger et al., 2021, 2022), spurious correlation detection (Veitch et al., 2021; Eisenstein, 2022), fairness (Kusner et al., 2017; Nabi and…”
Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond (2021, background): “To overcome the counterfactual generation problem, another class of approaches proposes to manipulate the representation of the text and not the text itself (Feder et al., 2021; Elazar et al., 2021; Ravfogel et al., 2021).”
Investigating Gender Bias in BERT (2020, background): “Amnesic probing [19] observes the change in the prediction of the classifier caused by the removal of property Z from the representation of the input.”
Causal Abstractions of Neural Networks (2021, background): “Probes are fundamentally unable to directly measure causal properties of neural representations, and Ravichander et al. [21], Elazar et al. [9], and Geiger et al. [11] have argued that probes are limited in their ability to provide even indirect evidence of causal properties.”