CausaLM: Causal Model Explanation Through Counterfactual Language Models

By Amir Feder, Nadav Oved, Uri Shalit, Roi Reichart. 2020

193 citations recorded by the source corpus.

Abstract Understanding predictions made by deep neural networks is notoriously difficult, but also crucial to their dissemination. As all machine learning–based methods, they are as good as their training data, and can also capture unwanted biases. While there are tools that can help understand whether such biases exist, they do not distinguish between correlation and causation, and might be ill-suited for text-based models and for reasoning about high-level language concepts. A key problem of estimating the causal effect of a concept of interest on a given model is that this estimation requires the generation of counterfactual examples, which is challenging with existing generation technology. To bridge that gap, we propose CausaLM, a framework for producing causal model explanations using counterfactual language representation models. Our approach is based on fine-tuning of deep contextualized embedding models with auxiliary adversarial tasks derived from the causal graph of the problem. Concretely, we show that by carefully choosing auxiliary adversarial pre-training tasks, language representation models such as BERT can effectively learn a counterfactual representation for a given concept of interest, and be used to estimate its true causal effect on model performance. A byproduct of our method is a language representation model that is unaffected by the tested concept, which can be useful in mitigating unwanted bias ingrained in the data.1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Safety, Robustness & Interpretability, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Safety, Robustness & Interpretability

In Causal Inference & Discovery

Citations in the corpus

Probing Classifiers: Promises, Shortcomings, and Advances (2021, methodology): “Giulianelli et al. (2018) use gradients from g to modify the representations in”
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals (2020, background): “Finally and concurrent to this work, Feder et al. (2020) have studied a similar question of a causal attribution of concepts to representations, using adversarial training guided by causal graphs.”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “For instance, recent work on counterfactual language representations by Feder et al. (2020) is a promising step in that direction.”
Probing Classifiers: Promises, Shortcomings, and Alternatives (2021, background): “In a similar vein, Feder et al. (2020) remove properties z from representations in f by training g adversarially, while continuing to train f .”
Explaining NLP Models via Minimal Contrastive Editing (MiCE) (2020, background): “Recent work within NLP derives causal explanations of models through counterfactual interventions (Feder et al., 2021; Vig et al., 2020).”
How to make causal inferences using texts (2018, unknown): “Following our posting of the preprint version of this article in 2018, there has been a burst of interest from the natural language processing community exploring various components of causal inference with text classifiers and language models ( 36 – 38 ).”
Mechanistic Interpretability for AI Safety - A Review (2024, methodology): “…network activations (Jenner et al., 2023), unifying various interpretability methods such as LIME (Ribeiro et al., 2016), causal effect estimation (Feder et al., 2021), causal mediation analysis (Vig et al., 2020), iterated nullspace projection (Ravfogel et al., 2020), and circuit-based…”
Towards Automated Circuit Discovery for Mechanistic Interpretability (2023, background): “Much prior research on understanding language models has drawn inspiration from causal inference (Pearl, 2009), leading to the development of frameworks that provide causal explanations for model outputs (Pearl, 2009; Feder et al., 2021; Geiger et al., 2021; Wu et al., 2021).”
Post-hoc Concept Bottleneck Models (2022, background): “Recent works evaluated the causal validity of explanations (Feder et al., 2021; Elazar et al., 2021; Goyal et al., 2019), e.”
Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond (2021, methodology): “Feder et al. (2021) compute the counterfactual representation by pre-training an additional instance of the language representation model employed by the classiﬁer, with an adversarial component designed to "forget" the concept of choice, while controlling for confounding concepts.”
Measuring Causal Effects of Data Statistics on Language Model's 'Factual' Predictions (2022, methodology): “…of a given human-interpretable concept of a pretrained model, and relating the intervention to the change in the model’s behavior (Giulianelli et al., 2018; Lakretz et al., 2019; Bau et al., 2019; Elazar et al., 2021b; Feder et al., 2021b; Ravfogel et al., 2021; Antverg and Be-linkov, 2022).”