Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias

By Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, Stuart M. Shieber. 2020

211 citations recorded by the source corpus.

Common methods for interpreting neural models in natural language processing typically examine either their structure or their behavior, but not both. We propose a methodology grounded in the theory of causal mediation analysis for interpreting which parts of a model are causally implicated in its behavior. It enables us to analyze the mechanisms by which information flows from input to output through various model components, known as mediators. We apply this methodology to analyze gender bias in pre-trained Transformer language models. We study the role of individual neurons and attention heads in mediating gender bias across three datasets designed to gauge a model's sensitivity to gender bias. Our mediation analysis reveals that gender bias effects are (i) sparse, concentrated in a small part of the network; (ii) synergistic, amplified or repressed by different components; and (iii) decomposable into effects flowing directly from the input and indirectly through the mediators.

Subfields: Artificial Intelligence, Natural Language Processing, Causal Inference & Discovery, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Causal Inference & Discovery

In AI Ethics, Fairness & Society

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, neutral): “…(2019); Prates et al. (2019); Prost et al. (2019); Qian et al. (2019); Swinger et al. (2019); Zhao et al. (2019); Zhou et al. (2019); Ethayarajh (2020); Huang et al. (2020); Jia et al. (2020); Popović et al. (2020); Pryzant et al. (2020); Vig et al. (2020); Wang et al. (2020); Zhao et al. (2020).”
Locating and Editing Factual Associations in GPT (2022, methodology): “As Vig et al. (2020b) have shown, this is a natural case for causal mediation analysis , which quantiﬁes the contribution of intermediate variables in causal graphs (Pearl, 2001).”
Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, methodology): “Unlike Vig et al. (2020a), we do not observe larger TEs for larger models.”
Editing Factual Knowledge in Language Models (2021, background): “As we discussed, the hyper-network may be regarded as a probe providing insights about the mechanism used by the model to encode the knowledge (Vig et al., 2020).”
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals (2020, background): “, 2020) or in specific neurons (Vig et al., 2020), our intervention is done on the representation layers.”
CausaLM: Causal Model Explanation Through Counterfactual Language Models (2020, methodology): “Vig et al. (2020) also suggest using ideas from causality for DNN explanations, but focus on understanding how information flows through different model components, while we are interested in understanding the effect of textual concepts on classification decisions.”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “It does not attempt to establish a causal link between attention and model behavior [28, 84], nor to explain model predictions [35, 87].”
Towards Continual Knowledge Learning of Language Models (2021, background): “, 2017) require identifying important parameters used for previous tasks, exactly how and where the knowledge is stored in the parameters of an LM is currently extremely difficult to identify and localize (Vig et al., 2020; De Cao et al., 2021).”
A Survey on Gender Bias in Natural Language Processing (2021, methodology): “This measure has been widely applied in numerous research [Bartl et al. 2020; Vig et al. 2020].”
Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance? (2020, background): “prediction also present another promising direction for future work (Giulianelli et al., 2018; Bau et al., 2018; Vig et al., 2020).”
What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge (2019, background): “We therefore see efforts to combine behavioral testing with various other analysis methods (Belinkov and Glass, 2019) that aim to uncover correlations and causal patterns between internal model representations and discrete structures (Chrupała and Alishahi, 2019; Vig et al., 2020; Geiger et al., 202”
Double-Hard Debias: Tailoring Word Embeddings for Gender Bias Mitigation (2020, background): “Recently, Vig et al. (2020) proposed causal mediation analysis as a way to interpret and analyze gender bias in neural models.”
Towards Automated Circuit Discovery for Mechanistic Interpretability (2023, methodology): “Neural network behaviors are implemented by algorithms within the model’s computational graph, and prior work has identified subgraphs (circuits, following Wang et al. (2023)’s definition) that capture the majority of particular behaviors.”
Towards a Comprehensive Understanding and Accurate Evaluation of Societal Biases in Pre-Trained Transformers (2021, methodology): “Kurita et al. (2019) and Vig et al. (2020) devise a pronoun-ranking test for BERT by comparing relative likelihoods of target words.”
Theories of “Gender” in NLP Bias Research (2022, background): “[151] Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber.”