Investigating Gender Bias in Language Models Using Causal Mediation Analysis

By Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, Stuart M. Shieber. 2020

725 citations recorded by the source corpus.

Subfields: Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Societal Biases in Language Generation: Progress and Challenges (2021, methodology): “…Transformer-based models, including GPT (Shwartz et al., 2020), GPT-2 (Solaiman et al., 2019; Sheng et al., 2019, 2020; Shwartz et al., 2020; Vig et al., 2020; Yeo and Chen, 2020; Huang et al., 2020; Dhamala et al., 2021; Schick et al., 2021), GPT-3 (Brown et al., 2020), CTRL (Dhamala et…”
An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models (2023, methodology): “Our metric is negatively correlated with CMA and SS metrics in auto-encoder models.”
Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Bias (2022, methodology): “We measure the occupational gender bias in 2https://www.bls.gov/cps/cpsaat11.htm 3This is consistent with previous findings (Nadeem et al., 2021; Vig et al., 2020). three models’ families, using two methods— prompt based method (Kurita et al., 2019) and Winogender schema (Rudinger et al., 2018).”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, background): “…measurement templates, which form the basis for prompts that can be used to measure bias (Rudinger et al., 2018; May et al., 2019; Sheng et al., 2019; Kurita et al., 2019; Webster et al., 2020; Gehman et al., 2020; Huang et al., 2020; Vig et al., 2020; Kirk et al., 2021; Perez et al., 2022).”
Locating and Editing Factual Associations in GPT (2022, methodology): “As Vig et al. (2020b) have shown, this is a natural case for causal mediation analysis , which quantiﬁes the contribution of intermediate variables in causal graphs (Pearl, 2001).”
Mass-Editing Memory in a Transformer (2022, methodology): “We then employ causal mediation analysis (Pearl, 2001; Vig et al., 2020; Meng et al., 2022), which implicates a specific range of layers in recalling factual knowledge.”
Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, methodology): “Unlike Vig et al. (2020a), we do not observe larger TEs for larger models.”
Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “Good probing performance is often taken to indicate several potential situations: 3 For chronological completeness, workshop and preprint versions of Hupkes, Veldhoen, and Zuidema (2018) and Adi et al. (2017) appeared earlier (Veldhoen, Hupkes, and Zuidema 2016; Adi et al. 2016).”
Transformer Feed-Forward Layers Are Key-Value Memories (2020, background): “…line of work targeted neuron functionality in general, extracting the properties that neurons and subsets of neurons capture (Durrani et al., 2020; Dalvi et al., 2019; Rethmeier et al., 2020; Mu and Andreas, 2020; Vig et al., 2020), regardless of the model architecture or neurons’ position in it.”
CausaLM: Causal Model Explanation Through Counterfactual Language Models (2020, methodology): “Vig et al. (2020) also suggest using ideas from causality for DNN explanations, but focus on understanding how information flows through different model components, while we are interested in understanding the effect of textual concepts on classification decisions.”
Attributing Fair Decisions with Attention Interventions (2021, methodology): “In addition, Vig et al. (2020) analyze the effect of the attention weights in transformer models for bias analysis in language models.”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “1 We note that all of the above analyses are purely associative and do not attempt to establish a causal link between attention and model behavior (Vig et al., 2020; Grimsley et al., 2020), nor to explain model predictions (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019).”
HONEST: Measuring Hurtful Sentence Completion in Language Models (2021, background): “…word embeddings (Chaloner and Maldon-ado, 2019; Zhou et al., 2019; Nissim et al., 2020) and more recently on pre-trained contextualized embeddings models (Kurita et al., 2019; May et al., 2019; Zhao et al., 2019; Field and Tsvetkov, 2019; Sheng et al., 2019; Nangia et al., 2020; Vig et al., 2020).”
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, background): “Most benchmarks focus on the output, but it is possible to take measurements of speciﬁc parts of the model which approximate harm, such as in Vig et al. [100] and as analyzed in K. et al. [61].”
Measuring Causal Effects of Data Statistics on Language Model's 'Factual' Predictions (2022, methodology): “Vig et al. (2020) have used causal mediation analysis to attribute gender bias to individual model components (neurons and attention heads), and in order to identify the source of that bias.”
Probing Classifiers: Promises, Shortcomings, and Alternatives (2021, methodology): “Vig et al. (2020) design interventions on inputs x and quantify the effect of intermediate variables in language models (in particular, neurons and attention heads) on gender-biased predictions, using causal mediation analysis (Pearl 2001).”
Explaining NLP Models via Minimal Contrastive Editing (MiCE) (2020, background): “Recent work within NLP derives causal explanations of models through counterfactual interventions (Feder et al., 2021; Vig et al., 2020).”
Improving Gender Fairness of Pre-Trained Language Models without Catastrophic Forgetting (2021, background): “Among various gender-biased behaviours of pre-trained models, bias on professions is the most prominent and wellstudied (de Vassimon Manela et al., 2021; Vig et al., 2020; Qian et al., 2019; Zhao et al., 2019).”