Gender Bias in Neural Natural Language Processing

By Kaiji Lu, Piotr (Peter) Mardziel, Fangjing Wu, Preetam Amancharla, Anupam Datta. 2018

408 citations recorded by the source corpus.

We examine whether neural natural language processing (NLP) systems reflect historical biases in training data. We define a general benchmark to quantify gender bias in a variety of neural NLP tasks. Our empirical evaluation with state-of-the-art neural coreference resolution and textbook RNN-based language models trained on benchmark datasets finds significant gender bias in how models view occupations. We then mitigate bias with CDA: a generic methodology for corpus augmentation via causal interventions that breaks associations between gendered and gender-neutral words. We empirically show that CDA effectively decreases gender bias while preserving accuracy. We also explore the space of mitigation strategies with CDA, a prior approach to word embedding debiasing (WED), and their compositions. We show that CDA outperforms WED, drastically so when word embeddings are trained. For pre-trained embeddings, the two methods can be effectively composed. We also find that as training proceeds on the original data set with gradient descent the gender bias grows as the loss reduces, indicating that the optimization encourages bias; CDA mitigates this behavior.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “developed for a breadth of tasks including language modeling (Lu et al., 2018; Bordia and Bowman, 2019), coreference resolution (Rudinger et al.”
Reducing Sentiment Bias in Language Models via Counterfactual Evaluation (2019, methodology): “In this work, we focus on language models which have been shown to exhibit systematic biases (Lu et al., 2018; Bordia & Bowman, 2019; Qian et al., 2019).”
The Woman Worked as a Babysitter: On Biases in Language Generation (2019, methodology): “- - Regard Although sentiment has been used as a proxy for bias in other works (Lu et al., 2018; Kiritchenko and Mohammad, 2018), there has been little analysis on the correlation of sentiment to human judgment of bias.”
Reducing Gender Bias in Word-Level Language Models with a Gender-Equalizing Loss Function (2019, methodology): “Two prior works focused on reducing bias in language modelling by data preprocessing (Lu et al., 2018) and word embedding debiasing (Bordia and Bowman, 2019).”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, neutral): “Analyzing Gender Bias in Models Recent work has identiﬁed the presence and ampliﬁcation of many social biases in data-driven machine learning models (Lu et al., 2018; Zhao et al., 2018, i.a.).”
Societal Biases in Language Generation: Progress and Challenges (2021, methodology): “CDA has been applied to datasets used for continued or fresh training in dialogue generation (Dinan et al., 2020a; Liu et al., 2020a) as well as machine translation (Saunders and Byrne, 2020; Costa-juss`a and de Jorge, 2020; Stafanoviˇcs et al., 2020).”
Persistent Anti-Muslim Bias in Large Language Models (2021, background): “Prior work has shown that language models can be debiased by preprocessing training datasets [7] or modifying the model training algorithm [10].”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, neutral): “Finally, we compare and contrast the effectiveness of several mitigation strategies, including counterfactual data augmentation (Lu et al., 2018; Hall Maudslay et al., 2019) to scramble names, a novel application of unlikelihood training to bias mitigation (Welleck et al.”
BBQ: A hand-built bias benchmark for question answering (2021, background): “In coreference resolution, much of the work on biases has focused on gender bias that relates to specific gender stereotypes (Lu et al., 2020) or on measuring gender-occupation associations (Rudinger et al., 2018; Zhao et al., 2018).”
Alignment of Language Agents (2021, background): “Language agents may give harmful and biased outputs, producing discriminatory content relating to people’s protected characteristics and other sensitive attributes such as someone’s socio-economic status, see e.g. (Jentzsch et al., 2019; Lu et al., 2020; Zhao et al., 2017).”
Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models (2021, background): “We ideally want these models to be † Equal contribution. free from unwanted bias and whilst it is true that they exhibit less gender bias than static word embeddings (Sun et al., 2019), they are by no means immune to this problem (Lu et al., 2018).”
Does Gender Matter? Towards Fairness in Dialogue Systems (2019, methodology): “Following the fairness definition proposed in [18], we define the fairness in dialogue systems as follows: Definition 2 Suppose D is a dialogue model that can be viewed as a function {D : C 7→ R} which maps a context C to a response R.”
Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, methodology): “Data For neuron intervention experiments, we augment the list of templates from Lu et al. (2018) with several other templates, instantiated with pro-fessions from Bolukbasi et al. (2016).”
Learning the Difference that Makes a Difference with Counterfactually-Augmented Data (2019, positive): “Lu et al. (2018) programmatically alter text to invert gender bias and combined the original and manipulated data yielding gender-balanced dataset for learning word embeddings.”
Mitigating Gender Bias in Natural Language Processing: Literature Review (2019, background): “Gender-swapping can be generalized to sentences by swapping each male-deﬁnitional word with its respective female equivalent and vice-versa (Zhao et al., 2018a; Lu et al., 2018; Kiritchenko and Mo-hammad, 2018).”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, background): “Minimal pairs of ed-its that perturb gender or racial attributes are also useful for evaluating social biases (Rudinger et al., 2018; Zhao et al., 2018; Lu et al., 2018).”
It’s All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitution (2019, methodology): “As an alternative to WED, Lu et al. (2018) pro-pose Counterfactual Data Augmentation (CDA), in which a text transformation designed to invert bias is performed on a text corpus, the result of which is then appended to the original, to form a new bias-mitigated corpus used for training embed-dings.”
Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology (2019, methodology): “For example, Bolukbasi et al. (2016) proposed a way of mitigating gender stereotypes in word embeddings while preserving meanings; Lu et al. (2018) studied gender stereotypes in language models; and Rudinger et al. (2018) introduced a novel Winograd schema for evaluating gender stereotypes in…”
A Multiscale Visualization of Attention in the Transformer Model (2019, background): “This suggests that the model’s coreference mechanism may encode gender bias (Zhao et al., 2018; Lu et al., 2018).”