Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation

By Shahar Levy, Koren Lazar, Gabriel Stanovsky. 2021

86 citations recorded by the source corpus.

Recent works have found evidence of gender bias in models of machine translation and coreference resolution using mostly synthetic diagnostic datasets. While these quantify bias in a controlled experiment, they often do so on a small scale and consist mostly of artificial, out-of-distribution sentences. In this work, we find grammatical patterns indicating stereotypical and non-stereotypical gender-role assignments (e.g., female nurses versus male dancers) in corpora from three domains, resulting in a first large-scale gender bias dataset of 108K diverse real-world English sentences. We manually verify the quality of our corpus and use it to evaluate gender bias in various coreference resolution and machine translation models. We find that all tested models tend to over-rely on gender stereotypes when presented with natural inputs, which may be especially harmful when deployed in commercial systems. Finally, we show that our dataset lends itself to finetuning a coreference resolution model, finding it mitigates bias on a held out set. Our dataset and models are publicly available at www.github.com/SLAB-NLP/BUG. We hope they will spur future research into gender bias evaluation mitigation techniques in realistic settings.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “, 2021), (2) improving coreference to mitigate gender bias in translation (Levy et al., 2021), and (3) using text entailment to develop loss functions to improve fidelity (Falke et al.”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, methodology): “…biases learned from data, previous works have used finetuning on balanced datasets to reduce the gender bias of language models with some success (Levy et al., 2021; Gira et al., 2022; Kirtane et al., 2022), yet little is known about the role of specific corpus statistics in the emergence of…”
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings (2022, methodology): “The majority of previous work on bias has focused par-694 ticularly on gender bias (Zhao et al., 2018; Stanovsky 695 et al., 2019; Levy et al., 2021; Sharma et al., 2021) and the very few works (Nadeem et al., 2020; Nangia et al., 2020) that take other dimensions of biases into account, have their…”
The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks (2022, background): “Lastly, we also encourage discussions on the complexity of the sentences used in benchmarks and their implications on what gets measured in relation to un-templated, naturally-occurring text (Levy et al., 2021), as an attempt to ground our measurements in experienced harms.”
Bias and Fairness in Large Language Models: A Survey (2023, background): “BUG (Levy et al., 2021) provides more syntactically diverse coreference templates, containing 108,419 sentences to measure stereotypical gender role assignments.”