Gender Bias in Coreference Resolution

By Rachel Rudinger, Jason Naradowsky, Brian Leonard, Benjamin Van Durme. 2018

750 citations recorded by the source corpus.

We present an empirical study of gender bias in coreference resolution systems. We first introduce a novel, Winograd schema-style set of minimal pair sentences that differ only by pronoun gender. With these “Winogender schemas,” we evaluate and confirm systematic gender bias in three publicly-available coreference resolution systems, and correlate this bias with real-world and textual gender statistics.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Language Models are Few-Shot Learners (2020, methodology): “We also carried out pronoun resolution on the Winogender dataset [RNLVD18] using two methods which further corroborated the model’s tendency to associate most occupations with males.”
Training Compute-Optimal Large Language Models (2022, neutral): “Here, we test if potential gender and occupation biases manifest in unfair outcomes on coreference resolutions, using the Winogender dataset (Rudinger et al., 2018) in a zero-shot setting.”
PaLM: Scaling Language Modeling with Pathways (2022, methodology): “…from BIG-bench (BIG-bench collaboration, 2021), reasoning tasks (Section 6.3), code completion tasks (Section 6.4), multilingual generation and question answering tasks (Section 6.6), translation tasks (Section 6.5), and bias and toxicity benchmarks (Rudinger et al., 2018; Gehman et al., 2020).”
LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “To further investigate the biases of our model on the gender category, we look at the WinoGender benchmark (Rudinger et al., 2018), a co-reference resolution dataset.”
Training language models to follow instructions with human feedback (2022, background): “InstructGPT does not significantly improve over GPT-3 on the Winogender (Rudinger et al., 2018) and CrowSPairs (Nangia et al., 2020) datasets.”
Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “…work on “bias” in systems developed for a breadth of tasks including language modeling (Lu et al., 2018; Bordia and Bowman, 2019), coreference resolution (Rudinger et al., 2018; Zhao et al., 2018a), machine translation (Vanmassenhove et al., 2018; Stanovsky et al., 2019), sentiment analysis…”
StereoSet: Measuring stereotypical bias in pretrained language models (2020, methodology): “Another popular method to evaluate bias of pre-trained representations is to measure bias on extrinsic applications like coreference resolution (Rudinger et al., 2018; Zhao et al., 2018) and sentiment analysis (Kiritchenko and Mohammad, 2018).”
Reducing Sentiment Bias in Language Models via Counterfactual Evaluation (2019, background): “Systematic imbalances were observed across NLP tasks, such as gender bias in coreference resolution (Zhao et al., 2018; Rudinger et al., 2018), visual semantic role labeling (Zhao et al., 2017), image captioning (Hendricks et al., 2018), and demographic biases in language generation (Sheng et al.,…”
The Woman Worked as a Babysitter: On Biases in Language Generation (2019, methodology): “We analyze these two bias contexts because the occupation context has been well-studied in other tasks (Bolukbasi et al., 2016; Rudinger et al., 2018; Zhao et al., 2018; Zhou et al., 2019), and the more descriptive language in respect contexts are a good contrast for the more subtle occupation…”
WinoGrande (2019, neutral): “Winogender (Rudinger et al. 2018) This dataset introduces 720 problems focusing on pronouns resolution with respect to people, with distinct goal of measuring gender bias in coreference resolution systems.”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, methodology): “Though all models obtain near perfect gender parity scores on Winogender, this is due to the fact that they are obtaining accuracy near that of random guessing.”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, methodology): “An important strand of this research focuses on unintended biases learnt by language models, including correlations between gender and profession (Bolukbasi et al., 2016; Rudinger et al., 2018; Zhao et al., 2018), negative sentiment about racial and religious groups (Li et al.”
Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies (2021, neutral): “There have been multiple attempts to understand social biases in language processing (Sheng et al., 2021; Caliskan et al., 2017), quantify them (Rudinger et al., 2018; Webster et al., 2018; De-Arteaga et al., 2019), and mitigate them (Zhao et al., 2019; Ravfo-gel et al., 2020; Sun et al., 2019).”
Toward Gender-Inclusive Coreference Resolution (2019, background): “2018] and [Rudinger et al. 2018] (as well as in contrast to ourMAP dataset), we focused on naturally occurring data, but sampled specifically to surface more gender-related phenomena than may be found in, say, the Wall Street Journal.”
On Measuring and Mitigating Biased Inferences of Word Embeddings (2019, methodology): “Coreference resolution, especially pronoun coreference, has been recently used as an extrinsic probe to measure bias in representations (e.g., (Rudinger et al. 2018), (Zhao et al. 2019), (Webster et al. 2018)).”
Evaluating Gender Bias in Machine Translation (2019, methodology): “As shown in Figure 1, we use data introduced by two recent coreference gender-bias studies: the Winogender (Rudinger et al., 2018), and the WinoBias (Zhao et al., 2018) datasets.”
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets (2021, background): “These datasets consist of tests that can take a variety of formats, including sentence templates where terms pertaining to groups or their attributes are perturbed (Rudinger et al., 2018; Zhao et al., 2018; Kurita et al., 2019), prompts designed to elicit problematic responses (Gehman et al.”
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “(2022), we also evaluate on the WinoGender Schemas (Rudinger et al., 2018) cast as a textual entailment task (Poliak et al.”
CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models (2020, background): “One line of work explores evaluation grounded to specific downstream tasks, such as coreference resolution (Rudinger et al., 2018; Webster et al., 2018; Dinan et al., 2020) and relation extraction (Gaut et al., 2019).”