Multi-Dimensional Gender Bias Classification

By Emily Dinan, Angela Fan, Ledell Yu Wu, Jason Aaron Edward Weston, Douwe Kiela, Adina Williams. 2020

140 citations recorded by the source corpus.

Machine learning models are trained to find patterns in data. NLP models can inadvertently learn socially undesirable patterns when training on gender biased text. In this work, we propose a general framework that decomposes gender bias in text along several pragmatic and semantic dimensions: bias from the gender of the person being spoken about, bias from the gender of the person being spoken to, and bias from the gender of the speaker. Using this fine-grained framework, we automatically annotate eight large scale datasets with gender information. In addition, we collect a novel, crowdsourced evaluation benchmark of utterance-level gender rewrites. Distinguishing between gender bias along multiple dimensions is important, as it enables us to train finer-grained gender bias classifiers. We show our classifiers prove valuable for a variety of important applications, such as controlling for gender bias in generative models, detecting gender bias in arbitrary text, and shed light on offensive language in terms of genderedness.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, background): “Additionally, measures of bias could be multi-dimensional (e.g., Dinan et al., 2020), include explanations (e.g., Sap et al., 2020), or be evolving over time (e.g., using similarity to toxic online content).”
Societal Biases in Language Generation: Progress and Challenges (2021, methodology): “…2019) There are also metrics for other bias evaluation setups in continuation generation tasks involving sentiment (Shwartz et al., 2020), the ratio of gendered words (Solaiman et al., 2019; Vig et al., 2020; Dinan et al., 2020a), and other novel metrics (Peng et al., 2020; Yeo and Chen, 2020).”
CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models (2020, background): “One line of work explores evaluation grounded to specific downstream tasks, such as coreference resolution (Rudinger et al., 2018; Webster et al., 2018; Dinan et al., 2020) and relation extraction (Gaut et al., 2019).”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, methodology): “In particular, many works focus on generative models (Dinan et al., 2020a,b; Xu et al., 2021b; Kirk et al., 2021; Sheng et al., 2021b; Nozza et al., 2021; Renduchintala et al., 2021; Baheti et al., 2021; Perez et al., 2022), which are well known to pose unique challenges for automatic evaluation…”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, methodology): “As in §4, we use the multi-dimensional gender bias classifier from Dinan et al. (2020b) to measure the amount of gender bias in conversation turns from Speaker A and B for 920,000 self-chats generated by each of our de-biased models.”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “The BST tasks are based off of the ConvAI2 (Dinan et al., 2020b), Wizard of Wikipedia (Zhang et al., 2018; Dinan et al., 2020b), and Empathetic dialogs (Rashkin et al., 2019) tasks.”
Recipes for Safety in Open-domain Chatbots (2020, background): “Gender bias can also be connected to toxic language, in that offensive utterances about a female are more likely to contain gendered or swear words than about a male (Dinan et al., 2020).”
Queens Are Powerful Too: Mitigating Gender Bias in Dialogue Generation (2019, background): “Note that other functions could be used as well, such as a bias classifier (Dinan et al., 2020).”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “Finally, we add the Funpedia task (Dinan et al., 2020b) – which involves learning to produce an engaging dialogue utterance given a wikipedia sentence – and the LIGHT (Urbanek et al., 2019) and LIGHT WILD (Shuster et al., 2021b) tasks – which are open-domain dialogue tasks grounded in a medieval…”
Mitigating Gender Bias for Neural Dialogue Generation with Adversarial Learning (2020, background): “The gender bias2 in dialogues comes from different dimensions – the gender of the person that speakers are talking about (speaking-about), and the gender of the speaker (speaking-as) and the addressee (speaking-to) (Dinan et al., 2020).”
Investigating Failures of Automatic Translation in the Case of Unambiguous Gender (2021, neutral): “…al., 2016; Hovy and Spruit, 2016; Caliskan et al., 2017; Rudinger et al., 2017; Garg et al., 2018; Garimella et al., 2019; Gonen and Goldberg, 2019; Dinan et al., 2020a,b), we further observe in our results yet a higher order type of stereotyping that negatively affects women, namely androcentrism…”
RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models (2021, background): “…of work that focuses on detecting and mitigating biases in conversational systems is surprisingly limited (Lee et al., 2019; Liu et al., 2020a,b; Dinan et al., 2020a,b), albeit some more research has recently emerged in the wider context of biases in general-purpose language generation models…”
Multi-Modal Open-Domain Dialogue (2020, methodology): “For dialogue-based datasets, we consider the same four as in Roller et al. (2020): ConvAI2 (Dinan et al., 2020b), EmpatheticDia-logues (Rashkin et al., 2019), Wizard of Wikipedia (Dinan et al., 2019c), and BlendedSkillTalk (Smith et al., 2020).”
Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models (2023, methodology): “The persistence of social bias and stereotypes in large language models (LLMs) is well-documented (Dinan et al., 2020; Weidinger et al., 2021, inter alia).”
The Importance of Modeling Social Factors of Language: Theory and Practice (2021, background): “In-corporating user networks into geolocation substantially improves performance (Rahimi et al., 2018; Fornaciari and Hovy, 2019) and Dinan et al. (2020) show that the different roles of speaking-as, speaking-to, and speaking-about affect gender bias in NLP models.”
Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking (2021, background): “, replacing “he” with “they” following [14, 13]) and (ii) by substituting names with others that are statistically predictive [70] of another race or ethnicity (e.”
Towards Ethics by Design in Online Abusive Content Detection (2020, background): “Dinan et al. (2020) decomposed gender bias in text along several pragmatic and semantic dimensions and proposed classiﬁers for controlling gender bias.”
Perturbation Augmentation for Fairer NLP (2022, methodology): “…used to measure and often lessen the severity of social bias in text data (Hall Maud-slay et al., 2019; Prabhakaran et al., 2019; Zmigrod et al., 2019; Dinan et al., 2020a,b; Webster et al., 2020; Ma et al., 2021; Smith and Williams, 2021; Renduch-intala and Williams, 2022; Emmery et al., 2022).”