Measuring and Mitigating Unintended Bias in Text Classification

By Lucas Dixon, John Li, Jeffrey Scott Sorensen, Nithum Thain, Lucy Vasserman. 2018

916 citations recorded by the source corpus.

We introduce and illustrate a new approach to measuring and mitigating unintended bias in machine learning models. Our definition of unintended bias is parameterized by a test set and a subset of input features. We illustrate how this can be used to evaluate text classifiers using a synthetic test set and a public corpus of comments annotated for toxicity from Wikipedia Talk pages. We also demonstrate how imbalances in training data can lead to unintended bias in the resulting models, and therefore potentially unfair applications. We use a set of common demographic identity terms as the subset of input features on which we measure bias. This technique permits analysis in the common scenario where demographic information on authors and readers is unavailable, so that bias mitigation must focus on the content of the text itself. The mitigation method we introduce is an unsupervised approach based on balancing the training dataset. We demonstrate that this approach reduces the unintended bias without compromising overall model quality.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, background): “We note that we are subject to the social bias of the Perspective API, which can assign higher toxicity to innocuous mentions of certain identity groups (Dixon et al., 2018; Röttger et al., 2021).”
Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “A.2 Full categorization: Motivations Allocational harms Hovy and Spruit (2016); Caliskan et al. (2017); Madnani et al. (2017); Dixon et al. (2018); Kiritchenko and Mohammad (2018); Shen et al. (2018); Zhao et al. (2018b); Bhaskaran and Bhallamudi (2019); Bordia and Bowman (2019); Brunet et al.…”
Reducing Sentiment Bias in Language Models via Counterfactual Evaluation (2019, background): “, 2018), or in text classification (Dixon et al., 2018; Garg et al., 2019).”
Model Cards for Model Reporting (2018, methodology): “A model card could even extend beyond these summary statistics to reveal other measures of differences between distributions such as cross entropy, perplexity, KL divergence and pinned area under the curve (pinned AUC) [13].”
Challenges in Detoxifying Language Models (2021, background): “Confirming previously identified identity-related biases in toxicity classifiers (Dixon et al., 2018), we observe that identity term mentions are disproportionately frequent among samples flagged as toxic by PERSPECTIVE API.”
Detoxifying Language Models Risks Marginalizing Minority Voices (2021, methodology): “Unfortunately, there are spurious correlations between the toxic label and the presence of AAE and minority identity mentions (Sap et al., 2019; Dixon et al., 2018).”
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, methodology): “…and identity mentions are known to co-occur with hateful and toxic language (specifically, minority identity mentions; Park and Fung, 2017; Dixon et al., 2018).20 We extract frequency of swearwords from a widely used list of profanity, and use the minority-majority word pairs from Sheng…”
HateCheck: Functional Tests for Hate Speech Detection Models (2020, methodology): “F18 – neutral statements using protected group identiﬁers : Oversensitiveness of hate speech detection models to terms such as “black” and “gay” (Dixon et al., 2018; Park et al., 2018; Kennedy et al., 2020).”
Social Biases in NLP Models as Barriers for Persons with Disabilities (2020, background): “…other historically marginalized groups (Bolukbasi et al., 2016; Caliskan et al., 2017; Garg et al., 2017; Barocas et al., 2017; Garg et al., 2019; Dixon et al., 2018; Noble, 2018; Manzini et al., 2019; Sap et al., 2019; May et al., 2019; Speer, 2017), bias with respect to different disability…”
The Risk of Racial Bias in Hate Speech Detection (2019, background): “Social media platforms are under increasing pressure to respond (Trindade, 2018), but automated removal of such content risks further suppressing alreadymarginalized voices (Yasin, 2018; Dixon et al., 2018).”
OpenAssistant Conversations - Democratizing Large Language Model Alignment (2023, background): “Additionally, these models tend to incorporate biases present in their training data, leading to unfair and discriminatory outputs [25].”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “Existing datasets are often biased due to their curation process (Dixon et al., 2018; Wiegand et al., 2019; Geva et al., 2019; Sap et al., 2021; Kryscinski et al., 2020) and can have unreliable annotations (Ross et al., 2017; Field and Tsvetkov, 2020b; Pagnoni et al., 2021).”
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, background): “Some of these efforts augment humans (through guidelines, templates, programmatic generation of attacks, and various combinations thereof) to devise test cases that cause systems to fail [45, 46, 29, 21, 30, 55, 6, 23].”
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (2022, background): “As such, minority mentions often co-occur with toxicity labels in datasets scraped from online platforms (Dixon et al., 2018).”
Understanding Dataset Difficulty with V-Usable Information (2021, positive): “Our findings corroborate prior work which shows that certain lexical items (e.g., swear words, identity mentions) are responsible for hate speech prediction (Dixon et al., 2018; Dinan et al., 2019).”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, background): “…standard training datasets for dialogue models in the AI literature contain social biases or demographic imbalances, and it has been established that models learn them (Dixon et al., 2018; Bordia and Bowman, 2019; Lee et al., 2019; Dinan et al., 2020a,b; Liu et al., 2020a,b; Sheng et al., 2021).”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “These biases can result in toxicity being associated with certain words, such as profanities or identity terms (Dinan et al., 2019b; Dixon et al., 2018), or language varieties, such as African American English (AAE) (Liu et al., 2019; Sap et al., 2019).”
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “However, these classifiers themselves can have issues with bias, e.g., by learning undesirable correlations that tie toxicity to identity terms (Dixon et al., 2018; Nozza et al., 2021, 2022), or language varieties, such as African American English (Liu et al., 2019; Sap et al., 2019).”