Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview

By Deven Santosh Shah, H. A. Schwartz, Dirk Hovy. 2019

291 citations recorded by the source corpus.

An increasing number of natural language processing papers address the effect of bias on predictions, introducing mitigation techniques at different parts of the standard NLP pipeline (data and models). However, these works have been conducted individually, without a unifying framework to organize efforts within the field. This situation leads to repetitive approaches, and focuses overly on bias symptoms/effects, rather than on their origins, which could limit the development of effective countermeasures. In this paper, we propose a unifying predictive bias framework for NLP. We summarize the NLP literature and suggest general mathematical definitions of predictive bias. We differentiate two consequences of bias: outcome disparities and error disparities, as well as four potential origins of biases: label bias, selection bias, model overamplification, and semantic bias. Our framework serves as an overview of predictive bias in NLP, integrating existing work into a single structure, and providing a conceptual baseline for improved frameworks.

Subfields: Artificial Intelligence, Information Systems, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “lyzing “bias” in NLP systems often state multiple motivations (e.g., Hovy and Spruit, 2016; Bender, 2019; Sun et al., 2019; Rozado, 2020; Shah et al., 2020).”
Societal Biases in Language Generation: Progress and Challenges (2021, neutral): “C L ] 2 2 Ju n 20 21 on mitigating gender biases and Shah et al. (2020) categorize sources of biases—both largely focus on natural language understanding (NLU) tasks, while we examine biases in NLG tasks.”
HateCheck: Functional Tests for Hate Speech Detection Models (2020, methodology): “by learning simple decision rules rather than encoding a more generalisable understanding of the task (e.g. Niven and Kao, 2019; Geva et al., 2019; Shah et al., 2020).”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “(2021), can make matters worse by overamplifying existing data bias (Zhao et al., 2017; Shah et al., 2020).”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, background): “…but one kind of systematic gap; there are also biases due to the writing styles of small groups of annotators (Geva et al., 2019), the distributional biases in the data that was chosen for annotation, as well as numerous other biases that are more subtle and harder to discern (Shah et al., 2020).”
A Survey on Bias and Fairness in Natural Language Processing (2022, neutral): “They can be grouped into 4 major groups (Shah et al., 2019) :- • Label bias Label bias emerges when the distribution of the labeled variable in the input dataset diverges substantially from the ideal distribution.”
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, background): “Numerous works have surveyed the landscape of potential language model harms, both broadly [11, 102] and specific to social bias [52, 93, 95, 35, 61].”
A Survey of Race, Racism, and Anti-Racism in NLP (2021, background): “…2020 Workshop Social Science/Media Survey/Position Nangia et al. (2020) 2020 EMNLP Text Representations Detect Bias Roy and Goldwasser (2020) 2020 EMNLP Social Science/Media Analyze Corpus Sap et al. (2020) 2020 ACL Abusive Language Collect Corpus Shah et al. (2020) 2020 ACL Ethics/Task-indep.”
Disembodied Machine Learning: On the Illusion of Objectivity in NLP (2021, methodology): “Jackson, 2013), the data sampling method (Shah et al., 2020), the annotation guidelines (Sap et al.”
Competency Problems: On Finding and Removing Artifacts in Language Data (2021, background): “Here, we ask whether typical NLP models learn to bias their predictions based on these artifacts for both the SNLI and RTE (Dagan et al., 2005) 9 datasets.”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, background): “Large neural models in general, and convAI models in particular, have been shown to replicate and even amplify negative, stereotypical, and derogatory associations in the data (Shah et al., 2020; Bender et al., 2021).”
RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models (2021, background): “For a comprehensive overview of work on bias in NLP, we refer the reader to (Sun et al., 2019; Blodgett et al., 2020; Shah et al., 2020).”
The Hitchhiker’s Guide to Computational Linguistics in Suicide Prevention (2021, background): “…the accuracy of CL-based predictions, but they can actually also cause harm to individuals, especially if these individuals already belong to a discriminated ethnical or sexual minority (for a comprehensive discussion of this issue, which is known in CL research as bias , see Shah et al., 2019).”
Neural Unsupervised Domain Adaptation in NLP—A Survey (2020, background): “Understanding bias sources and effects, besides effects only (Shah et al., 2020), and documenting the known are the first important steps (Bender and Friedman, 2018), as is building broader, more varied corpora (Ide and Suderman, 2004).”
Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots (2022, background): “These tools are not perfect and, in some cases, biased [34, 52].”
Large Language Model Alignment: A Survey (2023, background): “However, the biases (Shah et al., 2019), toxicity (Gehman et al.”
Benchmarking Intersectional Biases in NLP (2022, background): “Downstream tasks, such as sequence classification, also affect underrepresented groups, as these models show disparate impact on various demographic subsets, including women, African Americans, and the elderly (Blodgett et al., 2020; Bender et al., 2021; Shah et al., 2020).”