Five sources of bias in natural language processing

By Eduard H. Hovy, Shrimai Prabhumoye. 2021

348 citations recorded by the source corpus.

Abstract Recently, there has been an increased interest in demographically grounded bias in natural language processing (NLP) applications. Much of the recent work has focused on describing bias and providing an overview of bias in a larger context. Here, we provide a simple, actionable summary of this recent work. We outline five sources where bias can occur in NLP systems: (1) the data, (2) the annotation process, (3) the input representations, (4) the models, and finally (5) the research design (or how we conceptualize our research). We explore each of the bias sources in detail in this article, including examples and links to related work, as well as potential counter‐measures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Memory-assisted prompt editing to improve GPT-3 after deployment (2022, background): “Finally, large-language models generate text that might be biased and insensitive to a user’s socio-cultural context (Bordia and Bowman, 2019; Sharma et al., 2021; Hovy and Prabhumoye, 2021).”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “While a variety of past work has studied risks of language-only pretraining [125, 14, 8, 60], the video-centric pretraining that we explore in our work might have different benefits and risks.”
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings (2022, methodology): “Hovy and Prabhumoye (2021); Talat et al. (2021), and Cao and Daumé III (2020) argue that socially discriminatory biases can be encoded in several stages of the LLM development process (Biderman and Scheirer, 2020), including data sampling, annotation, selection of input representations or model,…”
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, background): “Numerous works have surveyed the landscape of potential language model harms, both broadly [11, 102] and speciﬁc to social bias [52, 93, 95, 35, 61].”
A Survey on Gender Bias in Natural Language Processing (2021, background): “Many concerns have been posed regarding modern NLP systems having been trained on potentially biased datasets, as as these biases can be perpetuated to downstream tasks and eventually society in the form of allocational harms [Hovy and Prabhumoye 2021].”
Pipelines for Social Bias Testing of Large Language Models (2022, background): “An overview of bias in NLP has been presented in several work (Blodgett et al., 2020; Shah et al., 2020; Hovy and Prabhumoye, 2021; Sheng et al., 2021; Stanczak and Augenstein, 2021).”
Biases in Large Language Models: Origins, Inventory, and Discussion (2023, background): “Ideally, when choosing a textual dataset to work with, one should also make decisions about the demographic groups represented in the data [64], and about how including, excluding, over-representing or under-representing a demographic group could afect language models.”
Undesirable biases in NLP: Averting a crisis of measurement (2022, background): “Researchers have identified many possible sources and explanations of model bias (see e.g., Sun et al., 2019; Hovy & Prabhumoye, 2021) and the black box nature of modern NLP systems makes it unlikely that a single (interpretable) representation exists where we could measure a “statistical bias”…”
Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases (2021, background): “The human labeling used to annotate these datasets can introduce bias and noisy labels (Hovy and Prabhumoye, 2021).”
Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (2022, background): “Hovy and Prabhumoye describe five sources of bias in NLP and potential ways to counteract it [170].”
Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models (2023, background): “For example, if an algorithm places more importance on certain features or data points, it may unintentionally introduce or amplify biases present in the data [30, 28, 31].”