Social Biases in NLP Models as Barriers for Persons with Disabilities

By Ben Hutchinson, Vinodkumar Prabhakaran, Emily L. Denton, Kellie Webster, Yu Zhong, Stephen Denuyl. 2020

378 citations recorded by the source corpus.

Building equitable and inclusive NLP technologies demands consideration of whether and how social attitudes are represented in ML models. In particular, representations encoded in models often inadvertently perpetuate undesirable social biases from the data on which they are trained. In this paper, we present evidence of such undesirable biases towards mentions of disability in two different English language models: toxicity prediction and sentiment analysis. Next, we demonstrate that the neural embeddings that are the critical first step in most NLP pipelines similarly contain undesirable biases towards mentions of disability. We end by highlighting topical biases in the discourse about disability which may contribute to the observed model biases; for instance, gun violence, homelessness, and drug addiction are over-represented in texts discussing mental illness.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, background): “Similarly, while our analyses focus on the biases in data and models, the downstream harms they cause will depend on how the models are used in practice (Blodgett et al., 2020; Hutchinson et al., 2020).”
Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, neutral): “…et al. (2019); Sun et al. (2019); Tan and Celis (2019); Webster et al. (2019); Zmigrod et al. (2019); Gyamfi et al. (2020); Hube et al. (2020); Hutchinson et al. (2020); Kim et al. (2020); Nadeem et al. (2020); Papakyriakopoulos et al. (2020); Ravfogel et al. (2020); Rozado (2020); Sen and…”
LaMDA: Language Models for Dialog Applications (2022, background): “, [58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72]), and inappropriately revealing personally identifying information (PII) from training data [73].”
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, background): “…has found that systems are overestimating the prevalence of toxicity in text that contains a minority identity mention (e.g., “I’m a gay man”; Dixon et al., 2018; Hutchinson et al., 2020) or text by racial minorities (e.g., text in African American English; Sap et al., 2019; Davidson et al., 2019).”
AI and the Everything in the Whole Wide World Benchmark (2021, methodology): “• System output analysis / behavioural testing (see Appendix A.2), including error analysis [e.g. van Miltenburg et al., 2021], disaggregated analysis [e.g. Raji and Buolamwini, 2019], and counterfactual analysis [e.g. Garg et al., 2019, Hutchinson et al., 2020].”
Re-imagining Algorithmic Fairness in India and Beyond (2021, background): “While other axes of discrimination and injustices such as disability status [91], age [59], and sexual orientation [76] have gotten some attention, biases relevant to other geographies and cultures are not explored (e.”
The Capacity for Moral Self-Correction in Large Language Models (2023, background): “Large language models exhibit harmful social biases [1, 6, 8, 11, 15, 24, 29, 50, 62] that can sometimes get worse for larger models [2, 18, 20, 43, 55].”
Re-contextualizing Fairness in NLP: The Case of India (2022, methodology): “…2016; Sun et al., 2019; Zhao et al., 2017) and racial biases (Sap et al., 2019; Davidson et al., 2019; Manzini et al., 2019), other axes of disparities such as ability (Hutchinson et al., 2020), age (Diaz et al., 2018), and sexual orientation (Garg et al., 2019) have also gotten some attention.”
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, background): “Large language models exhibit a wide range of harmful behaviors such as reinforcing social biases (e.g., [47, 28, 1, 33, 7]), generating offensive or toxic outputs [25], leaking personally identiﬁable information from the training data [13], aiding in disinformation campaigns [12], generating…”
GeDi: Generative Discriminator Guided Sequence Generation (2020, background): “Additionally, existing toxicity classifiers are often biased in that they overestimate the toxicity of text that mentions sexual orientations or racial minorities (Dixon et al., 2018; Sap et al., 2019; Hutchinson et al., 2020).”
Spacerini: Plug-and-play Search Engines with Pyserini and Hugging Face (2023, background): “This is especially problematic in light of the potential real-world harms that ensue [8, 9, 13, 31].”
Data and its (dis)contents: A survey of dataset development and use in machine learning research (2020, background): “…language processing datasets frequently reﬂect social biases and stereotypes relating to race, gender, (dis)ability, and more [Garg et al., 2018, Hutchinson et al., 2020] and correlations between gender and activities depicted in computer vision datasets have been shown to reﬂect common gender…”
Datasets: A Community Library for Natural Language Processing (2021, background): “…behavior of trained models in unexpected ways (Gururangan et al., 2018), 7 issues in the way the standard split was initially devised to test a model’s ability to adapt to new settings (Kr-ishna et al., 2021), or new understanding of the social biases exhibited therein (Hutchinson et al., 2020).”
Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure (2020, background): “udies have uncovered widespread prevalence of undesirable biases in machine learning datasets, such as the under-representation of minoritized groups [27, 40, 130] and stereotype aligned correlations [28, 51, 72, 153]. Datasets also frequently reflect historical patterns of social injustices, which ”
Bringing the People Back In: Contesting Benchmark Machine Learning Datasets (2020, background): “…systems abound in ways that they have failed people of color (Noble, 2018; Benjamin, 2019), women (Bolukbasi et al., 2016), LGBTQ+ communities (Scheuerman et al., 2019), people with disabilities (Hutchinson et al., 2020; Trewin, 2018), and the working class and those in poverty (Eubanks, 2018).”
DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts (2021, background): “…of large-scale evaluation required for systematic comparison of meth-ods, an abundance of work shows that their accuracy is far from ideal (Dixon et al., 2018; Sap et al., 2019; Davidson et al., 2019; Hutchinson et al., 2020) in part due to reliance on spurious features, which we discuss in §8.”
Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022, background): ““Clean” text additionally has been misrepresented as being “unbiased” against any community—a notion that has been strongly contested [22, 73, 130].”