Language (Technology) is Power: A Critical Survey of “Bias” in NLP

By Su Lin Blodgett, Solon Barocas, Hal Daumé Iii, Hanna M. Wallach. 2020

1,755 citations recorded by the source corpus.

We survey 146 papers analyzing “bias” in NLP systems, finding that their motivations are often vague, inconsistent, and lacking in normative reasoning, despite the fact that analyzing “bias” is an inherently normative process. We further find that these papers’ proposed quantitative techniques for measuring or mitigating “bias” are poorly matched to their motivations and do not engage with the relevant literature outside of NLP. Based on these findings, we describe the beginnings of a path forward by proposing three recommendations that should guide work analyzing “bias” in NLP systems. These recommendations rest on a greater recognition of the relationships between language and social hierarchies, encouraging researchers and practitioners to articulate their conceptualizations of “bias”---i.e., what kinds of system behaviors are harmful, in what ways, to whom, and why, as well as the normative reasoning underlying these statements—and to center work around the lived experiences of members of communities affected by NLP systems, while interrogating and reimagining the power relations between technologists and such communities.

Subfields: Artificial Intelligence, Communication, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Communication

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, background): “…et al., 2021b), there is a lack of standardization of fairness benchmarks, an understanding of what harms different bias measures in NLP relate to (Blodgett et al., 2020, 2021; Jacobs & Wallach, 2021), and coverage of identities in fluid, comprehensive ways (Cao & Daumé III, 2020; Dev et al.,…”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, background): “, 2021), including bias in representations (Blodgett et al., 2020), proper handling (Rogers, 2021) and documentation (Bender & Friedman, 2018) of training data, privacy (Abadi et al.”
Detoxifying Language Models Risks Marginalizing Minority Voices (2021, background): “Our results demonstrate that the current state of detoxification poses representational harms (Blodgett et al., 2020) to minority groups.”
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, methodology): “Following Blodgett et al. (2020), we recommend a reexamination of the relationship between NLP systems and their end users, using methods from humancentered design, such as value-sensitive (Friedman et al., 2008) or participatory design (Sanders, 2002; DiSalvo et al., 2012; Denton et al., 2020),…”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “A similar argument can be made for dialects; focusing on African American Vernacular English (AAVE), Blodgett et al. (2020) describe multiple studies showing a drop in performance on popular NLU tasks when applied to text with AAVE features (Jørgensen et al.”
Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies (2021, methodology): “In this paper, we conduct a detailed investigation into the representational and allocational harms (Barocas et al., 2017; Blodgett et al., 2020) related to the treatment of gender with binary predilections in English language technologies.”
Evaluating Large Language Models Trained on Code (2021, methodology): “Generative models have been shown to encode bias in modalities such as natural language (Brown et al., 2020; Blodgett et al., 2020) and images (Radford et al., 2021), and we ﬁnd that the same is true of models like Codex that generate code.”
Societal Biases in Language Generation: Progress and Challenges (2021, background): “We survey detrimental representational9 and allocational10 impacts (Crawford, 2017; Barocas et al., 2017; Blodgett et al., 2020) used to motivate”
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, background): “This exclusion is a form of allocational harms (Barocas et al., 2017; Blodgett et al., 2020) and exacerbates existing (language-based) racial inequality (Rosa, 2019) as well as stigmatization of LGBTQ+ identities (Pinsof and Haselton, 2017).”
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets (2021, background): “, 2020), and a lack of consensus about how to conceptualize or operationalize such harms (Blodgett et al., 2020; Jacobs and Wallach, 2021).”
CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models (2020, background): “Discussing Bias Upon surveying 146 NLP papers that analyze or mitigate bias, Blodgett et al. (2020) provide recommendations to guide such research.”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Technical evaluation suites are often specific to a type of system and harm; for example, biases in natural language processing systems [33].”
The Capacity for Moral Self-Correction in Large Language Models (2023, background): “There are many benchmarks for measuring stereotype bias that we do not use in our work [32, 37, 38, 64], along with cogent criticism [9, 10] of these benchmarks and the ones we do use.”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, positive): “Evidence for this behavior has been substantially corroborated in NLP literature (Nadeem et al., 2021; Nozza et al., 2021; Huang et al., 2020b; Hassan et al., 2021; Blodgett et al., 2020; Liang et al., 2021; Field et al., 2021, inter alia).”
Re-contextualizing Fairness in NLP: The Case of India (2022, background): “, 2022), it has also been demonstrated that language technologies may capture, propagate, and amplify societal biases (Blodgett et al., 2020).”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, background): “We acknowledge that works that attempt to measure bias often run into inadequate or incomplete definitions of bias (Blodgett et al., 2020): for instance, Devinney et al.”