Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings

By Thomas Manzini, Yao Chong Lim, Yulia Tsvetkov, Alan W. Black. 2019

341 citations recorded by the source corpus.

Online texts - across genres, registers, domains, and styles - are riddled with human stereotypes, expressed in overt or subtle ways. Word embeddings, trained on these texts, perpetuate and amplify these stereotypes, and propagate biases to machine learning models that use word embeddings as features. In this work, we propose a method to debias word embeddings in multiclass settings such as race and religion, extending the work of (Bolukbasi et al., 2016) from the binary setting, such as binary gender. Next, we propose a novel methodology for the evaluation of multiclass debiasing. We demonstrate that our multiclass debiasing is robust and maintains the efficacy in standard NLP tasks.

Subfields: Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, neutral): “…and Alvarez-Melis (2019); Kaneko and Bollegala (2019); Karve et al. (2019); Kurita et al. (2019); Lauscher and Glavaš (2019); Lee et al. (2019); Manzini et al. (2019); Mayfield et al. (2019); Précenth (2019); Pujari et al. (2019); Ruane et al. (2019); Stanovsky et al. (2019); Sun et al.…”
StereoSet: Measuring stereotypical bias in pretrained language models (2020, background): “Manzini et al. (2019) extend this to show that word embeddings capture several stereotypical biases such as racial and religious biases.”
Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor (2019, methodology): “terior_designer he architect 2 interior_designer architect she feminism he conservatism 4 feminism liberalism she lovely he brilliant 10 lovely magniﬁcent she sewing he carpentry 4 sewing woodworking Manzini et al. (2019b) black criminal caucasian lawful 13 legal statutory caucasian lawful black cri”
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, methodology): “8 Following previous work [7, 39], we use the list of 323 occupations shown in Tables 18 and 19 of Section B.”
Social Biases in NLP Models as Barriers for Persons with Disabilities (2020, background): “…groups (Bolukbasi et al., 2016; Caliskan et al., 2017; Garg et al., 2017; Barocas et al., 2017; Garg et al., 2019; Dixon et al., 2018; Noble, 2018; Manzini et al., 2019; Sap et al., 2019; May et al., 2019; Speer, 2017), bias with respect to different disability groups has been relatively…”
On Measuring and Mitigating Biased Inferences of Word Embeddings (2019, background): “To our knowledge, this is the among the ﬁrst demonstrations (Sweeney and Najaﬁan 2019; Manzini et al. 2019) of national or religious bias in word embeddings.”
Re-contextualizing Fairness in NLP: The Case of India (2022, background): “and racial biases (Sap et al., 2019; Davidson et al., 2019; Manzini et al., 2019), other axes of disparities such as ability (Hutchinson et al.”
Towards Understanding and Mitigating Social Biases in Language Models (2021, methodology): “Corresponding methods for debiasing these embeddings for both binary (Bolukbasi et al., 2016; Zhao et al., 2018) and multiclass (Manzini et al., 2019) attributes across gender, race, and religion have been devised.”
Measuring Bias in Contextualized Word Representations (2019, background): “There have been several efforts to investigate the amount of intrinsic bias within uncontextual-ized word embeddings in binary (Bolukbasi et al., 2016; Garg et al., 2018; Swinger et al., 2019) and multiclass (Manzini et al., 2019) settings.”
Mitigating Gender Bias in Natural Language Processing: Literature Review (2019, methodology): “With few exceptions (Manzini et al., 2019), work on debiasing in NLP has assumed that the protected attribute being discriminated against is binary.”
Towards Controllable Biases in Language Generation (2020, background): “g static embeddings (Bolukbasi et al., 2016; Zhao et al., 2018b), debiasing contextualized embeddings (Zhao et al., 2019; Kurita et al., 2019), extending bias analyses to multiple demographic groups (Manzini et al., 2019), and debiasing across languages (Zhou et al., 2019; Zmigrod et al., 2019)). Mu”
Towards Debiasing Sentence Representations (2020, methodology): “We therefore focus on post-hoc debiasing techniques which add a post-training debiasing step to these sentence representations before they are used in downstream tasks [5, 13].”
What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus (2021, background): “, 2020; Sweeney and Najafian, 2019) and to remove them (Bolukbasi et al., 2016; Zhao et al., 2018; Manzini et al., 2019), although the extent to which this is possible is still under debate (Gonen and Goldberg, 2019).”
Perturbation Sensitivity Analysis to Detect Unintended Model Biases (2019, background): “…al., 2016; Caliskan et al., 2017; Webster et al., 2018; D´ıaz et al., 2018; Dixon et al., 2018; De-Arteaga et al., 2019; Gonen and Goldberg, 2019; Manzini et al., 2019)) over a range of NLP tasks such as co-reference resolution and machine translation, as well as the tasks we studied — sentiment…”
Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview (2019, methodology): “Manzini et al. (2019), identiﬁed the bias subspace using principal component analysis and removed the biased components using hard Neutralize and Equalize debiasing and soft biasing methods proposed by Boluk-basi et al. (2016).”
Automatically Neutralizing Subjective Bias in Text (2019, background): “Such studies begin with identifying a direction or subspace that capture the bias and then removing this bias component to make representations fair across attributes like gender and age (Bolukbasi et al. 2016; Manzini et al. 2019).”
A Survey of Race, Racism, and Anti-Racism in NLP (2021, background): “These studies focus on how training data might describe racial minorities in biased ways, for example, by examining words associated with terms like ‘black’ or traditionally European/African American names (Caliskan et al., 2017; Manzini et al., 2019).”