Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
By Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, Adam Tauman Kalai. 2016
3,784 citations recorded by the source corpus.
The blind application of machine learning runs the risk of amplifying biases present in data. Such a danger is facing us with word embedding, a popular framework to represent text data as vectors which has been used in many machine learning and natural language processing tasks. We show that even word embeddings trained on Google News articles exhibit female/male gender stereotypes to a disturbing extent. This raises concerns because their widespread use, as we describe, often tends to amplify these biases. Geometrically, gender bias is first shown to be captured by a direction in the word embedding. Second, gender neutral words are shown to be linearly separable from gender definition words in the word embedding. Using these properties, we provide a methodology for modifying an embedding to remove gender stereotypes, such as the association between the words receptionist and female, while maintaining desired associations such as between the words queen and female. Using crowd-worker evaluation as well as standard benchmarks, we empirically demonstrate that our algorithms significantly reduce gender bias in embeddings while preserving the its useful properties such as the ability to cluster related concepts and to solve analogy tasks. The resulting embeddings can be used in applications without amplifying gender bias.
Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Luke S. Zettlemoyer
- Yejin Choi
- Jianfeng Gao
- Noah A. Smith
- Graham Neubig
- Yoshua Bengio
In Representation & Self-Supervised Learning
- Yoshua Bengio
- Ivan Vulic
- Yoav Goldberg
- Samuel R. Bowman
- Ruslan Salakhutdinov
- Yonatan Belinkov
- Sebastian Ruder
- Mohit Bansal
In Safety, Robustness & Interpretability
- Adina Williams
- Kai-Wei Chang
- Yulia Tsvetkov
- Eric Michael Smith
- Sunipa Dev
- Dirk Hovy
- Emily Sheng
- Isabelle Augenstein
Citations in the corpus
- Learning Transferable Visual Models From Natural Language Supervision (2021, positive): “This points to gendered associations similar to those that have previously been found for occupations (Schwemmer et al., 2020) (Nosek et al., 2002) (Bolukbasi et al., 2016).”
- Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “Other representational harms Hovy and Søgaard (2015); Blodgett et al. (2016); Bolukbasi et al. (2016b); Hovy and Spruit (2016); Blodgett and O’Connor (2017); Larson (2017); Schnoebelen (2017); Blodgett et al. (2018); Curry and Rieser (2018); Díaz et al. (2018); Dixon et al. (2018); Kiritchenko and…”
- StereoSet: Measuring stereotypical bias in pretrained language models (2020, positive): “The seminal works of Bolukbasi et al. (2016) and Caliskan et al. (2017) show that word embed-dings such as word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) contain stereo-typical biases using diagnostic methods like word analogies and association tests.”
- Reducing Sentiment Bias in Language Models via Counterfactual Evaluation (2019, methodology): “der across more than 200 systems. For word embeddings, occupational gender bias has been identified and addressed by measuring projections onto linear gender-related subspaces of word representations (Bolukbasi et al., 2016; Lemoine et al., 2018; Zhao et al., 2018b; Bordia & Bowman, 2019). Gonen ”
- The Woman Worked as a Babysitter: On Biases in Language Generation (2019, methodology): “We analyze these two bias contexts because the occupation context has been well-studied in other tasks (Bolukbasi et al., 2016; Rudinger et al., 2018; Zhao et al., 2018; Zhou et al., 2019), and the more descriptive language in respect contexts are a good contrast for the more subtle occupation…”
- Reducing Gender Bias in Word-Level Language Models with a Gender-Equalizing Loss Function (2019, methodology): “Bolukbasi et al. (2016) and Zhao et al. (2017) defined gender bias as a projection of gender-neutral words on a gender subspace and removed bias by minimizing this projection.”
- Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor (2019, methodology): “As shown in Equation (3), Bolukbasi et al. (2016) introduce a threshold δ to ensure that B and D are semantically similar.”
- Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them (2019, methodology): “For the HARD-DEBIASED embedding we get a Pearson correlation of 0.686 (compared to a correlation of 0.741 when checking neighbors according to the biased version).”
- Gender Bias in Coreference Resolution (2018, background): “A number of papers explore (gender) bias in English word embeddings: how they capture implicit human biases in modern (Caliskan et al., 2017) and historical (Garg et al., 2018) text, and methods for debiasing them (Bolukbasi et al., 2016).”
- Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, methodology): “8 Following previous work [7, 39], we use the list of 323 occupations shown in Tables 18 and 19 of Section B.”
- LaMDA: Language Models for Dialog Applications (2022, background): “, [58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72]), and inappropriately revealing personally identifying information (PII) from training data [73].”
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, background): “…hand, it can also be expected to capture associations present in written text collections including correlations between gender and profession (Bolukbasi et al., 2016; Rudinger et al., 2018b; Zhao et al., 2018), negative sentiment about different racial and religious groups (Li et al., 2020;…”
- Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies (2021, neutral): “…language model biases such as gender, race or religion-related stereotypes in static word embeddings (GloVe (Pennington et al., 2014)) and contextual (e.g., BERT (Devlin et al., 2019)) representations (Bolukbasi et al., 2016; De-Arteaga et al., 2019; Ravfogel et al., 2020; Dev et al., 2020b).”
- Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs (2021, background): “However, several pieces of work have demonstrated that AI systems can perform differently for different groups of people, often exhibiting especially poor performance for already-disadvantaged groups (e.g., Nicol, Casey, and MacFarlane 2002; Rodger and Pendharkar 2004; Tatman 2017; Buolamwini and Ge”
- Re-imagining Algorithmic Fairness in India and Beyond (2021, neutral): “, [42, 218]), alongside testing strategies, e.”
- Persistent Anti-Muslim Bias in Large Language Models (2021, background): “Previous analyses of language models have demonstrated that biases can be revealed by looking at the embeddings of words and phrases and forming analogies in the embedding space [2].”
- Social Biases in NLP Models as Barriers for Persons with Disabilities (2020, background): “While previous studies have studied unintended biases in NLP models against other historically marginalized groups (Bolukbasi et al., 2016; Caliskan et al., 2017; Garg et al., 2017; Barocas et al., 2017; Garg et al., 2019; Dixon et al., 2018; Noble, 2018; Manzini et al., 2019; Sap et al., 2019; May…”
- Toward Gender-Inclusive Coreference Resolution (2019, background): “…more broadly than just in coreference resolution, including, for instance, natural language inference [Rudinger et al. 2017], word embeddings [e.g., Bolukbasi et al. 2016; Gonen and Goldberg 2019; Romanov et al. 2019], sentiment analysis [Kir-itchenko and Mohammad 2018], machine translation [Font…”
- On Measuring and Mitigating Biased Inferences of Word Embeddings (2019, methodology): “Not only is this approach simple and outperforms other approaches on intrinsic measures (Dev and Phillips 2019), it also does not have the potential to leave residual information among associated words (Gonen and Goldberg 2019) unlike hard debiasing (Bolukbasi et al. 2016).”
- Evaluating Gender Bias in Machine Translation (2019, background): “…done by women, construction workers are stereotypically men; Zhao et al., 2017), lexical semantics (“man is to computer programmer as woman is to home-maker”; Bolukbasi et al., 2016), and natural language inference (associating women with gossiping and men with guitars; Rudinger et al., 2017).”