Quantifying and Reducing Stereotypes in Word Embeddings

By Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, Adam Tauman Kalai. 2016

63 citations recorded by the source corpus.

Machine learning algorithms are optimized to model statistical properties of the training data. If the input data reflects stereotypes and biases of the broader society, then the output of the learning algorithm also captures these stereotypes. In this paper, we initiate the study of gender stereotypes in {\em word embedding}, a popular framework to represent text data. As their use becomes increasingly common, applications can inadvertently amplify unwanted stereotypes. We show across multiple datasets that the embeddings contain significant gender stereotypes, especially with regard to professions. We created a novel gender analogy task and combined it with crowdsourcing to systematically quantify the gender bias in a given embedding. We developed an efficient algorithm that reduces gender stereotype using just a handful of training examples while preserving the useful geometric properties of the embedding. We evaluated our algorithm on several metrics. While we focus on male/female stereotypes, our framework may be applicable to other types of embedding biases.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “Other representational harms Hovy and Søgaard (2015); Blodgett et al. (2016); Bolukbasi et al. (2016b); Hovy and Spruit (2016); Blodgett and O’Connor (2017); Larson (2017); Schnoebelen (2017); Blodgett et al. (2018); Curry and Rieser (2018); Díaz et al. (2018); Dixon et al. (2018); Kiritchenko and…”
Assessing Social and Intersectional Biases in Contextualized Word Representations (2019, background): “Current de-biasing methods largely address biases in context-free word embeddings [3], but the imperative for de-biasing contextual word models increases as they become more widespread.”
Women’s Syntactic Resilience and Men’s Grammatical Luck: Gender-Bias in Part-of-Speech Tagging and Dependency Parsing (2019, background): “While data bias can be overcome by accounting for demographics, and can even improve classification performance (Volkova et al., 2013; Hovy, 2015; Bolukbasi et al., 2016; Benton et al., 2017; Zhao et al., 2017; Lynn et al., 2017), there is still little”
Addressing Age-Related Bias in Sentiment Analysis (2018, background): “Word embeddings have been shown to encode stereotypes and human biases (e.g., “computer programmer” – “man” + “woman” = “homemaker”) [Bolukbasi et al., 2016].”
Reducing sentiment polarity for demographic attributes in word embeddings using adversarial learning (2020, background): “Recent studies have shown that these pre-trained models contain many types of bias [4, 6].”
Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (2022, background): “Such a representation allows arithmetic (measurable comparisons) to be performed on words [143].”
deb2viz: Debiasing gender in word embedding data using subspace visualization (2020, background): “Given that fact, whole constellations of interventions are extensively developed into machine learning algorithms, to mutually support fair assessment of situations and mitigate bias in these textual data [5-8].”
Bias in word embeddings (2020, background): “Previous work has proven that word embeddings contain bias [13], and researchers have already developed methodologies for tracing, quantifying, and mitigating it [12, 16].”
Auditing Search Engines for Differential Satisfaction Across Demographics (2017, background): “Although machine learning can address many challenges in web search, there is also increasing evidence that suggests that these methods may systematically and inconspicuously underserve some groups of users [7, 3].”
A Transparent Framework for Evaluating Unintended Demographic Bias in Word Embeddings (2019, background): “Much of the focus for mitigating unintended bias in NLP is either targeted at reducing gender stereotypes in text (Bolukbasi et al., 2016b,a; Zhao et al., 2017; Zhang et al., 2018), or inequality of sentiment or toxicity for various protected groups (Caliskan-Islam et al., 2016; Bakarov, 2018;…”