What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes

By Alexey Romanov, Maria De-Arteaga, Hanna M. Wallach, Jennifer Tour Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Cem Geyik, Krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai. 2019

86 citations recorded by the source corpus.

There is a growing body of work that proposes methods for mitigating bias in machine learning systems. These methods typically rely on access to protected attributes such as race, gender, or age. However, this raises two significant challenges: (1) protected attributes may not be available or it may not be legal to use them, and (2) it is often desirable to simultaneously consider multiple protected attributes, as well as their intersections. In the context of mitigating bias in occupation classification, we propose a method for discouraging correlation between the predicted probability of an individual’s true occupation and a word embedding of their name. This method leverages the societal biases that are encoded in word embeddings, eliminating the need for access to protected attributes. Crucially, it only requires access to individuals’ names at training time and not at deployment time. We evaluate two variations of our proposed method using a large-scale dataset of online biographies. We find that both variations simultaneously reduce race and gender biases, with almost no reduction in the classifier’s overall true positive rate.

Subfields: Artificial Intelligence, General Social Sciences, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “…(2019); Font and Costa-jussà (2019); James-Sorenson and Alvarez-Melis (2019); Kurita et al. (2019); Mayfield et al. (2019); Pujari et al. (2019); Romanov et al. (2019); Ruane et al. (2019); Sedoc and Ungar (2019); Sun et al. (2019); Zmigrod et al. (2019); Hutchinson et al. (2020);…”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, background): “Here, we focus on names, which are comparatively under-explored as a linguistic proxy for gender in NLP (Hall Maudslay et al., 2019; Romanov et al., 2019; Webster et al., 2020), and have yet to have been explored systematically in the context of conversational dialogue.”
BBQ: A hand-built bias benchmark for question answering (2021, background): “Names are widely recognized to carry information about both gender and racial identity in the U.S. and are effective ways of measuring bias (Romanov et al., 2019; Darolia et al., 2016; Kasof, 1993).”
Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “…data (e.g., Lu et al., 2018; Hall Maudslay et al., 2019; Zhao et al., 2018a; Kaushik et al., 2019), the training process (e.g., Huang et al., 2019; Qian et al., 2019), or the model itself (e.g., Madras et al., 2019; Romanov et al., 2019; Gehrmann et al., 2019) to ensure counterfactual fairness.”
Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection (2020, methodology): “Following Romanov et al. (2019), we also calculate the root-mean square of GAP TPRg,y over all professions y, to get a single per-gender bias score: GAP TPR,RMSg = √ 1 |C| ∑ y∈C (GAP TPRg,y ) 2 (4) where C is the set of all labels (professions).”
Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview (2019, background): “Romanov et al. (2019) generalize this work to a multi-class setting, exploring methods to mitigate bias in an occupation classification task.”
Impact of Pretraining Term Frequencies on Few-Shot Reasoning (2022, background): “Techniques for mitigating biases in the presence of dataset artifacts are covered by Romanov et al. (2019) and Karimi Mahabadi et al. (2020).”
Measuring and Reducing Gendered Correlations in Pre-trained Models (2020, neutral): “Racial and gender bias has been documented in resume-job matching software (De-Arteaga et al., 2019; Romanov et al., 2019), sentiment analysis (Kiritchenko and Mohammad, 2018), coreference resolution (Rudinger et al., 2018; Zhao et al., 2018a; Webster et al., 2018), image captioning (Zhao et al.,…”
A Survey of Race, Racism, and Anti-Racism in NLP (2021, neutral): “Survey/Position Romanov et al. (2019) 2019 NAACL Sector-spec.”
Disembodied Machine Learning: On the Illusion of Objectivity in NLP (2021, background): “…one in which bias is imagined as a ﬁnite quantity in the model that can be minimised by altering the model’s representation (Agarwal et al. 2018; Romanov et al. 2019); 2 and one which, similar to our work, accepts the premise that ML, and more broadly optimisation systems, contain social biases…”
Towards Understanding Gender Bias in Relation Extraction (2019, background): “Existing studies have revealed gender bias in various NLP tasks (Zhao et al., 2017; Rudinger et al., 2018; Zhao et al., 2018; Dixon et al., 2018; Lu et al., 2018; Kir-itchenko and Mohammad, 2018; Romanov et al., 2019; Sheng et al., 2019; Sun et al., 2019).”
What We Can't Measure, We Can't Understand: Challenges to Demographic Data Procurement in the Pursuit of Fairness (2020, methodology): “, [32, 69, 90]), to only use demographic data during model training (e.”
Investigating Gender Bias in Language Models Using Causal Mediation Analysis (2020, background): “…data (e.g., Lu et al., 2018; Hall Maud-slay et al., 2019; Zhao et al., 2018a; Kaushik et al., 2019), the training process (e.g., Huang et al., 2019; Qian et al., 2019), or the model itself (e.g., Madras et al., 2019; Romanov et al., 2019; Gehrmann et al., 2019) to ensure counterfactual fairness.”
Theories of “Gender” in NLP Bias Research (2022, background): “07667 [116] Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram”
Benchmarking Intersectional Biases in NLP (2022, background): “For instance, all four of the aforementioned allocational harm papers measure and/or mitigate gender bias with respect to an NLP-based occupation classifier (De-Arteaga et al., 2019; Prost et al., 2019; Romanov et al., 2019; Zhao et al., 2020).”
Equity Beyond Bias in Language Technologies for Education (2019, background): “…et al., 2017; Garg et al., 2018) and is passed downstream, producing biased outcomes for tasks like coreference resolution (Zhao et al., 2018a), sentiment analysis (Kiritchenko and Mohammad, 2018), search (Romanov et al., 2019), and dialogue systems (Voigt et al., 2018; Henderson et al., 2018).”
“You Are Grounded!”: Latent Name Artifacts in Pre-trained Language Models (2020, background): “There is multiple evidence that word embeddings encode gender and racial bias (Bolukbasi et al., 2016; Caliskan et al., 2017; Manzini et al., 2019; Gonen and Goldberg, 2019), in particular in the representations of given names (Romanov et al., 2019).”
Toward Gender-Inclusive Coreference Resolution: An Analysis of Gender and Bias Throughout the Machine Learning Lifecycle* (2021, background): “…resolution, including, for instance, natural language inference (Rudinger, May, and Van Durme 2017), word embeddings (e.g., Bolukbasi et al. 2016; Romanov et al. 2019; Gonen and Goldberg 2019), sentiment analysis (Kiritchenko and Mohammad 2018), machine translation (Font and Costa-jussà 2019;…”