Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting

By Maria De-Arteaga, Alexey Romanov, Hanna M. Wallach, Jennifer Tour Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Cem Geyik, Krishnaram Kenthapadi, Adam Tauman Kalai. 2019

587 citations recorded by the source corpus.

We present a large-scale study of gender bias in occupation classification, a task where the use of machine learning may lead to negative outcomes on peoples' lives. We analyze the potential allocation harms that can result from semantic representation bias. To do so, we study the impact on occupation classification of including explicit gender indicators---such as first names and pronouns---in different semantic representations of online biographies. Additionally, we quantify the bias that remains when these indicators are "scrubbed," and describe proxy behavior that occurs in the absence of explicit gender indicators. As we demonstrate, differences in true positive rates between genders are correlated with existing gender imbalances in occupations, which may compound these imbalances.

Subfields: Artificial Intelligence, Sociology and Political Science, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “…et al. (2018); Zhao et al. (2018b); Bhaskaran and Bhallamudi (2019); Bordia and Bowman (2019); Brunet et al. (2019); Chaloner and Maldonado (2019); De-Arteaga et al. (2019); Dev and Phillips (2019); Font and Costa-jussà (2019); James-Sorenson and Alvarez-Melis (2019); Kurita et al. (2019);…”
Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies (2021, neutral): “There have been multiple attempts to understand social biases in language processing (Sheng et al., 2021; Caliskan et al., 2017), quantify them (Rudinger et al., 2018; Webster et al., 2018; De-Arteaga et al., 2019), and mitigate them (Zhao et al., 2019; Ravfo-gel et al., 2020; Sun et al., 2019).”
Social Biases in NLP Models as Barriers for Persons with Disabilities (2020, background): “This is important because NLP models are increasingly being used for tasks such as fighting online abuse (Jigsaw, 2017), measuring brand sentiment (Mostafa, 2013), and matching job applicants to job opportunities (De-Arteaga et al., 2019).”
Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Bias (2022, background): “However, the success of these models comes with a price—they are trained on vast amounts of mostly web-based data, which often contains social stereotypes and biases that the models might pick up (Bender et al., 2021; Dodge et al., 2021; De-Arteaga et al., 2019).”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, background): “…that given only a first name, humans can often infer the gender and race of the individual with that name and, from that, can draw stereotypical conclusions about them (Dasgupta et al., 2000; Bertrand and Mullainathan, 2004; Wheeler, 2016; De-Arteaga et al., 2019; Romanov et al., ar X iv :2 10 9.”
Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “, 2019); and reliance on stereotypical associations with race and gender (Caliskan et al., 2017; Rudinger et al., 2018; Zhao et al., 2018; De-Arteaga et al., 2019).”
CausaLM: Causal Model Explanation Through Counterfactual Language Models (2020, methodology): “The corpus-based estimator of CONEXP is defined as follows: CONEXPC0(O) = 1 I [∑ i∈I 〈~z(f(φO(xi)|Cj = 1)− ~z(f(φO(xi)|Cj = 0)〉 ] (13) The second baseline we consider is TPR-GAP, introduced in De-Arteaga et al. (2019) and used by Ravfogel et al. (2020).”
Racial Bias in Hate Speech and Abusive Language Detection Datasets (2019, background): “This includes both supervised (Blodgett and O’Connor, 2017; Tatman, 2017; Kiritchenko and Mohammad, 2018; De-Arteaga et al., 2019) and unsupervised natural language processing systems (Bolukbasi et al.”
Attributing Fair Decisions with Attention Interventions (2021, methodology): “We consider the biosbias dataset (De-Arteaga et al. 2019), and use our mitigation technique to reduce observed biases in the classification task performed on this dataset.”
Perturbation Sensitivity Analysis to Detect Unintended Model Biases (2019, background): “Additionally, De-Arteaga et al. (2019) study gender bias in occupation classiﬁcation using names in online biographies.”
Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection (2020, methodology): “To measure the bias in a classiﬁer, we follow De-Arteaga et al. (2019) and use the TPR-GAP measure.”
Learning to Deceive with Attention-Based Explanations (2019, methodology): “Transformer Models We use the Bidirectional Encoder Representations from Transformers (BERT) model (Devlin et al., 2019).”
What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes (2019, methodology): “…2017), where the task is to predict whether an individual earns more than $50k per year (i.e., whether their occupation is “high status”), and a dataset of online biographies (De-Arteaga et al., 2019), where the task is to predict an individual’s occupation from the text of their online biography.”
Measuring and Reducing Gendered Correlations in Pre-trained Models (2020, positive): “To measure how similar model predictions actually are, we follow Rudinger et al. (2017) and track the Pearson correlation ( r ) between the score difference and the representation in the BLS. Bias-in-Bios De-Arteaga et al. (2019) builds a dataset over biographies from the web by labeling each with t”