Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems

By Svetlana Kiritchenko, Saif M. Mohammad. 2018

523 citations recorded by the source corpus.

Automatic machine learning systems can inadvertently accentuate and perpetuate inappropriate human biases. Past work on examining inappropriate biases has largely focused on just individual systems. Further, there is no benchmark dataset for examining inappropriate biases in systems. Here for the first time, we present the Equity Evaluation Corpus (EEC), which consists of 8,640 English sentences carefully chosen to tease out biases towards certain races and genders. We use the dataset to examine 219 automatic sentiment analysis systems that took part in a recent shared task, SemEval-2018 Task 1 ‘Affect in Tweets’. We find that several of the systems show statistically significant bias; that is, they consistently provide slightly higher sentiment intensity predictions for one race or one gender. We make the EEC freely available.

Subfields: Artificial Intelligence, General Social Sciences, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “…(Greenwald et al., 1998) from the social psychology literature, while several techniques operationalize the well-studied “Angry Black Woman” stereotype (Kiritchenko and Mohammad, 2018; May et al., 2019; Tan and Celis, 2019) and the “double bind” faced by women (May et al., 2019; Tan and Celis,…”
StereoSet: Measuring stereotypical bias in pretrained language models (2020, methodology): “Another popular method to evaluate bias of pre-trained representations is to measure bias on extrinsic applications like coreference resolution (Rudinger et al., 2018; Zhao et al., 2018) and sentiment analysis (Kiritchenko and Mohammad, 2018).”
Reducing Sentiment Bias in Language Models via Counterfactual Evaluation (2019, neutral): “BERT-based classifier may themselves contain bias, which has been shown for many sentiment analysis systems (Kiritchenko and Mohammad, 2018).”
The Woman Worked as a Babysitter: On Biases in Language Generation (2019, methodology): “- - Regard Although sentiment has been used as a proxy for bias in other works (Lu et al., 2018; Kiritchenko and Mohammad, 2018), there has been little analysis on the correlation of sentiment to human judgment of bias.”
Detoxifying Language Models Risks Marginalizing Minority Voices (2021, background): “Unfortunately, since sentiment datasets are often biased against certain racial groups (Kiritchenko and Mohammad, 2018), controlling the sentiment of text will also affect which races are discussed.”
Toward Gender-Inclusive Coreference Resolution (2019, background): “, 2019; Gonen and Goldberg, 2019), sentiment analysis (Kiritchenko and Mohammad, 2018), machine translation (Font and Costa-jussà, 2019; Prates et al.”
Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Bias (2022, methodology): “Another method to evaluate bias in pretrained models is through downstream tasks, such as coreference resolution (Rudinger et al., 2018; Zhao et al., 2018) and sentiment analysis (Kiritchenko and Mohammad, 2018).”
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation (2021, background): “The Equity Evaluation Corpus (EEC) [16] presents a dataset to measure the difference in the intensity of sentiments predicted by sentiment analyzers across various gender and racial groups.”
FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age (2019, background): “Studies in algorithmic fairness have focused on either 1) discovering (auditing) existing bias in datasets or systems [50, 7, 30], 2) making a better dataset [39, 1], or 3) designing a better algorithm or model [12, 1, 47, 71, 70].”
Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models (2021, methodology): “Indeed, since contextual word embedding models such as BERT are optimised to capture the statistical properties of training data, they tend to pick up and amplify any social stereotypes that may be present (Kurita et al., 2019).”
Understanding and Evaluating Racial Biases in Image Captioning (2021, background): “Limitations include that sentiment analysis tools have been shown to encode societal biases themselves [43, 23], and may not generalize well to out-of-distribution machinegenerated text.”
On Measuring Social Biases in Sentence Encoders (2019, methodology): “…a sentence encoder makes with a given term rather than those it happens to make with the contexts of that term that are prevalent in the training data; a similar design was used in a recent sentiment analysis evaluation corpus stratiﬁed by race and gender (Kiritchenko and Mohammad, 2018).”
Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “Neural networks learn to replicate historical, societal biases from training data in various tasks such as natural language inference (Rudinger et al., 2017), coreference resolution (Cao and Daum´e III, 2019), and sentiment analysis (Kiritchenko and Mohammad, 2018).”
Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science (2018, neutral): “(See also Kiritchenko and Mohammad, 2018.)”
Learning the Difference that Makes a Difference with Counterfactually-Augmented Data (2019, background): “Across myriad tasks, researchers have demonstrated that models tend to rely on spurious associations [23, 8, 14, 16].”