Racial Bias in Hate Speech and Abusive Language Detection Datasets

By Thomas Davidson, Debasmita Bhattacharya, Ingmar Weber. 2019

530 citations recorded by the source corpus.

Technologies for abusive language detection are being developed and applied with little consideration of their potential biases. We examine racial bias in five different sets of Twitter data annotated for hate speech and abusive language. We train classifiers on these datasets and compare the predictions of these classifiers on tweets written in African-American English with those written in Standard American English. The results show evidence of systematic racial bias in all datasets, as classifiers trained on them tend to predict that tweets written in African-American English are abusive at substantially higher rates. If these abusive language detection systems are used in the field they will therefore have a disproportionate negative impact on African-American social media users. Consequently, these systems may discriminate against the groups who are often the targets of the abuse we are trying to detect.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “…associated with AAE than on text without these features (Jørgensen et al., 2015, 2016; Blodgett et al., 2016, 2018), and that toxicity detection systems score tweets containing features associated with AAE as more offensive than tweets without them (Davidson et al., 2019; Sap et al., 2019).”
Challenges in Detoxifying Language Models (2021, background): “A second type of bias considers disparate performance across dialects, where classifiers on average assign higher toxicity scores e.g. to African-American English (AAE) (Davidson et al., 2019; Sap et al., 2019a).”
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, background): “, 2020) or text by racial minorities (e.g., text in African American English; Sap et al., 2019; Davidson et al., 2019).”
Social Biases in NLP Models as Barriers for Persons with Disabilities (2020, background): “It is important to recognize that social norms around language are contextual and differ across groups (Castelle, 2018; Davidson et al., 2019; Vidgen et al., 2019).”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “For instance, automated hate speech detection driven by annotated data with an insensitivity to dialect differences can amplify harm to minority or marginalized groups by silencing their voices (see 4.2.2.1 Community Erasure) or incorrectly labeling their speech as offensive [67].”
Re-contextualizing Fairness in NLP: The Case of India (2022, background): “and racial biases (Sap et al., 2019; Davidson et al., 2019; Manzini et al., 2019), other axes of disparities such as ability (Hutchinson et al.”
BBQ: A hand-built bias benchmark for question answering (2021, background): “Similarly, Davidson et al. (2019) find differences in hate speech detection performance for tweets written in African American English in contrast with Standard American English.”
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “Hate speech detection has proven to be difficult, and for instance subject to unwanted bias [16, 63, 10].”
Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection (2021, background): “(2017) dataset has been shown to have several cases where non-hateful African American vernacular has been mislabelled (Davidson and Weber, 2019; Sap et al., 2019).”
Social Bias Frames: Reasoning about Social and Power Implications of Language (2019, background): “However, simple classifcations run the risk of discriminating against minority groups, due to high variation and identity-based biases in annotations (e.g., which cause models to learn associations between dialect and toxicity; Sap et al., 2019a; Davidson et al., 2019).”
DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts (2021, background): “…of large-scale evaluation required for systematic comparison of meth-ods, an abundance of work shows that their accuracy is far from ideal (Dixon et al., 2018; Sap et al., 2019; Davidson et al., 2019; Hutchinson et al., 2020) in part due to reliance on spurious features, which we discuss in §8.”
Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022, background): “Unsurprisingly, gendered and racial disparities have been documented in a number of language technologies [37, 79, 141], and processes of creating resources and technologies may further entrench such disparities [25, 38, 144].”
HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection (2020, background): “The target community information would allow researchers to delve into issues related to bias in hate speech (Davidson, Bhattacharya, and Weber 2019).”
Challenges in Automated Debiasing for Toxic Language Detection (2021, background): “We anonymize the usernames to protect user privacy. tems has grown, several biases have been found in dataset and models, spurring various debiasing efforts to mitigate these individual biases (e.g., gender bias, racial bias; Park et al., 2018; Sap et al., 2019; Davidson et al., 2019).”
Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection (2021, background): “While those annotation setups and annotator homogeneity could cause less variation in toxicity annotations of anti-Black, AAE, and vulgar posts, there is still empirical evidence of anti-AAE racial biases in some of these datasets (Sap et al., 2019; Davidson et al., 2019).”
A Survey of Race, Racism, and Anti-Racism in NLP (2021, background): “However, more recent work has show that existing hate speech classifiers are likely to falsely label text containing identity terms like ‘black’ or text containing linguistic markers of AAE as toxic (Dixon et al., 2018; Sap et al., 2019; Davidson et al., 2019; Xia et al., 2020).”