The Risk of Racial Bias in Hate Speech Detection

By Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, Noah A. Smith. 2019

1,043 citations recorded by the source corpus.

We investigate how annotators’ insensitivity to differences in dialect can lead to racial bias in automatic hate speech detection models, potentially amplifying harm against minority populations. We first uncover unexpected correlations between surface markers of African American English (AAE) and ratings of toxicity in several widely-used hate speech datasets. Then, we show that models trained on these corpora acquire and propagate these biases, such that AAE tweets and tweets by self-identified African Americans are up to two times more likely to be labelled as offensive compared to others. Finally, we propose *dialect* and *race priming* as ways to reduce the racial bias in annotation, showing that when annotators are made explicitly aware of an AAE tweet’s dialect they are significantly less likely to label the tweet as offensive.

Subfields: Artificial Intelligence, Political Science and International Relations, Sociology and Political Science, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Political Science and International Relations

In Sociology and Political Science

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, background): “However, this may have disproportionately excluded casual language, code-switching, or dialectal diversity (e.g., Sap et al. (2019)) and may limit PaLM’s capability to model the non-dominant dialects across the English-speaking regions globally.”
Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “…associated with AAE than on text without these features (Jørgensen et al., 2015, 2016; Blodgett et al., 2016, 2018), and that toxicity detection systems score tweets containing features associated with AAE as more offensive than tweets without them (Davidson et al., 2019; Sap et al., 2019).”
LaMDA: Language Models for Dialog Applications (2022, background): “, [58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72]), and inappropriately revealing personally identifying information (PII) from training data [73].”
Ethical and social risks of harm from Language Models (2021, background): “et al., 2021; Sap et al., 2019)6.”
Challenges in Detoxifying Language Models (2021, methodology): “Disparate Impacts on Dialect Coverage Figure 5c shows relative loss gaps between the detoxified and the standard models, for both AAE and WAE tweets.”
Detoxifying Language Models Risks Marginalizing Minority Voices (2021, methodology): “In particular, toxicity datasets often contain spurious correlations between the toxic label and the presence of AAE and minority identity mentions (Sap et al., 2019).”
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, methodology): “…and corpora exhibit biases against minorities and suffer from low agreement in annotations (Waseem, 2016; Ross et al., 2017), partially due to annotator identity influencing their perception of hate speech (Cowan and Khatchadourian, 2003) and differences in annotation task setup (Sap et al., 2019).”
Re-imagining Algorithmic Fairness in India and Beyond (2021, background): “The majority of fairness research looks at racial [46, 57, 121, 125, 183] and gender biases [42, 46, 197,”
HateCheck: Functional Tests for Hate Speech Detection Models (2020, background): “…in data source, sampling strategy and annotation process (Vidgen and Derczynski, 2020; Poletto et al., 2020), and are known to exhibit annotator biases (Waseem, 2016; Waseem et al., 2018; Sap et al., 2019) as well as topic and author biases (Wiegand et al., 2019; Nejadgholi and Kiritchenko, 2020).”
Social Biases in NLP Models as Barriers for Persons with Disabilities (2020, background): “…(Bolukbasi et al., 2016; Caliskan et al., 2017; Garg et al., 2017; Barocas et al., 2017; Garg et al., 2019; Dixon et al., 2018; Noble, 2018; Manzini et al., 2019; Sap et al., 2019; May et al., 2019; Speer, 2017), bias with respect to different disability groups has been relatively under-explored.”
Pretraining Language Models with Human Preferences (2023, background): “Toxicity LMs can generate highly harmful language, including insults, profanities and threats (Sap et al., 2019; Gehman et al., 2020; Abid et al., 2021).”
An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models (2023, background): “These harmful representations are usually overlooked by toxic language detectors (Sap et al., 2019; Breitfeller et al., 2019; Hartvigsen et al., 2022), while they can resurface in language technologies and disadvantage an already disadvantaged group of people.”
Re-contextualizing Fairness in NLP: The Case of India (2022, background): “and racial biases (Sap et al., 2019; Davidson et al., 2019; Manzini et al., 2019), other axes of disparities such as ability (Hutchinson et al.”
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (2022, background): “Importantly, such biases in toxicity detection risk further marginalizing or censoring minority groups (Yasin, 2018; Sap et al., 2019; Dias Oliva et al., 2020; Are, 2020; Díaz and Hecht-Felella, 2021).”
Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “In-group AAVE terms are also predictive of offensive speech in DWMW17 even when they are used non-offensively, hinting at possible bias in the dataset (Sap et al., 2019).”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “These biases can result in toxicity being associated with certain words, such as profanities or identity terms (Dinan et al., 2019b; Dixon et al., 2018), or language varieties, such as African American English (AAE) (Liu et al., 2019; Sap et al., 2019).”
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “Possible mitigations include using race and dialect priming (Sap et al., 2019), using adversarial training techniques (Xia et al., 2020), adding fairness constraints (Gencoglu, 2020), or relabeling data used during training (Zhou et al., 2021).”