Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection

By Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, Noah A. Smith. 2021

408 citations recorded by the source corpus.

The perceived toxicity of language can vary based on someone’s identity and beliefs, but this variation is often ignored when collecting toxic language datasets, resulting in dataset and model biases. We seek to understand the *who*, *why*, and *what* behind biases in toxicity annotations. In two online studies with demographically and politically diverse participants, we investigate the effect of annotator identities (*who*) and beliefs (*why*), drawing from social psychology research about hate speech, free speech, racist beliefs, political leaning, and more. We disentangle *what* is annotated as toxic by considering posts with three characteristics: anti-Black language, African American English (AAE) dialect, and vulgarity. Our results show strong associations between annotator identity and beliefs and their ratings of toxicity. Notably, more conservative annotators and those who scored highly on our scale for racist beliefs were less likely to rate anti-Black language as toxic, but more likely to rate AAE as toxic. We additionally present a case study illustrating how a popular toxicity detection system’s ratings inherently reflect only specific beliefs and perspectives. Our findings call for contextualizing toxicity labels in social variables, which raises immense implications for toxic language annotation and detection.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Accounting for the demographics and composition of human crowdworkers can also provide information [209] about subsequent impacts.”
Scaling Instruction-Finetuned Language Models (2022, background): “, 2019a), and challenges with evaluation given diverse perspectives on annotation (Goyal et al., 2022; Sap et al., 2021).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “Existing datasets are often biased due to their curation process (Dixon et al., 2018; Wiegand et al., 2019; Geva et al., 2019; Sap et al., 2021; Kryscinski et al., 2020) and can have unreliable annotations (Ross et al., 2017; Field and Tsvetkov, 2020b; Pagnoni et al., 2021).”
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (2022, background): “Still, toxicity is inherently subjective (Sap et al., 2021).”
PaLM 2 Technical Report (2023, background): “, 2022), and do not consider diverse perspectives (Goyal et al., 2022; Sap et al., 2021), and as discussed the performance of automated measures varies across language (Perspective API model cards).”
Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection (2022, background): “Language ideologies also surface in tools for toxicity detection; for example, the classification behavior of the PERSPECTIVE API (a popular hate speech detector) aligns with the attitudes of conservative, white, female annotators, who tend to perceive African-American dialects as more toxic (Sap et”
Mix and Match: Learning-free Controllable Text Generationusing Energy Language Models (2022, background): “These mechanisms for controlling the generation of language have the potential to mitigate undesirable biases encoded by the large language models and prevent the generation of hate speech and toxic language (Xu et al.; Gehman et al., 2020; Sap et al., 2021; Baheti et al., 2021; Mireshghallah and Be”
Towards Measuring the Representation of Subjective Global Opinions in Language Models (2023, positive): “However, such applications may require language models to make subjective judgments that could vary significantly across different social groups (Solaiman & Dennison, 2021; Bender et al., 2021; Johnson et al., 2022; Goyal et al., 2022; Sap et al., 2022).”
Towards Identifying Social Bias in Dialog Systems: Framework, Dataset, and Benchmark (2022, background): “Potential Mis-annotation Recently work revealed that bias underlying the annotation process can be enlarged by the system (Sap et al., 2021).”
ProsocialDialog: A Prosocial Backbone for Conversational Agents (2022, methodology): “Given the subjective nature of the safety labels which could lead to annotation variation (Sap et al., 2022), we ﬁnalize the context’s safety label according to the voting combination rather than the majority vote criterion.”
Data Feedback Loops: Model-driven Amplification of Dataset Biases (2022, background): “Though toxicity classifiers have shortcomings (Kumar et al., 2021; Sap et al., 2022), this work is primarily concerned with aggregate, relative changes in toxicity over time to measure amplification.”
From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models (2023, background): “It is commonly believed that annotator (Geva et al., 2019; Sap et al., 2019; Davani et al., 2022; Sap et al., 2022) and data bias (Park et al., 2018; Dixon et al., 2018; Dodge et al., 2021; Harris et al., 2022) are the cause of this impact, and some studies have investigated the connection between…”
Whose Opinions Do Language Models Reflect? (2023, background): “Prior works show that for many tasks—from toxicity detection (Gordon et al., 2021, 2022; Davani et al., 2022; Sap et al., 2022; Goyal et al., 2022), ethics judgements (Lourie et al., 2021), and inference (Pavlick & Kwiatkowski, 2019)—there is inherent variability in what different humans consider…”