Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation

By Nitesh Goyal, Ian Kivlichan, Rachel Rosen, Lucy Vasserman. 2022

136 citations recorded by the source corpus.

Machine learning models are commonly used to detect toxicity in online conversations. These models are trained on datasets annotated by human raters. We explore how raters' self-described identities impact how they annotate toxicity in online comments. We first define the concept of Specialized Rater Pools: rater pools formed based on raters' self-described identities, rather than at random. We formed three such rater pools for this study - specialized rater pools of raters from the U.S. who identify as African American, LGBTQ, and those who identify as neither. Each of these rater pools annotated the same set of comments, which contains many references to these identity groups. We found that rater identity is a statistically significant factor in how raters will annotate toxicity for identity-related annotations. Using preliminary content analysis, we examined the comments with the most disagreement between rater pools and found nuanced differences in the toxicity annotations. Next, we trained models on the annotations from each of the different rater pools, and compared the scores of these models on comments from several test sets. Finally, we discuss how using raters that self-identify with the subjects of comments can create more inclusive machine learning models, and provide more nuanced ratings than those by random raters.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Scaling Instruction-Finetuned Language Models (2022, background): “, 2019a), and challenges with evaluation given diverse perspectives on annotation (Goyal et al., 2022; Sap et al., 2021).”
PaLM 2 Technical Report (2023, background): “, 2022), and do not consider diverse perspectives (Goyal et al., 2022; Sap et al., 2021), and as discussed the performance of automated measures varies across language (Perspective API model cards).”
On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research (2023, unknown): “These are challenging first steps, though, because the perception of toxicity and hate-speech is known to vary among different identity groups (Goyal et al., 2022) and genders (Binns et al., 2017).”
Whose Opinions Do Language Models Reflect? (2023, background): “Prior works show that for many tasks—from toxicity detection (Gordon et al., 2021, 2022; Davani et al., 2022; Sap et al., 2022; Goyal et al., 2022), ethics judgements (Lourie et al., 2021), and inference (Pavlick & Kwiatkowski, 2019)—there is inherent variability in what different humans consider…”
Towards Measuring the Representation of Subjective Global Opinions in Language Models (2023, unknown): “However, such applications may require language models to make subjective judgments that could vary significantly across different social groups (Solaiman & Dennison, 2021; Bender et al., 2021; Johnson et al., 2022; Goyal et al., 2022; Sap et al., 2022).”