Towards a Comprehensive Taxonomy and Large-Scale Annotated Corpus for Online Slur Usage

By Jana Kurrek, Haji Mohammad Saleem, Derek Ruths. 2020

49 citations recorded by the source corpus.

Abusive language classifiers have been shown to exhibit bias against women and racial minorities. Since these models are trained on data that is collected using keywords, they tend to exhibit a high sensitivity towards pejoratives. As a result, comments written by victims of abuse are frequently labelled as hateful, even if they discuss or reclaim slurs. Any attempt to address bias in keyword-based corpora requires a better understanding of pejorative language, as well as an equitable representation of targeted users in data collection. We make two main contributions to this end. First, we provide an annotation guide that outlines 4 main categories of online slur usage, which we further divide into a total of 12 sub-categories. Second, we present a publicly available corpus based on our taxonomy, with 39.8k human annotated comments extracted from Reddit. This corpus was annotated by a diverse cohort of coders, with Shannon equitability indices of 0.90, 0.92, and 0.87 across sexuality, ethnicity, and gender. Taken together, our taxonomy and corpus allow researchers to evaluate classifiers on a wider range of speech containing slurs.

Subfields: Artificial Intelligence, Communication, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Communication

In Natural Language Processing

Citations in the corpus

HateCheck: Functional Tests for Hate Speech Detection Models (2020, background): “We identiﬁed different types of hate in taxonomies of abusive content (e.g. Zampieri et al., 2019; Banko et al., 2020; Kurrek et al., 2020).”
Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection (2021, background): “Similarly, many models overfit on the use of slurs and pejorative terms, treating them as hateful irrespective of how they are used (Waseem et al., 2018; Davidson et al., 2017; Kurrek et al., 2020; Palmer et al., 2020).”
Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation (2022, background): “[23] presents work where the focus is on detecting slurs in Reddit data by a team of 20 annotators that are a mix of genders, ethnicities, and sexual orientations.”