Detection of Abusive Language: the Problem of Biased Datasets

By Michael Wiegand, Josef Ruppenhofer, Thomas Kleinbauer. 2019

268 citations recorded by the source corpus.

We discuss the impact of data bias on abusive language detection. We show that classification scores on popular datasets reported in previous work are much lower under realistic settings in which this bias is reduced. Such biases are most notably observed on datasets that are created by focused sampling instead of random sampling. Datasets with a higher proportion of implicit abuse are more affected than datasets with a lower proportion.

Subfields: Artificial Intelligence, Communication, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Communication

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

HateCheck: Functional Tests for Hate Speech Detection Models (2020, background): “This reﬂects the importance of training data composition, which previous hate speech research has emphasised (Wiegand et al., 2019; Nejadgholi and Kiritchenko, 2020).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “Existing datasets are often biased due to their curation process (Dixon et al., 2018; Wiegand et al., 2019; Geva et al., 2019; Sap et al., 2021; Kryscinski et al., 2020) and can have unreliable annota-”
Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection (2021, background): “Such bootstrapping can substantially bias the nature and coverage of datasets (Wiegand et al., 2019).”
Racial Bias in Hate Speech and Abusive Language Detection Datasets (2019, background): “Wiegand et al. (2019) identify more general biases due to the sampling procedures used to collect training data, resulting in classifiers associating innocuous topics like”
Challenges and frontiers in abusive content detection (2019, background): “Wiegand et al. show that if most abusive content in a dataset comes from only a few users then including user-level information risks overfitting: the classifier just picks up on those authors’ linguistic traits (Wiegand, Ruppenhofer, & Kleinbauer, 2019).”
TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification (2020, background): “It is also important to highlight that these datasets do not represent their underlying tasks as a whole but only a subsample, and therefore contain biases - automatic models trained on them might not be able to generalize to other specific settings (Augenstein et al., 2017; Wiegand et al., 2019).”
Demoting Racial Bias in Hate Speech Detection (2020, background): “…to annotator bias, sampling strategies also result in topic and author bias in datasets of abusive language detection, leading to decreased classiﬁcation performance when testing in more realistic settings, necessitating the adoption of cross-domain evaluation for fairness (Wiegand et al., 2019).”
Latent Hatred: A Benchmark for Understanding Implicit Hate Speech (2021, background): “, which may introduce topic bias and artiﬁcially inﬂate model performance on implicit examples (Wiegand et al., 2019).”
Resources and benchmark corpora for hate speech detection: a systematic review (2020, background): “Recent studies are showing how the volatile nature of topics, especially on social media, can hinder the predictive capability of supervised models trained on data collected with particular keyword sets (Wiegand et al. 2019), or in restricted time spans (Florio et al. 2020).”
Handling Bias in Toxic Speech Detection: A Survey (2022, methodology): “Dataset Sampling Lexical Annotation Racial Gender E E M E E M E M Founta [34] [69, 113] [113] [120] [19, 20, 44] [85, 116, 120] [44] [70]”
Tackling Online Abuse: A Survey of Automated Abuse Detection Methods (2019, background): “Wulczyn et al. (2017a) crowd-sourced annotations for English comments from Wikipedia’s talk pages and released three datasets: one focusing on personal attacks (115, 864 comments; 11.”
Directions in abusive language training data, a systematic review: Garbage in, garbage out (2020, background): “show that 70% of the sexist tweets in the highly cited Waseem and Hovy dataset [32] come from two content creators and that 99% of the racist tweets come from just one [96].”
XHate-999: Analyzing and Detecting Abusive Language Across Domains and Languages (2020, unknown): “In this work, we combine three different abusive language variants – hatefulness (Gao and Huang, 2017), aggression (Kumar et al., 2018), and attack (Wulczyn et al., 2017) – spanning three distinct data sources (comments under Fox News stories, Twitter/Facebook posts, and Wikipedia edit messages,…”
Towards multidomain and multilingual abusive language detection: a survey (2021, background): “The cross-domain setting is also explored by Wiegand et al. [146] to prevent bias contained in the training data, as they experimentally found several biases in currently popular abusive language datasets, including topic bias and author bias.”
ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AI (2021, background): “However, this can lead to the creation of heavily biased datasets (Vidgen and Derczyn-ski, 2021; Wiegand et al., 2019).”
A BERT-Based Transfer Learning Approach for Hate Speech Detection in Online Social Media (2019, background): “ing datasets [2,21,24].”
Implicitly Abusive Language – What does it actually look like and why are we not getting there? (2021, background): “As Wiegand et al. (2019) showed, classiﬁers trained on these datasets may correctly detect implicitly abusive instances on unseen test instances of the same datasets.”
Contextualizing Hate Speech Classifiers with Post-hoc Explanation (2020, background): “Modern text classiﬁers thus struggle to learn a model of hate speech that generalizes to real-world applications (Wiegand et al., 2019).”