Challenges for Toxic Comment Classification: An In-Depth Error Analysis

By Betty van Aken, Julian Risch, Ralf Krestel, Alexander Löser. 2018

250 citations recorded by the source corpus.

Toxic comment classification has become an active research field with many recently proposed approaches. However, while these approaches address some of the task’s challenges others still remain unsolved and directions for further research are needed. To this end, we compare different deep learning and shallow approaches on a new, large comment dataset and propose an ensemble that outperforms all individual models. Further, we validate our findings on a second dataset. The results of the ensemble enable us to perform an extensive error analysis, which reveals open challenges for state-of-the-art methods and directions towards pending future research. These challenges include missing paradigmatic context and inconsistent dataset labels.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

HateCheck: Functional Tests for Hate Speech Detection Models (2020, methodology): “F11 – non-hateful uses of profanity : Oversensi-tiveness of hate speech detection models to profan-ity (Davidson et al., 2017; Malmasi and Zampieri, 2018; van Aken et al., 2018).”
The Risk of Racial Bias in Hate Speech Detection (2019, methodology): “To understand the racial and dialectic bias in toxic language detection, we focus our analyses on two corpora of tweets (Davidson et al., 2017; Founta et al., 2018) that are widely used in hate speech detection (Park et al., 2018; van Aken et al., 2018; Kapoor et al., 2018; Alorainy et al., 2018 1.”
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, methodology): “While earlier detectors commonly employed neural networks like CNNs, RNNs, and LSTMs (Georgakopoulos et al., 2018; van Aken et al., 2018; Gunasekara and Nejadgholi, 2018; Joulin et al., 2017; Kshirsagar et al., 2018; Mishra et al., 2018; Mitrovi´c et al., 2019; Sigurbergsson and Derczynski, 2020),…”
Challenges and frontiers in abusive content detection (2019, background): “Noticeably, van Aken suggests that Davidson et al.’s widely used hate and offensive language dataset has up to 10% of its data mislabeled (van Aken et al., 2018).”
Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, methodology): “Efficient methods that reliably quantify a deep neural network (DNN)’s predictive uncertainty are important for industrial-scale, real-world applications, which include examples such as object recognition in autonomous driving [21], ad click prediction in online advertising [74], and intent understa”
Finding Microaggressions in the Wild: A Case for Locating Elusive Phenomena in Social Media Posts (2019, background): “Instead, much of the recent work has focused on explicitly toxic language (e.g., Waseem et al., 2017), with surveys of the area also overlooking this important and challenging task of recognizing this subtle toxicity (van Aken et al., 2018; Salminen et al., 2018; Fortuna and Nunes, 2018).”
Detecting and Classifying Malevolent Dialogue Responses: Taxonomy, Data and Methodology (2020, methodology): “And van Aken et al. [54] use LSTMs and LSTMs+CNNs for toxic comment classification on the KTCDD dataset.”
Towards Robust Toxic Content Classification (2019, methodology): “Both van Aken et al. (2018) and Gunasekara and Nejadgholi (2018) have shown that among the various approaches, bidirectional RNNs with attention using pretrained fastText embeddings (Joulin et al. 2016) have strong performance, with Gunasekara and Nejadgholi (2018) acheiving the best single-model…”
A taxonomy, data set, and benchmark for detecting and classifying malevolent dialogue responses (2021, methodology): “Pretrained word embeddings, that is, GloVe, have been used in several studies (Arango et al., 2019; van Aken et al., 2018; Zampieri et al., 2019).”
Detecting Cross-Geographic Biases in Toxicity Modeling on Social Media (2021, background): “…that these models often encode various societal biases against marginalized groups (Waseem, 2016; Caliskan et al., 2017; Park et al., 2018; van Aken et al., 2018; Dixon et al., 2018; Sap et al., 2019; Davidson et al., 2019; Hutchinson et al., 2020; Zhou et al., 2021) and can be adversarially…”
Tackling Online Abuse: A Survey of Automated Abuse Detection Methods (2019, background): “, Aken et al. (2018)) noted that sarcastic comments are hard for abuse detection methods to deal with since surface features are not sufficient; typically the knowledge of the context or background of the user is also required. Mishra (2018) found that metaphors are more frequent in abusive sam-”
A comprehensive review on ensemble deep learning: Opportunities and challenges (2023, methodology): “In Haralabopoulos et al. (2020) applied both voting and stacking based on several deep learning models, namely LSTM, GRU, CNN, RCNN, and DNN, using two English tweets datasets (SemEval (Bethard et al., 2016), Toxic Comment (van Aken et al., 2018)).”
Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset (2022, background): “[120] Betty van Aken, Julian Risch, Ralf Krestel, and Alexander Löser.”
Implicitly Abusive Language – What does it actually look like and why are we not getting there? (2021, methodology): “The subtypes we used were either mentioned in previous work (van Aken et al., 2018) or frequently observed in the examined datasets.”
Towards Ethics by Design in Online Abusive Content Detection (2020, methodology): “For example, van Aken et al. (2018) questioned 10–15% of manually obtained labels on two widely used datasets, Kaggle Toxicity by Jigsaw and Google 2 and the one by Davidson et al. (2017).”