Studying Generalisability across Abusive Language Detection Datasets

By Steve Durairaj Swamy, Anupam Jamatia, Björn Gambäck. 2019

105 citations recorded by the source corpus.

Work on Abusive Language Detection has tackled a wide range of subtasks and domains. As a result of this, there exists a great deal of redundancy and non-generalisability between datasets. Through experiments on cross-dataset training and testing, the paper reveals that the preconceived notion of including more non-abusive samples in a dataset (to emulate reality) may have a detrimental effect on the generalisability of a model trained on that data. Hence a hierarchical annotation model is utilised here to reveal redundancies in existing datasets and to help reduce redundancy in future efforts.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

In Signal Processing

Citations in the corpus

Recipes for Safety in Open-domain Chatbots (2020, methodology): “We consider the same architecture and setup as in BlenderBot (Roller et al., 2020), which employs a Seq2Seq Transformer architecture (Vaswani et al., 2017), with an implementation based on the ParlAI version (Miller et al., 2017a).”
Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection (2021, background): “Another source of false negatives is when classification systems are applied to out-ofdomain settings, such as system trained on Twitter data being applied to data from Gab (Karan and Šnajder, 2018; Pamungkas et al., 2020; Swamy et al., 2019; Basile et al., 2019; Salminen et al., 2020).”
Misogyny Detection in Twitter: a Multilingual and Cross-Domain Study (2020, unknown): “…detection Davidson, Bhattacharya, and Weber (2019); Park, Shin, and Fung (2018); Wiegand, Ruppenhofer, and Kleinbauer (2019), and cross-domain abusive language detection Karan and Snajder (2018); Pamungkas and Patti (2019); Swamy, Jamatia, and Gambäck (2019); Waseem, Thorne, and Bingel (2018).”
Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective (2020, methodology): “Swamy, Jamatia, and Gambäck (2019) revealed that the dominance of benign examples in abusive language datasets, which is a common practice to emulate reality, might have a detrimental effect on the generalizability of classifiers.”
Toxic, Hateful, Offensive or Abusive? What Are We Really Classifying? An Empirical Analysis of Hate Speech Datasets (2020, background): “In another recent study (Swamy et al., 2019), it is also highlighted that more work must be done to identify similarities and differences in the publicly available datasets, as the data is more important than the model when tackling the problem of model generalization.”
XHate-999: Analyzing and Detecting Abusive Language Across Domains and Languages (2020, unknown): “This stands in contrast with insights from Swamy et al. (2019), which suggest that the high variation in abusive language typically precludes wide generalisations and domain adaptation.”
Towards multidomain and multilingual abusive language detection: a survey (2021, methodology): “[134] Neural model This study proposed several LSTM-based models that only focuses on using text information (char n-grams and word embedding) representation for building platform-agnostic hate speech detector, but they did not conduct any cross or multidomain experiment to evaluate their model.”
HateBERT: Retraining BERT for Abusive Language Detection in English (2020, methodology): “…(Waseem and Hovy, 2016; Ribeiro et al., 2018), neural network architectures (e.g., CNN or Bi-LSTM) (Kshirsagar et al., 2018; Mishra et al., 2018; Mitrovi´c et al., 2019), and, ﬁnally, ﬁne-tuning pre-trained language models, e.g., BERT, RoBERTa, among others (Liu et al., 2019; Swamy et al., 2019).”