Resources and benchmark corpora for hate speech detection: a systematic review

By Fabio Poletto, Valerio Basile, Manuela Sanguinetti, Cristina Bosco, Viviana Patti. 2020

496 citations recorded by the source corpus.

Hate Speech in social media is a complex phenomenon, whose detection has recently gained significant traction in the Natural Language Processing community, as attested by several recent review works. Annotated corpora and benchmarks are key resources, considering the vast number of supervised approaches that have been proposed. Lexica play an important role as well for the development of hate speech detection systems. In this review, we systematically analyze the resources made available by the community at large, including their development methodology, topical focus, language coverage, and other factors. The results of our analysis highlight a heterogeneous, growing landscape, marked by several issues and venues for improvement.

Subfields: Artificial Intelligence, Natural Language Processing, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In AI Ethics, Fairness & Society

Citations in the corpus

HateCheck: Functional Tests for Hate Speech Detection Models (2020, background): “speech datasets vary in data source, sampling strategy and annotation process (Vidgen and Derczynski, 2020; Poletto et al., 2020), and are known to exhibit annotator biases (Waseem, 2016; Sap et al.”
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “In recent years, the field of toxic language detection has seen notable advancements, supported by comprehensive research benchmarks (Poletto et al., 2021) and released tools such as Perspective API.”
Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection (2021, background): “hate speech training datasets, such as lacking linguistic variety, being inexpertly annotated and degrading over time (Vidgen et al., 2019a; Poletto et al., 2020).”
Dynabench: Rethinking Benchmarking in NLP (2021, background): “Few high-quality, varied and large training datasets are available for training hate detection systems (Vidgen and Derczynski, 2020; Poletto et al., 2020; Vidgen et al., 2019).”
On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark (2021, background): “Despite abundant research on toxic language and social bias in natural language (Schmidt and Wie-gand, 2017; Poletto et al., 2021), it is still challenging to directly transfer them onto open-domain dialogue safety tasks, for two major reasons.”
Towards multidomain and multilingual abusive language detection: a survey (2021, methodology): “The current availability of datasets in many languages [103], makes the time ripe for addressing the multilingual challenge.”
Human-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Countering (2022, methodology): “Regarding the systems built on top of these benchmarks, we refer the readers to the surveys by Poletto et al. (2020); Schmidt and Wiegand (2017); Fortuna and Nunes (2018) for detailed reviews.”
Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective (2020, methodology): “For more detail on language resources for detecting online abuse we refer the reader to recent surveys by Poletto et al. (2020), Vidgen and Derczynski (2020). Poletto et al. (2020) conducted a systematic review of text collections annotated for hate speech.”
Implicitly Abusive Language – What does it actually look like and why are we not getting there? (2021, background): “Since a survey would be beyond the scope of this section, we refer the reader to Poletto et al. (2020) and Vidgen and Derczynski (2020).”