SOLID: A Large-Scale Semi-Supervised Dataset for Offensive Language Identification

By Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Marcos Zampieri, Preslav Nakov. 2020

185 citations recorded by the source corpus.

The use of offensive language is a major problem in social media which has led to an abundance of research in detecting content such as hate speech, cyberbulling, and cyber-aggression. There have been several attempts to consolidate and categorize these efforts. Recently, the OLID dataset used at SemEval-2019 proposed a hierarchical three-level annotation taxonomy which addresses different types of offensive language as well as important information such as the target of such content. The categorization provides meaningful and important information for understanding offensive language. However, the OLID dataset is limited in size, especially for some of the low-level categories, which included only a few hundred instances, thus making it challenging to train robust deep learning models. Here, we address this limitation by creating the largest available dataset for this task, SOLID. SOLID contains over nine million English tweets labeled in a semi-supervised manner. We further demonstrate experimentally that using SOLID along with OLID yields improved performance on the OLID test set for two different models, especially for the lower levels of the taxonomy. Finally, we perform analysis of the models' performance on easy and hard examples of offensive language using data annotated in a semi-supervised way.

Subfields: Artificial Intelligence, Communication, Natural Language Processing, Evaluation & Benchmarks, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Communication

In Natural Language Processing

In Evaluation & Benchmarks

In AI Ethics, Fairness & Society

Citations in the corpus

SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020) (2020, background): “More details about the dataset can be found in Rosenthal et al. (2020).”
Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases (2021, background): “Recently, transformers have been fine-tuned for hate-speech detection (Caselli et al., 2020), detection of targeted offensive language (Rosenthal et al., 2021) and bias (Sap et al., 2020).”
ETHOS: a multi-label hate speech detection dataset (2020, methodology): “Similar directions have recently been investigated in the framework of HS detection [36].”
A Holistic Approach to Undesired Content Detection in the Real World (2022, unknown): “There is a long track record of work on the definition and detection of hateful, toxic, offensive, and abusive content (Kwok and Wang 2013; Nobata et al. 2016; Rosenthal et al. 2020; Lees et al. 2022; Pavlopoulos et al. 2020).”
WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (2024, background): “There exists an extensive body of work on detecting hateful, toxic, offensive and abusive content [12, 29, 21] on online social networking sites such as Twitter [38] and Reddit [14].”