SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020)

By Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, cCaugri cColtekin. 2020

533 citations recorded by the source corpus.

We present the results and the main findings of SemEval-2020 Task 12 on Multilingual Offensive Language Identification in Social Media (OffensEval-2020). The task included three subtasks corresponding to the hierarchical taxonomy of the OLID schema from OffensEval-2019, and it was offered in five languages: Arabic, Danish, English, Greek, and Turkish. OffensEval-2020 was one of the most popular tasks at SemEval-2020, attracting a large number of participants across all subtasks and languages: a total of 528 teams signed up to participate in the task, 145 teams submitted official runs on the test data, and 70 teams submitted system description papers.

Subfields: Artificial Intelligence, Communication, Law, Natural Language Processing, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Communication

In Law

In Natural Language Processing

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

Citations in the corpus

Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “, 2019; 2020), Facebook (Glavaš et al., 2020; Zampieri et al., 2020), or Reddit (Han & Tsvetkov, 2020; Zampieri et al.”
Recipes for Safety in Open-domain Chatbots (2020, methodology): “Waseem et al. (2017) advocate for partitioning abusive content according to what entity it is directed to, an approach adopted by the OLID/OffensEval datasets (Zampieri et al., 2019, 2020).”
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “, 2020), Facebook (Glavaš et al., 2020; Zampieri et al., 2020), or Reddit (Han and Tsvetkov, 2020; Zampieri et al.”
Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection (2021, background): “Benchmark datasets Several benchmark datasets have been put forward for online hate classification (Waseem and Hovy, 2016; Waseem, 2016; Davidson et al., 2017; Founta et al., 2018; Mandl et al., 2019; Zampieri et al., 2019, 2020; Vidgen et al., 2020, 2021).”
Resources and benchmark corpora for hate speech detection: a systematic review (2020, background): “OffensEval2020 , Multilingual Offensive Language Identiﬁcation in Social Media (Zampieri et al. 2020) is the second edition of a shared task on offensive language organized at SemEval 2020.”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, background): “, 2019, 2020), Facebook (Glavaš et al., 2020; Zampieri et al., 2020), or Reddit”
Text Detoxification using Large Pre-trained Neural Models (2021, background): “Identification of toxicity in user texts is an active area of research (Zampieri et al., 2020; D’Sa et al., 2020; Han and Tsvetkov, 2020).”
Implicitly Abusive Language – What does it actually look like and why are we not getting there? (2021, methodology): “…logistic regression, were among the most popular methods for this task (Warner and Hirschberg, 2012; Burnap et al., 2015; Nobata et al., 2016), at present, best results are obtained by deep-learning methods, particularly transformers (Struß et al., 2019; Kumar et al., 2020; Zampieri et al., 2020).”
Toxic Language Detection in Social Media for Brazilian Portuguese: New Dataset and Multilingual Analysis (2020, methodology): “OffensEval is a series of shared tasks focusing on offensive comments detection (Zampieri et al., 2019b, 2020).”
Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators’ Disagreement (2021, methodology): “However, system results reported for the last Offenseval shared task on offensive language identification in English tweets (Zampieri et al., 2020) show that the majority of submissions achieved an F1 score > 0.90 on the binary classification task.”
Probing Toxic Content in Large Pre-Trained Language Models (2021, background): “…different English datasets (Davidson et al., 2017; Founta et al., 2018; Ousidhoum et al., 2019; Zampieri et al., 2019), four others in Arabic (Ousid-houm et al., 2020; Albadi et al., 2018; Mulki et al., 2019; Zampieri et al., 2020), and the only one we know about in French (Ousidhoum et al., 2019).”
ToxCCIn: Toxic Content Classification with Interpretability (2021, background): “Leveraging the development of pre-trained language models (Devlin et al., 2019; Liu et al., 2019b) and domain transfer learning (Gururangan et al., 2020; Sotudeh et al., 2020), many models (Wiedemann et al., 2020) achieved high performance on toxicity detection (Zampieri et al., 2020).”
Directions in abusive language training data, a systematic review: Garbage in, garbage out (2020, background): “For example, although multilingual BERT supports dozens of languages, using this for building a language-specific classifier can be sub-optimal [142].”
Hybrid Emoji-Based Masked Language Models for Zero-Shot Abusive Language Detection (2020, background): “Then, to investigate whether our results can be generalised also when dealing with typologi-cally different languages, we will test our approach on additional abusive language datasets covering other languages (Ousidhoum et al., 2019; Zampieri et al., 2020).”
XHate-999: Analyzing and Detecting Abusive Language Across Domains and Languages (2020, unknown): “The recent OffenseEval task (Zampieri et al., 2020) introduced a multilingual data set for 5 languages (English, Arabic, Danish, Hebrew, Turkish), which was expanded to German and Italian by Casula (2020).”
GPT is an effective tool for multilingual psychological text analysis (2024, methodology): “We then examined GPT’s ability to detect a different psychological construct, offensiveness, in both English and Turkish (25, 68).”
Automatically Exposing Problems with Neural Dialog Models (2021, methodology): “To make machine learning models safe to use especially with language generation, there is a long literature in safety such as hate speech (Zampieri et al., 2020) and bias (Di-nan et al., 2019, 2020).”
SOLID: A Large-Scale Semi-Supervised Dataset for Offensive Language Identification (2020, methodology): “The tweets in the analysis set for Level A differ from those in Levels B and C to accommodate the Semeval-2020 Evaluation (Zampieri et al., 2020) task schedule.”
Introducing CAD: the Contextual Abuse Dataset (2021, background): “It builds on critical social scientific research (Marwick and Miller, 2014; Citron and Norton, 2011; Lenhart et al., 2016), and addresses issues in previous taxonomies, including those provided by Zampieri et al. (2020), Waseem et al. (2017), Founta et al. (2018) and Vidgen et al. (2019a).”