SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)

By Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, Ritesh Kumar. 2019

874 citations recorded by the source corpus.

We present the results and the main findings of SemEval-2019 Task 6 on Identifying and Categorizing Offensive Language in Social Media (OffensEval). The task was based on a new dataset, the Offensive Language Identification Dataset (OLID), which contains over 14,000 English tweets, and it featured three sub-tasks. In sub-task A, systems were asked to discriminate between offensive and non-offensive posts. In sub-task B, systems had to identify the type of offensive content in the post. Finally, in sub-task C, systems had to detect the target of the offensive posts. OffensEval attracted a large number of participants and it was one of the most popular tasks in SemEval-2019. In total, nearly 800 teams signed up to participate in the task and 115 of them submitted results, which are presented and analyzed in this report.

Subfields: Artificial Intelligence, Communication, Social Psychology, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Communication

In Social Psychology

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Challenges in Detoxifying Language Models (2021, background): “While detecting hate speech and offensive language (Warner and Hirschberg, 2012; Kwok and Wang, 2013; Davidson et al., 2017; Zampieri et al., 2019), mostly in the context of online community moderation, has long been a subject of research; the”
Recipes for Safety in Open-domain Chatbots (2020, methodology): “Waseem et al. (2017) advocate for partitioning abusive content according to what entity it is directed to, an approach adopted by the OLID/OffensEval datasets (Zampieri et al., 2019, 2020).”
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “…content – often centred on content written by humans on social media platforms, such as Twitter (e.g. Waseem and Hovy, 2016; Wang et al., 2020; Zampieri et al., 2019, 2020; Zhang et al., 2020), Facebook (Glavaš et al., 2020; Zampieri et al., 2020), or Reddit (Han and Tsvetkov, 2020;…”
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, background): “– TweetEval: Emotion Recognition (Mohammad et al., 2018), Emoji Prediction (Barbieri et al., 2018), Irony Detection (Van Hee et al., 2018), Hate Speech Detection (Basile et al., 2019), Offensive Language Identification (Zampieri et al., 2019), Sentiment Analysis (Rosenthal et al., 2017), Stance Dete”
Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack (2019, methodology): “Perhaps the most commonly studied class is hate speech, but work has also covered bullying, aggression, and toxic comments (Zampieri et al., 2019).”
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021, background): “…the euro" SOUTHERN AFRICA: “95/458/EC: Commission Regulation (EC) No 302/2006 of 20 February 2006 on import licences in respect of beef and veal products originating in Botswana, Kenya, Madagascar, Swaziland, Zimbabwe and Namibia" SemEval 2019 Task 6 (Barbieri et al., 2020) (Zampieri et al., 2019).”
Challenges and frontiers in abusive content detection (2019, background): “This distinction has been widely adopted (Zampieri et al., 2019).”
Predicting the Type and Target of Offensive Posts in Social Media (2019, background): “OLID is the ofﬁcial dataset of the shared task Se-mEval 2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval) (Zampieri et al., 2019).”
TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification (2020, methodology): “For our benchmark we rely on the SemEval2019 OffensE-val dataset (Zampieri et al., 2019).”
SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020) (2020, methodology): “One such example is SemEval-2019 Task 6: OffensEval(Zampieri et al., 2019b), which is the precursor to the present shared task.”
Resources and benchmark corpora for hate speech detection: a systematic review (2020, methodology): “Zampieri et al. (2019a) (ZMNRFK)—14,100 tweets in English, annotated with crowdsourcing using a multi-level scheme.”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, background): “…years, however, much of this focuses on human-produced content on social media platforms, such as Twitter (e.g. Waseem and Hovy, 2016; Wang et al., 2020; Zampieri et al., 2019, 2020), Facebook (Glavaš et al., 2020; Zampieri et al., 2020), or Reddit (Han and Tsvetkov, 2020; Zampieri et al., 2020).”
Dynamic Language Models for Continuously Evolving Content (2021, methodology): “For model evaluation, we use two downstream tasks, Country Hashtag Prediction (predicting the associated country hashtag for a tweet from 16 pre-selected country hashtags) and OffensEval 2019 [48] (a shared task from SemEval 2019 to predict if a tweet is offensive or not).”
Civil Rephrases Of Toxic Texts With Self-Supervised Transformers (2021, background): “…its strategy may involve shifting the original intent, since “republican” is not a non-offensive synonym of “moron”. et al., 2017; Pavlopoulos et al., 2017; Wulczyn et al., 2017; Gamb¨ack and Sikdar, 2017; Fortuna and Nunes, 2018; Zhang et al., 2018a; Van Hee et al., 2018; Zampieri et al., 2019).”
SOLID: A Large-Scale Semi-Supervised Dataset for Offensive Language Identification (2020, methodology): “3 The OLID Dataset The OLID (Zampieri et al., 2019a) dataset tackles the challenge of detecting offensive language us-ing a labeling schema that classiﬁes each example using the following three-level hierarchy: Level A: OTH Targets that does not fall into the previous categories, e.g., organizations”
Overview of the HASOC track at FIRE 2019: Hate Speech and Offensive Content Identification in Indo-European Languages (2019, methodology): “HASOC is primarily inspired by two previous evaluation forums, GermEval [44], and OffensEval [47], and tries to leverage the synergies of these initiatives.”
Toxic Language Detection in Social Media for Brazilian Portuguese: New Dataset and Multilingual Analysis (2020, methodology): “OffensEval is a series of shared tasks focusing on offensive comments detection (Zampieri et al., 2019b, 2020).”
Probing Toxic Content in Large Pre-Trained Language Models (2021, methodology): “We trained an LR classiﬁer on four relatively different English datasets (Davidson et al., 2017; Founta et al., 2018; Ousidhoum et al., 2019; Zampieri et al., 2019), four others in Arabic (Ousid-houm et al., 2020; Albadi et al., 2018; Mulki et al., 2019; Zampieri et al., 2020), and the only one we…”
Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings (2019, background): “The advantage of such contextually embedded token representations compared to static word embeddings has been shown for a number of tasks such as text classification (Zampieri et al., 2019) and sequence tagging (Akbik et al., 2018).”