SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter
By Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, Manuela Sanguinetti. 2019
1,079 citations recorded by the source corpus.
The paper describes the organization of the SemEval 2019 Task 5 about the detection of hate speech against immigrants and women in Spanish and English messages extracted from Twitter. The task is organized in two related classification subtasks: a main binary subtask for detecting the presence of hate speech, and a finer-grained one devoted to identifying further features in hateful contents such as the aggressive attitude and the target harassed, to distinguish if the incitement is against an individual rather than a group. HatEval has been one of the most popular tasks in SemEval-2019 with a total of 108 submitted runs for Subtask A and 70 runs for Subtask B, from a total of 74 different teams. Data provided for the task are described by showing how they have been collected and annotated. Moreover, the paper provides an analysis and discussion about the participant systems and the results they achieved in both subtasks.
Subfields: Artificial Intelligence, Communication, Law, Natural Language Processing, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yejin Choi
- Mohit Bansal
- Luke S. Zettlemoyer
- Noah A. Smith
- Percy Liang
- Kai-Wei Chang
- Jianfeng Gao
- Dirk Hovy
In Communication
- Valerio Basile
- Leon Derczynski
- Jelena Mitrović
- Michael Granitzer
- Viviana Patti
- Marcos Zampieri
- Verena Rieser
- Gavin Abercrombie
In Law
- Valerio Basile
- Marcos Zampieri
- Mai Elsherief
- Aida Mostafazadeh Davani
- Alexandra Sasha Luccioni
- Joseph D. Viviano
- Paula Fortuna
- Leon Derczynski
In Natural Language Processing
- Dirk Hovy
- Kai-Wei Chang
- Gavin Abercrombie
- Verena Rieser
- Valerio Basile
- Nanyun Peng
- Yejin Choi
- Diyi Yang
In Safety, Robustness & Interpretability
- Dirk Hovy
- Valerio Basile
- Emily Sheng
- Kai-Wei Chang
- Nanyun Peng
- Gavin Abercrombie
- Sunipa Dev
- Anne Lauscher
Citations in the corpus
- Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech (2023, background): “Hate Speech Explanation Generation To hinder the spread of toxic speech among online social platforms, researchers developed effective ML models in recent years [2, 5].”
- Refining Language Models with Compositional Explanations (2021, methodology): “For hate speech detection, we use Stormfront [2] and HatEval [1] as upstream datasets, and the Gab Hate Corpus (GHC) [20] as the downstream dataset.”
- Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021, background): “…and visualization. many tools have been developed to confront this challenge ..." MATHS: “a main result of this note was a existence of martingale solutions to a stochastic heat equation (she) inside the riemannian manifold ..." SemEval 2019 Task 5 (Barbieri et al., 2020) (Basile et al., 2019).”
- Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection (2021, neutral): “Another source of false negatives is when classification systems are applied to out-ofdomain settings, such as system trained on Twitter data being applied to data from Gab (Karan and Šnajder, 2018; Pamungkas et al., 2020; Swamy et al., 2019; Basile et al., 2019; Salminen et al., 2020).”
- What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus (2021, methodology): “Most of this research is carried out on data extracted from social media sources such as Twitter (Founta et al., 2018; Basile et al., 2019; Waseem and Hovy, 2016) and Reddit (Tadesse et al., 2019; Farrell et al., 2019), with both ML-based (Badjatiya et al., 2017) and count-based approaches…”
- HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection (2020, background): “2018b,a), and shared tasks (Basile et al. 2019; Bosco et al. 2018) have been made available in the recent years by the community, towards the development of automatic hate speech detection.”
- Detecting and Classifying Malevolent Dialogue Responses: Taxonomy, Data and Methodology (2020, methodology): “More recently, Basile et al. [5] release the dataset MDHS for detecting hate speech that targets multilingual research and hate against immigrants and women with 3,783 ‘hateful’ and 5,217 ‘not hateful’ tweets.”
- Latent Hatred: A Benchmark for Understanding Implicit Hate Speech (2021, background): “The majority are skewed towards explicitly abusive text since they were originally seeded with hate lexicons (Basile et al., 2019; Founta et al., 2018; Davidson et al., 2017; Waseem and Hovy, 2016), racial identifiers (Warner and Hirschberg, 2012), or explicitly hateful phrases such as “I hate…”
- SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020) (2020, background): “Examples include HASOC 2019 (Mandl et al., 2019) for English, German, and Hindi, HatEval 2019 (Basile et al., 2019) for English and Spanish, GermEval 2019 for German (Struß et al., 2019), and TRAC 2020 (Kumar et al., 2018b) for Bengali, Hindi, and English.”
- Resources and benchmark corpora for hate speech detection: a systematic review (2020, methodology): “In the HatEval English set, for instance, the number of keywords used for the two targets differ, and therefore the word distributions in the resulting corpora will be less natural.”
- Towards multidomain and multilingual abusive language detection: a survey (2021, background): “Some of them used offensive words [32, 39, 41, 89], which are usually a strong signal of abusive content, while other studies decided to use more neutral keywords to maintain a real-world approach to the problem [11], or even both offensive and neutral keywords [152].”
- Misogyny Detection in Twitter: a Multilingual and Cross-Domain Study (2020, positive): “The dataset augmentation experiment show that augmenting the data training coverage by adding external data only works when the additional data share similar targets or topics, as observed on the AMI, HatEvalM, and WaseemS datasets.”
- Measuring Harmful Sentence Completion in Language Models for LGBTQIA+ Individuals (2022, background): “Research in hate speech detection considering gender and sexuality have mostly focus on sexism (Fersini et al., 2018; Basile et al., 2019; Nozza et al., 2019; Chiril et al., 2020; Fersini et al., 2020a,b; Attanasio and Pastor, 2020; Zein-29 ert et al., 2021; Mulki and Ghanem, 2021; Nozza, 2021;…”
- A New Generation of Perspective API: Efficient Multilingual Character-level Transformers (2022, methodology): “We evaluate our models on the TweetEval hate content classification test split [3], which is taken from the SemEval2019 Hateval challenge [4].”
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations (2023, background): “Examples of these include Jigsaw (Jigsaw, 2017), Twitter (Zampieri et al., 2019; Basile et al., 2019), Stormfront (de Gibert et al., 2018), Reddit (Hada et al., 2021), Hateful Memes (Kiela et al., 2021).”
- Sentiment Analysis in the Era of Large Language Models: A Reality Check (2023, methodology): “For our analysis, we utilize the dataset from the SemEval2019 HatEval challenge (Basile et al., 2019).”