ETHOS: a multi-label hate speech detection dataset

By Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, Grigorios Tsoumakas. 2020

169 citations recorded by the source corpus.

Online hate speech is a recent problem in our society that is rising at a steady pace by leveraging the vulnerabilities of the corresponding regimes that characterise most social media platforms. This phenomenon is primarily fostered by offensive comments, either during user interaction or in the form of a posted multimedia context. Nowadays, giant corporations own platforms where millions of users log in every day, and protection from exposure to similar phenomena appears to be necessary to comply with the corresponding legislation and maintain a high level of service quality. A robust and reliable system for detecting and preventing the uploading of relevant content will have a significant impact on our digitally interconnected society. Several aspects of our daily lives are undeniably linked to our social profiles, making us vulnerable to abusive behaviours. As a result, the lack of accurate hate speech detection mechanisms would severely degrade the overall user experience, although its erroneous operation would pose many ethical concerns. In this paper, we present ‘ETHOS’ (multi-labEl haTe speecH detectiOn dataSet), a textual dataset with two variants: binary and multi-label, based on YouTube and Reddit comments validated using the Figure-Eight crowdsourcing platform. Furthermore, we present the annotation protocol used to create this dataset: an active sampling procedure for balancing our data in relation to the various aspects defined. Our key assumption is that, even gaining a small amount of labelled data from such a time-consuming process, we can guarantee hate speech occurrences in the examined material.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dirk Hovy
2. Federico Bianchi
3. Yejin Choi
4. Debora Nozza
5. Zeerak Talat
6. Noah A. Smith
7. Su Lin Blodgett
8. Mohit Bansal

In Information Systems

1. Zeerak Talat
2. Federico Bianchi
3. Dirk Hovy
4. Irene Solaiman
5. Chrisantha Fernando
6. Marie-Therese Png
7. Debora Nozza
8. Yacine Jernite

In Natural Language Processing

1. Dirk Hovy
2. Debora Nozza
3. Federico Bianchi
4. Zeerak Talat
5. Su Lin Blodgett
6. Alexandra Sasha Luccioni
7. Anne Lauscher
8. Irene Solaiman

In Safety, Robustness & Interpretability

1. Dirk Hovy
2. Federico Bianchi
3. Anne Lauscher
4. Su Lin Blodgett
5. Debora Nozza
6. Irene Solaiman
7. Zeerak Talat
8. Dylan Baker

In Signal Processing

1. Irene Solaiman
2. Simon Osindero
3. Yacine Jernite
4. Tim Rocktaschel
5. Margaret Mitchell
6. Sara Hooker

Citations in the corpus

• Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Abusive and offensive language are a large umbrella for unsafe content, which can also include abuse and hate speech[151, 236].”
• Measuring Harmful Sentence Completion in Language Models for LGBTQIA+ Individuals (2022, background): “Few recent works covered hate speech on the basis of sexual orientation (Ousidhoum et al., 2019; Mollas et al., 2022; Kennedy et al., 2022; Chakravarthi et al., 2022; Nozza, 2022).”
• Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution (2023, methodology): “, 2023), which used the ETHOS dataset (Mollas et al., 2022).”