Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter
By Zeerak Talat, Dirk Hovy. 2016
1,873 citations recorded by the source corpus.
Hate speech in the form of racist and sexist remarks are a common occurrence on social media. For that reason, many social media services address the problem of identifying hate speech, but the definition of hate speech varies markedly and is largely a manual effort (BBC, 2015; Lomas, 2015). We provide a list of criteria founded in critical race theory, and use them to annotate a publicly available corpus of more than 16k tweets. We analyze the impact of various extra-linguistic features in conjunction with character n-grams for hatespeech detection. We also present a dictionary based the most indicative words in our data.
Subfields: Artificial Intelligence, Communication, Law, Natural Language Processing, AI Ethics, Fairness & Society
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yejin Choi
- Noah A. Smith
- Mohit Bansal
- Percy Liang
- Yulia Tsvetkov
- Jianfeng Gao
- Luke S. Zettlemoyer
- Jason Aaron Edward Weston
In Communication
- Bertie Vidgen
- Isar Nejadgholi
- Svetlana Kiritchenko
- Leon Derczynski
- Valerio Basile
- Mladen Karan
- Jan Snajder
- Ziqi Zhang
In Law
- Bertie Vidgen
- Valerio Basile
- Michael Wiegand
- Ziqi Zhang
- Marco Guerini
- Serra Sinem Tekiroğlu
- Emily M. Bender
- Svetlana Kiritchenko
In Natural Language Processing
- Yejin Choi
- Bertie Vidgen
- Noah A. Smith
- Yulia Tsvetkov
- Svetlana Kiritchenko
- Isar Nejadgholi
- Dirk Hovy
- Mohit Bansal
In AI Ethics, Fairness & Society
- Bertie Vidgen
- Isar Nejadgholi
- Leon Derczynski
- Björn Gambäck
- Svetlana Kiritchenko
- Mladen Karan
- Marcos Zampieri
- Jan Snajder
Citations in the corpus
- Challenges in Detoxifying Language Models (2021, background): “This phenomenon has been linked to an increased relative prevalence of identity terms among toxic samples (Waseem and Hovy, 2016; Dixon et al., 2018; Park et al., 2018).”
- HateCheck: Functional Tests for Hate Speech Detection Models (2020, methodology): “…in data source, sampling strategy and annotation process (Vidgen and Derczynski, 2020; Poletto et al., 2020), and are known to exhibit annotator biases (Waseem, 2016; Waseem et al., 2018; Sap et al., 2019) as well as topic and author biases (Wiegand et al., 2019; Nejadgholi and Kiritchenko, 2020).”
- The Risk of Racial Bias in Hate Speech Detection (2019, methodology): “A.3 Bias in Waseem and Hovy (2016) We replicate our analyses in § 3 on the widely used dataset by Waseem and Hovy (2016, henceforth, WH 16), which categorizes tweets in three hate speech categories: racist , sexist , or none , shown in Table 2, along with their correlations with AAE.”
- Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “Much of this focuses on human-produced content on social media platforms, such as Twitter (e.g. Waseem & Hovy, 2016; Wang et al., 2020; Zampieri et al., 2019; 2020), Facebook (Glava ˇ s et al., 2020; Zampieri et al., 2020), or Reddit (Han & Tsvetkov, 2020; Zampieri et al., 2020).”
- Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “…rely on the ability to detect problematic content – often centred on content written by humans on social media platforms, such as Twitter (e.g. Waseem and Hovy, 2016; Wang et al., 2020; Zampieri et al., 2019, 2020; Zhang et al., 2020), Facebook (Glavaš et al., 2020; Zampieri et al., 2020),…”
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “Several text-only hate speech datasets have been released, mostly based on Twitter [83, 82, 11, 24, 20], and various architectures have been proposed for classifiers [45, 54, 53].”
- Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science (2018, methodology): “Waseem and Hovy (2016) provide the following complete list of terms used in their initial scrape: MKR, asian drive, feminazi, immigrant, nigger, sjw, WomenAgainstFeminism, blameonenotall, islam terrorism, notallmen, victimcard, victim card, arab terror, gamergate, jsil, racecard, race card. dataset…”
- Social Bias Frames: Reasoning about Social and Power Implications of Language (2019, background): “Bias and toxicity detection Detection of hateful, abusive, or other toxic language has received increased attention recently (Schmidt and Wiegand, 2017), and most dataset creation work has cast this detection problem as binary classifcation (Waseem and Hovy, 2016; Davidson et al., 2017; Founta et al”
- Ex Machina: Personal Attacks Seen at Scale (2016, background): “e, the current open datasets are limited to the Internet Argument Corpus [26], the CAW 2.0 dataset provided by the Fundacion Barcelona Media [6], the "Hate Speech Twitter Annotations" corpus[28], and the "Detecting Insults in Social Commentary" dataset released by Impermium via K”
- Automated Hate Speech Detection and the Problem of Offensive Language (2017, methodology): “This finding is consistent with previous work that has found amateur coders to often be un-reliable at identifying abusive content (Nobata et al. 2016; Waseem 2016).”
- Racial Bias in Hate Speech and Abusive Language Detection Datasets (2019, methodology): “We see similar results for Waseem and Hovy (2016) and Waseem (2016).”
- Mitigating Gender Bias in Natural Language Processing: Literature Review (2019, methodology): “Park et al. (2018) use transfer learning from a gender unbiased abusive tweets data set (Founta et al., 2018) and fine-tuning on a gender-biased sexist tweets data set (Waseem and Hovy, 2016) to train a Convolutional Neural Network (CNN).”
- Reducing Gender Bias in Abusive Language Detection (2018, background): “So far, many efforts were put into defining and constructing abusive language datasets from different sources and labeling them through crowd-sourcing or user moderation (Waseem and Hovy, 2016; Waseem, 2016; Founta et al., 2018; Wulczyn et al., 2017).”
- Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter (2016, methodology): “Waseem and Hovy (2016) and Ross et al. (2016) focus on building corpora which they annotate for containing hate speech.”
- A Survey on Hate Speech Detection using Natural Language Processing (2017, background): “…of meta-information are common, such as the number of posts by a user, the number of replies to a post, the average of the total number of replies per follower or the geographical origin, but most of these have not been found effective for classification (Zhong et al., 2016; Waseem and Hovy, 2016).”
- Hate Speech Dataset from a White Supremacy Forum (2018, background): “Waseem and Hovy (2016) published a collection of 16k tweets classified into racist, sexist or neither.”
- Challenges and frontiers in abusive content detection (2019, background): “For instance, in 2018 Pitsilis et al. trained a classification system on Waseem and Hovy’s 16,000 tweet dataset and achieved an F-Score of 0.932, compared against Waseem and Hovy’s original 0.739; a 20-point increase (Pitsilis, Ramampiaro, & Langseth, 2018; Waseem & Hovy, 2016).”