All You Need is "Love": Evading Hate Speech Detection

By Tommi Gröndahl, Luca Pajola, Mika Juuti, Mauro Conti, N. Asokan. 2018

261 citations recorded by the source corpus.

With the spread of social networks and their unfortunate use for hate speech, automatic detection of the latter has become a pressing problem. In this paper, we reproduce seven state-of-the-art hate speech detection models from prior work, and show that they perform well only when tested on the same type of data they were trained on. Based on these results, we argue that for successful hate speech detection, model architecture is less important than the type of data and labeling criteria. We further show that all proposed detection techniques are brittle against adversaries who can (automatically) insert typos, change word boundaries or add innocuous words to the original hate speech. A combination of these methods is also effective against Google Perspective - a cutting-edge solution from industry. Our experiments demonstrate that adversarial training does not completely mitigate the attacks, and using character-level features makes the models systematically more attack-resistant than using word-level features.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

HateCheck: Functional Tests for Hate Speech Detection Models (2020, background): “to hate speech since they can reflect intentional behaviour of users looking to avoid detection (Hosseini et al., 2017; Gröndahl et al., 2018; Vidgen et al., 2019).”
Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack (2019, background): “In that context, any attempt to automatically detect this behavior can be expected to be adversarially attacked by looking for weaknesses in the detection system, which currently can easily be exploited as shown in (Hosseini et al., 2017; Gr¨ondahl et al., 2018).”
Challenges and frontiers in abusive content detection (2019, background): “Spelling variations are also sometimes used adversarially to obfuscate and avoid detection (e.g. by using unusual punctuation or additional spaces) (Eger et al., 2019; Gröndahl, Pajola, Juuti, Conti, & Asokan, 2018).”
ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AI (2021, background): “Following initial hyperparameter optimization experiments, we use the following systems and settings: • Support Vector Machine : SVMs have been used is previous work on abuse detection in Twitter data, (e.g. Davidson et al., 2017), and have been shown to outperform neural systems (Niemann et al., 20”
Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective (2020, background): “Gröndahl et al. (2018) showed that adding a positive word such as ‘love’ to an abusive comment can flip the models predictions.”
HABERTOR: An Efficient and Effective Deep Hatespeech Detector (2020, background): “Another direction focuses on the testing generalization of the current hatespeech classifiers (Agrawal and Awekar, 2018; Dadvar and Eckert, 2018; Gröndahl et al., 2018), where those methods are tested in other datasets and domains such as Twitter data (Waseem and Hovy, 2016), Wikipedia data (Wulczyn”
Studying Generalisability across Abusive Language Detection Datasets (2019, methodology): “Additionally, Gr¨ondahl et al. (2018) showed how adversarial methods such as typos and word changes could bypass existing state-of-the-art abusive language detection systems.”
Fighting Hate Speech, Silencing Drag Queens? Artificial Intelligence in Content Moderation and Risks to LGBTQ Voices Online (2020, background): “…improves the technology to detect these changes, but it remains flawed—the study ‘All You Need is “Love”: Evading Hate Speech Detection’ published by Gröndahl et al. (2018) mentions, for instance, that if the word ‘love’ is inserted into the sentences, their perceived toxicity also diminishes.”
DeepHate: Hate Speech Detection via Multi-Faceted Text Representations (2020, methodology): “• LSTM: The LSTM model ,is another model that was commonly explored in previous hate speech detection studies [1, 3, 21].”
A New Generation of Perspective API: Efficient Multilingual Character-level Transformers (2022, methodology): “One common technique used to bypass toxicity classification models is to intentionally misspell words in a fashion that is understood by human readers, yet obfuscated to machine learning models [11].”
Detecting East Asian Prejudice on Social Media (2020, background): “hateful compared with Standard American [31, 32], penalising hate against certain targets more strongly than others [33], or being easily fooled by simple spelling changes that any human can identify [34]. These issues are considerable limitations as they mean that, if used in the ‘real world’, comp”
Machine Learning Suites for Online Toxicity Detection (2018, methodology): “Like the subtle changes of tone used by Hosseini, et al (2017) and Gröndahl, et al. (2018), such adversarial attacks can reveal brittleness in toxic speech detectors.”
Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset (2022, unknown): “The category of ‘toxic speech’ is deﬁned in multiple ways [47, 123, 4].”
Detecting Cross-Geographic Biases in Toxicity Modeling on Social Media (2021, background): “(Hosseini et al., 2017; Gröndahl et al., 2018; Kurita et al., 2019a), potentially furthering disproportionate harm to those in the margins.”