Towards Robust Toxic Content Classification

By Keita Kurita, Anna Belova, Antonios Anastasopoulos. 2019

41 citations recorded by the source corpus.

Toxic content detection aims to identify content that can offend or harm its recipients. Automated classifiers of toxic content need to be robust against adversaries who deliberately try to bypass filters. We propose a method of generating realistic model-agnostic attacks using a lexicon of toxic tokens, which attempts to mislead toxicity classifiers by diluting the toxicity signal either by obfuscating toxic tokens through character-level perturbations, or by injecting non-toxic distractor tokens. We show that these realistic attacks reduce the detection recall of state-of-the-art neural toxicity detectors, including those using ELMo and BERT, by more than 50% in some cases. We explore two approaches for defending against such attacks. First, we examine the effect of training on synthetically noised data. Second, we propose the Contextual Denoising Autoencoder (CDAE): a method for learning robust representations that uses character-level and contextual information to denoise perturbed tokens. We show that the two approaches are complementary, improving robustness to both character-level perturbations and distractors, recovering a considerable portion of the lost accuracy. Finally, we analyze the robustness characteristics of the most competitive methods and outline practical considerations for improving toxicity detectors.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “While such classifiers support quick development and are interpretable, they tend to overfit to lexical features, are prone to false positives, and are easy for malicious users to bypass (Kurita et al., 2019).”
A New Generation of Perspective API: Efficient Multilingual Character-level Transformers (2022, background): “While character-level features or modeling approaches have been explored in the context of toxicity detection [18], our paper proposes the first byte-level pretrained model that remains competitive with (or outperforms) subword models tailored to specific domains or languages.”
Detecting Cross-Geographic Biases in Toxicity Modeling on Social Media (2021, background): “(Hosseini et al., 2017; Gröndahl et al., 2018; Kurita et al., 2019a), potentially furthering disproportionate harm to those in the margins.”