Deceiving Google's Perspective API Built for Detecting Toxic Comments

By Hossein Hosseini, Sreeram Kannan, Baosen Zhang, Radha Poovendran. 2017

369 citations recorded by the source corpus.

Social media platforms provide an environment where people can freely engage in discussions. Unfortunately, they also enable several problems, such as online harassment. Recently, Google and Jigsaw started a project called Perspective, which uses machine learning to automatically detect toxic language. A demonstration website has been also launched, which allows anyone to type a phrase in the interface and instantaneously see the toxicity score [1]. In this paper, we propose an attack on the Perspective toxic detection system based on the adversarial examples. We show that an adversary can subtly modify a highly toxic phrase in a way that the system assigns significantly lower toxicity score to it. We apply the attack on the sample phrases provided in the Perspective website and show that we can consistently reduce the toxicity scores to the level of the non-toxic phrases. The existence of such adversarial examples is very harmful for toxic detection systems and seriously undermines their usability.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

HateCheck: Functional Tests for Hate Speech Detection Models (2020, background): “to hate speech since they can reflect intentional behaviour of users looking to avoid detection (Hosseini et al., 2017; Gröndahl et al., 2018; Vidgen et al., 2019).”
Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack (2019, neutral): “Nevertheless algorithmic approaches have been attempted, for example in text classiﬁcation (Ebrahimi et al., 2018), machine translation (Belinkov and Bisk, 2018), dialogue generation tasks (Li et al., 2017) and reading comprehension (Jia and Liang, 2017).”
Red Teaming Language Models with Language Models (2022, methodology): “Such methods find inputs that elicit inaccurate predictions from text classifiers (Hosseini et al., 2017; Ebrahimi et al., 2018; Behjati et al., 2019, inter alia) and offensive text from LMs (Wallace et al., 2019; He and Glass, 2019; Liu et al., 2019; Song et al., 2020; Liu et al., 2020; Yu and…”
Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection (2021, background): “Hosseini et al. (2017) also investigate how detection models can be attacked and report similar findings.”
Reducing Gender Bias in Abusive Language Detection (2018, background): “Hosseini et al. (2017) have shown how to easily cause false predictions with adversarial examples in Google’s API, and Dixon et al. (2017) show that classiﬁers can have unfair biases toward certain groups of people.”
Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks (2017, background): “proposed that correcting the spelling on inputs before they are provided to a character-based toxic text detection system can defend against adversarial examples [15].”
HotFlip: White-Box Adversarial Examples for Text Classification (2017, background): “Hosseini et al. (2017) show that simple modiﬁcations, such as adding spaces or dots between characters, can drastically change the toxicity score from Google’s perspective API 1 .”
Quark: Controllable Text Generation with Reinforced Unlearning (2022, background): “The API is itself imperfect and reflects some social biases [26, 45, 63].”
RL with KL penalties is better viewed as Bayesian inference (2022, background): “Reward models are vulnerable to adversarial examples (Grosse et al., 2016; Hosseini et al., 2017) and LMs optimised against them can exploit these adversarial examples (Pan et al., 2022).”
Towards Ethics by Design in Online Abusive Content Detection (2020, methodology): “Hosseini et al. (2017) demonstrated that an adversary can query the system multiple times and ﬁnd a way to subtly modify an abusive phrase resulting in signiﬁcantly lowering conﬁdence that the phrase is abusive.”
Towards Robust Toxic Content Classification (2019, background): “2017) generated adversarial attacks against the Google Perspective API, a public API for detecting toxic content, and showed the brittleness of this system.”
Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective (2020, background): “Hosseini et al. (2017) demonstrated that an adversary can query the system multiple times and find a way to subtly modify an abusive phrase resulting in a significantly lower confidence that the phrase is abusive.”
Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale (2022, methodology): “Moreover, automated methods to scan text for harm and toxicity, even with significant shortcomings, are still far more well-developed than those to scan images (Hosseini et al., 2017).”
Online Hate Ratings Vary by Extremes: A Statistical Analysis (2019, methodology): “An example is Perspective API, a tool by Alphabet for scoring toxicity of online comments, that is trained using crowd annotators [9].”
Google's Cloud Vision API is Not Robust to Noise (2017, background): “Several papers have presented attacks on various ML systems, such as voice interfaces [8], face-recognition systems [9], toxic comment detectors [10], and video annotation systems [11].”
Attacking Automatic Video Analysis Algorithms: A Case Study of Google Cloud Video Intelligence API (2017, methodology): “This property has been used to attack voice interfaces [15], face-recognition systems [40] and text classification systems [23, 39].”
Interpretable Adversarial Perturbation in Input Embedding Space for Text (2018, methodology): “Random character swaps can break the output of neural machine translation systems [ Belinkov and Bisk, 2018; Hosseini et al. , 2017 ] , and thus they proposed AdvT methods that generate random character swaps and utilize the generated input sentences as additional training data for their models.”
Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots (2022, background): “, sensitivity to adversarial text [25] and bias towards text mentioning marginalized groups or written in African-American English [51].”
Algorithmic content moderation: Technical and political challenges in the automation of platform governance (2020, background): “In addition to these manually constructed counter-examples, other researchers have demonstrated how to automatically construct adversarial examples (Hosseini et al., 2017).”