Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots

By Waiman Si, Michael Backes, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, Savvas Zannettou, Yang Zhang. 2022

86 citations recorded by the source corpus.

Chatbots are used in many applications, e.g., automated agents, smart home assistants, interactive characters in online games, etc. Therefore, it is crucial to ensure they do not behave in undesired manners, providing offensive or toxic responses to users. This is not a trivial task as state-of-the-art chatbot models are trained on large, public datasets openly collected from the Internet. This paper presents a first-of-its-kind, large-scale measurement of toxicity in chatbots. We show that publicly available chatbots are prone to providing toxic responses when fed toxic queries. Even more worryingly, some non-toxic queries can trigger toxic responses too. We then set out to design and experiment with an attack, ToxicBuddy, which relies on fine-tuning GPT-2 to generate non-toxic queries that make chatbots respond in a toxic manner. Our extensive experimental evaluation demonstrates that our attack is effective against public chatbot models and outperforms manually-crafted malicious queries proposed by previous work. We also evaluate three defense mechanisms against ToxicBuddy, showing that they either reduce the attack performance at the cost of affecting the chatbot's utility or are only effective at mitigating a portion of the attack. This highlights the need for more research from the computer security and online safety communities to ensure that chatbot models do not hurt their users. Overall, we are confident that ToxicBuddy can be used as an auditing tool and that our work will pave the way toward designing more effective defenses for chatbot safety.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “Moreover, some works completely remove human involvement and solely rely on large language models to generate new data (Perez et al., 2022; Si et al., 2022; Deng et al., 2023).”
Exploring AI Ethics of ChatGPT: A Diagnostic Analysis (2023, background): “[24] demonstrated that BlenderBot[25] and TwitterBot [26] can easily trigger toxic responses, though with low toxicity.”
Low-Resource Languages Jailbreak GPT-4 (2023, background): “As large language models (LLMs) are deployed in user-facing applications like chatbots and writing tools, safety-training and red-teaming are crucial to avoid AI safety failures [27, 32, 36, 48].”
Prompt Injection attack against LLM-integrated Applications (2023, background): “Jailbreak [33,55,58,60] involves eliciting model-generated content that divulges training data specifics, which can lead to serious privacy breaches, particularly when training data include sensitive or private information.”