On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark

By Hao Sun, Guangxuan Xu, Deng Jiawen, Jiale Cheng, Chujie Zheng, Hao Zhou, Nanyun Peng, Xiaoyan Zhu, Minlie Huang. 2021

117 citations recorded by the source corpus.

Dialogue safety problems severely limit the real-world deployment of neural conversational models and have attracted great research interests recently. However, dialogue safety problems remain under-defined and the corresponding dataset is scarce. We propose a taxonomy for dialogue safety specifically designed to capture unsafe behaviors in human-bot dialogue settings, with focuses on context-sensitive unsafety, which is under-explored in prior works. To spur research in this direction, we compile DiaSafety, a dataset with rich context-sensitive unsafe examples. Experiments show that existing safety guarding tools fail severely on our dataset. As a remedy, we train a dialogue safety classifier to provide a strong baseline for context-sensitive dialogue unsafety detection. With our classifier, we perform safety evaluations on popular conversational models and show that existing dialogue systems still exhibit concerning context-sensitive safety problems.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation (2022, background): “McGuffie and Newhouse, 2020; Bender et al., 2021; Sun et al., 2022), which greatly hinders such models from the real-world application, especially in the interactive scenarios such as chit-chatting.”
Using In-Context Learning to Improve Dialogue Safety (2023, background): “DiaSafety (Sun et al., 2022): DiaSafety is a collection of adversarial utterances which can illicit unsafe responses from conversational models.”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, background): “Note that this taxonomy has already inspired further work in this area (Sun et al., 2021).”
Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots (2022, background): “[58] build a dataset with six unsafe categories, while Ung et al.”
CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility (2023, unknown): “Different from the previous Chinese LLM benchmark work (Huang et al., 2023; Liu et al., 2023; Sun et al., 2022, 2023), we are the ﬁrst to introduce a new human value benchmark considering two ascending levels of assessment criteria, namely safety and responsibility.”
SafeText: A Benchmark for Exploring Physical Safety in Language Models (2022, background): “Within the context of natural language processing, some work analyzes safety as a whole (Sun et al., 2022; Dinan et al., 2022) but may underrep-resent or completely miss certain types of safety.”
Mirages: On Anthropomorphism in Dialogue Systems (2023, background): “It is precisely due to their lack of agency and responsibility that developers have invested significant efforts to avoiding contentious topics (e.g. Glaese et al., 2022; Sun et al., 2022; Xu et al., 2021) leading to the creation of taboos for such systems, another particularly human phenomenon.”
COLD: A Benchmark for Chinese Offensive Language Detection (2022, methodology): “However, as [Sun et al., 2021; Dinan et al., 2021] pointed out, in dialogue scenarios, the system tends to cater to users and generate responses of toxicity agreement.”