SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems

By Emily Dinan, Gavin Abercrombie, A. S. Bergman, Shannon L. Spruit, Dirk Hovy, Y-Lan Boureau, Verena Rieser. 2022

67 citations recorded by the source corpus.

The social impact of natural language processing and its applications has received increasing attention. In this position paper, we focus on the problem of safety for end-to-end conversational AI. We survey the problem landscape therein, introducing a taxonomy of three observed phenomena: the Instigator, Yea-Sayer, and Impostor effects. We then empirically assess the extent to which current tools can measure these effects and current systems display them. We release these tools as part of a “first aid kit” (SafetyKit) to quickly assess apparent safety concerns. Our results show that, while current tools are able to provide an estimate of the relative safety of systems in various settings, they still have several shortcomings. We suggest several future directions and discuss ethical considerations.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “…situations, by directly generating toxic or otherwise harmful content, by agreeing with offensive statements uttered by the conversation partner (Dinan et al., 2022), or by responding defensively or dismissively when provided with corrective feedback by the conversation partner (Ung et al.,…”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, background): “…al., 2021; Hendrycks et al., 2021; Weidinger et al., 2021), and in particular the tendency for conversational models specifically to generate harmful content, respond inappropriately to harmful content, or falsely portray themselves as an authority when giving sensitive advice (Dinan et al., 2022).”
Mirages: On Anthropomorphism in Dialogue Systems (2023, background): “…of anthropomorphic tools Given our inherent nature to attribute meaning to signals, one must consider the appropriateness of the tool and use cases (Dinan et al., 2022; Bender et al., 2021) when designing dialogue systems, in order to avoid the (over-)integration of anthropomorphic cues.”
DialGuide: Aligning Dialogue Model Behavior with Developer Guidelines (2022, background): “Dialogue Safety is an important concern for conversational models, as they can generate harmful content, exhibit social biases, and align themselves with offensive statements (Xu et al., 2021b; Ba-heti et al., 2021; Barikeri et al., 2021; Dinan et al., 2022).”
SafeText: A Benchmark for Exploring Physical Safety in Language Models (2022, background): “Within the context of natural language processing, some work analyzes safety as a whole (Sun et al., 2022; Dinan et al., 2022) but may underrep-resent or completely miss certain types of safety.”
XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models (2023, background): “Dinan et al. (2022), introduced similar tests for open-domain conversational systems.”
ProsocialDialog: A Prosocial Backbone for Conversational Agents (2022, background): “Although such design choice aims to create uplifting user-bot interaction experiences, lacking appropriate strategies to cope with problematic user queries poses serious safety concerns for the real-world deployment of conversational AIs (Dinan et al., 2022; Weidinger et al., 2021).”