Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling

By Emily Dinan, Gavin Abercrombie, A. S. Bergman, Shannon L. Spruit, Dirk Hovy, Y-Lan Boureau, Verena Rieser. 2021

117 citations recorded by the source corpus.

Over the last several years, end-to-end neural conversational agents have vastly improved in their ability to carry a chit-chat conversation with humans. However, these models are often trained on large datasets from the internet, and as a result, may learn undesirable behaviors from this data, such as toxic or otherwise harmful language. Researchers must thus wrestle with the issue of how and when to release these models. In this paper, we survey the problem landscape for safety for end-to-end conversational AI and discuss recent and related work. We highlight tensions between values, potential positive impact and potential harms, and provide a framework for making decisions about whether and how to release these models, following the tenets of value-sensitive design. We additionally provide a suite of tools to enable researchers to make better-informed decisions about training and releasing end-to-end conversational AI models.

Subfields: Artificial Intelligence, Safety Research, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, background): “Work from Roller et al. (2020) and Dinan et al. (2021) also illuminates the difficulties tied to chatbot-oriented LLMs, with concerns ranging from privacy to misleading expertise claims.”
LaMDA: Language Models for Dialog Applications (2022, background): “Similar issues have also been discussed specifically for dialog models [53].”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…appropriate for interactive systems like dialog systems which are out of scope for this work and which require evaluation considerations such as the distinction between turn- and dialoglevel metrics (Smith et al., 2022) and which are much more susceptible to antropomorphism (Dinan et al., 2021).”
OPT: Open Pre-trained Transformer Language Models (2022, background): “As shown in Section 4, we also ﬁnd OPT-175B has a high propensity to generate toxic language and reinforce harmful stereotypes, even when provided with a relatively innocuous prompt (Gehman et al., 2020), and adversarial prompts are trivial to ﬁnd (Dinan et al., 2021).”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Hate, Toxicity, and Targeted Violence and safe to hurtful outputs can be evaluated in context of safe discussions, toxicity metrics [87, 182], hurtfulness [165], and level of offense [71] for language.”
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “For instance, recent studies have emphasized the mental harm caused by inappropriate advice and the public opinion risks brought about by controversial re-marks (Dinan et al., 2021; Sun et al., 2022; Levy et al., 2022).”
The Capacity for Moral Self-Correction in Large Language Models (2023, background): “Large language models exhibit harmful social biases [1, 6, 8, 11, 15, 24, 29, 50, 61] that can sometimes get worse for larger models [2, 18, 20, 43].”
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, background): “…social biases (e.g., [47, 28, 1, 33, 7]), generating offensive or toxic outputs [25], leaking personally identiﬁable information from the training data [13], aiding in disinformation campaigns [12], generating extremist texts [37], spreading falsehoods [35], and more [9, 10, 18, 57, 22, 51].”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “We also note that these benchmarks have limitations with respect to their ability to measure safety concerns: the datasets used therein are static and crowd-sourced, and cannot guarantee safety in all situations (Dinan et al., 2021).”
Predictability and Surprise in Large Generative Models (2022, background): “Although we focus on scaling laws, many of our points complement existing views on the societal risks of deploying large models [7, 67, 9, 19, 72, 41].”
Using In-Context Learning to Improve Dialogue Safety (2023, methodology): “1 Following the taxonomy introduced by Dinan et al. (2021), our work investigates reducing the I NSTI - GATOR and Y EA -”
SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures (2021, background): “…to mimic human English-language open-domain conversations have become engaging (Adiwardana et al., 2020; Roller et al., 2020b), but are still prone to uttering problematic language, e.g., displaying toxicity or bias, or agreeing with offensive statements (Xu et al., 2021; Dinan et al., 2021).”
DEMix Layers: Disentangling Domains for Modular Language Modeling (2021, background): “This design principle emphasizes the ability to rapidly adapt the LM after training, a need that has been broadly advocated for language systems (Dinan et al., 2021; Lazaridou et al., 2021).”
Taxonomy of Risks posed by Language Models (2022, background): “Progress has been made in developing technical risk mitigation tools, yet more innovation and stress-testing of these mitigations is needed [37, 48, 171, 194].”
DialGuide: Aligning Dialogue Model Behavior with Developer Guidelines (2022, unknown): “, 2021), or provide unsafe counsel (Dinan et al., 2021).”
Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls (2022, background): “While much work has centered on detecting undesirable behavior such as toxic language (Xu et al., 2020; Dinan et al., 2021), trolling (Tomaiuolo et al., 2020; Mihaylov and Nakov, 2019) and bias (Dinan et al., 2019a), less work has studied robust learning from organic conversations with potentially…”
Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations (2023, background): “Dinan et al. (2021) surveys the safety landscape and proposes a framework to determine launch decisions for these systems.”
Robust Conversational Agents against Imperceptible Toxicity Triggers (2022, background): “We acknowledge that our attack can be used by unethical adversaries to force the models to generate toxic responses which is undesirable as also previously observed in chatbots (Wolf et al., 2017; Henderson et al., 2018; Dinan et al., 2021).”
Cultural Incongruencies in Artificial Intelligence (2022, background): “• Safety gaps : With increasing adoption of AI systems, there are also increasing efforts on ensuring the AI systems are safe and fair [8, 25].”