Bot-Adversarial Dialogue for Safe Conversational Agents

By Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Aaron Edward Weston, Emily Dinan. 2021

179 citations recorded by the source corpus.

Conversational agents trained on large unlabeled corpora of human interactions will learn patterns and mimic behaviors therein, which include offensive or otherwise toxic behavior. We introduce a new human-and-model-in-the-loop framework for evaluating the toxicity of such models, and compare a variety of existing methods in both the cases of non-adversarial and adversarial users that expose their weaknesses. We then go on to propose two novel methods for safe conversational agents, by either training on data from our new human-and-model-in-the-loop framework in a two-stage system, or ”baking-in” safety to the generative model itself. We find our new techniques are (i) safer than existing models; while (ii) maintaining usability metrics such as engagingness relative to state-of-the-art chatbots. In contrast, we expose serious safety issues in existing standard systems like GPT2, DialoGPT, and BlenderBot.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

OPT: Open Pre-trained Transformer Language Models (2022, methodology): “There has been a great deal of work on mitigations for toxicity and biases (Dathathri et al., 2019; Dinan et al., 2019a; Sheng et al., 2019; Dinan et al., 2020a; Liu et al., 2019a; Krause et al., 2020; Xu et al., 2020; Liang et al., 2021; Dinan et al., 2021; Xu et al., 2021a; Dhamala et al., 2021; S”
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, methodology): “Apart from our previous work, our approach is most similar to [60] & [53], who have crowdworkers attempt to elicit offensive outputs from dialogue agents in open-ended dialogues, then use the resulting data to create effective safety interventions.”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, methodology): “In particular, many works focus on generative models (Dinan et al., 2020a,b; Xu et al., 2021b; Kirk et al., 2021; Sheng et al., 2021b; Nozza et al., 2021; Renduchintala et al., 2021; Baheti et al., 2021; Perez et al., 2022), which are well known to pose unique challenges for automatic evaluation…”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “Xu et al. (2021a) show that popular methods for mitigating toxic generation in LLMs decreases the utility of these models on marginalized groups.”
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “While several recent efforts have been made to describe and mitigate unsafe behaviour of conversational models (e.g. Dinan et al., 2019; Xu et al., 2021; Ouyang et al., 2022; Thoppilan et al., 2022; Perez et al., 2022; Dinan et al., 2022), this work aims to provide a framework to help practitioners…”
Red Teaming Language Models with Language Models (2022, methodology): “Our generated questions compare favorably to questions written by paid, human adversaries from the BAD dataset (Xu et al., 2021b), as shown in Fig.”
Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “, 2021), using scorers or safety filters while decoding (Roller et al., 2021; Xu et al., 2021), supervised fine-tuning on curated data (Solaiman & Dennison, 2021; Ngo et al.”
Predictability and Surprise in Large Generative Models (2022, methodology): “…adversarial datasets to probe for weaknesses in computer vision systems [36]), as well as continuous evaluation by humans carrying out multi-step interactions (e.g, conversations [3, 88]) with these models, as well as plans for how to update the models in response to what these evaluations find.”
Using In-Context Learning to Improve Dialogue Safety (2023, methodology): “One popular approach for creating safer dialogue systems involves using safety filters (Xu et al., 2021b; Shuster et al., 2022).”
Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022, background): “Unsurprisingly, gendered and racial disparities have been documented in a number of language technologies [37, 79, 141], and processes of creating resources and technologies may further entrench such disparities [25, 38, 144].”
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, methodology): “Indeed, though no group based analysis was included, both BAD and PALMS considered different demographic groups when building their dataset (e.g., to source questions).”
Director: Generator-Classifiers For Supervised Language Modeling (2022, positive): “We use the training set of WTC, in addition to the safety data from (Dinan et al., 2019; Xu et al., 2021), as positively and negatively labeled data to train supervised models (reranker, FUDGE, PACER, DIRECTOR).”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, background): “Xu et al. (2021a) show that popular methods for mitigating toxic generation in LLMs decreases the utility of these models on marginalized groups, potentially resulting in harms such as forcing marginalized users to code-switch.”
BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset (2023, background): “BAD [28] by MetaAI is a dialogue dataset in which annotators attempt to elicit unsafe behaviors from the chat-bot using unsafe utterances, and all utterances are annotated as offensive or safe.”
DialGuide: Aligning Dialogue Model Behavior with Developer Guidelines (2022, background): “The safety classiﬁer is a Deberta-large classiﬁer trained on the BAD (Bot Adversarial Dialogue) dataset (Xu et al., 2021a), which consists of dialogue safety data and labels collected through an adversarial human-and-model-in-the-loop framework.”
COLD: A Benchmark for Chinese Offensive Language Detection (2022, background): “, 2021b), the safety issues due to offensive generation continue to be exposed (Gehman et al., 2020; Xu et al., 2021; Bender et al., 2021; Mi et al., 2022), attracting widespread attention from researchers and pushing the research boom on this task to new heights”