Recipes for Safety in Open-domain Chatbots

By Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Aaron Edward Weston, Emily Dinan. 2020

257 citations recorded by the source corpus.

Models trained on large unlabeled corpora of human interactions will learn patterns and mimic behaviors therein, which include offensive or otherwise toxic behavior and unwanted biases. We investigate a variety of methods to mitigate these issues in the context of open-domain generative dialogue models. We introduce a new human-and-model-in-the-loop framework for both training safer models and for evaluating them, as well as a novel method to distill safety considerations inside generative models without the use of an external classifier at deployment time. We conduct experiments comparing these methods and find our new techniques are (i) safer than existing models as measured by automatic and human evaluations while (ii) maintaining usability metrics such as engagingness relative to the state of the art. We then discuss the limitations of this work by analyzing failure cases of our models.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, background): “Importantly, this allows Llama 2-Chat to generalize more effectively during safety tuning with fewer examples (Welbl et al., 2021; Korbak et al., 2023; Xu et al., 2021).”
Training language models to follow instructions with human feedback (2022, background): “There are many ways to mitigate these harms, including by fine-tuning on a small, value-targeted dataset (Solaiman and Dennison, 2021), filtering the pretraining dataset (Ngo et al., 2021), or human-in-the-loop data collection (Dinan et al., 2019; Xu et al., 2020).”
Constitutional AI: Harmlessness from AI Feedback (2022, background): “4In some contexts this could be a virtue [Xu et al., 2020], but in this paper we view it as a problem since it reduces transparency and helpfulness.”
Challenges in Detoxifying Language Models (2021, background): “Other work on toxicity in generated text includes Xu et al. (2020), who investigate safety specifically in a dialogue setting, and translating existing offensive text into non-offensive variants (Nogueira dos Santos et al., 2018; Laugier et al., 2021).”
TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “Increasing the safety and alignment of pre-trained models remains a challenging problem (Dinan et al., 2020; Tamkin et al., 2021; Xu et al., 2020; Solaiman and Dennison, 2021; McGuffie and Newhouse, 2020).”
Detoxifying Language Models Risks Marginalizing Minority Voices (2021, methodology): “Following past work (Gehman et al., 2020; Xu et al., 2020), we use four techniques that provide state-of-the-art levels of detoxification.”
OPT: Open Pre-trained Transformer Language Models (2022, neutral): “…for toxicity and biases (Dathathri et al., 2019; Dinan et al., 2019a; Sheng et al., 2019; Dinan et al., 2020a; Liu et al., 2019a; Krause et al., 2020; Xu et al., 2020; Liang et al., 2021; Dinan et al., 2021; Xu et al., 2021a; Dhamala et al., 2021; Schick et al., 2021; Ouyang et al., 2022).”
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, methodology): “Dinan et al. (2019) and Xu et al. (2020) both investigated the human-in-the-loop method to make adversarial attacks on the dialogue models.”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “Relying on the detectors, the most straightforward way a user-facing application can prevent harm is to not display the text at all (redacting) or to display it with a warning sign (flagging) (Xu et al., 2020).”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, methodology): “The same general ethical considerations apply to these de-biased dialogue models as to other dialogue models, including their potential for unsafe responses (as investigated in Xu et al. 2020) and their use of large amounts of energy to train and deploy (Bender et al., 2021).”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “It is comprised of 311M parameters and was fine-tuned on dialog safety classification tasks, including the multi-turn Bot-Adversarial dialog (Xu et al., 2020) dataset.”
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “Automatically detecting un-safe user utterances is still a challenge, both for system directed abuse (Cercas Curry et al., 2021) and general toxic statements (Xu et al., 2020).”
Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets (2021, methodology): “…methods include [6]’s foundational work to debias word embeddings; [29]’s use of product of experts to train a model to avoid dataset biases; [39]’s human-and-model-in-the-loop technique to better train and evaluate models without toxic and unwanted behavior; [23]’s use of toxic experts to…”
GeDi: Generative Discriminator Guided Sequence Generation (2020, background): “Xu et al. (2020) develop a human in the loop method for adversarially probing toxic responses in conversational agents, and train a model to give preset responses when encountering potentially unsafe probes.”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “…works have also focused on the potential of conversational models for bias, either based on gender and its intersections (Dinan et al., 2020a,b; Xu et al., 2020; Smith and Williams, 2021) or several axes of demographic axis more broadly (Barikeri et al., 2021; Perez et al., 2022; Smith et al.,…”
Internet-Augmented Dialogue Generation (2021, background): “For dialogue in particular, see Xu et al. (2020) for a review of the literature and evaluation of recent methods that try to mitigate these safety issues.”