Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts

By Ashutosh Baheti, Maarten Sap, Alan Ritter, Mark O. Riedl. 2021

101 citations recorded by the source corpus.

Dialogue models trained on human conversations inadvertently learn to generate toxic responses. In addition to producing explicitly offensive utterances, these models can also implicitly insult a group or individual by aligning themselves with an offensive statement. To better understand the dynamics of contextually offensive language, we investigate the stance of dialogue model responses in offensive Reddit conversations. Specifically, we create ToxiChat, a crowd-annotated dataset of 2,000 Reddit threads and model responses labeled with offensive language and stance. Our analysis reveals that 42% of human responses agree with toxic comments, whereas only 13% agree with safe comments. This undesirable behavior is learned by neural dialogue models, such as DialoGPT, which we show are two times more likely to agree with offensive comments. To enable automatic detection of offensive language, we fine-tuned transformer-based classifiers on ToxiChat that achieve 0.71 F1 for offensive labels and 0.53 Macro-F1 for stance labels. Finally, we quantify the effectiveness of controllable text generation (CTG) methods to mitigate the tendency of neural dialogue models to agree with offensive comments. Compared to the baseline, our best CTG model achieves a 19% reduction in agreement with offensive comments and produces 29% fewer offensive replies. Our work highlights the need for further efforts to characterize and analyze inappropriate behavior in dialogue models, in order to help make them safer.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, background): “…generative models (Dinan et al., 2020a,b; Xu et al., 2021b; Kirk et al., 2021; Sheng et al., 2021b; Nozza et al., 2021; Renduchintala et al., 2021; Baheti et al., 2021; Perez et al., 2022), which are well known to pose unique challenges for automatic evaluation (Lowe et al., 2017; Howcroft et…”
Using In-Context Learning to Improve Dialogue Safety (2023, background): “For example, these models often exhibit social biases (Dinan et al., 2020; Barikeri et al., 2021) and inappropriately align themselves with offensive statements during conversation (Baheti et al., 2021).”
Revealing Persona Biases in Dialogue Systems (2021, positive): “Our metric for harmful agreements is also motivated by the work of Baheti et al. (2021), who find that popular language generation models such as DialoGPT have a learned tendency to agree with offensive statements.”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, methodology): “Baheti et al. (2021) extend this approach by adding a “stance” (agree, disagree, neutral) towards a previous utterance.”
Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models (2022, unknown): “Baheti et al. [32] focus on mitigating the offensive behavior in dialogue systems.”
Mix and Match: Learning-free Controllable Text Generationusing Energy Language Models (2022, background): “These mechanisms for controlling the generation of language have the potential to mitigate undesirable biases encoded by the large language models and prevent the generation of hate speech and toxic language (Xu et al.; Gehman et al., 2020; Sap et al., 2021; Baheti et al., 2021; Mireshghallah and Be”
Robust Conversational Agents against Imperceptible Toxicity Triggers (2022, background): “In addition, there is a body of work in detecting toxic behavior in conversational agents (Zhang et al., 2018; Almerekhi et al., 2019; Baheti et al., 2021) that can be utilized to design ethically aligned systems.”
On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark (2021, background): “As shown in Table 3, the overall annotations closely achieve moderate agreement, which is already quite impressive given the difficulty of the task, and in comparison of related contextual tasks (Baheti et al., 2021).”
DialGuide: Aligning Dialogue Model Behavior with Developer Guidelines (2022, background): “Other approaches include strategies for responding to problematic contexts, such as steering away from toxicity (Baheti et al., 2021; Arora et al., 2022), using apologies (Ung et al., 2022), and non-sequiturs (Xu et al., 2021b).”
PERSONACHATGEN: Generating Personalized Dialogues using GPT-3 (2022, background): “…as an alternate way, many studies have created datasets by leveraging pre-trained language models with designed prompt instructions (Yoo et al., 2021; Baheti et al., 2021; Hartvigsen et al., 2022) due to their enormous ability to produce more human-like text (Clark et al., 2021; Dou et al., 2021).”
Towards Identifying Social Bias in Dialog Systems: Framework, Dataset, and Benchmark (2022, background): “Dialog Safety and Social Bias Inheriting from pre-trained language models, dialog safety issues, including toxicity and offensiveness (Baheti et al., 2021; Cercas Curry and Rieser, 2018; Dinan et al., 2021), bias (Henderson et al., 2018; Liu et al., 2020; Barikeri et al., 2021; Lee et al., 2019),…”
ProsocialDialog: A Prosocial Backbone for Conversational Agents (2022, methodology): “Almost all workers have lived in US for more than 10 years (97%); 57% of them live in suburban areas, 25% in urban areas, and 18% in rural areas.”