Detecting and Classifying Malevolent Dialogue Responses: Taxonomy, Data and Methodology

By Yangjun Zhang, Pengjie Ren, Maarten de Rijke. 2020

12 citations recorded by the source corpus.

Conversational interfaces are increasingly popular as a way of connecting people to information. Corpus-based conversational interfaces are able to generate more diverse and natural responses than template-based or retrieval-based agents. With their increased generative capacity of corpusbased conversational agents comes the need to classify and filter out malevolent responses that are inappropriate in terms of content and dialogue acts. Previous studies on the topic of recognizing and classifying inappropriate content are mostly focused on a certain category of malevolence or on single sentences instead of an entire dialogue. In this paper, we define the task of Malevolent Dialogue Response Detection and Classification (MDRDC). We make three contributions to advance research on this task. First, we present a Hierarchical Malevolent Dialogue Taxonomy (HMDT). Second, we create a labelled multi-turn dialogue dataset and formulate the MDRDC task as a hierarchical classification task over this taxonomy. Third, we apply stateof-the-art text classification methods to the MDRDC task and report on extensive experiments aimed at assessing the performance of these approaches.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “Consequently, based on related studies (Zhang et al., 2020; Dinan et al., 2021; Wei-dinger et al., 2021; Sun et al., 2022) and recently widely discussed safety risks (Hendrycks et al., 2023), we first introduce the scope of safety risks and discuss them from six perspectives, including Toxicity and…”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “…“human-likeness” or “engagingness”, either by automatically comparing with a human ground-truth reference, e.g., by using similarity metrics such as BLEURT (Sellam et al., 2020) or BERTscore (Zhang et al., 2020a), or by asking humans to evaluate this manually (Deriu et al., 2020; Li et al., 2019).”
Recipes for Safety in Open-domain Chatbots (2020, background): “Zhang et al. (2020) propose a wider-ranging hierarchical taxonomy of malevolent dialogue, deﬁned as “a system-generated response that is grounded in negative emotion, inappropriate behavior or unethical value basis in terms of content and dialogue acts.”
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “…– often centred on content written by humans on social media platforms, such as Twitter (e.g. Waseem and Hovy, 2016; Wang et al., 2020; Zampieri et al., 2019, 2020; Zhang et al., 2020), Facebook (Glavaš et al., 2020; Zampieri et al., 2020), or Reddit (Han and Tsvetkov, 2020; Zampieri et al., 2020).”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, methodology): “At decoding time, the neural generative models (the 90M and 2.7B parameter variants of BlenderBot (Roller et al., 2021), as well as DialoGPT (Zhang et al., 2020b) and GPT-2 (Radford et al., 2019)) use beam search with a beam size of 10 , context and label 3 -gram blocking (Paulus et al., 2017), and ”