A taxonomy, data set, and benchmark for detecting and classifying malevolent dialogue responses

By Yangjun Zhang, Pengjie Ren, Maarten de Rijke. 2021

14 citations recorded by the source corpus.

Conversational interfaces are increasingly popular as a way of connecting people to information. With the increased generative capacity of corpus‐based conversational agents comes the need to classify and filter out malevolent responses that are inappropriate in terms of content and dialogue acts. Previous studies on the topic of detecting and classifying inappropriate content are mostly focused on a specific category of malevolence or on single sentences instead of an entire dialogue. We make three contributions to advance research on the malevolent dialogue response detection and classification (MDRDC) task. First, we define the task and present a hierarchical malevolent dialogue taxonomy. Second, we create a labeled multiturn dialogue data set and formulate the MDRDC task as a hierarchical classification task. Last, we apply state‐of‐the‐art text classification methods to the MDRDC task, and report on experiments aimed at assessing the performance of these approaches.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Minlie Huang
2. Xiaoyan Zhu
3. Chujie Zheng
4. Fei Mi
5. Hao Sun
6. Nanyun Peng
7. Zhexin Zhang
8. Deng Jiawen

In Natural Language Processing

1. Minlie Huang
2. Chujie Zheng
3. Hao Sun
4. Xiaoyan Zhu
5. Fei Mi
6. Zhexin Zhang
7. Deng Jiawen
8. Guangxuan Xu

In Safety, Robustness & Interpretability

1. Zhexin Zhang
2. Hao Sun
3. Deng Jiawen
4. Minlie Huang
5. Guangxuan Xu
6. Jiawen Deng
7. Jiale Cheng
8. Jingyan Zhou

Citations in the corpus

• Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “…2021) or more covert offenses (Wang and Potts, 2019; Breitfeller et al., 2019; Han and Tsvetkov, 2020; Price et al., 2020), the interaction between LMs and users is growing more frequent, resulting in increasingly complex generated content (Sheng et al., 2021; Zhang et al., 2021; Sun et al., 2022).”
• On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark (2021, background): “Meanwhile, more and more researchers started to pay attention to the influence of context (Dinan et al., 2019; Xu et al., 2020; Zhang et al., 2021) and find context-aware data is more challenging for annotation and detection (Pavlopou-los et al., 2020; Menini et al., 2021).”
• COLD: A Benchmark for Chinese Offensive Language Detection (2022, methodology): “We evaluate the versions of CPM-Generate and CPM-Generate-distill.”