Fatal or not? Finding errors that lead to dialogue breakdowns in chat-oriented dialogue systems

By Ryuichiro Higashinaka, Masahiro Mizukami, Kotaro Funakoshi, Masahiro Araki, Hiroshi Tsukahara, Yuka Kobayashi. 2015

32 citations recorded by the source corpus.

This paper aims to find errors that lead to dialogue breakdowns in chat-oriented dialogue systems. We collected chat dialogue data, annotated them with dialogue breakdown labels, and collected comments describing the error that led to the breakdown. By mining the comments, we first identified error types. Then, we calculated the correlation between an error type and the degree of dialogue breakdown it incurred, quantifying its impact on dialogue breakdown. This is the first study to quantitatively analyze error types and their effect in chat-oriented dialogue systems.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Elizabeth Clark
2. Sebastian Gehrmann
3. Thibault Sellam
4. Stefan Wermter
5. Cornelius Weber
6. Nicolás Navarro-Guerrero
7. Anabel Martín-González
8. Chikara Hashimoto

In Natural Language Processing

1. Thibault Sellam
2. Elizabeth Clark
3. Sebastian Gehrmann
4. Nicolás Navarro-Guerrero
5. Anabel Martín-González
6. Stefan Wermter
7. Chikara Hashimoto
8. Cornelius Weber

In Safety, Robustness & Interpretability

1. Elizabeth Clark
2. Thibault Sellam
3. Sebastian Gehrmann

Citations in the corpus

• Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “To facilitate error analyses, Higashinaka et al. (2015) propose a process of error annotations and taxonomy creation, which they demonstrate using a dialog system.”
• Detecting and Classifying Malevolent Dialogue Responses: Taxonomy, Data and Methodology (2020, neutral): “This option directly comes from previous research [4, 23].”
• Detecting Absurd Conversations from Intelligent Assistant Logs by Exploiting User Feedback Utterances (2018, methodology): “Our method can be seen as an online end-to-end metric, which is further categorized as one based on reference responses [3, 12–14, 18, 23] or one using external knowledge [5, 11, 15, 25].”
• A taxonomy, data set, and benchmark for detecting and classifying malevolent dialogue responses (2021, unknown): “This option directly comes from previous research (Ashktorab et al., 2019; Higashinaka et al., 2015). d. Abandonment —”
• Survey on reinforcement learning for language processing (2021, background): “conversational system, including context identification, dynamic context adaptation, user intention [19], and domain knowledge [49].”