DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulations

By Sarik Ghazarian, Nuan Wen, Aram Galstyan, Nanyun Peng. 2022

45 citations recorded by the source corpus.

Automatic evaluation metrics are essential for the rapid development of open-domain dialogue systems as they facilitate hyper-parameter tuning and comparison between models. Although recently proposed trainable conversation-level metrics have shown encouraging results, the quality of the metrics is strongly dependent on the quality of training data. Prior works mainly resort to heuristic text-level manipulations (e.g. utterances shuffling) to bootstrap incoherent conversations (negative examples) from coherent dialogues (positive examples). Such approaches are insufficient to appropriately reflect the incoherence that occurs in interactions between advanced dialogue models and humans. To tackle this problem, we propose DEAM, a Dialogue coherence Evaluation metric that relies on Abstract Meaning Representation (AMR) to apply semantic-level Manipulations for incoherent (negative) data generation. AMRs naturally facilitate the injection of various types of incoherence sources, such as coreference inconsistency, irrelevancy, contradictions, and decrease engagement, at the semantic level, thus resulting in more natural incoherent samples. Our experiments show that DEAM achieves higher correlations with human judgments compared to baseline methods on several dialog datasets by significant margins. We also show that DEAM can distinguish between coherent and incoherent dialogues generated by baseline manipulations, whereas those baseline models cannot detect incoherent examples generated by DEAM. Our results demonstrate the potential of AMR-based semantic manipulations for natural negative example generation.

Subfields: Evaluation & Benchmarks, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

In Dialogue & Conversational AI

Citations in the corpus

GPTScore: Evaluate as You Desire (2023, methodology): “For example, DEAM (Ghazarian et al., 2022) and Quan-tiDCE (Ye et al., 2021) were proposed for the evaluation of the coherence of the dialogue system; several evaluators (Cao et al., 2020; Durmus et al., 2020; Wang et al., 2020a) are designed for the evaluation of the consistency of text…”
MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation (2022, methodology): “Following (Tao et al., 2018; Gao et al., 2021; Ghazarian et al., 2022), we randomly selected 200 contexts from the test set and solicited ratings from three English-speaking volunteers on a 1-5 Likert scale, reflecting their satisfaction with the response (retrieved or generated) to the given contex”
Evaluating Very Long-Term Conversational Memory of LLM Agents (2024, background): “…employs lexical overlap (Papineni et al., 2002) and semantic overlap (Zhang et al., 2019) between ground truth and the agent response, or consistency (Ghazarian et al., 2022), contradiction (Nie et al., 2021; Welleck et al., 2019), and empathy (Zhang et al., 2021a, 2022) of the agent response.”
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022, background): “In addition, existing works on model-based dialogue-level metrics (Li et al., 2021; Zhang et al., 2021a; Ghazarian et al., 2022; Zhao et al., 2022)”