DynaEval: Unifying Turn and Dialogue Level Evaluation

By Chen Zhang, Yiming Chen, Luis Fernando D'Haro, Yan Zhang, Thomas Friedrichs, Grandee Lee, Haizhou Li. 2021

81 citations recorded by the source corpus.

A dialogue is essentially a multi-turn interaction among interlocutors. Effective evaluation metrics should reflect the dynamics of such interaction. Existing automatic metrics are focused very much on the turn-level quality, while ignoring such dynamics. To this end, we propose DynaEval, a unified automatic evaluation framework which is not only capable of performing turn-level evaluation, but also holistically considers the quality of the entire dialogue. In DynaEval, the graph convolutional network (GCN) is adopted to model a dialogue in totality, where the graph nodes denote each individual utterance and the edges represent the dependency between pairs of utterances. A contrastive loss is then applied to distinguish well-formed dialogues from carefully constructed negative samples. Experiments show that DynaEval significantly outperforms the state-of-the-art dialogue coherence model, and correlates strongly with human judgements across multiple dialogue evaluation aspects at both turn and dialogue level.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Dialogue & Conversational AI

Citations in the corpus

GPTScore: Evaluate as You Desire (2023, background): “DynaEval (Zhang et al., 2021) is a unified automatic evaluation framework for dialogue response generation tasks on the turn level and dialogue level.”
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022, methodology): “Furthermore, both Yeh et al. (2021) and Zhang et al. (2021d) highlight that the combination of different metrics leads to better correlations with human evaluation than individual specialized metrics.”
Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: A Preliminary Empirical Study (2023, background): “, 2020); FED (Mehri and Eskenazi, 2020a); DynaEval (Zhang et al., 2021); SelfEval (Ma et al.”
Evaluating Very Long-Term Conversational Memory of LLM Agents (2024, background): “…employs lexical overlap (Papineni et al., 2002) and semantic overlap (Zhang et al., 2019) between ground truth and the agent response, or consistency (Ghazarian et al., 2022), contradiction (Nie et al., 2021; Welleck et al., 2019), and empathy (Zhang et al., 2021a, 2022) of the agent response.”
A Comprehensive Assessment of Dialog Evaluation Metrics (2021, background): “Amongst the metrics assessed here, there are several reference-free evaluation metrics: HolisticEval, MAUDE, GRADE, USR, FED, FlowSore, USL-H, QuestE-val, DEB, DynaEval, PredictiveEngage and Dialo-gRPT.”
DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulations (2022, unknown): “Previous works pursue different models such as graph-based (Vakulenko et al., 2018; Zhang et al., 2021) or text-based (Mesgar et al., 2020) approaches to develop automatic trainable coherence evaluation metrics.”