D-Score: Holistic Dialogue Evaluation Without Reference

By Chen Zhang, Grandee Lee, Luis Fernando D'Haro, Haizhou Li. 2021

27 citations recorded by the source corpus.

In artistic gymnastics, difficulty score or D-score is used for judging performance. Starting from zero, an athlete earns points from different aspects such as composition requirement, difficulty, and connection between moves. The final score is a composition of the quality of various performance indicators. Similarly, when evaluating dialogue responses, human judges generally follow a number of criteria, among which language fluency, context coherence, logical consistency, and semantic appropriateness are on top of the agenda. In this paper, we propose an automatic dialogue evaluation framework called D-score that resembles the way gymnastics is evaluated. Following the four human judging criteria above, we devise a range of evaluation tasks and model them under a multi-task learning framework. The proposed framework, without relying on any human-written reference, learns to appreciate the overall quality of human-human conversations through a representation that is shared by all tasks without over-fitting to individual task domain. We evaluate D-score by performing comprehensive correlation analyses with human judgement on three dialogue evaluation datasets, among which two are from past DSTC series, and benchmark against state-of-the-art baselines. D-score not only outperforms the best baseline by a large margin in terms of system-level Spearman correlation but also represents an important step towards explainable dialogue scoring.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, background): “Recently, several works have proposed evaluation metrics for measuring consistency, such as using natural language inference (NLI) [40, 202], training learnable evaluation metrics [225], or releasing an additional test set for coherence [10].”
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022, methodology): “Furthermore, both Yeh et al. (2021) and Zhang et al. (2021d) highlight that the combination of different metrics leads to better correlations with human evaluation than individual specialized metrics.”