A Comprehensive Assessment of Dialog Evaluation Metrics

By Yi-Ting Yeh, Maxine Eskénazi, Shikib Mehri. 2021

122 citations recorded by the source corpus.

Automatic evaluation metrics are a crucial component of dialog systems research. Standard language evaluation metrics are known to be ineffective for evaluating dialog. As such, recent research has proposed a number of novel, dialog-specific metrics that correlate better with human judgements. Due to the fast pace of research, many of these metrics have been assessed on different datasets and there has as yet been no time for a systematic comparison between them. To this end, this paper provides a comprehensive assessment of recently proposed dialog evaluation metrics on a number of datasets. In this paper, 23 different automatic evaluation metrics are evaluated on 10 different datasets. Furthermore, the metrics are assessed in different settings, to better qualify their respective strengths and weaknesses. This comprehensive assessment offers several takeaways pertaining to dialog evaluation metrics in general. It also suggests how to best assess evaluation metrics and indicates promising directions for future work.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (2024, background): “In closed-ended tasks, such as multiple-choice QA, such evaluations are straightforward to implement and trust (Novikova et al., 2017; Yeh et al., 2021).”
Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark (2021, result): “These results are in line with the robustness issues reported for other natural language generation metrics, despite the high correlation of those metrics with human judgments (Durmus et al., 2022; Gehrmann et al., 2021; Gabriel et al., 2021; Yeh et al., 2021).”
Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback (2023, methodology): “We further include BLEURT (Sellam et al., 2020), BERTScore (Zhang et al., 2019), chrF (Popovi´c, 2015), which have been shown to be among the best-performing text generation metrics on dialog(Yeh et al., 2021; Peng et al., 2022).”
Think Before You Speak: Using Self-talk to Generate Implicit Commonsense Knowledge for Response Generation (2021, methodology): “, 2020) shown to have consistent correlation (Yeh et al., 2021) with human judgements to ensure the validity of experimental results.”
Spurious Correlations in Reference-Free Evaluation of Text Generation (2022, background): “This may be a potential explanation for the robustness issues that are observed in recent work, despite the seemingly high reported correlations with human judgements (Gabriel et al., 2021; Yeh et al., 2021).”
What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation (2022, methodology): “As described earlier, most previous work in automatic evaluation focuses on text-based open-domain dialog systems (Yeh et al., 2021; Lan et al., 2020; Sinha et al., 2020; Huang et al., 2020; Ghazarian et al., 2020).”
Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents (2022, background): “For a comprehensive comparison of automatic metrics – both standard and learned metrics – see Yeh et al. (2021).”
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022, background): “Furthermore, both Yeh et al. (2021) and Zhang et al. (2021d) highlight that the combination of different metrics leads to better correlations with human evaluation than individual specialized metrics.”
DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulations (2022, unknown): “However, these metrics cannot appropriately model the whole conversation ﬂow (Yeh et al., 2021), and thus are insufﬁcient for dialogue-level evaluation.”
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning (2022, result): “We compare our model with reference-free models studied in Yeh et al. (2021).”
Think Before You Speak: Explicitly Generating Implicit Commonsense Knowledge for Response Generation (2021, methodology): “We also use GRADE (Huang et al., 2020), a reference-free metric shown to have consistent correlation with human judgements (Yeh et al., 2021) to ensure the validity of experimental results.”