How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

By Chia-Wei Liu, Ryan J. Lowe, Iulian Serban, Michael Noseworthy, Laurent Charlin, Joelle Pineau. 2016

1,399 citations recorded by the source corpus.

We investigate evaluation metrics for dialogue response generation systems where supervised labels, such as task completion, are not available. Recent works in response generation have adopted metrics from machine translation to compare a model's generated response to a single target response. We show that these metrics correlate very weakly with human judgements in the non-technical Twitter domain, and not at all in the technical Ubuntu domain. We provide quantitative and qualitative results highlighting specific weaknesses in existing metrics, and provide recommendations for future development of better automatic evaluation metrics for dialogue systems.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

A Deep Reinforced Model for Abstractive Summarization (2017, background): “It is possible to game such discrete metrics and increase their score without an actual increase in readability or relevance (Liu et al., 2016).”
Experience Grounds Language (2020, background): “Despite the current, passive way generation is created and evaluated (Liu et al., 2016), the ability to change the world through other people is the rich learning signal that natural language generation offers.”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, background): “Some text generation tasks fail to meet this criteria due to issues with automatic metrics like ROUGE and BLEU (Callison-Burch et al., 2006; Liu et al., 2016, i.a.).”
Towards a Human-like Open-Domain Chatbot (2020, background): “, 2002), ROUGE (Lin, 2004), or other related metrics in translation and summarization, while popular and easy to compute, have been shown to be unsuitable for dialog (Liu et al., 2016) or more broadly language gener-”
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (2021, background): “Sometimes the generated text is explicitly compared to human-authored text (e.g., Liu et al., 2016; Zellers et al., 2021; Zhang et al., 2020), but even when no human-authored text is evaluated, evaluators implicitly compare the generated text to their knowledge of language and norms within specific…”
Recipes for Safety in Open-domain Chatbots (2020, background): “We note that all automatic metrics have flaws (Liu et al., 2016), hence we also report human judgments as described in the next section.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “…metrics based on word-overlap with reference responses, as frequently used in question-answering (Rajpurkar et al., 2016) or machine translation (Papineni et al., 2002), do not work well, and have poor correlation with human judgments (Liu et al., 2016; Novikova et al., 2017; Lowe et al., 2017).”
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons (2019, background): “As in many other natural language generation (NLG) tasks, automatic metrics have not been shown to have a clear correlation with human evaluations (Liu et al. 2016; Lowe et al. 2017).”
Retrieve and Refine: Improved Sequence Generation Models For Dialogue (2018, background): “Perplexity Dialogue is known to be notoriously hard to evaluate with automated metrics (Liu et al., 2016).”
A Knowledge-Grounded Neural Conversation Model (2017, methodology): “While (Liu et al., 2016) B LEU , which is known to better correlate with human judgments (Przybocki et al., 2008) including for response generation (Galley et al., 2015).”
Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models (2020, background): “evaluation (Liu et al. 2016) in open domain text generation tasks including story generation and dialogue response generation because two equally good generated texts may have no n-gram overlap.”
Plug and Play Language Models: A Simple Approach to Controlled Text Generation (2019, background): “Perplexity is an automated measure of ﬂuency, though its effectiveness has been questioned in open-domain text generation (Liu et al., 2016).”
Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators (2019, background): “While the shortcomings of these metrics are well known for MT (Graham, 2015; Espinosa et al., 2010; Cahill, 2009), the problem is aggravated for dialog generation evaluation because of the much larger output space (Liu et al., 2016).”
Ethical Challenges in Data-Driven Dialogue Systems (2017, background): “However, the subjective nature of dialogue [7, 23], and the onset of data-driven components [37, 41], leads to several potential dangers with the widespread use of these systems.”
Red Teaming Language Models with Language Models (2022, neutral): “Though BLEU has limitations (CallisonBurch et al., 2006; Liu et al., 2016), we find similar results with other diversity metrics in Appendix §C.”