Survey on evaluation methods for dialogue systems

By Jan Deriu, Álvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, Mark Cieliebak. Artificial Intelligence Review, 2019

360 citations recorded by the source corpus.

In this paper, we survey the methods and concepts developed for the evaluation of dialogue systems. Evaluation, in and of itself, is a crucial part during the development process. Often, dialogue systems are evaluated by means of human evaluations and questionnaires. However, this tends to be very cost- and time-intensive. Thus, much work has been put into finding methods which allow a reduction in involvement of human labour. In this survey, we present the main concepts and methods. For this, we differentiate between the various classes of dialogue systems (task-oriented, conversational, and question-answering dialogue systems). We cover each class by introducing the main technologies developed for the dialogue systems and then present the evaluation methods regarding that class.

Subfields: Anthropology, Archeology

View Paper: https://www.semanticscholar.org/paper/14597394f68463424c52fbb565c61b096434b3e1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Yang Liu
2. Jianfeng Gao
3. Xiangnan He
4. Pei Zhou
5. Tat-seng Chua
6. Yi Tay
7. Mohit Bansal
8. Asli Celikyilmaz

Citations in the corpus

• FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022, background): “Evaluation is a long-lasting problem in dialogue system research (Deriu et al., 2021; Yeh et al., 2021; Mehri et al., 2022; Smith et al., 2022).”
• What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation (2022, background): “However, standard word-overlap metrics (BLEU, ROUGE, Perplexity) do not correlate well with human judgements of open-domain dialog systems (Deriu et al., 2020; Liu et al., 2016) because of the diverse set of outputs that can be relevant given a dialog context.”
• Spot the Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems (2020, background): “There exist various methods to evaluate dialogue systems, both automatic and human-based, but no single evaluation metric is widely agreed upon in the scientific community (Deriu et al., 2020).”
• Offline RL for Natural Language Generation with Implicit Language Q Learning (2022, background): “The lack of easily configurable task settings and reliable simulated evaluations has arguably down progress in applying sophisticated RL algorithms to language and dialogue tasks [49, 50, 51].”
• Evaluation of Text Generation: A Survey (2020, background): “, 2020), and dialog response generation (Liu et al., 2016; Novikova et al., 2017; Deriu et al., 2019; Dusek et al., 2019; Gao et al., 2019), to name a few.”
• Unsupervised Evaluation of Interactive Dialog with DialoGPT (2020, background): “, BLEU, METEOR) have been shown to be ineffective for dialog (Deriu et al., 2019; Liu et al., 2016), human evaluation is often used.”
• Commonsense-Focused Dialogues for Response Generation: An Empirical Study (2021, background): “with human judgement of these systems (Deriu et al., 2020; Liu et al., 2016).”
• A Comprehensive Assessment of Dialog Evaluation Metrics (2021, methodology): “Due to the fact that standard automatic metrics have been shown to be ineffective for dialog evaluation (Liu et al., 2016b; Deriu et al., 2021), dialog research typically relies on human evaluation.”
• LLMs Get Lost In Multi-Turn Conversation (2025, neutral): “Conversational AI was typically implemented as specialized systems that leveraged language models as components [36], and were evaluated through human protocols [17, 42, 21, 54], or competitions like Amazon’s Alex Prize [67].”
• Perturbation CheckLists for Evaluating NLG Evaluation Metrics (2021, background): “However, this is a time consuming and expensive process (Freitag et al., 2021; Deriu et al., 2019; Howcroft et al., 2020).”
• USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation (2020, background): “Depending on the task and the data, the desired qualities of a dialog system may differ (Walker et al., 1997; Deriu et al., 2019).”
• Advances and Challenges in Conversational Recommender Systems: A Survey (2021, methodology): “Also, there are survey papers that summarize new frontiers in conversational AI systems, such as the advanced methods [23, 250, 55] and the evaluation issues [18, 41] in dialogue systems.”