Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation

By Weixin Liang, James Y. Zou, Zhou Yu. 2020

36 citations recorded by the source corpus.

Open Domain dialog system evaluation is one of the most important challenges in dialog research. Existing automatic evaluation metrics, such as BLEU are mostly reference-based. They calculate the difference between the generated response and a limited number of available references. Likert-score based self-reported user rating is widely adopted by social conversational systems, such as Amazon Alexa Prize chatbots. However, self-reported user rating suffers from bias and variance among different users. To alleviate this problem, we formulate dialog evaluation as a comparison task. We also propose an automatic evaluation model CMADE (Comparison Model for Automatic Dialog Evaluation) that automatically cleans self-reported user ratings as it trains on them. Specifically, we first use a self-supervised method to learn better dialog feature representation, and then use KNN and Shapley to remove confusing samples. Our experiments show that CMADE achieves 89.2% accuracy in the dialog comparison task.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

ALICE: Active Learning with Contrastive Natural Language Explanations (2020, methodology): “However, previous work neglects the fact that there is usually a limited time budget to interact with domain experts (e.g., medical experts, biologists) (Liang et al., 2019, 2020) and high-quality natural language explanations are expensive, by nature.”
Neural Generation Meets Real People: Towards Emotionally Engaging Mixed-Initiative Conversations (2020, background): “These results may also reﬂect the inherent noise in user Likert-scale ratings (Liang et al., 2020).”
Evaluating the Morphosyntactic Well-formedness of Generated Texts (2021, methodology): “, 2002), METEOR (Banerjee and Lavie, 2005) and chrF (Popović, 2015) are used to evaluate MT, ROUGE (Lin, 2004) is a metric widely used in summarization, and various task-based metrics are used in dialogue (Liang et al., 2020).”
Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents (2022, background): “2018), or pairwise by comparing models (Li et al., 2019; Liang et al., 2020; Vinyals and Le, 2015; Li et al., 2016; Lee et al., 2020).”
Recent advances in deep learning based dialogue systems: a systematic survey (2021, background): “Liang et al. (2020) proposed a three-stage framework to denoise the self-rating process.”
Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons (2022, background): “Specifically, human evaluations using direct assessment have been shown to suffer from annotator bias, high variance and sequence effects where the annotation of one item is influenced by preceding items (Kulikov et al., 2019; Sudoh et al., 2021; Liang et al., 2020; See et al., 2019; Mathur et al., ”