RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems

By Chongyang Tao, Lili Mou, Dongyan Zhao, Rui Yan. 2017

227 citations recorded by the source corpus.

Open-domain human-computer conversation has been attracting increasing attention over the past few years. However, there does not exist a standard automatic evaluation metric for open-domain dialog systems; researchers usually resort to human annotation for model evaluation, which is time- and labor-intensive. In this paper, we propose RUBER, a Referenced metric and Unreferenced metric Blended Evaluation Routine, which evaluates a reply by taking into consideration both a groundtruth reply and a query (previous user-issued utterance). Our metric is learnable, but its training does not require labels of human satisfaction. Hence, RUBER is flexible and extensible to different datasets and languages. Experiments on both retrieval and generative dialog systems show that RUBER has a high correlation with human annotation, and that RUBER has fair transferability over different datasets.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Towards a Human-like Open-Domain Chatbot (2020, background): “Past works have attempted to build learnable metrics, either in a supervised fashion (Lowe et al., 2017), which requires human labels, or with unsupervised approaches (Tao et al., 2017; Ghazarian et al., 2019), that are more complex and need separate training, e.g., of a ranking system.”
Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators (2019, background): “Tao et al. (2017) tries to address the expensive annotation process by adding in unsupervised data.”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, methodology): “[136] proposed an evaluation model, called RUBER, which does not rely on human judged scores.”
Style Transfer in Text: Exploration and Evaluation (2017, background): “RUBER (Tao et al. 2017) was proposed to evaluate dialog system, it divides evaluation into referenced and unreferenced part.”
A Survey on Dialogue Systems: Recent Advances and New Frontiers (2017, background): “[81] combined referenced and unreferenced metrics, where the former measured the similarity between reply and the groundtruth through word embedding, and the latter scored the correlation be-tween reply and query trained with a max-margin objective function, where the negative reply is randomly…”
Commonsense-Focused Dialogues for Response Generation: An Empirical Study (2021, methodology): “Recently open domain dialog systems have been modeled using end-to-end approaches, more speciﬁcally encoder-decoder architectures (Sordoni et al., 2015; Serban et al., 2017, 2016; Vinyals and Le, 2015).”
Explaining Dialogue Evaluation Metrics using Adversarial Behavioral Analysis (2022, background): “Given our results, we hypothesize that other metrics like (Mehri and Eskenazi, 2020b; Lowe et al., 2017; Tao et al., 2018) trained similarly have deeper problems which need to be highlighted using a fine-grained methodology similar to ours.”
CLIPScore: A Reference-free Evaluation Metric for Image Captioning (2021, background): “…to the machine translation cases highlighted in the introduction, reference-free evaluations have been proposed for other generation tasks, including summarization (Louis and Nenkova, 2013; Peyrard and Gurevych, 2018; Sun and Nenkova, 2019) and dialogue (Tao et al., 2018; Mehri and Eskenazi, 2020).”
Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddings (2019, background): “The Referenced metric and Unreferenced metric Blended Evaluation Routine (RUBER) (Tao et al., 2018) stands out from recent work in automatic dialogue evaluation, relying minimally on human-annotated datasets of response quality for training.”
LSDSCC: a Large Scale Domain-Specific Conversational Corpus for Response Generation with Diversity Oriented Evaluation Metrics (2018, background): “Thus, efforts are devoted to simulate the human subjective judgment, which is similar with the response ranking task in retrieval-based chat agents (Lowe et al., 2017; Tao et al., 2018), but un-avoidable uncertainty and errors are brought into the systems (Hu et al., 2014).”
The statistical advantage of automatic NLG metrics at the system level (2021, background): “Metrics are an active research area in many NLG subﬁelds, including summarization (Lin, 2004), dialogue (Tao et al., 2018), and image captioning (Anderson et al., 2016), which seek to realize the goal of quick and reliable automatic evaluation.”
UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation (2020, methodology): “Furthermore, RUBER and its variants (Tao et al., 2018; Ghazarian et al., 2019) evaluate a response by directly averaging a non-learnable referenced embedding similarity score and a learnable unreferenced post-response relatedness score that is learned by applying negative sampling without human…”
Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretraining (2020, methodology): “Given the shortcomings of context-agnostic n - gram and embedding based metrics, the focus has now shifted to building neural network based, trainable dialogue evaluation models (Lowe et al., 2017; Tao et al., 2018; Shimanaka et al., 2019; Ghazarian et al., 2019).”
AVA: an Automatic eValuation Approach to Question Answering Systems (2020, background): “For dialog systems, neural-based automatic evaluators are also studied (Ghazarian et al., 2019; Lowe et al., 2017; Tao et al., 2017; Kannan and Vinyals, 2017) QA has been traditionally studied early in literature (Green et al., 1961).”
CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation (2022, unknown): “3) Hybrid metrics contain both referenced and reference-free scores, such as RUBER (Tao et al., 2018; Ghaz-arian et al., 2019), BLEURT (Sellam et al., 2020) and BARTScore (Yuan et al., 2021).”
Linguistically-Informed Specificity and Semantic Plausibility for Dialogue Generation (2019, background): “TEOR (Banerjee and Lavie, 2005) and perplexity have poor correlations with human judgment (Liu et al., 2016; Tao et al., 2018)4 or are modeldependent (Liu et al.”
A Survey of Evaluation Metrics Used for NLG Systems (2020, methodology): “RUBER (Referenced metric and Unreferenced metric Blended Evaluation Routine [156]): RUBER is another RNN-based dialogue evaluation model like ADEM.”