USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation

By Shikib Mehri, Maxine Eskénazi. 2020

276 citations recorded by the source corpus.

The lack of meaningful automatic evaluation metrics for dialog has impeded open-domain dialog research. Standard language generation metrics have been shown to be ineffective for evaluating dialog models. To this end, this paper presents USR, an UnSupervised and Reference-free evaluation metric for dialog. USR is a reference-free metric that trains unsupervised models to measure several desirable qualities of dialog. USR is shown to strongly correlate with human judgment on both Topical-Chat (turn-level: 0.42, system-level: 1.0) and PersonaChat (turn-level: 0.48 and system-level: 1.0). USR additionally produces interpretable measures for several desirable properties of dialog.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

LaMDA: Language Models for Dialog Applications (2022, background): “Many automated metrics for dialog models have been studied, including perplexity [16, 17], F1, Hits@1/N [25], USR [46], or BLEU/ROUGE [47, 15, 27].”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “On the other hand, there is a long tradition of “reference free” metrics which estimate the overall quality of a conversation from observable dialog behavior, e.g. (Walker et al., 1997; Rieser & Lemon, 2008; Mehri & Eskenazi, 2020).”
Retrieval Augmentation Reduces Hallucination in Conversation (2021, background): “Research has indicated that standard automated metrics useful in related fields, such as BLEU/ROUGE for Machine Translation and F1/EM for QA are not totally correlated with how well neural conversational models perform in the wild (Liu et al., 2016; Dinan et al., 2019a; Mehri and Eskenazi, 2020).”
GPTScore: Evaluate as You Desire (2023, methodology): “To explore the correlation between aspects, we conducted an empirical analysis with INT ( interesting ) Aspect Baseline GPTScore BT BTC BTCP FED DE a01 b01 c01 d01 d03 FED dialogue-level COH 1.7 -14.9 -18.9 25.7 43.7 18.7 15.0 22.5 56.9 13.4 ERR 9 .”
CLIPScore: A Reference-free Evaluation Metric for Image Captioning (2021, unknown): “7516 (Louis and Nenkova, 2013; Peyrard and Gurevych, 2018; Sun and Nenkova, 2019) and dialogue (Tao et al., 2018; Mehri and Eskenazi, 2020).”
ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (2023, methodology): “We then take the human annotation results from Mehri & Eskenazi (2020) where they carry out the annotations on 60 dialogue contexts with each response generated by 6 different systems.”
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (2023, methodology): “It is built on the CNN/DailyMail dataset (Hermann et al., 2015) Topical-Chat (Mehri and Eskenazi, 2020) is a testbed for meta-evaluating different evaluators on dialogue response generation systems that use knowledge.”
Explaining Dialogue Evaluation Metrics using Adversarial Behavioral Analysis (2022, background): “Recent research focuses on the use of neural networks to tackle this problem (Kreyssig et al., 2018; Sun et al., 2021a; Lowe et al., 2017; Jang et al., 2020; Shim et al., 2021; Mehri and Eskenazi, 2020a; Pang et al., 2020; Gao et al., 2020; Kachuee et al., 2021).”
Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: A Preliminary Empirical Study (2023, methodology): “Furthermore, different types of text necessitate evaluation of distinct aspects, such as coherence, ﬂu-ency, and consistency (Fabbri et al., 2021a; Mehri and Eskenazi, 2020a), which makes it hard to de-sign metrics for each type of text and dimension separately.”
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning (2022, unknown): “TU, PZ, DZ, CG, PU, DGU, DGR, FT, EG and FD are abbreviations for TopicalChat-USR (Mehri and Eskenazi, 2020b), PersonaChat-Zhao (Zhao et al.”
Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR (2020, result): “Thus, the high agreement with BERT SCORE could support the view that BERT SCORE may be more form-orientated than perhaps one would assume (Mehri and Eskenazi, 2020).”
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022, background): “, 2016) and poor interpretability (Mehri and Eskenazi, 2020b).”
Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022, background): “Concretely, compared to the best uniﬁed evaluators (Yuan et al., 2021; Mehri and Eskenazi, 2020), U NI E VAL improves the correlation with human judgment by 23% on text summarization, and the improvement exceeds 43% on dialogue response generation.”
A Comprehensive Assessment of Dialog Evaluation Metrics (2021, methodology): “RoBERTa, which is employed in USR (Mehri and Eskenazi, 2020b), improves the training techniques in BERT and trains the model on a much larger corpus which includes the CommonCrawl News dataset (Mackenzie et al., 2020) and text extracted from Reddit.”
Identifying Untrustworthy Samples: Data Filtering for Open-domain Dialogues with Bayesian Optimization (2021, background): “There exists several work focusing on dialogue attributes, and some use these features to influence the training and inference processes of dialoguemodels [7, 38, 57], while others apply them asmetrics for evaluation [28, 29, 32, 49, 58].”
Spot the Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems (2020, background): “Automatic evaluation metrics for chatbots are known to correlate poorly with human ratings (Liu et al., 2016; Lowe et al., 2017; Mehri and Eskenazi, 2020), so we focus on human-based approaches, which can be classified in two dimensions: 1) single-turn vs.”