Learning an Unreferenced Metric for Online Dialogue Evaluation

By Koustuv Sinha, Prasanna Parthasarathi, Jasmine Wang, Ryan J. Lowe, William L. Hamilton, Joelle Pineau. 2020

88 citations recorded by the source corpus.

Evaluating the quality of a dialogue interaction between two agents is a difficult task, especially in open-domain chit-chat style dialogue. There have been recent efforts to develop automatic dialogue evaluation metrics, but most of them do not generalize to unseen datasets and/or need a human-generated reference response during inference, making it infeasible for online evaluation. Here, we propose an unreferenced automated evaluation metric that uses large pre-trained language models to extract latent representations of utterances, and leverages the temporal transitions that exist between them. We show that our model achieves higher correlation with human annotations in an online setting, while not requiring true responses for comparison during inference.

Subfields: Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Dialogue & Conversational AI

Citations in the corpus

Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (2024, methodology): “Reference-free evaluation metrics Reference-free evaluation metrics have a long history, including classic methods (Louis & Nenkova, 2013) and more recent neural supervised learning methods (Kryscinski et al., 2020; Sinha et al., 2020; Goyal & Durrett, 2020).”
A Survey of Evaluation Metrics Used for NLG Systems (2020, background): “[143] propose MaUde to score dialogue responses without any need for reference responses.”
Recent advances in deep learning based dialogue systems: a systematic survey (2021, methodology): “Likewise, Sinha et al. (2020) trained a BERT-based model to discriminate between true and false responses, where false responses were automatically generated.”
Spurious Correlations in Reference-Free Evaluation of Text Generation (2022, methodology): “Sinha et al. (2020) propose a model that encodes each utterance in the dialog context using a pre-trained BERT model and leverages the temporal transitions between them to score a response.”
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022, background): “Most of them focus on turnlevel response quality (Tao et al., 2018; Ghazarian et al., 2019; Huang et al., 2020; Sai et al., 2020; Sinha et al., 2020; Zhang et al., 2021b).”
A Comprehensive Assessment of Dialog Evaluation Metrics (2021, methodology): “Compared to BERT-RUBER, MAUDE performs better on USR-PersonaChat and GRADE-ConvAI2 and has worse performance on the other quality-annotated datasets.”
What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation (2022, background): “As described earlier, most previous work in automatic evaluation focuses on text-based opendomain dialog systems (Yeh et al., 2021; Lan et al., 2020; Sinha et al., 2020; Huang et al., 2020; Ghazarian et al., 2020).”
Perturbation CheckLists for Evaluating NLG Evaluation Metrics (2021, unknown): “…0.02 0.11 0.09 0.34 0.02 0.56 0.26 0.09 0.32 0.48 0.31 0.29 0.53 0.03 0.18 0.11 0.18 0.02 0.58 0.27 0.09 (f) Image Captioning proposed metrics such as DEB (Sai et al., 2020a), BLANC (Vasilyev et al., 2020) and MaUde (Sinha et al., 2020) do not correlate well with human judgements on other criteria.”
Identifying Untrustworthy Samples: Data Filtering for Open-domain Dialogues with Bayesian Optimization (2021, background): “Recently, unreferenced metrics are also proposed to assess different dialogue attributes, such as Distinct [18], Entropy [32], MAUDE [47], USR [29], and FED [28].”
DynaEval: Unifying Turn and Dialogue Level Evaluation (2021, methodology): “A remedy to this problematic aspect of DynaEval is to introduce perturbation strategies targeting the token level, such as word drop, word shufﬂing and word replacement (Sinha et al., 2020; Park et al., 2021).”