Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddings

By Sarik Ghazarian, Johnny Tian-Zheng Wei, Aram Galstyan, Nanyun Peng. 2019

91 citations recorded by the source corpus.

Despite advances in open-domain dialogue systems, automatic evaluation of such systems is still a challenging problem. Traditional reference-based metrics such as BLEU are ineffective because there could be many valid responses for a given context that share no common words with reference responses. A recent work proposed Referenced metric and Unreferenced metric Blended Evaluation Routine (RUBER) to combine a learning-based metric, which predicts relatedness between a generated response and a given query, with reference-based metric; it showed high correlation with human judgments. In this paper, we explore using contextualized word embeddings to compute more accurate relatedness scores, thus better evaluation metrics. Experiments show that our evaluation metrics outperform RUBER, which is trained on static embeddings.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Towards a Human-like Open-Domain Chatbot (2020, background): “Past works have attempted to build learnable metrics, either in a supervised fashion (Lowe et al., 2017), which requires human labels, or with unsupervised approaches (Tao et al., 2017; Ghazarian et al., 2019), that are more complex and need separate training, e.g., of a ranking system.”
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (2023, background): “…to assess the consistency of the generated summaries (Kry´s-ci´nski et al., 2020; Wang et al., 2020; Cao et al., 2020; Durmus et al., 2020), and dialogue response generation tasks need to assess the coherence of the generated responses (Dziri et al., 2019; Ye et al., 2021; Ghazarian et al., 2019).”
DynaEval: Unifying Turn and Dialogue Level Evaluation (2021, background): “Metrics to Compare The recently proposed reference-free state-of-the-art dialogue metrics, including USR (Mehri and Eskenazi, 2020b), BERT-RUBER (Ghazarian et al., 2019) (BERT-R), GPT-2 based coherence metric (Pang et al., 2020) (GPT-2) and FED (Mehri and Eskenazi, 2020a) 6 , serve as the baseline…”
A Survey of Evaluation Metrics Used for NLG Systems (2020, methodology): “ations from each speaker, which is necessary to obtain the speaker sensitive samples. Manuscript submitted to ACM 36 Sai, et al. Transformer based metrics RUBER with BERT embeddings: Ghazarian et al. [51] propose enhancing RUBER with contextualized embeddings. The modification to the referenced part”
Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: A Preliminary Empirical Study (2023, methodology): “…(Radford et al., 2019); USR (Mehri and Eskenazi, 2020b); S-DiCoh (Mesgar et al., 2020); FED (Mehri and Eskenazi, 2020a); DynaEval (Zhang et al., 2021); SelfEval (Ma et al., 2022); PPL (Guan et al., 2021); iBLEU (Sun and Zhou, 2012); BERT-iBLEU (Niu et al., 2021); ParaScore (Shen et al., 2022).”
Towards Quantifiable Dialogue Coherence Evaluation (2021, background): “To overcome this issue, researchers turned to develop learnable metrics based on neural networks that incorporate the semantic information, such as RUBER (Tao et al., 2018), BERT-RUBER (Ghazarian et al., 2019) and GRADE (Huang et al., 2020).”
Towards Holistic and Automatic Evaluation of Open-Domain Dialogue Generation (2020, background): “Word-overlap metrics (Papineni et al., 2002; Banerjee and Lavie, 2005; Lin, 2004) or ad-hoc classiﬁers (Tao et al., 2018; Ghazarian et al., 2019) are designed for measuring the quality.”
Identifying Untrustworthy Samples: Data Filtering for Open-domain Dialogues with Bayesian Optimization (2021, methodology): “Inspired by the automatic evaluation conducted in other fields, referenced automatic metrics are applied [12, 33, 48].”
Adversarial Language Games for Advanced Natural Language Intelligence (2019, background): “also is not well-established, which has received much attention from the community (Tao et al., 2018; Ghazarian et al., 2019).”
What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation (2022, methodology): “This matches the conclusion in (Mehri and Eske-nazi, 2020) who point out that FED captures the dialog-level qualities from its training data Reddit better than turn-level qualities.”
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022, background): “Most of them focus on turnlevel response quality (Tao et al., 2018; Ghazarian et al., 2019; Huang et al., 2020; Sai et al., 2020; Sinha et al., 2020; Zhang et al., 2021b).”
CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation (2022, unknown): “3) Hybrid metrics contain both referenced and reference-free scores, such as RUBER (Tao et al., 2018; Ghaz-arian et al., 2019), BLEURT (Sellam et al., 2020) and BARTScore (Yuan et al., 2021).”
Ditch the Gold Standard: Re-evaluating Conversational Question Answering (2021, background): “, 2016; Reiter, 2018), and have proposed new evaluation protocols to mitigate the problems and align more with how humans would evaluate language systes in a real-world setting (Ghazarian et al., 2019; Gehrmann et al., 2021).”
DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulations (2022, methodology): “Random utterance substitution from other conversations is a simple way to inject incoherence sources in dialogues, which has been frequently used in prior work (Tao et al., 2018; Ghazarian et al., 2019; Mesgar et al., 2020; Zhang et al., 2021).”
A Comprehensive Assessment of Dialog Evaluation Metrics (2021, methodology): “Compared to BERT-RUBER, MAUDE performs better on USR-PersonaChat and GRADE-ConvAI2 and has worse performance on the other quality-annotated datasets.”
UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation (2020, methodology): “Furthermore, RUBER and its variants (Tao et al., 2018; Ghazarian et al., 2019) evaluate a response by directly averaging a non-learnable referenced embedding similarity score and a learnable unreferenced post-response relatedness score that is learned by applying negative sampling without human…”
AVA: an Automatic eValuation Approach to Question Answering Systems (2020, background): “For dialog systems, neural-based automatic evaluators are also studied (Ghazarian et al., 2019; Lowe et al., 2017; Tao et al., 2017; Kannan and Vinyals, 2017) QA has been traditionally studied early in literature (Green et al., 1961).”