Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretraining

By Ananya B. Sai, Akash Kumar Mohankumar, Siddharth Arora, Mitesh M. Khapra. 2020

83 citations recorded by the source corpus.

Abstract There is an increasing focus on model-based dialog evaluation metrics such as ADEM, RUBER, and the more recent BERT-based metrics. These models aim to assign a high score to all relevant responses and a low score to all irrelevant responses. Ideally, such models should be trained using multiple relevant and irrelevant responses for any given context. However, no such data is publicly available, and hence existing models are usually trained using a single relevant response and multiple randomly selected responses from other contexts (random negatives). To allow for better training and robust evaluation of model-based metrics, we introduce the DailyDialog++ dataset, consisting of (i) five relevant responses for each context and (ii) five adversarially crafted irrelevant responses for each context. Using this dataset, we first show that even in the presence of multiple correct references, n-gram based metrics and embedding based metrics do not perform well at separating relevant responses from even random negatives. While model-based metrics perform better than n-gram and embedding based metrics on random negatives, their performance drops substantially when evaluated on adversarial examples. To check if large scale pretraining could help, we propose a new BERT-based evaluation metric called DEB, which is pretrained on 727M Reddit conversations and then finetuned on our dataset. DEB significantly outperforms existing models, showing better correlation with human judgments and better performance on random negatives (88.27% accuracy). However, its performance again drops substantially when evaluated on adversarial responses, thereby highlighting that even large-scale pretrained evaluation models are not robust to the adversarial examples in our dataset. The dataset1 and code2 are publicly available.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Dialogue & Conversational AI

Citations in the corpus

Using In-Context Learning to Improve Dialogue Safety (2023, methodology): “We use five open-text generation metrics to evaluate response relevance: R OUGE -1 (Lin, 2004), F1, M ETEOR (Banerjee and Lavie, 2005), D EB (Sai et al., 2020), and S ELF -”
Towards Quantifiable Dialogue Coherence Evaluation (2021, methodology): “…experiments of the two-level setting, where both the adversarial irrelevant responses and random selected responses of the dailydialog++ dataset (Sai et al., 2020) are treated as negative samples and labeled as 0, while the golden reference responses are treated as positive samples and labeled…”
Perturbation CheckLists for Evaluating NLG Evaluation Metrics (2021, methodology): “In case of dialogue, several n-gram-based and embedding-based metrics have been shown to fall short in capturing the diversity of the valid responses (Liu et al., 2016; Sai et al., 2020a).”
DialGuide: Aligning Dialogue Model Behavior with Developer Guidelines (2022, methodology): “Coher-ence is measured using a Bert-large model trained on a combination of conversations from the DEB dataset (Sai et al., 2020) and BST and Prosocial dialogue, taking a pair of context and response as input and predicting if the response is coherent.”
A Comprehensive Assessment of Dialog Evaluation Metrics (2021, background): “DEB replaces the BERT pretraining data with a dialog corpus, and further ﬁnetunes the model on a manually-created dataset consisting of adversarial responses designed for the NSP objective.”
Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons (2022, background): “In particular, 042 many studies have shown that they do not correlate 043 well with human judgments (Novikova et al., 2017; 044 Elliott and Keller, 2014; Sai et al., 2019, 2020a,b).”
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022, background): “Most of them focus on turnlevel response quality (Tao et al., 2018; Ghazarian et al., 2019; Huang et al., 2020; Sai et al., 2020; Sinha et al., 2020; Zhang et al., 2021b).”
Ditch the Gold Standard: Re-evaluating Conversational Question Answering (2021, methodology): “In recent years, the NLP community has cautiously re-evaluated and identiﬁed ﬂaws in many popular automated evaluation strategies of dialogue systems (Liu et al., 2016; Sai et al., 2019), and have proposed new evaluation protocols to align more with human evaluation in a real-world setting: Huang et”