Unsupervised Evaluation of Interactive Dialog with DialoGPT

By Shikib Mehri, Maxine Eskénazi. 2020

215 citations recorded by the source corpus.

It is important to define meaningful and interpretable automatic evaluation metrics for open-domain dialog research. Standard language generation metrics have been shown to be ineffective for dialog. This paper introduces the FED metric (fine-grained evaluation of dialog), an automatic evaluation metric which uses DialoGPT, without any fine-tuning or supervision. It also introduces the FED dataset which is constructed by annotating a set of human-system and human-human conversations with eighteen fine-grained dialog qualities. The FED metric (1) does not rely on a ground-truth response, (2) does not require training data and (3) measures fine-grained dialog qualities at both the turn and whole dialog levels. FED attains moderate to strong correlation with human judgement at both levels.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Self-Refine: Iterative Refinement with Self-Feedback (2023, methodology): “Feedback and Refinement Error Analysis for Response Generation We perform a detailed error analysis of SELF-REFINE feedback and refinement process for Dialogue Response Generation, which we summarize in Tables Table 11 and Table 12.”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “, 2018), and that reducing repetition and increasing specificity on the part of the model can improve conversation quality (See et al., 2019; Mehri and Eskenazi, 2020).”
GPTScore: Evaluate as You Desire (2023, methodology): “To explore the correlation between aspects, we conducted an empirical analysis with INT ( interesting ) Aspect Baseline GPTScore BT BTC BTCP FED DE a01 b01 c01 d01 d03 FED dialogue-level COH 1.7 -14.9 -18.9 25.7 43.7 18.7 15.0 22.5 56.9 13.4 ERR 9 .”
AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors in Agents (2023, unknown): “To assess the agents’ general understanding and reasoning capabilities, we use four datasets: FED (Mehri & Esk ´ enazi, 2020), Commongen Challenge (Madaan et al., 2023), MGSM (Shi et al., 2023), and Logic Grid Puzzles (Srivastava et al., 2022).”
Identifying Untrustworthy Samples: Data Filtering for Open-domain Dialogues with Bayesian Optimization (2021, background): “Some automatic metrics [28, 29, 32, 46, 48, 56] that correlate more with human judgment have been proposed recently, but they are not accepted thoroughly by the entire community.”
Task-specific Objectives of Pre-trained Language Models for Dialogue Adaptation (2020, methodology): “According to the FED results in (Mehri and Esk´enazi 2020), the inter-annotator agreement is high for all of the dialogue qualities (with Spearman correlation in 0.75-0.85), indicating that these qualities are highly correlated.”
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning (2022, unknown): “TU, PU, PZ, DZ, CG, DGU, DGR, EG, FT and FD are abbreviations for TopicalChat-USR, PersonaChat-USR (Mehri and Eskenazi, 2020b), PersonaChat-Zhao (Zhao et al., 2020a), DailyDialog-Zhao (Zhao et al., 2020a), ConvAI2-GRADE (Huang et al., 2020), DailyDialog-Gupta (Gupta et al., 2019), DailyDialog-GRADE ”
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022, background): “In this section, we analyze the human ratings of FED (Mehri and Eskenazi, 2020a), a high-quality dialogue-level evaluation benchmark.”
Conversations Are Not Flat: Modeling the Dynamic Information Flow across Dialogue Utterances (2021, methodology): “Mehri and Eskénazi (2020) propose the FED score, an automatic dialogue evaluation metric using pre-trained DialoGPT-large, which works with pre-set common human comments, like “It is interesting to talk with you.””
Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation (2021, methodology): “We compare with all the diverse metrics studied in (Mehri and Eskenazi, 2020b) and FED (Mehri and Eskenazi, 2020a), a set of latest unsupervised dialogue metrics based on the DialoGPT model (Zhang et al., 2020b).”
A Comprehensive Assessment of Dialog Evaluation Metrics (2021, methodology): “RoBERTa, which is employed in USR (Mehri and Eskenazi, 2020b), improves the training techniques in BERT and trains the model on a much larger corpus which includes the CommonCrawl News dataset (Mackenzie et al., 2020) and text extracted from Reddit.”
DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulations (2022, unknown): “We also note that the performance difference between DSTC9 and FED could Figure 7: The accuracy of evaluation metrics to distinguish coherent/incoherent conversations in test data (y-axis) generated using baseline manipulations (x-axis). be due to the long conversations in DSTC9 that mostly include ”
What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation (2022, methodology): “It is also important to get some holistic evaluation at the dialog level in order to assess the dialogs as a whole (Zhang et al., 2021a; Li et al., 2021; Mehri and Eskenazi, 2020; Finch et al., 2021).”
Explaining Dialogue Evaluation Metrics using Adversarial Behavioral Analysis (2022, background): “Recent research focuses on the use of neural networks to tackle this problem (Kreyssig et al., 2018; Sun et al., 2021a; Lowe et al., 2017; Jang et al., 2020; Shim et al., 2021; Mehri and Eskenazi, 2020a; Pang et al., 2020; Gao et al., 2020; Kachuee et al., 2021).”
Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: A Preliminary Empirical Study (2023, methodology): “We conduct experiments on the dialogue-level FED dataset (Mehri and Eskenazi, 2020a), which contains fine-grained human judgments for 124 conversations.”
DynaEval: Unifying Turn and Dialogue Level Evaluation (2021, background): “This may be because dialoGPT, the backbone of FED, was trained on a large amount of Reddit data, which contain diverse amount of topics and variation of expressions while DynaEval is trained on a single dialogue domian.”
Commonsense-Focused Dialogues for Response Generation: An Empirical Study (2021, methodology): “For comparison, we consider three baselines: our MLP with only symbolic features, our MLP with only neural features, and FED (Mehri and Eskenazi, 2020a), which uses Di-aloGPT to score how likely the next turn after the response expresses confusion.”