Towards Holistic and Automatic Evaluation of Open-Domain Dialogue Generation

By Bo Pang, Erik Nijkamp, Wenjuan Han, A. Zhou, Yixian Liu. 2020

90 citations recorded by the source corpus.

Open-domain dialogue generation has gained increasing attention in Natural Language Processing. Its evaluation requires a holistic means. Human ratings are deemed as the gold standard. As human evaluation is inefficient and costly, an automated substitute is highly desirable. In this paper, we propose holistic evaluation metrics that capture different aspects of open-domain dialogues. Our metrics consist of (1) GPT-2 based context coherence between sentences in a dialogue, (2) GPT-2 based fluency in phrasing, (3) n-gram based diversity in responses to augmented queries, and (4) textual-entailment-inference based logical self-consistency. The empirical validity of our metrics is demonstrated by strong correlations with human judgments. We open source the code and relevant materials.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Dialogue & Conversational AI

Citations in the corpus

A Comprehensive Assessment of Dialog Evaluation Metrics (2021, methodology): “HolisticEval (Pang et al., 2020) evaluates several qualities of dialog: context coherence , lan-4 guage ﬂuency , response diversity , and logical self-consistency .”
What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation (2022, background): “∗Work done while Sarik Ghazarian was an intern at Amazon Alexa AI contradiction (Pang et al., 2020; Nie et al., 2021) amongst others.”
Identifying Untrustworthy Samples: Data Filtering for Open-domain Dialogues with Bayesian Optimization (2021, methodology): “Some automatic metrics [28, 29, 32, 46, 48, 56] that correlate more with human judgment have been proposed recently, but they are not accepted thoroughly by the entire community.”
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022, background): “The idea of decomposing overall dialogue quality into ﬁne-grained dimensions has been explored in prior works (Mehri and Eskenazi, 2020b; Phy et al., 2020; Pang et al., 2020; Zhang et al., 2021c) for turn-level evaluation.”
Recent advances in deep learning based dialogue systems: a systematic survey (2021, background): “Pang et al. (2020) combined four metrics in their automatic evaluation framework: the context coherence metric based on GPT-2; phrase fluency metric based on GPT-2; diversity metric based on n-grams; logical self-consistency metric based on textual-entailmentinference.”
Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks (2020, background): “The issue is even higher in dialogue generation where there are a plenitude of acceptable responses (Pang et al., 2020).”
Explaining Dialogue Evaluation Metrics using Adversarial Behavioral Analysis (2022, methodology): “We use three different evaluation metrics Dialo-gRPT (Gao et al., 2020), Pang-Evaluation (Pang et al., 2020) and User Satisfaction (Sun et al., 2021b) during the training while components of Pang-Evaluation rely on pre-trained neural backbones.”
DynaEval: Unifying Turn and Dialogue Level Evaluation (2021, methodology): “For example, the GPT-2 based coherence metric (Pang et al., 2020) computes a turn-level score based on averaging the token-wise conditional log probabilities in the corresponding response.”