Evaluating Coherence in Dialogue Systems using Entailment

By Nouha Dziri, Ehsan Kamalloo, Kory Wallace Mathewson, Osmar R Zaiane. 2019

106 citations recorded by the source corpus.

Evaluating open-domain dialogue systems is difficult due to the diversity of possible correct answers. Automatic metrics such as BLEU correlate weakly with human annotations, resulting in a significant bias across different models and datasets. Some researchers resort to human judgment experimentation for assessing response quality, which is expensive, time consuming, and not scalable. Moreover, judges tend to evaluate a small number of dialogues, meaning that minor differences in evaluation configuration may lead to dissimilar results. In this paper, we present interpretable metrics for evaluating topic coherence by making use of distributed sentence representations. Furthermore, we introduce calculable approximations of human judgment based on conversational coherence by adopting state-of-the-art entailment techniques. Results show that our metrics can be used as a surrogate for human judgment, making it easy to evaluate dialogue systems on large-scale datasets and allowing an unbiased estimate for the quality of the responses. This paper has been accepted in NAACL 2019.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, background): “With increased attention to hallucination/faithfulness problems, task-specific datasets are constructed to improve from the NLI-based metrics.”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, background): “[27] characterized the contextual consistency as a natural language inference (NLI) problem [22].”
I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling (2020, methodology): “Dziri et al. (2019a) attempted to use entailment models trained on synthetic datasets for dialogue topic coherence evaluation.”
DynaEval: Unifying Turn and Dialogue Level Evaluation (2021, background): “Besides these two major aspects, there are many metrics for other qualities, such as adequacy (D’Haro et al., 2019; Zhang et al., 2021a), consistency (Welleck et al., 2019; Dziri et al., 2019), engagement (Ghazarian et al., 2020).”
Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals (2022, background): “[32] Nouha Dziri, Ehsan Kamalloo, Kory W Mathewson, and Osmar Zaiane.”
Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022, methodology): “One way to achieve this ﬁne-grained evaluation is to develop multiple eval-uators dedicated to every single dimension (Dziri et al., 2019; Kry ´ sci ´ nski et al., 2020).”
Improving Factual Consistency Between a Response and Persona Facts (2021, background): “Persona consistency sub-reward ( R 1 ) Recent studies (Welleck et al., 2019; Dziri et al., 2019) show that consistency with factual information, such as persona facts, can be characterized as a nat-ural language inference (NLI) problem, where entailment labels can be taken as consistent labels and…”
Hierarchical Reinforcement Learning for Open-Domain Dialog (2019, methodology): “2018) as it better correlates with human judgment when evaluating dialog quality (Dziri et al. 2019).”
Automatically Exposing Problems with Neural Dialog Models (2021, background): “In terms of consistency detection, Dziri et al. (2019); Welleck et al. (2019); Li et al. (2020) introduce and suggest using natural language inference to model conversation coherence.”
Generating Persona Consistent Dialogues by Exploiting Natural Language Inference (2019, methodology): “Dziri et al. (2019) has shown that entailment techniques can be used as a surrogate for human judgment in evaluating dialogue consistency.”
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (2023, background): “…to assess the consistency of the generated summaries (Kry´s-ci´nski et al., 2020; Wang et al., 2020; Cao et al., 2020; Durmus et al., 2020), and dialogue response generation tasks need to assess the coherence of the generated responses (Dziri et al., 2019; Ye et al., 2021; Ghazarian et al., 2019).”