Towards Quantifiable Dialogue Coherence Evaluation

By Zheng Ye, Liucun Lu, Lishan Huang, Liang Lin, Xiaodan Liang. 2021

33 citations recorded by the source corpus.

Automatic dialogue coherence evaluation has attracted increasing attention and is crucial for developing promising dialogue systems. However, existing metrics have two major limitations: (a) they are mostly trained in a simplified two-level setting (coherent vs. incoherent), while humans give Likert-type multi-level coherence scores, dubbed as “quantifiable”; (b) their predicted coherence scores cannot align with the actual human rating standards due to the absence of human guidance during training. To address these limitations, we propose Quantifiable Dialogue Coherence Evaluation (QuantiDCE), a novel framework aiming to train a quantifiable dialogue coherence metric that can reflect the actual human rating standards. Specifically, QuantiDCE includes two training stages, Multi-Level Ranking (MLR) pre-training and Knowledge Distillation (KD) fine-tuning. During MLR pre-training, a new MLR loss is proposed for enabling the model to learn the coarse judgement of coherence degrees. Then, during KD fine-tuning, the pretrained model is further finetuned to learn the actual human rating standards with only very few human-annotated data. To advocate the generalizability even with limited fine-tuning data, a novel KD regularization is introduced to retain the knowledge learned at the pre-training stage. Experimental results show that the model trained by QuantiDCE presents stronger correlations with human judgements than the other state-of-the-art metrics.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Pengfei Liu
2. Peng Liu
3. Yang Liu
4. Heng Ji
5. Chenguang Zhu
6. Graham Neubig
7. Jinlan Fu
8. Jiawei Han

In Evaluation & Benchmarks

1. See-Kiong Ng
2. Jinlan Fu
3. Pengfei Liu
4. Peng Liu
5. Chenguang Zhu
6. Yang Liu
7. Zhengbao Jiang
8. Heng Ji

Citations in the corpus

• GPTScore: Evaluate as You Desire (2023, methodology): “For example, DEAM (Ghazarian et al., 2022) and Quan-tiDCE (Ye et al., 2021) were proposed for the evaluation of the coherence of the dialogue system; several evaluators (Cao et al., 2020; Durmus et al., 2020; Wang et al., 2020a) are designed for the evaluation of the consistency of text…”
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (2023, background): “…to assess the consistency of the generated summaries (Kry´s-ci´nski et al., 2020; Wang et al., 2020; Cao et al., 2020; Durmus et al., 2020), and dialogue response generation tasks need to assess the coherence of the generated responses (Dziri et al., 2019; Ye et al., 2021; Ghazarian et al., 2019).”
• Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022, background): “, 2020) and coherence in dialogue response generation (Dziri et al., 2019; Huang et al., 2020; Ye et al., 2021).”