DialSummEval: Revisiting Summarization Evaluation for Dialogues

By Mingqi Gao, Xiaojun Wan. 2022

48 citations recorded by the source corpus.

Dialogue summarization is receiving increasing attention from researchers due to its extraordinary difficulty and unique application value. We observe that current dialogue summarization models have flaws that may not be well exposed by frequently used metrics such as ROUGE. In our paper, we re-evaluate 18 categories of metrics in terms of four dimensions: coherence, consistency, fluency and relevance, as well as a unified human evaluation of various models for the first time. Some noteworthy trends which are different from the conventional summarization tasks are identified. We will release DialSummEval, a multi-faceted dataset of human judgments containing the outputs of 14 models on SAMSum.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Summarization, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Pengfei Liu
2. Dragomir R. Radev
3. Caiming Xiong
4. Yixin Liu
5. Shafiq R. Joty
6. Chien-Sheng Wu
7. Graham Neubig
8. Alexander R. Fabbri

In Evaluation & Benchmarks

1. Pengfei Liu
2. Dragomir R. Radev
3. Alexander R. Fabbri
4. Caiming Xiong
5. Yixin Liu
6. Shafiq R. Joty
7. Ruilin Han
8. Chien-Sheng Wu

In Summarization

1. Yixin Liu
2. Pengfei Liu
3. Dragomir R. Radev
4. Graham Neubig
5. Peng Liu

In Dialogue & Conversational AI

1. Caiming Xiong

Citations in the corpus

• Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, background): “∗ Equal contribution for manual evaluation (Bhandari et al., 2020; Fabbri et al., 2022a; Gao and Wan, 2022).”
• Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, background): “Recent studies show that traditional automatic metrics for summarization such as ROUGE [78] and BERTScore [79] show sub-optimal performance on factual consistency measurement [80].”