Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries

By Xiangru Tang, Alexander R. Fabbri, Ziming Mao, Griffin Adams, Borui Wang, Haoran Li, Yashar Mehdad, Dragomir R. Radev. 2021

27 citations recorded by the source corpus.

Current pre-trained models applied for summarization are prone to factual inconsistencies that misrepresent the source text. Evaluating the factual consistency of summaries is thus necessary to develop better models. However, the human evaluation setup for evaluating factual consistency has not been standardized. To determine the factors that affect the reliability of the human evaluation, we crowdsource evaluations for factual consistency across state-of-the-art models on two news summarization datasets using the rating-based Likert Scale and ranking-based Best-Worst Scaling. Our analysis reveals that the ranking-based Best-Worst Scaling offers a more reliable measure of summary quality across datasets and that the reliability of Likert ratings highly depends on the target dataset and the evaluation design. To improve crowdsourcing reliability, we extend the scale of the Likert rating and present a scoring algorithm for Best-Worst Scaling that we call value learning. Our crowdsourcing guidelines will be publicly available to facilitate future work on factual consistency in summarization.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Best-worst scaling was also more recently been shown as a more effective approach than Likert scales for assessing factual consistency of summaries (Tang et al., 2022).”
Does Summary Evaluation Survive Translation to Other Languages? (2021, background): “The human an-029 notation alone is a complicated effort, requiring030 careful planning and setup (Kryscinski et al., 2020;031 Tang et al., 2021; Iskender et al., 2021).032 What purpose do the human annotations serve?”
CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning (2021, background): “Tang et al. (2021) found that Likert scales are a more consistent measure of factuality for ab-stractive dialogue summarization than Best-Worst Scaling.”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, background): “However, this benchmark only covers a single dataset (CNNDM) without a focus on similarly-performing state-of-the-art systems, which may skew metric analysis (Tang et al., 2022a) and not fully reflect realistic scenar- ios (Deutsch et al., 2022b).”