Crowdsourcing Lightweight Pyramids for Manual Summary Evaluation

By Ori Shapira, David Gabay, Yang Gao, H. Ronen, Ramakanth Pasunuru, Mohit Bansal, Yael Amsterdamer, Ido Dagan. 2019

70 citations recorded by the source corpus.

Conducting a manual evaluation is considered an essential part of summary evaluation methodology. Traditionally, the Pyramid protocol, which exhaustively compares system summaries to references, has been perceived as very reliable, providing objective scores. Yet, due to the high cost of the Pyramid method and the required expertise, researchers resorted to cheaper and less thorough manual evaluation methods, such as Responsiveness and pairwise comparison, attainable via crowdsourcing. We revisit the Pyramid approach, proposing a lightweight sampling-based version that is crowdsourcable. We analyze the performance of our method in comparison to original expert-based Pyramid evaluations, showing higher correlation relative to the common Responsiveness method. We release our crowdsourced Summary-Content-Units, along with all crowdsourcing scripts, for future evaluations.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “However, this finding may be confounded by the collection approach as well (Shapira et al., 2019).”
CTRLsum: Towards Generic Controllable Text Summarization (2020, background): “We note that obtaining reliable human judgement for summarization from non-expert crowdsourcing workers remains an active research direction (Shapira et al., 2019), and it has been shown before that non-expert human judgement of summarization can exhibit poor correlation with expert judgement…”
Finding a Balanced Degree of Automation for Summary Evaluation (2021, methodology): “We use a Semantic Role Labeling (SRL) model (Carreras and Màrquez, 2005; Palmer et al., 2010; He et al., 2017; Shi and Lin, 2019) to achieve this goal.”
Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics (2022, methodology): “REALSumm has Mjud = 100 summaries annotated with a Lightweight Pyramid score (Shapira et al., 2019) for N = 25 systems.”
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023, background): “…(1) breaking a generation into a series of atomic facts—short statements that each contain one piece of information (Nenkova and Passonneau, 2004; Shapira et al., 2019; Zhang and Bansal, 2021; Liu et al., 2022), and (2) assigning a binary label to each atomic fact, allowing a fine-grained…”
The statistical advantage of automatic NLG metrics at the system level (2021, unknown): “On the human side, we may focus on creating larger effect sizes or reducing noise by adopting new annotation schemes (L¨aubli et al., 2018; Shapira et al., 2019) or employing professional linguists (Fabbri et al., 2020; Toral et al., 2018).”
G ENIE A Leaderboard for Human-in-the-Loop Evaluation of Text Generation (2021, result): “(2) G ENIE facilitates the study of human evaluation interfaces (Nenkova and Passonneau, 2004; Shapira et al., 2019), addressing challenges such as annotator training, inter-annotator agreement, and reproducibility, all of which can be integrated into G ENIE to compare against other evaluation…”
Revisiting Automatic Evaluation of Extractive Summarization Task: Can We Do Better than ROUGE? (2022, methodology): “We collected six pre-trained models: BERT base (Liu and Lapata, 2019), MobileBERT (Sun et al., 2020), DistilBERT (Sanh et al., 2019), RoBERTa (Liu et al., 2019), XLNet (Yang et al., 2019), GPT-2 (Radford et al., 2019), from hugging-face (Wolf et al., 2020) that were ﬁne-tuned on the CNN/DailyMail da”
Re-evaluating Evaluation in Text Summarization (2020, methodology): “Recently, Shapira et al. (2019) proposed a lightweight and crowdsourceable version of the original Pyramid, and demonstrated it on the DUC 2005 (Dang, 2005) and 2006 (Dang, 2006) multi-document summarization datasets.”
Best Practices for Crowd-based Evaluation of German Summarization: Comparing Crowd, Expert and Automatic Evaluation (2020, background): “…- also called relevance assessment - which examines how useful the source document is to satisfy the given goal; 3) “Summary informativeness” measuring how much information from the source document is preserved in the extracted summary (Mani, 2001; Conroy and Dang, 2008; Shapira et al., 2019).”
SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization (2020, background): “Recently, systems have been developed to ease the construction of Pyramid scores, e.g. (Hirao et al., 2018; Yang et al., 2016; Gao et al., 2019b; Shapira et al., 2019), but they still require human-annotated Summary Content Units (SCUs) to produce reliable scores.”
Multi-document Summarization via Deep Learning Techniques: A Survey (2020, background): “Based on Pyramid [117] calculation, the abstract sentences are manually divided into several Summarization Content Units (SCUs), each representing a core concept formed from a single word or phrase/sentence.”
Understanding the Extent to which Content Quality Metrics Measure the Information Quality of Summaries (2021, methodology): “Although it relies heavily on annotations by experts, there have been attempts to crowdsource (Shapira et al., 2019) or automate all or parts of the Pyra-mid Method (Passonneau et al., 2013; Yang et al., 2016; Hirao et al., 2018) including PyrEval (Gao et al., 2019), which we analyzed in §6.”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, methodology): “…reference summary is inspired by Semantic Content Units (SCUs) first introduced in the Pyramid annotation protocol (Nenkova and Passonneau, 2004) and reframed later in LitePyramid (Shapira et al., 2019) and subsequent annotation collection efforts (Bhandari et al., 2020; Zhang and Bansal, 2021).”
Exploring Content Selection in Summarization of Novel Chapters (2020, methodology): “We use a lightweight, sampling based approach for pyramid analysis that relies on crowd-sourcing, proposed by Shapira et al. (2019), and correlates well with the original pyramid method (Nenkova et al., 2007).”
Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary (2020, methodology): “This result is very promising for QA-based evaluation metrics because it indicates that the QA pairs cover nearly all of the information that is used by the Pyramid Method, the best-performing manual content quality evaluation.”