Evaluating Content Selection in Summarization: The Pyramid Method

By Ani Nenkova, Rebecca J. Passonneau. 2004

747 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “…(Bangalore et al., 2000), overlap between “basic elements” (Hovy et al., 2005),5 automatically constructed content units (Tauchmann and Mieskes, 2020) using the Pyramid framework by Nenkova and Passonneau (2004), dependency parses (Pratapa et al., 2021), or sequence alignment (Deng et al., 2021).”
On Faithfulness and Factuality in Abstractive Summarization (2020, methodology): “Pyramid method (Nenkova and Passonneau, 2004) requires summaries to be annotated by experts for salient information.”
CIDEr: Consensus-based image description evaluation (2014, background): “Interestingly, similar ideas have been used previously to evaluate text summarization [31].”
Neural Text Summarization: A Critical Evaluation (2019, background): “Manual and semi-automatic (Nenkova and Passonneau, 2004; Passonneau et al., 2013) evaluation of large-scale summarization models is costly and”
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries (2020, background): “Other metrics have focused on specific aspects of summarization quality, including content selection (Nenkova and Passonneau, 2004), relevance prediction (Daume III and”
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (2019, methodology): “Metrics such as Pyramid (Nenkova and Passonneau, 2004) and BE (Hovy et al.”
Evaluation of Text Generation: A Survey (2020, methodology): “The PYRAMID method is a semi-automatic evaluation method (Nenkova & Passonneau, 2004) for evaluating the performance of document summarization models.”
Assessing The Factual Accuracy of Generated Text (2019, background): “nslation 1https://github.com/tensorflow/tensor2tensor 2https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/data_ generators/wikifact and ROUGE [13], Basic Elements [8] & Pyramid [22] for text summarization. In Steinberger and Jezek [33], the authors explain the different kinds ”
BARTScore: Evaluating Generated Text as Text Generation (2021, background): “Semantic Coverage (COV): How many semantic content units from reference texts are covered by the generated hypothesis [41].”
The limits of automatic summarisation according to ROUGE (2017, background): “The metric returns an average of ROUGE scores over multiple reference summaries in order to avoid bias (Nenkova and Passonneau, 2004).”
Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints (2018, background): “Conceptually, the likelihood term governs the grammatical structure of the response while the topic and semantic constraints drive content selection (Nenkova and Passonneau, 2004; Barzi-lay and Lapata, 2005).”
The Impact of Frequency on Summarization (2005, methodology): “We now briefly describe content units as introduced by (Nenkova and Passonneau, 2004) and then turn to studying content unit frequency.”
Multi-document Summarization via Deep Learning Techniques: A Survey (2020, methodology): “After sorting SCUs in the order of importance to form the Pyramid [93], the quality of automatic summarization is evaluated by calculating the number and importance of SCUs included in the document [94].”
Long story short - Global unsupervised models for keyphrase based meeting summarization (2010, methodology): “Evaluation measures for summarization performance like ROUGE (Lin, 2004) or Pyramid (Nenkova and Passonneau, 2004) and later developments like Basic Elements (Hovy et al.”
DialSummEval: Revisiting Summarization Evaluation for Dialogues (2022, methodology): “(2020) used pyramid (Nenkova and Passonneau, 2004), a widely used human evaluation method on several conventional summarization datasets to obtain relevance scores for some of the system outputs and re-evaluated the”
A compositional context sensitive multi-document summarizer: exploring the factors that influence summarization (2006, methodology): “One of the issues studied since the inception of automatic summarization in the 60s is that of human agreement [24]: different people choose different content for their summaries [27, 23, 19].”