The TIPSTER SUMMAC Text Summarization Evaluation

By Inderjeet Mani, David House, Gary Klein, Lynette Hirschman, Therese Firmin, Beth M. Sundheim. 1999

The TIPSTER Text Summarization Evaluation (SUMMAC) has established definitively that automatic text summarization is very effective in relevance assessment tasks. Summaries as short as 17% of full text length sped up decision-making by almost a factor of 2 with no statistically significant degradation in F-score accuracy. SUMMAC has also introduced a new intrinsic method for automated evaluation of informative summaries.

Subfields: Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “While some work exists that aims to standardize extrinsic evaluations (e.g., Mani et al., 1999; Gehrmann et al., 2019a), the design space is much larger.”
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries (2020, background): “The idea of evaluating summaries by their ability to answer a set of questions is also long-standing (Mani et al., 1999).”
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020, background): “To the best of our knowledge, Mani et al. (1999) ﬁrst use QA as an extrinsic evaluation for summarization: A good summary should answer key questions a reader might have about an article.”
Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation (2019, background): “(Mani et al., 1999) perform extrinsic summary evaluation by comparing the functional value of a summary vs.”
Discourse Constraints for Document Compression (2010, methodology): “Our methodology for constructing Q&A pairs and for scoring documents is inspired by the SUMMAC evaluation exercise (Mani et al. 2002).”
Centroid-based summarization of multiple documents: sentence extraction, utility-based evaluation, and user studies (2000, methodology): “The extrinsic evaluation, also called task-based evaluation, has received more attention recently at the DARPA Summarization Evaluation Conference [Mani et al., 1998].”
Introduction to the Special Issue on Summarization (2002, methodology): “In the Document Understanding Conference (DUC)-01 and DUC-02 summarization competitions (Harman and Marcu 2001; Hahn and Harman 2002), NIST used the Summary Evaluation Environment (SEE) interface (Lin 2001) to record values for precision and recall.”
Manual and automatic evaluation of summaries (2002, background): “Previous efforts in large-scale evaluation of text summarization include TIPSTER SUMMAC (Mani et al. 1998) and the Document Understanding Conference (DUC) sponsored by the National Institute of Standards and Technology (NIST).”
Articles Summarizing Scientific Articles: Experiments with Relevance and Rhetorical Status (2002, background): “Extracts are often useful in an information retrieval environment since they give users an idea as to what the source document is about (Tombros and Sanderson 1998; Mani et al. 1999), but they are texts of relatively low quality.”
DUC in context (2007, methodology): “A third task, that of question answering, was tried on a small scale in SUMMAC, and readers are referred to the report (Mani et al., 1999) for more on this pilot evaluation.”