SUMMAC: a text summarization evaluation

By Inderjeet Mani, Gary Klein, David House, Lynette Hirschman, Therese Firmin, Beth M. Sundheim. 2002

163 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Summarization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Summarization

Citations in the corpus

Ranking Sentences for Extractive Summarization with Reinforcement Learning (2018, background): “Our second experiment assessed the degree to which our model retains key information from the document following a question-answering (QA) paradigm which has been previously used to evaluate summary quality and text compression (Morris et al., 1992; Mani et al., 2002; Clarke and Lapata, 2010).”
Detecting Information-Dense Texts in Multiple News Domains (2014, background): “It is well-established that in the news domain the lead of the article forms a great summary (Mani et al. 2002), which is indeed the intention when the writer follows the inverted pyramid style.”
A Study of Translation Edit Rate with Targeted Human Annotation (2006, background): “Post editing measures have also been shown effective for text summarization evaluation (Mani et al., 2002) and natural language generation (Sripada et al., 2004). is the deﬁnitive MT measure.”
Text Summarization Techniques: A Brief Survey (2017, background): “They include SUMMAC (19961998) [81], DUC (the Document Understanding Conference, 2000-2007) [82], and more recently TAC (the Text Analysis Conference, 2008-present) 1.”
A Methodology for Extrinsic Evaluation of Text Summarization: Does ROUGE Correlate? (2005, methodology): “Section 4 demonstrates that this is a more reliable measure than that of previous gold standard methods, e.g., the LDC-Agreement method used for SUMMAC-style evaluations, and that this reliability allows us to make stronger statistical statements about the beneﬁts of summarization.”
Automatic summarising: The state of the art (2007, methodology): “This was assumed in Pollock and Zamora (1975), and tested in Brandow et al. (1995), Mani and Bloedorn (1997) and Jing et al. (1998), in SUMMAC (SUMMAC 1998, Mani et al. 2002), and by Wasson (2002).”
Automatic Summarization (2001, unknown): “” [Mani et al., 2002]”
A STUDY OF TRANSLATION ERROR RATE WITH TARGETED HUMAN ANNOTATION (2005, background): “Post editing measures have also been shown effective for text summarization evaluation [6] and natural language generation [9].”
Automatic Text Summarization: Past, Present and Future (2013, background): “These campaigns are essentially SUMMAC (1996-1998) [37], DUC (the Document Understanding Con-”
Choosing words in computer-generated weather forecasts (2005, methodology): “Although the techniques used to summarise textual input are quite different from the techniques used to summarise data (systems that summarise text generally use information retrieval techniques to identify key phrases and sentences, and then stitch these together into a coherent summary using limit”
Discourse Constraints for Document Compression (2010, methodology): “Our methodology for constructing Q&A pairs and for scoring documents is inspired by the SUMMAC evaluation exercise (Mani et al. 2002).”
Effective User Interaction for High-Recall Retrieval: Less is More (2018, methodology): “The TIPSTER Text Summarization Evaluation (SUMMAC) [19] found that short summaries of documents could reduce relevance assessment time by 40-43%.”
Multilingual Summarization Evaluation without Human Models (2010, methodology): “SUMMAC (Mani et al., 2002), in 1998, provided the ﬁrst system independent framework for summary evaluation; the Document Understanding Conference (DUC) (Over et al., 2007) was the main evaluation forum from 2000 until 2007; nowadays, the Text Analysis Conference (TAC) 1 provides a forum for…”