The Effects of Human Variation in DUC Summarization Evaluation

By Donna K. Harman, Paul Over. 2004

80 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “2004, Harman and Over (2004) find that averaged human ratings can yield meaningful insights, but”
MLSUM: The Multilingual Summarization Corpus (2020, background): “Document Understanding Conference Several small and high-quality summarization datasets in English (Harman and Over, 2004; Dang, 2006) have been produced in the context of the Document Understanding Conference (DUC).”
Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies (2018, background): “Datasets produced for the Document Understanding Conference (DUC)1 are small, high-quality datasets developed to evaluate summarization systems (Harman and Over, 2004; Dang, 2006).”
WikiHow: A Large Scale Text Summarization Dataset (2018, methodology): “On the other hand, the existing datasets may not be large enough (DUC) to train a sequence-to-sequence model, the summaries may be limited to only headlines (Gigaword), they may be more useful as an extractive summarization dataset (New York Times) and their abstraction level might be limited (CNN/D”
Overview of DUC 2005 (2005, background): “For example, (Harman and Over, 2004) noted that a human summary can vary in its level of granularity, whether the summary has a very high-level analysis or primarily contains details.”
Automatic Text Summarization of Newswire: Lessons Learned from the Document Understanding Conference (2005, methodology): “Other possible sources of variation such as evaluator, or system/evaluator interactions and the like, are not included in the model, since they have been controlled for in the evaluation set up as discussed in (Harman & Over 2004).”
TLDR: Extreme Summarization of Scientific Documents (2020, background): “3 Most summarization datasets provide a single gold summary for a given document, despite early work in summarization evaluation identifying variety in human generated summaries (Zechner, 1996; Harman and Over, 2004).”
Automatic summarising: The state of the art (2007, background): “(2001), Daumé and Marcu (2004) and Harman and Over (2004) point out, model summary variations may swamp system differences, so model-system differences have no clear implications for summary, especially system summary, value.”
Revisiting Automatic Evaluation of Extractive Summarization Task: Can We Do Better than ROUGE? (2022, background): “…Some human raters have the tendency to give higher ratings than deserved, this is known as the Leniency problem, which results in higher variance (Harman and Over, 2004), 3) Natural language descriptions are noisy and ambiguous, which makes manual ordering of sentences by annotators even harder…”
DUC in context (2007, result): “Overall coverage evaluation results were stable (Harman & Over, 2004); i.e., despite large variations in the human-generated model summaries and large variations in the manual judgments of single-model coverage, the ranking of systems remained comparatively constant when averaged over dozens of…”