DUC 2005: Evaluation of Question-Focused Summarization Systems

By Hoa Trang Dang. 2006

92 citations recorded by the source corpus.

The Document Understanding Conference (DUC) 2005 evaluation had a single user-oriented, question-focused summarization task, which was to synthesize from a set of 25--50 documents a well-organized, fluent answer to a complex question. The evaluation shows that the best summarization systems have difficulty extracting relevant sentences in response to complex questions (as opposed to representative sentences that might be appropriate to a generic summary). The relatively generous allowance of 250 words for each answer also reveals how difficult it is for current summarization systems to produce fluent text from multiple documents.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “However, this choice was not empirically motivated, and from DUC 2005 onwards, the recall scores of ROUGE-2 and ROUGE-SU4 were even used instead (Dang, 2006).6 On top of the disconnect between the past and present choices, both of them are suboptimal. Rankel et al. (2013) find that rarely used confi”
MLSUM: The Multilingual Summarization Corpus (2020, background): “Document Understanding Conference Several small and high-quality summarization datasets in English (Harman and Over, 2004; Dang, 2006) have been produced in the context of the Document Understanding Conference (DUC).”
Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies (2018, background): “Datasets produced for the Document Understanding Conference (DUC) 1 are small, high-quality datasets developed to evaluate summarization systems (Harman and Over, 2004; Dang, 2006).”
Aspect-Controllable Opinion Summarization (2021, background): “, 2018), style (Cao and Wang, 2021), or textual queries (Dang, 2006).”
From Local to Global: A Graph RAG Approach to Query-Focused Summarization (2024, background): “Instead, a more appropriate task framing is query-focused summarization (QFS, Dang, 2006), and in particular, query-focused abstractive summarization that generates natural language summaries and not just concatenated excerpts (Baumel et al., 2018; Laskar et al., 2020; Yao et al., 2017) .”
QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization (2021, background): “Alternatively, summarization systems should adopt a more ﬂexible and interactive approach that allows people to express their interests and caters to their diverse intents when generating summaries (Dang, 2005, 2006; Litvak and Vanetik, 2017; Baumel et al., 2018).”
An Investigation into the Validity of Some Metrics for Automatically Evaluating Natural Language Generation Systems (2009, methodology): “Viterbi generation: Do a Viterbi search of the generation forest for a given input, which maximizes the joint likelihood of all decisions taken in the generation process.”
Comparing Rating Scales and Preference Judgements in Language Evaluation (2010, background): “It is not surprising then that stable averages of quality judgements, let alone high levels of agreement, are hard to achieve, as has been observed for MT (Turian et al., 2003; Lin and Och, 2004), text summarisation (Trang Dang, 2006), and language generation (Belz and Reiter, 2006).”
Generating Referring Expressions in Context: The GREC Task Evaluation Challenges (2010, background): “It is not surprising then that stable averages of quality judgements, let alone high levels of agreement, are hard to achieve, as has been observed for mt [36, 22], text summarisation [35], and language generation [2].”