Best Practices for Crowd-based Evaluation of German Summarization: Comparing Crowd, Expert and Automatic Evaluation

By Neslihan Iskender, Tim Polzehl, Sebastian Möller. 2020

18 citations recorded by the source corpus.

One of the main challenges in the development of summarization tools is summarization quality evaluation. On the one hand, the human assessment of summarization quality conducted by linguistic experts is slow, expensive, and still not a standardized procedure. On the other hand, the automatic assessment metrics are reported not to correlate high enough with human quality ratings. As a solution, we propose crowdsourcing as a fast, scalable, and cost-effective alternative to expert evaluations to assess the intrinsic and extrinsic quality of summarization by comparing crowd ratings with expert ratings and automatic metrics such as ROUGE, BLEU, or BertScore on a German summarization data set. Our results provide a basis for best practices for crowd-based summarization evaluation regarding major influential factors such as the best annotation aggregation method, the influence of readability and reading effort on summarization evaluation, and the optimal number of crowd workers to achieve comparable results to experts, especially when determining factors such as overall quality, grammaticality, referential clarity, focus, structure & coherence, summary usefulness, and summary informativeness.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “In their work on German summarization evaluation, Iskender et al. (2020) find that non-redundancy and usefulness are very hard to assess using crowdworkers and suggest that experts should be used for them, while crowdworkers are suitable for other dimensions of text quality as long as results are…”
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation (2021, background): “Human evaluation of text quality: Most previous studies on human evaluation concentrate on constrained generation domains, such as machine translation (Guzmán et al., 2015; Graham et al., 2017; Toral et al., 2018; Castilho, 2021) or summarization (Gillick and Liu, 2010; Iskender et al., 2020).”