Non-Expert Evaluation of Summarization Systems is Risky

By D. Gillick, Yang Liu. 2010

104 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Gillick and Liu (2010) find that MTurk judges were better at measuring generated summaries’ linguistic quality than their content or overall quality and had a much higher correlation”
Learning-Based Single-Document Summarization with Compression and Anaphoricity Constraints (2016, methodology): “Gillick and Liu (2010) showed that for linguistic quality judgments (as opposed to content judgments), Turkers reproduced the ranking of systems according to expert judgments.”
CTRLsum: Towards Generic Controllable Text Summarization (2020, background): “Thus we set g control =“ the purpose of the present invention is ” to yield a concise summary on patent purpose.”
Automatically Assessing Machine Summary Content Without a Gold Standard (2013, background): “Gillick and Liu (2010) hypothesize that at least 17.5 hours are needed to evaluate two systems under this set up on a standard test set.”
Can Large Language Models Be an Alternative to Human Evaluations? (2023, methodology): “Human evaluation is an important method to understand the performance of an NLP model or algorithm (Guzmán et al., 2015; Gillick and Liu, 2010).”
Best Practices for Crowd-based Evaluation of German Summarization: Comparing Crowd, Expert and Automatic Evaluation (2020, background): “Contrary to the ﬁndings of Gillick and Liu (2010) and Fabbri et al. (2020), we achieve signiﬁcant correlations between the crowd and expert ratings ranging from moderate to very strong magnitude, as well as no signiﬁcant difference in absolute mean rating in between the crowd and expert assessment…”
Annotating and Modeling Fine-grained Factuality in Summarization (2021, background): “Human evaluation similarly focused on content selection (Gillick and Liu, 2010).”
Revisiting Automatic Evaluation of Extractive Summarization Task: Can We Do Better than ROUGE? (2022, unknown): “As mentioned in (Gillick and Liu, 2010), non-expert annotation can be risky, so we only considered expert annotations as followed by (Fabbri et al., 2021) Table 7 shows that our proposed metric outperforms ROUGE in terms of consistency (the most crucial dimension perhaps) for all 3 types of…”
Jointly Learning to Extract and Compress (2011, methodology): “To evaluate linguistic quality, we sent all the summaries to Mechanical Turk (with two times redun-System dancy), using the template and instructions designed by Gillick and Liu (2010).”
Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand (2021, background): “Recent work, however, points out problems when evaluations are done by crowdworkers even when extensive quality controls are performed (Gillick and Liu, 2010; Toral et al., 2018; Freitag et al., 2021; Clark et al., 2021; Fabbri et al., 2021).”
A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods (2021, methodology): “Some studies report having difficulty obtaining high-quality judgments from crowdworkers (Gillick and Liu, 2010; Fabbri et al., 2021), whereas others have been successful using the crowdsourced Lightweight Pyramid Score (Shapira et al.”
Crowdsourcing Lightweight Pyramids for Manual Summary Evaluation (2019, methodology): “Also, this method may tend to suffer from transitivity inconsistencies when comparing multiple system pairs (Gillick and Liu, 2010).”
Towards Summarization for Social Media - Results of the TL;DR Challenge (2019, background): “Gillick and Liu (2010) cautioned that workers have difﬁculties distinguishing the content of a summary from its text quality.”
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation (2021, unknown): “, 2018; Castilho, 2021) or summarization (Gillick and Liu, 2010; Iskender et al., 2020).”
SummEval: Re-evaluating Summarization Evaluation (2020, methodology): “Considering the concerns raised in previous work (Gillick and Liu, 2010) about the quality differences between crowd-sourced and expert annotations we study this issue using the human annotations collected as part of this work.”
Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries (2021, background): “This reﬂects similar observations made by Gillick and Liu (2010).”
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization (2021, background): “Prior work (Gillick and Liu, 2010) shows that there can be divergence in annotations between experts and non-experts in summarization, and because the authors of the paper are NLP researchers familiar with the limitations of automatic summarizations, we expect that FactCC annotations differs in qual”