Automatic Evaluation of Linguistic Quality in Multi-Document Summarization

By Emily Pitler, Annie Louis, Ani Nenkova. 2010

105 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “The lack of correlation between language and content quality was also noted by Pitler et al. (2010) who find correlations between some evaluation categories.”
System Combination for Multi-document Summarization (2015, methodology): “Redundancy: Redundancy correlates negatively with content quality (Pitler et al., 2010).”
Decontextualization: Making Sentences Stand-Alone (2021, background): “It has been observed that disambiguating entity mentions and correctly resolving anaphora is crucial for automatic summarization (Otterbacher et al., 2002; Steinberger et al., 2007) and for evaluation of summarization systems (Pitler et al., 2010).”
Re-evaluating Automatic Summarization with BLEU and 192 Shades of ROUGE (2015, background): “…used to compute human coverage scores, it has been shown that the two dimensions of quality do not correlate with one another (no signiﬁcant correlation) (Pitler et al., 2010), providing evidence that coverage scores alone do not fully represent human judgment of the overall quality of summaries.”
A Neural Local Coherence Model (2017, background): “…grid and its extensions have been successfully applied to many downstream tasks including coherence rating (Barzilay and Lapata, 2008), essay scoring (Burstein et al., 2010), story generation (McIntyre and Lapata, 2010), and readability assessment (Pitler et al., 2010; Barzilay and Lapata, 2008).”