Assessing the Effect of Inconsistent Assessors on Summarization Evaluation
By Karolina Owczarzak, Peter A. Rankel, Hoa Trang Dang, John M. Conroy. 2012
9 citations recorded by the source corpus.
Subfields: Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Natural Language Processing
- Caiming Xiong
- Richard Socher
- Pengfei Liu
- Bryan McCann
- Sebastian Gehrmann
- Elizabeth Clark
- Dragomir R. Radev
- Graham Neubig
In Evaluation & Benchmarks
- Elizabeth Clark
- Sebastian Gehrmann
- Caiming Xiong
- Pengfei Liu
- Thibault Sellam
- Alexander R. Fabbri
- Dragomir R. Radev
- Shafiq R. Joty
Citations in the corpus
- Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “scores with other annotators (Owczarzak et al., 2012), unless we know the source of the disagreement is not subjectivity.”
- Neural Text Summarization: A Critical Evaluation (2019, background): “Owczarzak et al. (2012) inspect how inconsistencies in human annotator judgments affect the ranking of summaries and correlations with automatic evaluation metrics.”
- Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, background): “, 2022b), and noted the effects of human annotation inconsistencies on system rankings (Owczarzak et al., 2012).”