Shared Task on Evaluating Accuracy
By Ehud Reiter, Craig Thomson. 2020
9 citations recorded by the source corpus.
We propose a shared task on methodologies and algorithms for evaluating the accuracy of generated texts, specifically summaries of basketball games produced from basketball box score and other game data. We welcome submissions based on protocols for human evaluation, automatic metrics, as well as combinations of human evaluations and metrics.
Subfields: Computer Vision and Pattern Recognition, Economics and Econometrics, Management Science and Operations Research, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Thibault Sellam
- Sebastian Gehrmann
- Elizabeth Clark
- Craig Thomson
- Ehud Reiter
- Idan Szpektor
- Yossi Matias
- Or Honovich
In Economics and Econometrics
In Management Science and Operations Research
In Evaluation & Benchmarks
- Sebastian Gehrmann
- Elizabeth Clark
- Thibault Sellam
- Craig Thomson
- Ehud Reiter
- Vered Cohen
- Idan Szpektor
- Yossi Matias
Citations in the corpus
- Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “This choice was popularized by Rush et al. (2015), who picked a subset of the options used in DUC 2004, which also included 3, 4, and LW (Over & Yen, 2004).”
- A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems (2020, background): “The shared task is described in a companion paper (Reiter and Thomson, 2020).”
- TRUE: Re-evaluating Factual Consistency Evaluation (2022, background): “(3)We focus on English text-to-text tasks, and leave data-totext (Parikh et al., 2020; Reiter and Thomson, 2020), multilingual and multimodal tasks to future work.”