Generation Challenges: Results of the Accuracy Evaluation Shared Task
By Craig Thomson, Ehud Reiter. 2021
21 citations recorded by the source corpus.
The Shared Task on Evaluating Accuracy focused on techniques (both manual and automatic) for evaluating the factual accuracy of texts produced by neural NLG systems, in a sports-reporting domain. Four teams submitted evaluation techniques for this task, using very different approaches and techniques. The best-performing submissions did encouragingly well at this difficult task. However, all automatic submissions struggled to detect factual errors which are semantically or pragmatically complex (for example, based on incorrect computation or inference).
Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Elizabeth Clark
- Sebastian Gehrmann
- Thibault Sellam
- Yejin Choi
- Percy Liang
- Noah A. Smith
- Mohit Iyyer
- Jianfeng Gao
In Information Systems
In Natural Language Processing
- Thibault Sellam
- Elizabeth Clark
- Sebastian Gehrmann
- Yejin Choi
- Yonatan Belinkov
- Tatsunori B. Hashimoto
- Greg Durrett
- Percy Liang
In Evaluation & Benchmarks
Citations in the corpus
- Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “…used in the accuracy shared task at INLG 2021 (Reiter and Thomson, 2020; Thomson and Reiter, 2020) produced high inter-annotator agreement, but Thomson and Reiter (2021) reported that each 300-word text took an annotator 20-30 minutes to evaluate and the annotation cost for a single generated…”