The ReproGen Shared Task on Reproducibility of Human Evaluations in NLG: Overview and Results

By Anya Belz, Anastasia Shimorina, Shubham Agarwal, Ehud Reiter. 2021

49 citations recorded by the source corpus.

The NLP field has recently seen a substantial increase in work related to reproducibility of results, and more generally in recognition of the importance of having shared definitions and practices relating to evaluation. Much of the work on reproducibility has so far focused on metric scores, with reproducibility of human evaluation results receiving far less attention. As part of a research programme designed to develop theory and practice of reproducibility assessment in NLP, we organised the first shared task on reproducibility of human evaluations, ReproGen 2021. This paper describes the shared task in detail, summarises results from each of the reproduction studies submitted, and provides further comparative analysis of the results. Out of nine initial team registrations, we received submissions from four teams. Meta-analysis of the four reproduction studies revealed varying degrees of reproducibility, and allowed very tentative first conclusions about what types of evaluation tend to have better reproducibility.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “To achieve this goal, we need to work toward reusability and replicability in human evaluations, for example by filling in human evaluation datasheets (Shimorina & Belz, 2021; Belz et al., 2021) or CrowdWorkSheets (Dı́az et al., 2022) and by contributing and using projects that standardize parts of ”
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap (2023, background): “However, human evaluation is costly and also lacks established practices about what and how to evaluate, leading to critiques about lack of standardization, reproducibility, validity, and generalizability to real-world settings [14, 37, 59, 80].”