The Human Evaluation Datasheet 1.0: A Template for Recording Details of Human Evaluation Experiments in NLP

By Anastasia Shimorina, Anya Belz. 2021

38 citations recorded by the source corpus.

This paper introduces the Human Evaluation Datasheet, a template for recording the details of individual human evaluation experiments in Natural Language Processing (NLP). Originally taking inspiration from seminal papers by Bender and Friedman (2018), Mitchell et al. (2019), and Gebru et al. (2020), the Human Evaluation Datasheet is intended to facilitate the recording of properties of human evaluations in sufficient detail, and with sufficient standardisation, to support comparability, meta-evaluation, and reproducibility tests.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sebastian Gehrmann
2. Elizabeth Clark
3. Thibault Sellam
4. Percy Liang
5. Yejin Choi
6. Mohit Bansal
7. Jianfeng Gao
8. Ondrej Dusek

In Evaluation & Benchmarks

1. Elizabeth Clark
2. Sebastian Gehrmann
3. Thibault Sellam
4. Ehud Reiter
5. Ondrej Dusek
6. Anya Belz
7. Saad Mahamood
8. Shashi Narayan

Citations in the corpus

• Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “…achieve this goal, we need to work toward reusability and replicability in human evaluations, for example by filling in human evaluation datasheets (Shimorina and Belz, 2021; Belz et al., 2021) and by contributing and using projects that standardize parts of the process (e.g., Khashabi et al.,…”
• The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, methodology): “The detailed setups will be described in a evaluation datasheet (Shimorina and Belz, 2021).”
• A Review of Human Evaluation for Style Transfer (2021, background): “59 Inspired by recent critiques of human evaluations of Natural Language Generation (NLG) systems (Howcroft et al., 2020; Lee, 2020; Belz et al., 2020, 2021; Shimorina and Belz, 2021), we conduct a structured review of human evaluation for neural style transfer systems as their evaluation is primari”
• Reusable Templates and Guides For Documenting Datasets and Models for Natural Language Processing and Generation: A Case Study of the HuggingFace and GEM Data and Model Cards (2021, background): “Thus, Shimorina and Belz (2021) develop a Human Evaluation Datasheet, with the goal of describing human evaluation experiments rather than naturally occurring language data.”
• The ReproGen Shared Task on Reproducibility of Human Evaluations in NLG: Overview and Results (2021, methodology): “For full details regarding HEDS questions and possible values, see Shimorina and Belz (2021).”