Twenty Years of Confusion in Human Evaluation: NLG Needs Evaluation Sheets and Standardised Definitions

By David M. Howcroft, Anya Belz, Miruna Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, Verena Rieser. 2020

263 citations recorded by the source corpus.

Human assessment remains the most trusted form of evaluation in NLG, but highly diverse approaches and a proliferation of different quality criteria used by researchers make it difficult to compare results and draw conclusions across papers, with adverse implications for meta-evaluation and reproducibility. In this paper, we present (i) our dataset of 165 NLG papers with human evaluations, (ii) the annotation scheme we developed to label the papers for different aspects of evaluations, (iii) quantitative analyses of the annotations, and (iv) a set of recommendations for improving standards in evaluation reporting. We use the annotations as a basis for examining information included in evaluation reports, and levels of consistency in approaches, experimental design and terminology, focusing in particular on the 200+ different terms that have been used for evaluated aspects of quality. We conclude that due to a pervasive lack of clarity in reports and extreme diversity in approaches, human evaluation in NLG presents as extremely confused in 2020, and that the field is in urgent need of standard methods and terminology.

Subfields: Sociology and Political Science, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Sociology and Political Science

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “However, if researchers do not report the details of their studies, no one can judge whether any of these biases would apply; surveys of NLG papers find as few as 35% (Howcroft et al., 2020) and 16% (Schoch et al., 2020) of papers share the questions used in their human evaluations.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, methodology): “…human evaluations of generated texts can provide complementary insights to automatic evaluation (Manning et al., 2020), it can also lead to contradicting results since studies often omit crucial replication details and assume different definitions of the measured quantities (Howcroft et al., 2020).”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, background): “, 2022), which are well known to pose unique challenges for automatic evaluation (Lowe et al., 2017; Howcroft et al., 2020; Celikyilmaz et al., 2021).”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, neutral): “, 2018), which makes them hard to replicate and compare (Howcroft et al., 2020), we recommend using the provided tooling as a way to compare human evaluation results to those from previous work.”
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (2021, methodology): “This, along with efforts to standardize human evaluation design (Belz et al., 2020; Howcroft et al., 2020) and deployment (Khashabi et al.”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, positive): “…small alterations to instructions or the provided UI (Xu et al., 2020; Li et al., 2019; Novikova et al., 2018), which makes them hard to replicate and compare (Howcroft et al., 2020), we recommend using the provided tooling as a way to compare human evaluation results to those from previous work.”
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation (2021, background): “Evaluation criteria: As in the survey of Howcroft et al. (2020), we observe a variety of different evaluation criteria and definitions of these criteria across the 45 papers.”
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap (2023, background): “However, human evaluation is costly and also lacks established practices about what and how to evaluate, leading to critiques about lack of standardization, reproducibility, validity, and generalizability to real-world settings [14, 37, 59, 80].”
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022, methodology): “We have thus taken the approach to develop better human evaluation for only a subset of tasks, solving issues pointed out by (Howcroft et al., 2020) and van der Lee et al. (2019), and we will release detailed instructions separately.”
Perturbation CheckLists for Evaluating NLG Evaluation Metrics (2021, background): “However, this is a time consuming and expensive process (Freitag et al., 2021; Deriu et al., 2019; Howcroft et al., 2020).”
The ReproGen Shared Task on Reproducibility of Human Evaluations in NLG: Overview and Results (2021, background): “, Clarity and Fluency are compound measures each derived from two separately assessed quality criteria, which map to the normalised quality criterion names shown in rows 4–7 in Table 3, following the taxonomy of normalised quality criteria proposed by Howcroft et al. (2020).”
The Human Evaluation Datasheet 1.0: A Template for Recording Details of Human Evaluation Experiments in NLP (2021, background): “Questions 2.1–2.5 relating to evaluated system, and 4.3.1–4.3.8 relating to response elicitation, are based on Howcroft et al. (2020), with some significant changes.”
What happens if you treat ordinal ratings as interval data? Human evaluations in NLP are even more under-powered than you think (2021, methodology): “Note that the datasets we had access to for this study are mostly use 6-points, while (van der Lee et al., 2019) found that the most frequently used rating scale in NLG research is the 5-point scale, which is also conﬁrmed by (Howcroft et al., 2020).”