Best practices for the human evaluation of automatically generated text

By Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, Emiel Krahmer. 2019

276 citations recorded by the source corpus.

Currently, there is little agreement as to how Natural Language Generation (NLG) systems should be evaluated. While there is some agreement regarding automatic metrics, there is a high degree of variation in the way that human evaluation is carried out. This paper provides an overview of how human evaluation is currently conducted, and presents a set of best practices, grounded in the literature. With this paper, we hope to contribute to the quality and consistency of human evaluations in NLG.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Automatic Detection of Generated Text is Easiest when Humans are Fooled (2019, background): “The related (but not identical) task of asking human raters to judge the quality of machine-generated excerpts remains the gold-standard for evaluating open-domain generation systems (van der Lee et al., 2019).”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Expanding on the suggestions by van der Lee et al. (2019), we need a wider adoption of effect size estimates, power analysis, statistical significance tests, and emphasize the importance of analysis of the validity of human evaluation results.”
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (2021, methodology): “Though evaluations with end-users in an applied setting are encouraged (Belz and Reiter, 2006), in practice, most human evaluations instead ask people to rate generated text’s intrinsic quality (van der Lee et al., 2019; Howcroft et al., 2020).”
Twenty Years of Confusion in Human Evaluation: NLG Needs Evaluation Sheets and Standardised Definitions (2020, background): “…criteria cause more variation in the responses, because different participants may weigh the importance of one of the quality criteria differently in their response; such complex quality criteria may best be measured using multiple items rather than a single question (van der Lee et al., 2019).”
Evaluation of Text Generation: A Survey (2020, background): “Furthermore, papers often leave out important details on how the human evaluations were run, such as who the evaluators are and how many people evaluated the text (van der Lee et al., 2019).”
With Little Power Comes Great Responsibility (2020, background): “But, as similarly found by van der Lee et al. (2019), many key details were not reported in this sample of experiments.”
CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022, background): “Then, we describe potential use cases of CoAuthor in HCI and NLP research, exemplifying the potential of CoAuthor serving as a boundary object.”
Data-to-text Generation with Macro Planning (2021, methodology): “They exper-iment with the WebNLG dataset (Gardent et al., 2017) which consists of RDF Subject, Object, Predicate triples paired with corresponding text.”
ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (2023, background): “Recognizing this, best practices often involve multiple human annotators collaborating in the evaluation (Van Der Lee et al., 2019).”
G ENIE A Leaderboard for Human-in-the-Loop Evaluation of Text Generation (2021, methodology): “For a broader discussion on best practices for measuring AI systems, we refer the reader to the existing studies, such as Welty et al. (2019); van der Lee et al. (2019).”
Shared Task on Evaluating Accuracy (2020, methodology): “This should include experimental design, guidance on number and type of subjects, and recommended statistical analysis (van der Lee et al., 2019).”
Controlling hallucinations at word level in data-to-text generation (2021, unknown): “Finally, we perform qualitative evaluations of the results obtained on WikiBIO and ToTTo, following the best practices outlined by van der Lee et al. [30].”
Spot the Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems (2020, methodology): “In order to perform interactive multi-turn evaluations, the standard method is to let humans converse with a chatbot and rate it afterward (Ghandeharioun et al., 2019), typically using Likert scales (van der Lee et al., 2019).”
A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems (2020, methodology): “Human evaluation of NLG systems is usually done using Likert scales or ratings (van der Lee et al., 2019).”
Disentangling the Properties of Human Evaluation Methods: A Classification System to Support Comparability, Meta-Evaluation and Reproducibility Testing (2020, background): “Human evaluations in NLG currently paint a confused picture3 with very poor standards for designing and reporting evaluations (van der Lee et al., 2019).”