Human evaluation of automatically generated text: Current trends and best practice guidelines

By Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Emiel Krahmer. 2021

216 citations recorded by the source corpus.

Abstract Currently, there is little agreement as to how Natural Language Generation (NLG) systems should be evaluated, with a particularly high degree of variation in the way that human evaluation is carried out. This paper provides an overview of how (mostly intrinsic) human evaluation is currently conducted and presents a set of best practices, grounded in the literature. These best practices are also linked to the stages that researchers go through when conducting an evaluation research (planning stage; execution and release stage), and the specific steps in these stages. With this paper, we hope to contribute to the quality and consistency of human evaluations in NLG.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “However, we did not find a single paper that estimated how many annotations should be collected, and most opted for the “typical” 100 data points which, as pointed out above, may be insufficient (van der Lee et al., 2021).”
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (2021, methodology): “…like the evaluation design (Novikova et al., 2018; Santhanam and Shaikh, 2019), question framing (Schoch et al., 2020), and evaluation measures like agreement (Amidei et al., 2018), as well as analyses of past NLG papers’ human evaluations (van der Lee et al., 2021; Howcroft et al., 2020).”
Evaluation of Text Generation: A Survey (2020, background): “Note that while these dimensions are common, they may be referred to by other names, explained to evaluators in different terms, or measured in different ways (Lee et al., 2021).”
The Human Evaluation Datasheet 1.0: A Template for Recording Details of Human Evaluation Experiments in NLP (2021, methodology): “HEDS was also informed by van der Lee et al. (2019, 2021) and by Gehrmann et al. (2021)’s1 data card guide.”
Can Large Language Models Provide Feedback to Students? A Case Study on ChatGPT (2023, unknown): “, readability [11], [13] to examine the quality of the machinegenerated text.”
Underreporting of errors in NLG output, and what to do about it (2021, background): “Hence, it may be useful to preregister your study, so as to make your thoughts and plans explicit (Haven and Grootel, 2019; van Miltenburg et al., 2021).”
Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions (2023, background): “A minimal solution of this kind would be for authors to always clarify, in their articles, what they mean by certain concepts.”
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation (2021, result): “…2018-2020 and find many critical details often go unreported (e.g., worker qualifications, payment, task descriptions, annotator agreement), a finding in line with prior reproducibility studies outside open-ended text generation (Card et al., 2020; Howcroft et al., 2020; van der Lee et al., 2021).”