Disentangling the Properties of Human Evaluation Methods: A Classification System to Support Comparability, Meta-Evaluation and Reproducibility Testing

By Anya Belz, Simon Mille, David M. Howcroft. 2020

78 citations recorded by the source corpus.

Current standards for designing and reporting human evaluations in NLP mean it is generally unclear which evaluations are comparable and can be expected to yield similar results when applied to the same system outputs. This has serious implications for reproducibility testing and meta-evaluation, in particular given that human evaluation is considered the gold standard against which the trustworthiness of automatic metrics is gauged. %and merging others, as well as deciding which evaluations should be able to reproduce each other’s results. Using examples from NLG, we propose a classification system for evaluations based on disentangling (i) what is being evaluated (which aspect of quality), and (ii) how it is evaluated in specific (a) evaluation modes and (b) experimental designs. We show that this approach provides a basis for determining comparability, hence for comparison of evaluations across papers, meta-evaluation experiments, reproducibility testing.

Subfields: Artificial Intelligence, Information Systems, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Evaluation & Benchmarks

Citations in the corpus

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, methodology): “They thus suggest reporting standards for criteria and methods, following a classification system proposed by Belz et al. (2020).”
Measuring Attribution in Natural Language Generation Models (2021, background): “Human evaluation studies are varied from paper to paper and detailed, reproducible annotation instructions are unavailable (Belz et al., 2020).”
Twenty Years of Confusion in Human Evaluation: NLG Needs Evaluation Sheets and Standardised Definitions (2020, methodology): “For the top three levels of branches in the taxonomy we used the quality criterion properties from Belz et al. (2020): (i) goodness vs. correctness vs. features; (ii) quality of output in its own right vs. quality of output relative to input vs. quality of output relative to an external frame of…”
Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation (2021, background): “Belz et al. (2020) proposed a categorization for different NLG quality aspects.”
A Review of Human Evaluation for Style Transfer (2021, methodology): “Inspired by recent critiques of human evaluations of Natural Language Generation ( NLG ) systems (Howcroft et al., 2020; Lee, 2020; Belz et al., 2020, 2021; Shimorina and Belz, 2021), we conduct a structured review of human evaluation for neural style transfer systems as their evaluation is…”
A Study of Automatic Metrics for the Evaluation of Natural Language Explanations (2021, background): “It has been brought into question whether a single automatic measure is able to capture multiple aspects of subjective human evaluation (Belz et al., 2020).”
Underreporting of errors in NLG output, and what to do about it (2021, background): “…different kinds of errors that appear in the output of different kinds of Natural Language Generation (NLG) systems, and to develop a general taxonomy of NLG output errors, based on the publications that have appeared at previous INLG conferences (similar to Howcroft et al. 2020; Belz et al. 2020).”
MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers (2021, background): “• Each of the criteria is a “Goodness” criteria as per the classification of [3].”