Agreement is overrated: A plea for correlation to assess human evaluation reliability

By Jacopo Amidei, Paul Piwek, Alistair Willis. 2019

45 citations recorded by the source corpus.

Inter-Annotator Agreement (IAA) is used as a means of assessing the quality of NLG evaluation data, in particular, its reliability. According to existing scales of IAA interpretation – see, for example, Lommel et al. (2014), Liu et al. (2016), Sedoc et al. (2018) and Amidei et al. (2018a) – most data collected for NLG evaluation fail the reliability test. We confirmed this trend by analysing papers published over the last 10 years in NLG-specific conferences (in total 135 papers that included some sort of human evaluation study). Following Sampson and Babarczy (2008), Lommel et al. (2014), Joshi et al. (2016) and Amidei et al. (2018b), such phenomena can be explained in terms of irreducible human language variability. Using three case studies, we show the limits of considering IAA as the only criterion for checking evaluation reliability. Given human language variability, we propose that for human evaluation of NLG, correlation coefficients and agreement coefficients should be used together to obtain a better assessment of the evaluation data reliability. This is illustrated using the three case studies.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “However, Amidei et al. (2019a) survey 10 years of annotation agreement measures and show that almost all studies fail reliability tests.”
Evaluation of Text Generation: A Survey (2020, background): “However, while this is a common way people evaluate agreement in NLG evaluations (Amidei et al., 2019a), it does not take into account the fact that the evaluators may agree purely by chance, particularly in cases where the number of scoring categories are low or some scoring categories are much…”
Can Large Language Models Be an Alternative to Human Evaluations? (2023, methodology): “We choose to use the correlation coefficient instead of the inter-annotator agreement score because IAA mainly cares if two annotators agree on the exact ratings, while the correlation coefficient focus on the question: "when annotator A rates one story higher, does annotator B also rate the story h”
Best Practices for Crowd-based Evaluation of German Summarization: Comparing Crowd, Expert and Automatic Evaluation (2020, methodology): “Also, we calculated Krippendorff’s α , which is technically a measure of evaluator disagreement rather than agreement and the most common of the measures in the set NLG papers surveyed in Amidei et al. (2019).”
Spot the Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems (2020, background): “Measuring and reporting IAA is not yet a standard practice in evaluating chatbots (Amidei et al., 2019a), and producing annotations with high IAA on open-domain conversations is prone to be impeded by subjective interpretation of feature deﬁnitions and idiosyncratic annotator behavior (Bishop and…”