Rethinking the Agreement in Human Evaluation Tasks

By Jacopo Amidei, Paul Piwek, Alistair Willis. 2018

54 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (2021, methodology): “evaluation measures like agreement (Amidei et al., 2018), as well as analyses of past NLG papers’ human evaluations (van der Lee et al.”
Evaluation of Text Generation: A Survey (2020, background): “However, as Amidei et al. (2018) point out, given the richness and variety of natural language, pushing for the highest possible inter-annotator agreement may not be the right choice when it comes to NLG evaluation.”
Agreement is overrated: A plea for correlation to assess human evaluation reliability (2019, background): “However, Sampson and Babarczy (2008), Lom-mel et al. (2014), Joshi et al. (2016) and Amidei et al. (2018b) suggest that a low level of IAA can be explained with human language variability.”
A Survey of Evaluation Metrics Used for NLG Systems (2020, background): “Achieving a high-enough IAA is more difficult on some NLG tasks which have room for subjectivity [2].”
Best practices for the human evaluation of automatically generated text (2019, background): “Gkatzia and Mahamood (2015) focused on NLG papers from 2005-2014; Amidei et al. (2018a) provided a 2013-2018 overview of evaluation in question generation; and Gatt and Krahmer (2018) provided a more general survey of the state-of-the-art in NLG.”
Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries (2021, background): “Various methodologies have been proposed to measure inter- and intra-annotator consistency in human evaluation (Amidei et al., 2018).”
Can Large Language Models Be an Alternative to Human Evaluations? (2023, background): “However, it is important to note that these limitations and potential harms also apply to human evaluation: the bias of human evaluators can affect the human evaluation result (Lentz and De Jong, 1997; Amidei et al., 2018).”