The use of rating and Likert scales in Natural Language Generation human evaluation tasks: A review and some recommendations

By Jacopo Amidei, Paul Piwek, Alistair Willis. 2019

61 citations recorded by the source corpus.

Rating and Likert scales are widely used in evaluation experiments to measure the quality of Natural Language Generation (NLG) systems. We review the use of rating and Likert scales for NLG evaluation tasks published in NLG specialized conferences over the last ten years (135 papers in total). Our analysis brings to light a number of deviations from good practice in their use. We conclude with some recommendations about the use of such scales. Our aim is to encourage the appropriate use of evaluation methodologies in the NLG community.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “However, Amidei et al. (2019a) survey 10 years of annotation agreement measures and show that almost all studies fail reliability tests.”
Evaluation of Text Generation: A Survey (2020, background): “However, while this is a common way people evaluate agreement in NLG evaluations (Amidei et al., 2019a), it does not take into account the fact that the evaluators may agree purely by chance, particularly in cases where the number of scoring categories are low or some scoring categories are much…”
Spot the Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems (2020, background): “Measuring and reporting IAA is not yet a standard practice in evaluating chatbots (Amidei et al., 2019a), and producing annotations with high IAA on open-domain conversations is prone to be impeded by subjective interpretation of feature deﬁnitions and idiosyncratic annotator behavior (Bishop and…”
What happens if you treat ordinal ratings as interval data? Human evaluations in NLP are even more under-powered than you think (2021, background): “And even when researchers do include signiﬁcance tests, they often apply the tests incorrectly (Dror et al., 2018; Amidei et al., 2019), with Amidei et al. (2019) reporting that the majority of recent papers incorrectly interpret rating and Likert scales as interval data (up to 84% for Likert…”
Think Before You Speak: Explicitly Generating Implicit Commonsense Knowledge for Response Generation (2021, methodology): “We choose to conduct pairwise comparison since multiple previous work has shown that it produces a more reliable evaluation than directly asking humans to score the response, which is a highly subjective task (Amidei et al., 2019; CallisonBurch et al., 2007; Celikyilmaz et al., 2020)”
“This is a Problem, Don’t You Agree?” Framing and Bias in Human Evaluation for Natural Language Generation (2020, background): “…et al., 2018; Gkatzia and Mahamood, 2015; van der Lee et al., 2019) with corresponding recommendations for best practices (van der Lee et al., 2019) and empirical studies or overviews investigating the effects of different question types and scales (Amidei et al., 2019; Novikova et al., 2018).”
Best practices for the human evaluation of automatically generated text (2019, background): “Do note that Likert and rating scales are treated indistinctly here (for a distinction, see Amidei et al., 2019).”