Comparing Rating Scales and Preference Judgements in Language Evaluation

By Anya Belz, Eric Kow. 2010

45 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “However, Belz and Kow (2010) argue that rating scales like those used in direct assessments (i.e., evaluating a generated text alone, without referencing other candidates) have many issues: they are unintuitive, agreement numbers are low, and most statistical measures are inappropriate for ordinal…”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “…and proceed to make comparisons of each item to a ‘modulus’, which serves as a comparison point throughout the experiment (see Bard et al., 1996).11 Belz and Kow (2010) compared a preference-based paradigm to a standard rating scale to evaluate systems from two different domains (weather…”
RankME: Reliable Human Ratings for Natural Language Generation (2018, methodology): “We draw upon previous studies in language generation, e.g. (Belz and Kow, 2010, 2011; Siddharthan and Katsos, 2012), as well as in the related ﬁeld of machine translation (MT), e.g. (Bojar et al., 2016, 2017).”
What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation (2022, unknown): “can have lower inter-annotator agreement (Belz and Kow, 2010).”
Can Large Language Models Transform Computational Social Science? (2023, background): “The ranking-style evaluation is more reliable and less variable than scoring for generation tasks (Harzing et al., 2009; Belz and Kow, 2010).”
Towards Best Experiment Design for Evaluating Dialogue System Output (2019, methodology): “terestingness (Asghar et al., 2017; Santhanam and Shaikh, 2019), ﬂuency or readability (Zhang et al., 2018), engagement (Venkatesh et al., 2018). While experiment designs primarily use Likert scales, Belz and Kow (2010) argue that discrete scales, such as the Likert scales, can be unintuitive and ce”
Disentangling the Properties of Human Evaluation Methods: A Classification System to Support Comparability, Meta-Evaluation and Reproducibility Testing (2020, unknown): “Others veer towards ﬂuency: “how ﬂuent and readable [the text is]” (Belz and Kow, 2010); “readability concerns ﬂuency of the textual data” (Mahapatra et al., 2016).”