Discrete vs. Continuous Rating Scales for Language Evaluation in NLP

By Anya Belz, Eric Kow. 2011

36 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, neutral): “Belz and Kow (2011) further compare continuous and discrete rating scales and found that both lead to similar results, but raters preferred continuous scales, consistent with prior findings (Svensson, 2000).14 Contrary to these findings, Bojar et al. (2016) and Novikova et al. (2018) compare direct…”
Why We Need New Evaluation Metrics for NLG (2017, background): “…to SFREST, where we previously got our worst re- 9Note that this mismatch can also be accounted for by continuous rating scales, as suggested by Belz and Kow (2011). sults, we can see an improvement for predicting informativeness, where all WBMs now perform significantly better than the…”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, background): “Recent studies show that continuous scales allow subjects to give more nuanced judgements (Belz and Kow 2011; Graham et al. 2013; Bojar et al. 2017).”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, neutral): “While discrete, ordinal scales are the dominant method, a continuous scale – for example, one involving a visually presented slider (Gatt & Belz, 2010; Belz & Kow, 2011) – might give subjects the possibility of giving more nuanced judgements.”
RankME: Reliable Human Ratings for Natural Language Generation (2018, methodology): “It combines advances suggested by previous research, such as continuous scales (Belz and Kow, 2011), magnitude estimation (Siddharthan et al., 2012) and relative assessment (Callison-Burch et al., 2007).”
The Third Multilingual Surface Realisation Shared Task (SR’20): Overview and Evaluation Results (2020, methodology): “We used continuous sliders as rating tools, the evidence being that raters tend to prefer them (Belz and Kow, 2011).”
The ReproGen Shared Task on Reproducibility of Human Evaluations in NLG: Overview and Results (2021, unknown): “Bearing in mind we had just one reproduction study for each original study available to us, and that as discussed we have to be cautious drawing conclusions based on sample sizes of 2, there are very few tentative ﬁrst conclusions concerning re-producibility of human evaluation in NLG we have been w”
The aNALoGuE Challenge: Non Aligned Language GEneration (2016, methodology): “Here, we will explore different experimental setups for evaluation following previous shared tasks, e.g. (Belz and Kow, 2011).”
The First Surface Realisation Shared Task: Overview and Evaluation Results (2011, methodology): “We used continuous sliders as rating tools (see Figures 1 and 2), because raters tend to prefer them (Belz and Kow, 2011).”
The First Multilingual Surface Realisation Shared Task (SR’18): Overview and Evaluation Results (2018, methodology): “, 2011), we used continuous sliders as rating tools, because raters tend to prefer them (Belz and Kow, 2011).”
A Survey of Evaluation Metrics Used for NLG Systems (2020, background): “However, some works [9, 46, 51] have also experimented with a dynamic/movable continuous scale that can allow the evaluator to give more nuanced judgements.”
Disentangling the Properties of Human Evaluation Methods: A Classification System to Support Comparability, Meta-Evaluation and Reproducibility Testing (2020, background): “…and the differences between them, are intuitively clear to evaluators, but how certain is this when good intra and inter-evaluator agreement is so hard to achieve (Belz and Kow, 2011), and correlations between apparently very different criteria are so often in the high nineties (Section 3)?”
Towards Best Experiment Design for Evaluating Dialogue System Output (2019, methodology): “We chose these questions types to investigate as these are commonly used across various language evaluation tasks (Belz and Kow, 2011; Asghar et al., 2018; Novikova et al., 2018; Kiritchenko and Mohammad, 2017) .”