What happens if you treat ordinal ratings as interval data? Human evaluations in NLP are even more under-powered than you think

By David M. Howcroft, Verena Rieser. 2021

22 citations recorded by the source corpus.

Previous work has shown that human evaluations in NLP are notoriously under-powered. Here, we argue that there are two common factors which make this problem even worse: NLP studies usually (a) treat ordinal data as interval data and (b) operate under high variance settings while the differences they are hoping to detect are often subtle. We demonstrate through simulation that ordinal mixed effects models are better able to detect small differences between models, especially in high variance settings common in evaluations of generated texts. We release tools for researchers to conduct their own power analysis and test their assumptions. We also make recommendations for improving statistical power.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Howcroft and Rieser (2021) demonstrate that treating ordinal data as interval data makes tests even more underpowered, which is what most papers do when analyzing rating and Likert scales (68 out of 85 recent papers, according to Amidei et al. (2019b)).”