RankME: Reliable Human Ratings for Natural Language Generation

By Jekaterina Novikova, Ondrej Dusek, Verena Rieser. 2018

124 citations recorded by the source corpus.

Human evaluation for natural language generation (NLG) often suffers from inconsistent user ratings. While previous research tends to attribute this problem to individual user preferences, we show that the quality of human judgements can also be improved by experimental design. We present a novel rank-based magnitude estimation method (RankME), which combines the use of continuous scales and relative assessments. We show that RankME significantly improves the reliability and consistency of human ratings compared to traditional evaluation methods. In addition, we show that it is possible to evaluate NLG systems according to multiple, distinct criteria, which is important for error analysis. Finally, we demonstrate that RankME, in combination with Bayesian estimation of system quality, is a cost-effective alternative for ranking multiple NLG systems.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, neutral): “…but raters preferred continuous scales, consistent with prior findings (Svensson, 2000).14 Contrary to these findings, Bojar et al. (2016) and Novikova et al. (2018) compare direct assessments and relative rankings and find that the rankings produced were very similar, but Novikova et al.…”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “…that human evaluation results can differ signiﬁcantly with small alterations to instructions or the provided UI (Xu et al., 2020; Li et al., 2019; Novikova et al., 2018), which makes them hard to replicate and compare (Howcroft et al., 2020), we recommend using the provided tooling as a way to…”
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (2021, background): “There are critiques and recommendations for different aspects of human evaluations, like the evaluation design (Novikova et al., 2018; Santhanam and Shaikh, 2019), question framing (Schoch et al.”
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons (2019, methodology): “Novikova, Du ˇ sek, and Rieser (2018) provide a method that combines continuous scales and relative assessments, but in single-turn, rather than multi-turn evaluation.”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, methodology): “18The original study (Novikova et al., 2018) was limited to comparing 3 similar systems on 100 utterances.”
Twenty Years of Confusion in Human Evaluation: NLG Needs Evaluation Sheets and Standardised Definitions (2020, background): “…2018), but come with their own issues, such as cost and time involved, the need for domain expertise (Celikyilmaz et al., 2020), and the fact that the experimental setup has a substantial impact on the reliability of human quality judgements (Novikova et al., 2018; Santhanam and Shaikh, 2019).”
Findings of the E2E NLG Challenge (2018, methodology): “Rank-based Magnitude Estimation (RankME) (Novikova et al., 2018) was used for evaluation, where crowd workers compared outputs of 5 systems for the same MR and assigned scores on a continuous scale.”
Evaluation of Text Generation: A Survey (2020, methodology): “One way to address this is to use a method like RankME (Novikova et al., 2018), which adds magnitude estimation (Bard et al.”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, methodology): “…that human evaluation results can differ significantly with small alterations to instructions or the provided UI (Xu et al., 2020; Li et al., 2019; Novikova et al., 2018), which makes them hard to replicate and compare (Howcroft et al., 2020), we recommend using the provided tooling as a way to…”
A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language Models (2022, methodology): “A simple way is to make a binary decision, i.e., good or bad, and a more complex way is to use finer-grained decisions: e.g., Likert scale as shown in Figure 8(a), and RankME in Figure 8(b), etc. [12, 46, 93].”
Scarecrow: A Framework for Scrutinizing Machine Text (2021, background): “Text is typically evaluated using a few global criteria, such as ﬂuency and relevance, using discrete (e.g., 5-point) (Sai et al., 2020) or continuous scales (Novikova et al., 2018).”
Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons (2022, background): “…ings into ( 5 2 ) pairwise comparisons of systems.1040 Grammatical Error Correction: The Gram-1041 marly evaluation datasets follow the RankME1042 (Novikova et al., 2018) annotation style where an-1043 notators were shown 8 outputs side by side for each1044 input and were asked to provide a…”
Findings of the 2021 Conference on Machine Translation (WMT21) (2021, methodology): “This year we extended the bilingual source-based human evaluation with contrastive evaluation using segment-level pairwise direct assessments (Novikova et al., 2018; Sakaguchi and Van Durme, 2018).”
A Crowd-based Evaluation of Abuse Response Strategies in Conversational Agents (2019, methodology): “We collect appropriateness ratings given a stimulus (the prompt) and four randomly sampled responses from our corpus that the worker is to label following the methodology described in (Novikova et al., 2018), where each utterance is rated relatively to a reference on a userdefined scale.”
A Study of Automatic Metrics for the Evaluation of Natural Language Explanations (2021, background): “According to the literature, Informativeness can provide “timely, relevant and useful information” (Novikova et al., 2018) and “make information immediately accessible” (Maxwell et al., 2017).”
Rethinking the Agreement in Human Evaluation Tasks (2018, methodology): “In recent work, Novikova et al. (2018), use a rank-based magnitude estimation methodology in order to improve the agreement of human judgements.”
Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity (2020, background): “Following ﬁndings from (Novikova et al., 2018; Van Der Lee et al., 2019) for acquiring more consistent human ratings, texts from different systems generated for the same meaning representation are presented in every task for annotators to score them relative to each other.”
“This is a Problem, Don’t You Agree?” Framing and Bias in Human Evaluation for Natural Language Generation (2020, background): “significant overlap, such as with naturalness, grammaticality, and fluency (Mir et al., 2019; Novikova et al., 2018).”