Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation

By Svetlana Kiritchenko, Saif M. Mohammad. 2017

218 citations recorded by the source corpus.

Rating scales are a widely used method for data annotation; however, they present several challenges, such as difficulty in maintaining inter- and intra-annotator consistency. Best–worst scaling (BWS) is an alternative method of annotation that is claimed to produce high-quality annotations while keeping the required number of annotations similar to that of rating scales. However, the veracity of this claim has never been systematically established. Here for the first time, we set up an experiment that directly compares the rating scale method with BWS. We show that with the same total number of annotations, BWS produces significantly more reliable results than the rating scale.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “Kiritchenko and Mohammad (2017) demonstrated that best-worst scaling (asking evaluators to choose the best and the worst items in a set) is an efficient and reliable method for collecting annotations, and this approach has been used to collect comparative evaluations of generated text (e.g., Liu…”
Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (2018, positive): “2015), a less labor-intensive alternative to paired comparisons that has been shown to produce more reliable results than rating scales (Kiritchenko and Mohammad, 2017).”
Text Summarization with Pretrained Encoders (2019, methodology): “Specifically, we followed the Best-Worst Scaling (Kiritchenko and Mohammad, 2017) method where participants were presented with the output of two systems (and the original document) and”
Data-to-Text Generation with Content Selection and Planning (2018, positive): “Our study used Best-Worst Scaling (BWS; Louviere, Flynn, and Marley 2015), a technique shown to be less labor-intensive and providing more reliable results as compared to rating scales (Kiritchenko and Mohammad 2017).”
Towards Ethics by Design in Online Abusive Content Detection (2020, methodology): “It has been shown that Best–Worst Scaling produces more reliable annotations as compared to the traditional rating scales, especially on linguistically more complex items (Kiritchenko and Mohammad, 2017).”
PSP: Pre-trained Soft Prompts for Few-Shot Abstractive Summarization (2022, unknown): “Scoring followed the Best-Worst Scaling method (Kiritchenko and Mohammad, 2017).”
Crowdsourcing Subjective Tasks: The Case Study of Understanding Toxicity in Online Discussions (2019, result): “The research results of [17] support the hypothesis that "humans may often be able to make more accurate ratings using comparative measures" [26], [40], [34], [33].”
Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries (2021, methodology): “Prior research has claimed that Best-Worst Scaling produces higher-quality evaluations than rating scales such as the Likert Scale for tasks such as sentiment analysis (Kiritchenko and Mohammad, 2017).”
Towards Best Experiment Design for Evaluating Dialogue System Output (2019, methodology): “Although the Best-Worst scaling method has been shown to be effective in obtaining crowdsourced ratings of emotions (Kiritchenko and Mohammad, 2017), we did not find it to be effective in this study.”
Sentiment Analysis: Detecting Valence, Emotions, and Other Affectual States from Text (2016, methodology): “In fact, recent work has shown that a large majority of sentiment and emotion analysis machine learning systems consistently give different emotionality scores to sentences mentioning different races and genders (Kiritchenko and Mohammad, 2018).”
Choose Your Own Adventure: Paired Suggestions in Collaborative Writing for Evaluating Story Generation Models (2021, result): “Comparative evaluations have been shown to produce more reliable and consistent results than Likert-scale ratings (Callison-Burch et al., 2007; Kiritchenko and Mohammad, 2017), and they have been used to evaluate natural language generation systems for translation and dialogue (Otani et al.”
Hierarchical Transformers for Multi-Document Summarization (2019, methodology): “Previous solutions include model transfer (Zhang et al., 2018; Lebanoff and Liu, 2018), where a sequence-to-sequence model is pretrained on single-document summarization data and finetuned on DUC (multi-document) benchmarks, or unsupervised models relying on reconstruction objectives (Ma et al.,…”