Best-worst scaling: theory and methods

By Terry N. Flynn, A. A. J. Marley. 2014

204 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/9ee7513116a8fa37c4bc7ff2e5b904117695aa23

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

Citations in the corpus

Adaptive Wavenet Vocoder for Residual Compensation in GAN-Based Voice Conversion (2018, methodology): “To alleviate such a problem, in this paper we also use an evaluation technique called Best-Worst Scaling (BWS) [41, 42], which can handle a long list of options and always generates discriminating results as the respondents are asked to choose the BEST and WORST option in a choice set.”
An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning (2020, background): “Scaling (BWS) [255] is proposed for voice conversion quality assessment [22], where listeners are presented only with a few randomly selected options each time.”
SemEval-2018 Task 1: Affect in Tweets (2018, methodology): “9 Annotation Aggregation: The intensity scores were calculated from the BWS responses using a simple counting procedure (Orme, 2009; Flynn and Marley, 2014): For each item, the score is the proportion of times the item was chosen as having the most intensity minus the percentage of times the item wa”
Obtaining Reliable Human Ratings of Valence, Arousal, and Dominance for 20,000 English Words (2018, methodology): “Real-valued scores of association between the items and the property of interest can be determined using simple arithmetic on the number of times an item was chosen best and number of times it was chosen worst (as described in Section 3) (Orme, 2009; Flynn and Marley, 2014).”
Towards Ethics by Design in Online Abusive Content Detection (2020, background): “5 × N to 2 × N such n -tuples (where N is the total number of textual instances to be annotated), a real-valued score of severity can be calculated for each textual instance, and a ranked list of instances by severity can be obtained (Flynn and Marley, 2014; Kiritchenko and Mohammad, 2016).”
Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation (2017, methodology): “We applied the counting procedure to obtain real-valued term–sentiment scores from the BWS annotations (Orme, 2009; Flynn and Marley, 2014): the term’s score was calculated as the percentage of times the term was chosen as most positive minus the percentage of times the term was chosen as most negat”