Best-Worst Scaling: Theory, Methods and Applications

By Jordan J. Louviere, Terry N. Flynn, A. A. J. Marley. 2015

561 citations recorded by the source corpus.

Subfields: Economics and Econometrics, Plant Science, Tourism, Leisure and Hospitality Management, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Plant Science

In Tourism, Leisure and Hospitality Management

In Evaluation & Benchmarks

Citations in the corpus

Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (2018, methodology): “The study was conducted on the Amazon Mechanical Turk platform using Best-Worst Scaling (BWS; Louviere and Woodworth 1991; Louviere et al. 2015), a less labor-intensive alternative to paired comparisons that has been shown to produce more reliable results than rating scales (Kiritchenko and…”
Leveraging Pre-trained Checkpoints for Sequence Generation Tasks (2019, methodology): “The study was conducted on the Amazon Mechanical Turk platform using Best-Worst Scaling, a less labor-intensive alternative to paired comparisons (Louviere and Woodworth, 1991; Louviere et al., 2015).”
Evaluation of Text Generation: A Survey (2020, methodology): “For example, best-worst scaling (Louviere et al., 2015) has been used in NLG tasks (Kiritchenko & Mohammad, 2016; Koncel-Kedziorski et al.”
Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation (2017, methodology): “5N–2N tuples need to be annotated) (Louviere et al., 2015; Kiritchenko and Mohammad, 2016a).”
Data-to-Text Generation with Content Selection and Planning (2018, positive): “Our study used Best-Worst Scaling (BWS; Louviere, Flynn, and Marley 2015), a technique shown to be less labor-intensive and providing more reliable results as compared to rating scales (Kiritchenko and Mohammad 2017).”
HighRES: Highlight-based Reference-less Evaluation of Summarization (2019, methodology): “The relative assessment is often done using the paired comparison (Thurstone, 1994) or the best-worst scaling (Woodworth and G, 1991; Louviere et al., 2015), to improve inter-annotator agreement.”
Text Generation from Knowledge Graphs with Graph Transformers (2019, methodology): “We used Best-Worst Scaling (BWS; (Louviere and Woodworth, 1991; Louviere et al., 2015)), a less labor-intensive alternative to paired comparisons that has been shown to produce more reliable results than rating scales (Kiritchenko and Mohammad, 2016).”
A Framework for the Computational Linguistic Analysis of Dehumanization (2020, background): “While it is challenging to directly solicit opinions from human annotators about such a complex and subjective concept such as dehumanization, annotation strategies such as best-worst scaling can be promising approaches (Louviere et al., 2015), particularly if they primarily consider opinions from m”
Towards Ethics by Design in Online Abusive Content Detection (2020, methodology): “An efﬁcient comparative technique, widely used in marketing research, is Best–Worst Scaling (Lou-viere and Woodworth, 1990; Louviere et al., 2015).”
Aspect-Controllable Opinion Summarization (2021, methodology): “The ﬁrst study assessed the quality of general opinion summaries using Best-Worst Scaling (BWS; Louviere et al., 2015).”
Obtaining Reliable Human Ratings of Valence, Arousal, and Dominance for 20,000 English Words (2018, methodology): “Notably, we use a comparative annotation technique called Best-Worst Scaling (BWS) that addresses the limitations of traditional rating scales (Louviere, 1991; Cohen, 2003; Louviere et al., 2015).”
Few-Shot Learning for Opinion Summarization (2020, methodology): “Best-Worst Scaling We performed human evaluation with the Best-Worst scaling (Louviere and Woodworth, 1991; Louviere et al., 2015; Kiritchenko and Mohammad, 2016) on the Amazon and Yelp test sets using the AMT platform.”
Discrete Choice Experiments: A Guide to Model Specification, Estimation and Software (2017, background): “[15] for further description and comparison of the three cases.”
Why school students choose and reject science: a study of the factors that students consider when selecting subjects (2017, methodology): “The BWS method is a well-validated technique that allows factors identified as impacting a decision-making process to be ranked according to their importance (Louviere et al., 2015).”