Analyzing ordinal data with metric models: What could possibly go wrong?
By Torrin M. Liddell, John K. Kruschke. Journal of Experimental Social Psychology, 2018
620 citations recorded by the source corpus.
Abstract We surveyed all articles in the Journal of Personality and Social Psychology (JPSP), Psychological Science (PS), and the Journal of Experimental Psychology: General (JEP:G) that mentioned the term “Likert,” and found that 100% of the articles that analyzed ordinal data did so using a metric model. We present novel evidence that analyzing ordinal data as if they were metric can systematically lead to errors. We demonstrate false alarms (i.e., detecting an effect where none exists, Type I errors) and failures to detect effects (i.e., loss of power, Type II errors). We demonstrate systematic inversions of effects, for which treating ordinal data as metric indicates the opposite ordering of means than the true ordering of means. We show the same problems — false alarms, misses, and inversions — for interactions in factorial designs and for trend analyses in regression. We demonstrate that averaging across multiple ordinal measurements does not solve or even ameliorate these problems. A central contribution is a graphical explanation of how and when the misrepresentations occur. Moreover, we point out that there is no sure-fire way to detect these problems by treating the ordinal values as metric, and instead we advocate use of ordered-probit models (or similar) because they will better describe the data. Finally, although frequentist approaches to some ordered-probit models are available, we use Bayesian methods because of their flexibility in specifying models and their richness and accuracy in providing parameter estimates. An R script is provided for running an analysis that compares ordered-probit and metric models.
Subfields: Management Science and Operations Research, Statistics and Probability
View Paper: https://www.semanticscholar.org/paper/015e20463de5c1ba4bcf58a1ef0f07d6d98c5992
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Management Science and Operations Research
- Verena Rieser
- Alistair Willis
- Paul Piwek
- Jacopo Amidei
- Jan Balaguer
- Matthew Botvinick
- John Aslanides
In Statistics and Probability
- Verena Rieser
- David M. Howcroft
- Alistair Willis
- Paul Piwek
- Jacopo Amidei
- Michael Henry Tessler
- Matthew Botvinick
- John Aslanides
Citations in the corpus
- Fine-tuning language models to find agreement among humans with diverse preferences (2022, methodology): “To further assess the reliability of the human preference for our SFT-Utilitarian model’s consensus generations over those of the ablated models, we constructed a maximal mixed-effects logistic generalized linear regression model [22], the standard in confirmatory hypothesis testing [6].”
- What happens if you treat ordinal ratings as interval data? Human evaluations in NLP are even more under-powered than you think (2021, background): “…as interval has been fought for several decades (Glass et al., 1972; Knapp, 1990; Jamieson, 2004; Carifio and Perla, 2007; Wu and Leung, 2017; Liddell and Kruschke, 2018), we argue that ordinal data needs to be analysed as ordinal in NLP: in this paper we demonstrate that this…”
- The use of rating and Likert scales in Natural Language Generation human evaluation tasks: A review and some recommendations (2019, background): “Liddell and Kruschke (2018) present several problematic cases where parametric statistic were used for ordinal data.”