Transformation of Mean Opinion Scores to Avoid Misleading of Ranked Based Statistical Techniques

By Babak Naderi, Sebastian Möller. 2020

18 citations recorded by the source corpus.

The rank correlation coefficients and the ranked-based statistical tests (as a subset of non-parametric techniques) might be misleading when they are applied to subjectively collected opinion scores. Those techniques assume that the data is measured at least at an ordinal level and define a sequence of scores to represent a tied rank when they have precisely an equal numeric value. In this paper, we show that the definition of tied rank, as mentioned above, is not suitable for Mean Opinion Scores (MOS) and might be misleading conclusions of rank-based statistical techniques. Furthermore, we introduce a method to overcome this issue by transforming the MOS values considering their 95% Confidence Intervals. The rank correlation coefficients and ranked-based statistical tests can then be safely applied to the transformed values. We also provide open-source software packages in different programming languages to utilize the application of our transformation method in the quality of experience domain.

Subfields: Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

1. Ross Cutler
2. Babak Naderi

Citations in the corpus

• An Open source Implementation of ITU-T Recommendation P.808 with Validation (2020, methodology): “[5] B. Naderi, T. Hossfeld, M. Hirth, F. Metzger, S. Möller, and R. Z. Jiménez, “Impact of the number of votes on the reliability and validity of subjective speech quality assessment in the crowdsourcing approach,” in 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX”
• A Crowdsourcing Extension of the ITU-T Recommendation P.835 with Validation (2020, methodology): “Rank transformation is recommended in case of small number of conditions for Spearman correlation[17].”
• Subjective Evaluation of Noise Suppression Algorithms in Crowdsourcing (2021, background): “Table 3 gives the PCC, SRCC, and the SRCC rank after transformation [23] for N=5 runs, which shows very good reproducibility.”