How Robust Are Multirater Interrater Reliability Indices to Changes in Frequency Distribution?

By David Quarfoot, Richard A. Levine. 2016

104 citations recorded by the source corpus.

ABSTRACT Interrater reliability studies are used in a diverse set of fields. Often, these investigations involve three or more raters, and thus, require the use of indices such as Fleiss’s kappa, Conger’s kappa, or Krippendorff’s alpha. Through two motivating examples—one theoretical and one from practice—this article exposes limitations of these indices when the units to be rated are not well-distributed across the rating categories. Then, using a Monte Carlo simulation and information visualizations, we argue for the use of two alternative indices, the Brennan–Prediger coefficient and Gwet’s AC2, because the agreement levels reported by these indices are more robust to variation in the distribution of units that raters encounter. The article concludes by exploring the complex, interwoven relationship between the number of levels in a rating instrument, the agreement level present among raters, and the distribution of units that are to be scored. Supplementary materials for this article are available online.

Subfields: Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/323ef0379f55673663d910a0860eb905b7cf0c11

Citations in the corpus

• Garbage in, garbage out?: do machine learning application papers in social computing report where human-labeled training data comes from? (2019, unknown): “However, this requires assuming a uniform prior possibility of such a random distribution, which generally only applies if each possible response by raters is equally likely [43, 48].”