The measurement of interrater agreement

1,271 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Evaluating the use of search engine development tools in IT education (2010, positive): “The first one considers the κ coefficient (Fleiss et al., 2003) provided by Social Sciences.”
Towards Sustainable Energy: A Systematic Review of Renewable Energy Sources, Technologies, and Public Opinions (2019, methodology): “The result of data extraction was assessed by using interrater agreement between two researchers using the κ coefficient [37].”
WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (2024, background): “Fleiss Kappa [10] scores are 0.55, 0.72, and 0.50 for the three tasks, indicating moderate to substantial agreement.”
On the Stability of Feature Selection Algorithms (2017, methodology): “…interesting relationship that could be used in future work is that Fleiss’ Kappa has also been linked to the Intra-Class Correlation Coefficient (ICC) in the binary case (Fleiss et al., 2004), so any result that applies to the ICC can also be applied to the proposed stability estimator. q = 2.”
Statistical validation of image segmentation quality based on a spatial overlap index. (2004, background): “75, as recommended by Fleiss (14).”