Computing inter-rater reliability and its variance in the presence of high agreement.

By K. Gwet. 2008

1,823 citations recorded by the source corpus.

Subfields: Management Science and Operations Research, Statistics and Probability, Surgery, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Statistics and Probability

In Surgery

In Evaluation & Benchmarks

Citations in the corpus

Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “We used Gwet’s AC1/2 statistic (Gwet, 2008, 2014) to measure inter-rater reliability (IRR), as we found it to be the most stable metric across different measurement scenarios.”
Interrater Reliability in Systematic Review Methodology: Exploring Variation in Coder Decision-Making (2018, background): “Reasons for this contradiction can be explained by the k’s paradox (Cicchetti and Feinstein 1990; Gwet 2008).”
Interrater agreement for Critical Care EEG Terminology (2014, methodology): “Gwet’s j statistics were recently developed to overcome certain well-known shortcomings of conventional IRA coefficients such as Cohen’s and Fleiss’ j statistics; specifically, these latter j statistics perform badly (exhibit “paradoxes”) when raters exhibit a high or low degree of agreement, and wh”
To Type or Not to Type: Quantifying Detectable Bugs in JavaScript (2017, neutral): “Instead, we use Gwet’s AC 1 agreement coefﬁcient, because it accommodates more than two raters and is more stable than Fleiss κ when the distribution of ratings is highly skewed (as in our case where over 80% of cases were rated as undetectable) [20, 21].”
A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples (2013, methodology): “Gwet has also proved the validity of the multiplerater version of the AC1 and the Fleiss’ Kappa statistics, using a Monte-Carlo simulation approach with various estimators [10].”
Development of the Patient Education Materials Assessment Tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information. (2014, methodology): “The kappa range for the understandability items was 0.40–0.84 and for the actionability items was 0.35–0.76 (see Table 2) For Round 4 IRR, we also calculated the Gwet’s AC1 and found strong agreement for both scales and material types.”
Protocol and quality assurance for carotid imaging in 100,000 participants of UK Biobank: development and assessment (2017, neutral): “However, Cohen’s kappa and its direct extensions tend to show paradoxical behaviour at the extremes of prevalence (for example, as in our case, if a large majority of images passed QA assessment), with very high agreement between observers sometimes even leading to negative kappas.(16,17) We therefo”
How Robust Are Multirater Interrater Reliability Indices to Changes in Frequency Distribution? (2016, background): “Indeed, under a fixed level of agreement between two raters, it is possible to get almost any IR value between the observed percentage of agreement and zero simply by adjusting the frequency with which certain ratings appear (Guggenmoos-Holzmann1993;Gwet 2002).”
Head-to-head Comparison of Transrectal Ultrasound-guided Prostate Biopsy Versus Multiparametric Prostate Resonance Imaging with Subsequent Magnetic Resonance-guided Biopsy in Biopsy-naïve Men with Elevated Prostate-specific Antigen: A Large Prospective Multicenter Clinical Study. (2019, methodology): “parametric MRI and histopathology reader agreement was calculated using Gwet's agreement coefficient (AC) [20].”
Three Decades of Reliability in Communication Content Analyses (2016, methodology): “” The use of the term “chance” is unfortunate because such false agreements are not necessarily the results of random behavior (Gwet, 2008; Riffe, Lacy, & Fico, 2005).”
Variance Estimation of Nominal-Scale Inter-Rater Reliability with Random Selection of Raters (2008, methodology): “The procedures discussed by Gwet (2008) are valid for inferring to the universe of subject only for a given sample of raters.”
Online Hate Ratings Vary by Extremes: A Statistical Analysis (2019, methodology): “Thus, an additional measure was used, namely, Gwet’s AC2, that can be used in the presence of high expected agreement by chance, as it does not assume independence between raters [7].”