Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters

By K. Gwet. 2014

1,558 citations recorded by the source corpus.

Subfields: Statistics and Probability, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

In Evaluation & Benchmarks

Citations in the corpus

Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “We used Gwet’s AC1/2 statistic (Gwet, 2008, 2014) to measure inter-rater reliability (IRR), as we found it to be the most stable metric across different measurement scenarios.”
ReCode: Robustness Evaluation of Code Generation Models (2022, background): “364], indicating that our annotation reaches “moderate agreement” and thus our annotations are reliable (Gwet, 2014).”
A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples (2013, methodology): “Therefore, as is done with the Kappa statistic, Gwet adjusted for chance agreement by using the AC1 tool, such that the AC1 between two or multiple raters is defined as the conditional probability that two randomly selected raters will agree, given that no agreement will occur by chance [9].”
Understanding Why Consumers Don't Skip Pre-Roll Video Ads (2017, methodology): “Following completion of this process, interrater consistency was assessed on a per-ad basis by calculating Pearson correlations (Gwet 2014; LeBreton and Senter 2008).”
ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AI (2021, methodology): “We measured agreement with Krippendorf’s alpha ( α ), which can take account for multiple annotators, missing values, and ordinal ratings (Gwet, 2014).”
The Paradigm-Shift of Social Spambots: Evidence, Theories, and Tools for the Arms Race (2017, methodology): “Given that each account under investigation has been clas-siﬁed by at least 3 diﬀerent contributors, we have also computed the Fleiss’ kappa ( κ ) inter-rater agreement metric [20].”
How Robust Are Multirater Interrater Reliability Indices to Changes in Frequency Distribution? (2016, background): “Indeed, under a fixed level of agreement between two raters, it is possible to get almost any IR value between the observed percentage of agreement and zero simply by adjusting the frequency with which certain ratings appear (Guggenmoos-Holzmann1993;Gwet 2002).”
Interrater agreement for Critical Care EEG Terminology (2014, methodology): “Precision of the overall group estimates for IRA was quantified by 95% confidence intervals (Cis) for the estimated j values calculated by the Jackknife method.(17,19) Overall individual performance scores were calculated as average percentage of correct answers, averaged over all 407 questions (11 ”
Teachers’ Views of Students’ Mathematical Capabilities: Challenges and Possibilities for Ambitious Reform (2017, methodology): “In addition, we calculated kappas for each of the relevant dimensions; the kappa for Explanations was 0.78 and the kappa for Supports was 0.65, both of which are generally considered to indicate high levels of agreement (Gwet, 2010).”
Network Structure, Collaborative Context, and Individual Creativity (2019, background): “and satisfactory values for the ICC, which is a measure of agreement between the two raters (Gwet, 2014).”
What is frugal innovation? Three defining criteria (2016, methodology): “Since it is customary to require α ≥ .800, our results are reliable (Gwet 2014; Krippendorff 2004).”
Improving Content Knowledge and Comprehension for English Language Learners: Findings From a Randomized Control Trial (2017, methodology): “As in prior PACT studies, interrater reliability on the fidelity instrument was established by using a gold-standard method (Gwet, 2012).”