Interrater reliability: the kappa statistic
By M. McHugh. 2012
16,963 citations recorded by the source corpus.
The kappa statistic is frequently used to test interrater reliability. The importance of rater reliability lies in the fact that it represents the extent to which the data collected in the study are correct representations of the variables measured. Measurement of the extent to which data collectors (raters) assign the same score to the same variable is called interrater reliability. While there have been a variety of methods to measure interrater reliability, traditionally it was measured as percent agreement, calculated as the number of agreement scores divided by the total number of scores. In 1960, Jacob Cohen critiqued use of percent agreement due to its inability to account for chance agreement. He introduced the Cohen’s kappa, developed to account for the possibility that raters actually guess on at least some variables due to uncertainty. Like most correlation statistics, the kappa can range from −1 to +1. While the kappa is one of the most commonly used statistics to test interrater reliability, it has limitations. Judgments about what level of kappa should be acceptable for health research are questioned. Cohen’s suggested interpretation may be too lenient for health related studies because it implies that a score as low as 0.41 might be acceptable. Kappa and percent agreement are compared, and levels for both kappa and percent agreement that should be demanded in healthcare studies are suggested.
Subfields: Management Science and Operations Research, Statistics and Probability, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Management Science and Operations Research
- Mitesh M. Khapra
- Clíodhna O’Connor
- Brandon Yang
- Iwona Mochol
- Joseph P. Vitta
- Philip Plettig
- M. de Vos
- Helle Holmgren
In Statistics and Probability
In Evaluation & Benchmarks
- Matthew P. Lungren
- Hannaneh Hajishirzi
- Xiaosong Wang
- Ronald M. Summers
- Ali Farhadi
- Emiel Krahmer
- Mohit Bansal
- Mitesh M. Khapra
Citations in the corpus
- Towards a Better Metric for Evaluating Question Generation Systems (2018, methodology): “Based on guidelines in (McHugh, 2012) we note that we have a strong inter-annotator agreement for WikiMovies and moderate agreement for SQuAD and VQA.”
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, neutral): “4 which indicates “moderate” agreement [55].”
- SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization (2019, methodology): “For the rest, we calculated the linear weighted Cohen’s kappa coefficient (McHugh, 2012) between annotators’ scores.”
- Detecting and Classifying Malevolent Dialogue Responses: Taxonomy, Data and Methodology (2020, background): “Kappa value greater than 0.8 is nearly perfect, 0.6–0.8 is substantial and 0.4–0.6 is moderate [36].”
- Robust fine-tuning of zero-shot models (2021, background): “Cohen’s kappa coefficient is a measure of agreement between two annotators (McHugh, 2012).”
- Toward defining deep brain stimulation targets in MNI space: A subcortical atlas based on multimodal MRI, histology and structural connectivity (2016, methodology): “The overlap of each warped nucleus with its ground truth (defined by manual segmentations) was assessed computing Cohen's κ (Cohen, 1960; McHugh, 2012).”
- NegBio: a high-performance tool for negation and uncertainty detection in radiology reports (2017, methodology): “In this paper, we used the inter-rater agreement (IRA) to measure the level of agreement between two experts 32 .”
- Why Do Multi-Agent LLM Systems Fail? (2025, neutral): “Note that Cohen’s Kappa score of more than 0.8 is considered strong and more than 0.9 is considered almost perfect alignment (McHugh, 2012).”
- Game-based Learning and 21st century skills: A review of recent research (2016, background): “According to Cohen (1960) and McHugh (2012), our interrater reliability was high ( ҡ ¼ 0.82), and we resolved all disagreements to consensus through discussion.”
- Measurement of Inter-Rater Reliability in Systematic Review (2015, neutral): “6 이하인 경우 그 자료 는 적절치 않은 것으로 간주하고 있다[14].”
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (2023, background): “(McHugh, 2012) which gauges the agreement between results from models and human annotators while taking into account the possibility of agreement occurring by chance.”
- Cultural intelligence, global mindset, and cross-cultural competencies: a systematic review using bibliometric methods (2020, background): “The overall intercoder reliability between the two coders is 0.813, which is within the upper level of “perfect agreement” (McHugh, 2012).”