A Coefficient of Agreement for Nominal Scales

By Jacob Cohen. 1960

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Findings of the 2016 Conference on Machine Translation (2016, methodology): “We measured pairwise agreement among annotators using Cohen’s kappa coefficient (κ) (Cohen, 1960).”
Findings of the 2015 Workshop on Statistical Machine Translation (2015, methodology): “We measured pairwise agreement among annotators using Cohen’s kappa coefficient () (Cohen, 1960).”
Findings of the 2014 Workshop on Statistical Machine Translation (2014, methodology): “We measured pair-wise agreement among annotators using Cohen’s kappa coefﬁcient ( κ ) (Cohen, 1960).”
Are We Done with MMLU? (2024, neutral): “…0.79 (8) → 0.93 (3) 0.56 (2) → 0.88 (7) 0.71 (1) → 0.74 (1) 0.55 (9) → 0.68 (7) 0.89 (6) → 0.96 (4) Llama 3 70b 0.83 (3) → 0.93 (4) 0.55 (8) → 0.91 (4) 0.55 (10) → 0.56 (10) 0.56 (8) → 0.67 (8) 0.85 (10) → 0.96 (6) Gemini 1.5 Flash 0.82 (6) → 0.91 (8) 0.53 (10) → 0.88 (9) 0.58 (9) → 0.57 (9) 0.60…”
Hate Speech Dataset from a White Supremacy Forum (2018, background): “Table 1 shows the agreements obtained in terms of the average percent agreement ( avg % ), average Cohen’s kappa coefﬁcient (Cohen, 1960) ( avg k ), and Fleiss’ kappa coefﬁcient (Fleiss, 1971) ( fleiss ).”
Using deep learning to detect price change indications in financial markets (2017, methodology): “To measure the performance of our model we use Kohen’s kappa [18], which is used to measure the concordance between sets of given answers, taking into consideration the possibility of random agreements happening.”
Continual Few-Shot Learning for Text Classification (2021, methodology): “We calculate the percentage agreement and Cohen’s kappa (Cohen, 1960) for each category independently and report the average scores across all categories.”
On the relation between accuracy and fairness in binary classification (2015, methodology): “Therefore, we suggest using for comparison a normalized accuracy, such as Cohen’s Kappa (Cohen, 1960), which indicates by how much a classiﬁer in question is better than a random classiﬁer: where A is the accuracy of the classiﬁer in question, and R is the accuracy of a random classiﬁer, in our case”
ERASER: A Benchmark to Evaluate Rationalized NLP Models (2019, background): “All datasets have a high Cohen κ (Cohen, 1960); with substantial or better agreement.”
Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation (2017, methodology): “We computed Cohen’s kappa scores (Cohen, 1960) between the human users in pairs of two.”
Evaluation of Text Generation: A Survey (2020, background): “Cohen’s κ (Cohen, 1960) is an agreement measure that can capture evaluator agreements that may happen by chance.”
Learning Attitudes and Attributes from Multi-aspect Reviews (2012, background): “Cohen’s kappa statistic is a standard accuracy measure for document labeling tasks [7].”
Towards Debiasing Fact Verification Models (2019, background): “They agreed with the dataset labels in 94% of cases, attaining a Cohen κ of 0.88 (Cohen, 1960).”
Language to Code: Learning Semantic Parsers for If-This-Then-That Recipes (2015, background): “Unlike Cohen’s κ (Cohen, 1960), the α statistic does not require that coders be the same for each unit of analysis.”