Squibs and Discussions: The Kappa Statistic: A Second Look

By Barbara Maria Di Eugenio, Michael Glass. International Conference on Computational Logic, 2004

545 citations recorded by the source corpus.

In recent years, the kappa coefficient of agreement has become the de facto standard for evaluating intercoder agreement for tagging tasks. In this squib, we highlight issues that affect and that the community has largely neglected. First, we discuss the assumptions underlying different computations of the expected agreement component of . Second, we discuss how prevalence and bias affect the measure.

Subfields: Management Science and Operations Research, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/be5bf416308b898de16041e1c5c8f89cb8faf55a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Domestic violence crimes and children: A population-based investigation of direct sensory exposure and the nature of involvement (2009, unknown): “%) represents strong agreement (DiEugenio & Glass, 2004).”
Synthesis Lectures on Human Language Technologies (2009, methodology): “In such cases, Krippendorff ’s alpha can be used, because it makes chance agreement independent of individual preferences (DiEugenio and Glass, 2004).”
Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators’ Disagreement (2021, background): “While there has been an extensive discussion on minimal standards for inter-annotator agreement to ensure data quality (Di Eugenio and Glass, 2004; Passonneau, 2004; Artstein and Poesio, 2008), recently an increasing number of works argue that disagreement is unavoidable because language is…”
Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial (2012, background): “Di Eugenio and Glass (2004) show how two variants of Cohen’s (1960) kappa (Byrt, Bishop, & Carlin, 1993; Siegel & Castellan, 1988, pp. 284-291) may be selected based on problems of prevalence and bias in the marginal distributions.”
Death to Kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment (2011, background): “…be pervasive in spite of harsh criticisms for decades 45 from many authors (Brennan and Prediger 1981, Aickin 1990, Foody 1992, Ma and Redmond 1995, Stehman 1997, Stehman and Czaplewski 1998, Foody 2002, Turk 2002, Jung 2003, Di Eugenio and Glass 2004, Foody 2004, Allouche et al. 2006, Foody 2008).”
What Determines Inter-Coder Agreement in Manual Annotations? A Meta-Analytic Investigation (2011, background): “…a rising number of publications on methodological and statistical issues of how to measure and interpret inter-annotator agreement (Carletta 1996; DiEugenio and Glass 2004; Passoneau 2004; Versley 2006; Artstein and Poesio 2008) as well as a rising number of reports on annotation quality and…”
FactBank: a corpus annotated with event factuality (2009, background): “As pointed out in Di Eugenio and Glass (2004), however, Cohen's kappa suffers from skewed distributions of categories (the so-called prevalence problem), as well as from the degree to which the annotators disagree (the bias problem).”
A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples (2013, background): “Di Eugenio B, Glass M: The Kappa Statistic: A Second Look.”
Computational Sociolinguistics: A Survey (2016, background): “Reproducibility of results is highly valued in both fields, which requires reliability wherever human judgment is involved, such as in the production of a gold standard (Carletta 1996; Di Eugenio and Glass 2004).”
Bias decreases in proportion to the number of annotators (2005, background): “We should note that a single coder’s bias cannot be measured in and of itself—it can only be measured by comparing the coder’s distribution of judgments to some other distribution.”
Can all tags be used for search? (2008, background): “The partial disagreement observed may be caused by ambiguity of the classification scheme, but also because the Topic category is so prevalent: Since this category is very frequently assigned (264 out of 300 ratings) chance agreement (expected frequency) is enormously high which leads to a reduced κ”
Automatic detection of rumor on Sina Weibo (2012, background): “The κ statistic is deﬁned as where Pr( a ) is the relative observed agreement among an-notators, and Pr( e ) is the probability of chance agreement [2] [4].”
Harshness in image classification accuracy assessment (2008, methodology): “In addition, widely used scales to interpret the kappa coefficient are problematic and arbitrary (Manel et al. 2001, Di Eugenio and Glass 2004).”
A review of recent advances in learner and skill modeling in intelligent learning environments (2012, background): “In addition, kappa varies substantially based on changes in the proportion in the original data labels (DiEugenio and Glass 2004), an issue when student performance is not close to 50%.”