Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.

By Jacob Cohen. 1968

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Towards Exploiting Background Knowledge for Building Conversation Systems (2018, methodology): “We computed inter-annotator agreement using the mean linearly weighted Cohen’s κ (Cohen, 1968) and mean Krippendorff’s α (Hayes and Krippendorff, 2007).”
Towards a Better Metric for Evaluating Question Generation Systems (2018, methodology): “In Table 3, we report the average inter-annotator agreement between the ratings using Cohen’s kappa ( κ ) score (Cohen, 1968).”
How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation (2016, background): “2 w.r.t. the other respondents, which is a standard measure for inter-rater agreement (Cohen, 1968).”
Evaluation of Text Generation: A Survey (2020, background): “For example, weighted κ (Cohen, 1968) extends Cohen’s κ by adding weights to each possible pair of score assignments.”
Overview of the 1st Workshop on Asian Translation (2014, background): “The weights for the weighted κ (Cohen, 1968) is defined as |Evaluation1− Evaluation2|/4.”
Material Parenting: How the Use of Goods in Parenting Fosters Materialism in the Next Generation (2015, methodology): “First, weighted kappa (Cohen 1968) was calculated and found to be significant for all variables.”
Learning to Think Critically (2014, methodology): “Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.”
A boost of positive affect (2013, methodology): “Weighted kappa coefficients (Cohen, 1968) were used to determine inter-rater reliability for the three positive emotions, including love ( w ¼ .49), appreciation ( w ¼ .33), and happiness ( w ¼ .47).”
Meta-analysis of Cohen’s kappa (2011, background): “In fact, Cohen’s j has been extended to estimate inter-rater reliability for many complicated rating scenarios (e.g., Berry and Mielke 1998; Cohen 1968, 1972; Davies and Fleiss 1982; Fleiss 1971; Gross 1986; Janson and Olsson 2001, 2004; Kraemer 1980; Kraemer et al. 2004; Landis and Koch 1977b;…”
An Informal Science Education Program’s Impact on STEM Major and STEM Career Outcomes (2020, methodology): “To assess consistency across coders (reliability), we calculated inter-coder agreement using both percent agreement (Lombard et al. 2002) and Cohen’s kappa (Cohen 1968) applying methodologies established for semi-structured interviews (Campbell et al. 2013).”
AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. (2009, unknown): “For each re-viewer, the data set extracted contained three quality ratings for each review, yielding a total of six ratings per review.”
Can Automatic Post-Editing Improve NMT? (2020, background): “Hence, measures such as a weighted Kappa κw (Cohen, 1968), which assigns partial credit to smaller disagreements and works better with ordinal data (such as our adequacy judgments), is more suitable.”