The measurement of observer agreement for categorical data.

By J. Landis, Gary G. Koch. 1977

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Findings of the 2016 Conference on Machine Translation (2016, background): “The exact interpretation of the kappa coefficient is difficult, but according to Landis and Koch (1977), 0–0.2 is slight, 0.2–0.4 is fair, 0.4–0.6 is moderate, 0.6–0.8 is substantial, and 0.8–1.0 is almost perfect.”
Findings of the 2014 Workshop on Statistical Machine Translation (2014, background): “, Lopez (2012); Bojar et al.”
HateCheck: Functional Tests for Hate Speech Detection Models (2020, neutral): “93, which indicates “almost perfect” agreement (Landis and Koch, 1977).”
On Faithfulness and Factuality in Abstractive Summarization (2020, background): “All models showed substantial agreement (0.61 ≤ k ≤ 0.80; Landis and Koch, 1977) among their annotations.”
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2021, methodology): “We ﬁnd Fleiss’ kappa (Fleiss, 1971) of 40.8 indicating moderate agreement (Landis and Koch, 1977), and 90.5% accuracy agreement.”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, neutral): “889, indicating almost perfect agreement (Landis and Koch, 1977).”
Constructing Datasets for Multi-hop Reading Comprehension Across Documents (2017, positive): “ likely to be a confounding effect on the other judgments. Inter-annotator agreement as measured by Fleiss’ kappa is 0.253 in (2), and 0.281 in (3) – indicating a fair overall agreement, according to Landis and Koch (1977). Overall, 9.5% of samples have no clear majority in (2). Among samples with a”
Why We Need New Evaluation Metrics for NLG (2017, background): “measures inter-observer reliability on ordinal data for more than two raters (Landis and Koch, 1977).”
Breaking NLI Systems with Sentences that Require Simple Lexical Inferences (2018, neutral): “61 (Landis and Koch, 1977).”
ToTTo: A Controlled Table-To-Text Generation Dataset (2020, positive): “00 ) according to (Landis and Koch, 1977).”
Continual Few-Shot Learning for Text Classification (2021, background): “80 is considered as substantial agreement (Cohen, 1960; Landis and Koch, 1977).”
Unsupervised Commonsense Question Answering with Self-Talk (2020, methodology): “The annotations yielded moderate levels of agreement, with Fleiss Kappa κ = 0.43 (Landis and Koch, 1977).”
(Meta-) Evaluation of Machine Translation (2007, background): “The interpretation of Kappa varies, but according to Landis and Koch (1977) The K values for ﬂuency and adequacy should give us pause about using these metrics in the future.”
Hurdles to Progress in Long-form Question Answering (2021, neutral): “The experiment had “fair agreement” with a Fleiss κ of 0.29 (Fleiss, 1971; Landis and Koch, 1977).”
Identifying Well-formed Natural Language Questions (2018, neutral): “52 which shows moderate agreement (Landis and Koch, 1977).”
Findings of the 2009 Workshop on Statistical Machine Translation (2009, background): “The interpretation of Kappa varies, but according to Landis and Koch (1977), 0 − .”