Measuring nominal scale agreement among many raters.

By J. Fleiss. 1971

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Findings of the 2016 Conference on Machine Translation (2016, positive): “481 according to Fleiss’ Kappa (Fleiss, 1971).”
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (2022, neutral): “…moderately and are higher than or equal rates to prior work on hate speech annotation (Ross et al., 2017; Sap et al., 2020), with a Fleiss’ κ =0.46 (Fleiss, 1971) and Krippendorff’s α =0.64 (Krippen-8 Speciﬁcally, we take the max of the HARMFUL I F AI and HARMFUL I F H UMAN scores and map it into…”
FEVER: a Large-scale Dataset for Fact Extraction and VERification (2018, methodology): “We calculated the Fleiss κ score (Fleiss, 1971) to be 0.6841 which we consider encouraging given the complexity of the task.”
Can Neural Machine Translation be Improved with User Feedback? (2018, background): “12 (Fleiss, 1971).”
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2021, background): “Results are included in Table 4: For binary acceptability, we find Fleiss’ kappa (Fleiss, 1971) of 40.”
CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation (2022, neutral): “25 [15]).”
ToTTo: A Controlled Table-To-Text Generation Dataset (2020, methodology): “We compute annotator agreement and Fleiss’ kappa (Fleiss, 1971) and (3) ﬁnal sentence after the secondary grammar correction task.”
Hate Speech Dataset from a White Supremacy Forum (2018, neutral): “15 (avg k), and Fleiss’ kappa coefficient (Fleiss, 1971) (fleiss).”
Lawyers are Dishonest? Quantifying Representational Harms in Commonsense Knowledge Resources (2021, methodology): “In addition, we measure the inter-annotator agreement and report the Fleiss’ kappa scores (Fleiss, 1971) to be 0.”
MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling (2018, methodology): “As this requires a multi-annotator metric over a multi-label task, we used Fleiss’ kappa metric (Fleiss, 1971) per single dialogue act.”
TabFact: A Large-scale Dataset for Table-based Fact Verification (2019, methodology): “We follow the previous works (Thorne et al., 2018; Bowman et al., 2015) to adopt the Fleiss Kappa (Fleiss, 1971) as an indicator, where Fleiss κ = ¯ p c − ¯ p e 1 − ¯ p e is computed from from the observed agreement ¯ p c and the agreement by chance ¯ p e .”
Hurdles to Progress in Long-form Question Answering (2021, neutral): “The experiment had “fair agreement” with a Fleiss κ of 0.29 (Fleiss, 1971; Landis and Koch, 1977).”
Identifying Well-formed Natural Language Questions (2018, neutral): “The Fleiss’ kappa (Fleiss, 1971) for measuring agreement among multiple annotators is computed to be κ = 0 .”
Evaluation of Text Generation: A Survey (2020, background): “Fleiss’ κ (Fleiss, 1971) can measure agreement between multiple evaluators.”
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence (2021, background): “The Fleiss κ score (Fleiss, 1971) between the four annotations is 0.7065, which means substantial agreement.”
Detecting Hallucinated Content in Conditional Neural Sequence Generation (2020, methodology): “We compute the Fleiss’s Kappa (Fleiss, 1971) (FK) scores of our annotations for MT and the processed annotations from (Maynez et al., 2020) on abstractive summarization (Tab.”