Free-Marginal Multirater Kappa (multirater K[free]): An Alternative to Fleiss' Fixed-Marginal Multirater Kappa.

By Justus J. Randolph. 2005

625 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Mohit Bansal
2. Xu Han
3. Zhiyuan Liu
4. Caiming Xiong
5. Zhe Gan
6. Yi Tay
7. Percy Liang
8. Maosong Sun

Citations in the corpus

• FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation (2018, methodology): “For the remaining 105 relations, we calculate the inter-annotator agreement for each relation using the free-marginal multirater kappa (Randolph, 2005), and keep the top 100 relations.”
• Explainable Automated Fact-Checking for Public Health Claims (2020, methodology): “For the human evaluation we computed Ran-dolph’s free-marginal κ (Randolph, 2005) and over-all agreement (O.A.) for all multiple choice questions.”
• Towards Expert-Level Medical Question Answering with Large Language Models (2023, methodology): “MMLU clinical topics On MMLU clinical topics, Med-PaLM 2 significantly improves over previously reported results in Med-PaLM [1] and is the state-of-the-art on 3 out 6 topics, with GPT-4-base reporting better numbers in the other three.”
• A framework for automatic question generation from text using deep reinforcement learning (2018, methodology): “We measure inter-rater agreement using Ran-dolph’s free-marginal multirater kappa (Randolph, 2005).”
• SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents (2023, unknown): “503 (Randolph, 2005).”
• Low-Resource Generation of Multi-hop Reasoning Questions (2020, methodology): “We used Randolph’s free-marginal kappa (Randolph, 2005) to measure the agreements among the raters.”
• All You May Need for VQA are Image Captions (2022, methodology): “For each dataset, 50 examples were rated by all raters, resulting in a free-margin Kappa (Randolph, 2005) of 0.71 for VQ 2 A -COCO and 0.59 for VQ 2 A -CC3M, corresponding to high inter-rater agreement.”
• Putting the Horse before the Cart: A Generator-Evaluator Framework for Question Generation from Text (2018, methodology): “We measure inter-rater agreement using Ran-dolph’s free-marginal multirater kappa (Randolph, 2005).”
• How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation (2010, methodology): “The kappa statistic used in the following analysis is called free-marginal kappa (see [3, 21]) and can be utilized when the annotators are not forced to assign a certain number of documents to each concept.”
• FeTaQA: Free-form Table Question Answering (2021, background): “We report % of samples that have score ≥ 4 to show high quality of FeTaQA, and report percent agreement and Randolph’s Kappa (Randolph, 2005) (with 95% CI) to show that our human evaluation has high inter-annotator agreement.”
• Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction (2017, background): “Interannotator agreement was also good at 0.724 κfree (Randolph, 2005).”
• Context Sensitivity Estimation in Toxicity Detection (2021, background): “2 The free-marginal kappa (Randolph, 2010) of the CCC annotations is 83.93%, while the average (mean pairwise) percentage agreement is 92%.”