Computing Krippendorff's Alpha-Reliability

By Klaus Krippendorff. 2011

1,455 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Towards a Human-like Open-Domain Chatbot (2020, methodology): “We measure crowd worker consistency for every model benchmark using agreement and Krippen-dorff’s alpha (Krippendorff, 2011), shown in Table 1.”
Improving alignment of dialogue agents via targeted human judgements (2022, methodology): “The IAA is computed as Krippendorff’s Alpha (Krippen- dorff, 2011), by binarising the Likert scale for rule violations into break and follow, discarding unsure ratings.”
Reframing Human-AI Collaboration for Generating Free-Text Explanations (2021, methodology): “Table 15: Inter-annotator agreement for absolute-comparison AMT studies, using Krippendorff’s α computed on an interval scale from -1 to 1.”
A Massive Collection of Cross-Lingual Web-Document Pairs (2019, neutral): “pendorff Alpha (Krippendorff, 2011) of the annotations.”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, neutral): “Furthermore, while we found relatively high Inter-Annotator Agreement (IAA) among the three annotators – Krippendorff’s α (Krippendorff, 2011) = 0 .”
DEFT: A corpus for definition extraction in free- and semi-structured text (2019, methodology): “Inter-annotator agreement (IAA) is measured us-ing a modiﬁed version of Krippendorff’s alpha (Krippendorff, 2011) with the MASI distance metric (Passonneau, 2006) in order to account for and score partial sequential overlaps of text: Where c = k and the text spans match exactly, the MASI distance…”
ProsocialDialog: A Prosocial Backbone for Conversational Agents (2022, methodology): “We measure the inter-annotator agreement for safety annotation using Krippendorff’s α (Krippendorff, 2011), which is 0.49; implying good agreement.”
COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics (2022, neutral): “The average ordinal Krippendorff alpha (0 ≤ α ≤ 1) (Krippendorff, 2007) is 0.”
BEEP! Korean Corpus of Online News Comments for Toxic Speech Detection (2020, methodology): “The inter-annotator agreement (IAA) of the label is calculated based on Krippendorff’s alpha (Krippendorff, 2011) that takes into account an arbitrary number of annotators labeling any number of instances.”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, methodology): “To calculate inter-annotator agreement, we use Krippendorff’s alpha (Krippendorff, 2011).”
Re-evaluating Evaluation in Text Summarization (2020, methodology): “After this ﬁltering, we obtain an average inter-annotator agreement (Krippendorff’s alpha (Krippendorff, 2011)) of 0.66.”
NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics (2021, neutral): “Agreement by ordinal Krippendorff alpha (0 ≤ α ≤ 1) (Krippendorff, 2007) is 0.”
GODEL: Large-Scale Pre-Training for Goal-Directed Dialog (2022, methodology): “This is computed using Krippendorff’s alpha (Krippendorff, 2011) which is suitable for Likert-scale crowd-sourced evaluation as it is applicable to any number of coders, to missing data, and to ordinal ratings.”
Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts (2021, methodology): “The inter-rater agreement was measured using Krippendorff’s alpha (Krippendorff, 2011) and pairwise agreement, which was found to be α = 0.”
Designing and Applying an ‘Extremist Media Index’ (2015, methodology): “Three coders thus graded the sample batch blindly and reliability scores were calculated using established reliability tests [19].”