Answering the Call for a Standard Reliability Measure for Coding Data

By Andrew F. Hayes, Klaus Krippendorff. 2007

4,391 citations recorded by the source corpus.

Subfields: Artificial Intelligence, General Social Sciences, Statistics and Probability, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Statistics and Probability

In Evaluation & Benchmarks

Citations in the corpus

On Using Twitter to Monitor Political Sentiment and Predict Election Results (2011, methodology): “The inter-annotator agreement for the four relevant classes is 0.478 according to Krippendorff’s Alpha, a standard measure of inter-annotator agreement for many annotators (Hayes and Krippendorff, 2007).”
Gender Roles and Humor in Advertising: The Occurrence of Stereotyping in Humorous and Nonhumorous Advertising and Its Consequences for Advertising Effectiveness (2014, methodology): “Intercoder reliability was assessed by Krippendorff ’s a,which can easily be compared to standard reliability measures (Hayes and Krippendorff 2007).”
The Effects of Message Framing and Visual Image on Persuasion (2013, methodology): “When the reliability of the emotion versus cognition coding was estimated using Krippendorff’s aK (Hayes & Krippendorff, 2007), the resulting values were .62 for flossing and .62 for sunscreen.”
CORRELATES OF VIOLENT POLITICAL EXTREMISM IN THE UNITED STATES (2018, methodology): “…10 percent of the individuals in the data to allow for iterative reliability tests of the coding instrument for each stage of the coding process, and they used the Krippendorf’s alpha procedure to test for inter-rater reliability across the double-coded cases (Hayes and Krippendorf, 2007).”
Values associated with luxury brand consumption and the role of gender (2017, methodology): “Krippendorff's alpha (Hayes & Krippendorff, 2007) reveals a good intercoder reliability (≤0.81; 95% confidence interval ranging from 0.76 to 0.86).”
Experiences of U.S. military couples during the post-deployment transition (2012, background): “…to other content analytic reliability measures because it (a) assesses agreement between any number of observers, (b) is independent of the distribution of categories, (c) generalizes across any level of measurement, and (d) possesses a computable sampling distribution (Hayes & Krippendorff, 2007).”
Contributing to Theory and Knowledge in Quantitative Communication Science (2012, methodology): “We assessed 50% of the sample for reliability using Krippendorff’s alpha (Hayes & Krippendorff, 2007) with two coders.”
Feedback providing improvement strategies and reflection on feedback use: Effects on students’ writing motivation, process, and performance (2012, background): “Inter-rater agreement (Krippendorff’s alpha; Hayes & Krippendorff, 2007) was sufficient (ak> 0.65) for all aspects.”