Reliability in Content Analysis: Some Common Misconceptions and Recommendations

By Klaus Krippendorff. 2004

2,691 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Statistics and Probability, Social Sciences & Humanities

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Social Sciences & Humanities

Citations in the corpus

Recipes for Safety in Open-domain Chatbots (2020, neutral): “Using Krippendorff’s alpha (Krippendorff, 2004) as inter-annotator agreement (IAA), the multi-label annotation task has a reliability co-efﬁcient of 0.41, and 0.53 in binary case (offensive/safe), close to the value (0.45) reported by (Wulczyn et al., 2017).”
Bot-Adversarial Dialogue for Safe Conversational Agents (2021, methodology): “Using Krippendorff’s alpha (Krippendorff, 2004) as inter-annotator agreement (IAA), the multi-label annotation task has a reliability coefﬁcient of 0.41, and 0.53 in binary case (offensive/safe), close to the value (0.45) reported by (Wulczyn et al., 2017).”
Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good (2019, methodology): “Content analysis method (Krippendorff, 2004) was employed to create the annotation scheme.”
Qualitative content analysis: theoretical foundation, basic procedures and software solution (2014, background): “Coefficients should be higher than 0,8 with a minimum for acceptance of 0,67 (Krippendorff, 2004).”
CaSiNo: A Corpus of Campsite Negotiation Dialogues for Automatic Negotiation Systems (2021, methodology): “We follow the conceptual content analysis procedure (Krippendorff, 2004) to design the scheme.”
Boosting the Potency of Resistance: Combining the Motivational Forces of Inoculation and Psychological Reactance. (2013, background): “Negative cognition and negative affect were only coded after establishing acceptable reliabilities (Krippendorff, 2005) concerning argument relevance and argument position for all arguments.”
Commentary: A Dissenting View on So-Called Paradoxes of Reliability Coefficients (2013, background): “They cite my earlier objection to this scale (Krippendorff, 2004b, p. 412) but counter it with the belief that an understanding of their paradoxes and assumptions would be helpful in making appropriate choices.”
The Effects of Message Framing and Visual Image on Persuasion (2013, background): “Although these coefficients were below the recommended minimum of .66 (Krippendorff, 2004), close inspection led us to believe that they resulted from uneven distribution of data rather than a problem with the coding scheme itself.1 Accordingly, we instructed coders to resolve disagreements, then…”
Survey Article: Inter-Coder Agreement for Computational Linguistics (2008, neutral): “This argument has been presented recently in much detail by Krippendorff (2004b) and reiterated by Craggs and McGee Wood (2005).”
A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda (2020, result): “At this stage, we compared the results and discussed discrepancies in the way we coded the articles until we reached consensus, instead of calculating the discrepancies quantitatively as suggested by Krippendorff (2004) or Holsti (1969).”
Handling Bias in Toxic Speech Detection: A Survey (2022, background): “They observed that Krippendorff’s α [45] did not consistently increase across the three questions posed with the guidelines.”
Constructing Gender Stereotypes Through Social Roles in Prime-Time Television (2008, methodology): “All intercoder reliability coefficients were above .80, the minimum specified as acceptable by by Krippendorff (2004).”
Teacher Scaffolding in Small-Group Work: An Intervention Study (2014, methodology): “With a Krippendorff’s alpha of .74, the interrater reliability for coding the steps of contingent teaching was acceptable; Krippendorff (2004) considered an alpha between .66 and .80 as acceptable and one above .80 as good.”
Cohen's kappa is a weighted average (2011, neutral): “been applied to square cross-classifications encountered in psychometrics, educationalmeasurement, epidemiology, diagnostic imaging [15],map comparison [24] and content analysis [14].”
Free Association in Psychology and the Grid Elaboration Method (2014, background): “However, Krippendorff (2004) notes that the meaning of talk or text only arises in the process of somebody conceptually engaging with them; some level of interpretation is therefore necessary to discern their meanings.”