Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research

By Daniel J. Hruschka, Deborah J. Schwartz, Daphne Cobb St.John, Erin Picone-Decaro, R. Jenkins, James W. Carey. 2004

806 citations recorded by the source corpus.

Subfields: Epidemiology, Sociology and Political Science, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/f62c0497ffb0096e5200b43b827b9887812f3138

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Epidemiology

H. Joffe

In Sociology and Political Science

In Statistics and Probability

Citations in the corpus

"Accessibility Came by Accident": Use of Voice-Controlled Intelligent Personal Assistants by People with Disabilities (2018, methodology): “To ensure coding reliability, we used a multi-phase process [21].”
Beyond a coefficient: an interactive process for achieving inter-rater consistency in qualitative coding (2020, methodology): “First, and most basically, our descriptions answer the call for researchers to be more transparent in relaying the steps taken in their treatment of qualitative data (e.g. Burla et al., 2008; Church et al., 2019; Hill et al., 2005; Hruschka et al., 2004; O’Connor and Joffe, 2020; Roberts et al., 201”
Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines (2020, background): “Theoretical relevance or meaning should never be sacrificed for reliability (Hruschka et al., 2004).”
The Coding Manual for Qualitative Researchers (2009, background): “For these types of collaborative ventures, intercoder agreement or interpretive convergence (the percentage at which different coders agree and remain consistent with their assignment of particular codes to particular data) is an important part of the process (for formulas and discussions see Bernar”
Explaining success and failure in the commons: the configural nature of Ostrom's institutional design principles (2016, methodology): “However, it is appropriate to utilize this technique in conjunction with other mea(Hruschka et al., 2004) sures if the variables analyzed are nominal (Feng, 2014).”
Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis (2021, methodology): “In order to ensure the reliability of the emerging codes and categories, we applied both qualitative and quantitative measures of reliability (Campbell, Quincy, Osserman, & Pedersen, 2013; Kurasaki, 2000; Hruschka et al., 2004).”
Coding In-depth Semistructured Interviews (2013, methodology): “…codes in order to reduce the number of codes that coders need to remember, clarify coding definitions, modify codes and/or coding instructions, repeat the reliability test, and then repeat the exercise until an acceptable level of reliability is achieved (Hodson 1999:24-26; Hruschka et al. 2004).”
A Mindfulness-Based Social and Emotional Learning Curriculum for School-Aged Children: The MindUP Program (2016, neutral): “Themes were reﬁ ned over the cycles of coding until the two coders achieved a 90 % inter-rater reliability on a subsample of data (Hruschka et al., 2004 ).”
Qualitative research and content validity: developing best practices based on science and experience (2009, methodology): “As with collecting data, the sorting process is iterative until both coder and facilitator are comfortable that they have appropriately captured the flavor and content of the interviews [ 46 ].”
Ecological influences on teachers’ well-being and “fitness” (2015, methodology): “Independent coding of data was employed for reliability and to reduce variations in interpretation, error, and bias (Hruschka et al., 2004).”
Guidelines for Establishing Reliability When Coding Narrative Data (2015, background): “A large number of codes allows for greater complexity to be captured in the data but comes at the possible cost of decreased reliability due to the complexity of the scheme (Hruschka et al., 2004).”