Reliability and Inter-rater Reliability in Qualitative Research

By Nora Mcdonald, Sarita Yardi Schoenebeck, Andrea Forte. Proc. ACM Hum. Comput. Interact., 2019

940 citations recorded by the source corpus.

What does reliability mean for building a grounded theory? What about when writing an auto-ethnography? When is it appropriate to use measures like inter-rater reliability (IRR)? Reliability is a familiar concept in traditional scientific practice, but how, and even whether to establish reliability in qualitative research is an oft-debated question. For researchers in highly interdisciplinary fields like computer-supported cooperative work (CSCW) and human-computer interaction (HCI), the question is particularly complex as collaborators bring diverse epistemologies and training to their research. In this article, we use two approaches to understand reliability in qualitative research. We first investigate and describe local norms in the CSCW and HCI literature, then we combine examples from these findings with guidelines from methods literature to help researchers answer questions like: "should I calculate IRR?" Drawing on a meta-analysis of a representative sample of CSCW and HCI papers from 2016-2018, we find that authors use a variety of approaches to communicate reliability; notably, IRR is rare, occurring in around 1/9 of qualitative papers. We reflect on current practices and propose guidelines for reporting on reliability in qualitative research using IRR as a central example of a form of agreement. The guidelines are designed to generate discussion and orient new CSCW and HCI scholars and reviewers to reliability in qualitative research.

Subfields: Sociology and Political Science, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/4d20f6fdbea91aa15256e6c1efba854ba4143d2b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Sociology and Political Science

1. R. Stuart Geiger
2. Lora Aroyo
3. Nithya Sambasivan
4. Rob Kitchin

In Human-Computer Interaction

1. R. Stuart Geiger
2. Lora Aroyo
3. Nithya Sambasivan

Citations in the corpus

• “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI (2021, methodology): “Since codes are our process, not product [80], IRR was not used.”
• Garbage in, garbage out?: do machine learning application papers in social computing report where human-labeled training data comes from? (2019, positive): “Having multiple independent annotators is typically a foundational best practice in structured content analysis, so that the integrity of the annotations and the schema can be evaluated (although see [35]). For multiple annotator overlap, our definitions required papers state whether all or some of ”