Best Practices in Interrater Reliability Three Common Approaches

By Steven E. Stemler, Jessica W. Tsai. 2008

135 citations recorded by the source corpus.

Subfields: Emergency Medical Services

View Paper: https://www.semanticscholar.org/paper/249e532cc81f8184c3d904e92a5678cec2514825

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Emergency Medical Services

1. S. Senaratne
2. Alistair Willis
3. Paul Piwek
4. T. Cooray
5. Jacopo Amidei
6. A. Gunarathne

Citations in the corpus

• Agreement is overrated: A plea for correlation to assess human evaluation reliability (2019, methodology): “More specifically, regarding the use of correlation coefficients in annotation tasks we refer to Stemler and Tsai (2008), LeBreton and Senter (2008) and Gisev et al. (2013).”
• Does Corporate Governance Affect the Quality of Integrated Reporting? (2020, methodology): “The multiple trained raters method was used to avoid subjectivity and improve the accuracy of assessing IRQ while ensuring inter-rater reliability [115].”
• Rater Experience, Rating Scale Length, and Judgments of L2 Pronunciation: Revisiting Research Conventions (2013, background): “However, Stemler and Tsai (2008) argued that relying on a single interrater reliability estimate could be misleading.”
• Teacher Study Group (2010, methodology): “…in the field on the relative merits of various approaches for calculating interobserver reliability estimates (e.g., Hayes & Hatch, 1999; Lei, Smith, & Suen, 2007), we chose to use the percentage agreement method for its strong face validity and ease of interpretation (Stemler & Tsai, 2008).”