A Comparison of Consensus, Consistency, and Measurement Approaches to Estimating Interrater Reliability

By Steven E. Stemler. 2004

1,088 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/be5e22d62f45a7c6d1c5f41304469701e22bc0f1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Education

Citations in the corpus

Measuring and promoting inter-rater agreement of teacher and principal performance ratings. (2012, background): “One rule of thumb suggested by various experts contends that, when using percentage of absolute agreement, values from 75% to 90% demonstrate an acceptable level of agreement (Hartmann, 1977; Stemler, 2004).”
Mixing interview and questionnaire methods: Practical problems in aligning data (2010, neutral): “Thus, this study explored research questions around the similarities and differences of participant conceptions of assessment across questionnaire and semi-structured interview formats.”
Impact of license choice on Open Source Software development activity (2009, methodology): “If two raters can agree exactly on a rubric score to assign to a student’s work, then the two raters “may be said to share a common interpretation of the construct” (Stemler, 2004).”
The use of scoring rubrics: Reliability, validity, and educational consequences (2007, background): “This means that many estimates failed to reach the criterion of 70% or greater, which is needed if exact agreement is to be considered reliable (Stemler, 2004).”
Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. (2007, background): “The CVI, andmost other indexes that have been proposed for content validity, fall into the category that Stemler (2004) called consensus estimates.”
The Effect of Common Ground on How Speakers Use Gesture and Speech to Represent Size Information (2007, methodology): “…is much harder to reach using a 7-point scale than when using a considerably more limited number of scale points or coding categories, we relaxed the criterion for agreement so that directly adjacent points on the scale were considered as the judges having reached consensus (cf. Stemler, 2004).”
The relationship between vocabulary and writing quality in three genres (2012, background): “70 is expected for all consensus or consistency inter-rater agreements (Brown, Glasswell, & Harland, 2004; Stemler, 2004).”
Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives (2006, background): “Agreement concerns the extent of exact consensus among reviewers on writing evaluation, whereas consistency concerns the extent to which reviewers consistently apply a scoring rubric to writing evaluation (Stemler, 2004).”
Interpretation of standards with Bloom’s revised taxonomy: a comparison of teachers and assessment experts (2009, background): “These measures are all useful when treating nominal variables (Stemler 2004) and the categories in at least the knowledge dimension of the taxonomy can only be assumed to be nominal variables.”