Monitoring Rater Performance Over Time: A Framework for Detecting Differential Accuracy and Differential Scale Category Use

By C. Myford, E. Wolfe. 2009

98 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Rater Effects on Essay Scoring: A Multilevel Analysis of Severity Drift, Central Tendency, and Rater Experience (2011, methodology): “…this topic: one study showed increased leniency over time (Lunz and Stahl, 1990), two studies showed increased severity (Congdon & McQueen, 2000; Pinot de Moira et al., 2002), and another study showed both positive and negative drift for a small proportion of their raters (Myford and Wolfe, 2009).”
Effect of Observation Mode on Measures of Secondary Mathematics Teaching (2013, background): “Recent research has discussed time trends in rater effects, specifically rater severity drift and changes in score scale category use (Harik et al., 2009; Leckie & Baird, 2011; Myford & Wolfe, 2009).”
Operational Rater Types in Writing Assessment: Linking Rater Cognition to Rater Behavior (2012, background): “Questions like these may suitably be addressed within Myford and Wolfe's (2009) MFRM framework for studying various aspects of rating behavior over time.”
Measuring and promoting inter-rater agreement of teacher and principal performance ratings. (2012, background): “Research suggests that even extensive training will not ensure that every observer agrees with a standard or with her/his peers (Myford & Wolfe, 2009).”