Detecting and measuring rater effects using many-facet Rasch measurement: part I.

By C. Myford, E. Wolfe. Journal of Applied Measurement, 2003

482 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/a0bb722335ac826bd0dba8413b8fbb9bcc47ad4a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Diagnostic Writing Assessment: The Development and Validation of a Rating Scale (2009, background): “A number of different studies have identified a variety of ways in which raters can vary (McNamara, 1996; Myford & Wolfe, 2003, 2004).”
Judgment Confidence and Judgment Accuracy of Teachers in Judging Self-Concepts of Students (2013, methodology): “As illustrated by research on rater bias (for summary, see Myford & Wolfe, 2003), persons who are unsure of their own judgments avoid choosing the extremities on an answer scale.”
A comprehensive review of Rasch measurement in language assessment: Recommendations and guidelines for research (2020, background): “It also allows examination of other aspects of the rating situation, such as the mode of test delivery, and the effect of interactions of various factors on the quality of the eventual assessment (Myford & Wolfe, 2003, 2004).”
Automated Essay Scoring in the Presence of Biased Ratings (2018, background): “While there is sizeable literature on rater effects in general settings (Myford and Wolfe, 2003), it remains unknown how biased ratings affect automated essay scoring models.”
Rater types in writing performance assessments: A classification approach to rater variability (2008, methodology): “Since the raters did not evaluate real examinee performance, but rated the general importance of familiar scoring criteria (forming a rater criterion factorial design), some FACETS-based methods of checking the psychometric quality of ratings were not applicable or took on different meanings here (f”
Monitoring Rater Performance Over Time: A Framework for Detecting Differential Accuracy and Differential Scale Category Use (2009, methodology): “The single rater-rest of the raters correlation (SR-ROR) (rSR−ROR) (Myford & Wolfe, 2003), a generalization of the Pearson correlation, depicts accuracy as a static characteristic of rater performance within the adopted frame of reference.”
Examining Rater Effects in TestDaF Writing and Speaking Performance Assessments: A Many-Facet Rasch Analysis (2005, background): “The finding that raters did not function interchangeably in any of the scoring sessions agreed well with related research on the degree of severity exercised in language performance assessments (see, for reviews, Engelhard, 2002; McNamara, 1996; Myford & Wolfe, 2003).”
Operational Rater Types in Writing Assessment: Linking Rater Cognition to Rater Behavior (2012, methodology): “Within-rater consistency was assessed by means of mean-square fit statistics, which have an expected value of 1 and can range from 0 to infinity (e.g., Eckes, 2011; Engelhard, 2002; Myford & Wolfe, 2003).”
Rater bias patterns in an EFL writing assessment (2008, background): “Myford and Wolfe (2004b) note that MFRM provides specific information on how each individual element of a facet, including raters, is functioning.”
A multifaceted approach to investigating pre-task planning effects on paired oral test performance (2014, methodology): “The rater reliability we obtained was the best one could obtain from FACETS (i.e., 0.0), suggesting that the two raters were interchangeable (Myford & Wolfe, 2004).”
An investigation into native and non-native teachers' judgments of oral English performance: A mixed methods approach (2009, methodology): “…analysis were carried out to investigate teachers’ internal consistency, based on: 1) fit statistics; 2) proportions of large standard residuals between observed and expected scores (Myford & Wolfe, 2000); and 3) a single rater–rest of the raters (SR/ROR) correlation (Myford & Wolfe, 2004a).”
Generalizability and Dependability of Behavior Assessment Methods to Estimate Academic Engagement: A Comparison of Systematic Direct Observation and Direct Behavior Rating (2010, background): “Suggestions for minimizing halo error include ensuring that ratings occur temporally as close as possible to the observed behavior (Feeley, 2002) and limiting the number of ratings required so that raters do not become overwhelmed or fatigued (Myford, 2003).”