Assessment criteria in a large-scale writing test: what do they really mean to the raters?

By Tom Lumley. 2002

320 citations recorded by the source corpus.

Subfields: Language and Linguistics, Literature and Literary Theory

View Paper: https://www.semanticscholar.org/paper/f0f87b4c05a9bc036352832633c78fa6b7bf145c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Language and Linguistics

In Literature and Literary Theory

Citations in the corpus

Score generalizability of academic writing tasks: Does one test method fit it all? (2009, result): “This result is consistent with many studies that stressed the importance of rater experience (Song & Caruso, 1996) and rater training (Lumley, 2002; Weigle, 1994, 1998) in establishing rater reliability.”
Rater types in writing performance assessments: A classification approach to rater variability (2008, background): “According to Lumley (2002; 2005), individual monitoring processes come into play when raters try to reconcile their overall impression of the text, specific, hard-to-evaluate features of the text and the rating scale descriptors.”
Reliability and validity of rubrics for assessment through writing (2010, unknown): “Judgmentsultimatelyturnonadjectivesthatare murkyandendupbeinglefttotheteacher’sdiscretion(Kohn,2006;Lumley,2002).”
Discourse Characteristics of Writing and Speaking Task Types on the "TOEFL iBT"® Test: A Lexico-Grammatical Analysis. "TOEFL iBT"® Research Report. TOEFL iBT-19. Research Report. RR-13-04. (2013, methodology): “Studies of rater cognition generally employ think-aloud methods to identify the considerations that are most influential for raters (Lumley, 2002; Cumming, Kantor, & Powers, 2002).”
Diagnostic assessment of writing: A comparison of two rating scales (2009, background): “Some studies have in fact found that raters display halo effects only when encountering problems in the rating process (e.g. Lumley, 2002; Vaughan, 1991).”
Generalizability of writing scores: an application of structural equation modeling (2005, background): “The process of rating text qualities is quite complex and is not easily guided by general scoring instructions, as Lumley (2002) showed.”
Rater bias patterns in an EFL writing assessment (2008, methodology): “On the other hand, other researchers have found that rater variability persists in spite of extensive rater training and screening (Engelhard, 1992, 1994; Lumley, 2002, 2005; Lumley & McNamara, 1995; McNamara, 1996).”
Measuring and promoting inter-rater agreement of teacher and principal performance ratings. (2012, background): “Lumley (2002) found that when evaluators are unable to decide between two score points, other extraneous factors often creep into their decision-making, such as over-weighting one factor, adjudicating based on factors not included in the rubric, or comparing the current subject with previously…”
EFL classroom peer assessment: Training effects on rating and commenting (2008, background): “Not surprisingly, however, experience does not always lead to a strong consensus among raters (Lumley, 2002).”