When Rater Reliability Is Not Enough

By H. Hill, Charalambos Y. Charalambous, M. Kraft. 2012

342 citations recorded by the source corpus.

Subfields: Education, Information Systems and Management

View Paper: https://www.semanticscholar.org/paper/83a23ec879fcee4f817ebd0c2a0fa7e5ea8e0edb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Education

1. H. Hill
2. Charalambos Y. Charalambous
3. David Blazar
4. Julie Cohen
5. Dan McGinn
6. Erica Litke
7. M. Kraft
8. Dan Goldhaber

In Information Systems and Management

1. David Blazar
2. H. Hill
3. Julie Cohen
4. M. Kraft
5. Charalambos Y. Charalambous
6. Dan Goldhaber
7. Michael Hansen
8. Roddy J. Theobald

Citations in the corpus

• Validating Arguments for Observational Instruments: Attending to Multiple Sources of Variation (2012, background): “Previously (Hill et al., 2012), we have argued for viewing classroom observation measures not as instruments but as systems, which we defined as a collection of elements that together produce scores representing individual teachers’ instructional quality.”
• Effective teaching in elementary mathematics: Identifying classroom practices that support student achievement (2015, background): “Comparatively, in this study, raters provided scores for the whole lesson, which is in line with recommendations made by Hill, Charalambous, and Kraft (2012) in a formal generalizability study.”
• Building a More Complete Understanding of Teacher Evaluation Using Classroom Observations (2016, background): “Indeed, a number of studies suggest multiple raters are needed for reliable estimates of practice (Casabianca et al., 2013; Hill et al., 2012; Kane & Staiger, 2012), and while training multiple raters or hiring outside raters may be sensible from a measurement perspective, it represents a costly and”
• One lesson is all you need? Stability of instructional quality across lessons. (2014, background): “At least two things can be learned from these results: First, as instructional variables show considerable differences regarding their temporal stability (see also Calkins et al., 1997; Erlich & Shavelson, 1978; Hill et al., 2012; Kunter, 2005; Rakoczy, 2008), the stability of instructional quality ”
• Mathematics Classroom Observation Protocol for Practices (MCOP2): A validation study (2017, neutral): “RTOP IQA MQI M-SCAN MCOP2 Affordances General indictors of reformoriented teaching Specific to cognitivechallenging tasks Not biased toward instructional type Multiple dimensions of measure Multiple dimensions of measure (teachers and students) for holistic view of classroom Use in science and math ”