Automated Scoring of Constructed-Response Science Items: Prospects and Obstacles

By Ou Lydia Liu, C. Brew, John Blackmore, Libby F. Gerard, Jacquie Madhok, Marcia C. Linn. Educational Measurement: Issues and Practice, 2014

104 citations recorded by the source corpus.

Educational Measurement: Issues and Practice Summer 2014, Vol. 33, No. 2, pp. 19–28 Automated Scoring of Constructed-Response Science Items: Prospects and Obstacles Ou Lydia Liu, Educational Testing Service, Chris Brew, Nuance, John Blackmore, Educational Testing Service, Libby Gerard, Jacquie Madhok, and Marcia C. Linn, University of California, Berkeley Content-based automated scoring has been applied in a variety of science domains. However, many prior applications involved simplified scoring rubrics without considering rubrics representing multiple levels of understanding. This study tested a concept-based scoring tool for content-based scoring, c-rater TM , for four science items with rubrics aiming to differentiate among multiple levels of understanding. The items showed moderate to good agreement with human scores. The findings suggest that automated scoring has the potential to score constructed-response items with complex scoring rubrics, but in its current design cannot replace human raters. This article discusses sources of disagreement and factors that could potentially improve the accuracy of concept-based automated scoring. Keywords: automated scoring, constructed-response items, c-rater TM , science assessment onstructed-response items are an important tool to C elicit students’ in-depth understanding of science con- tent and measure students’ ability to communicate scientific ideas. Both science education researchers (e.g., Lane, 2004; Shepard, 2000) and the Next Generation Science Standards (NGSS; National Research Council, 2012) stress that scien- tific explanation is a critical component of science education. As the NGSS points out, multiple-choice items can provide snapshots of some types of science knowledge, but open- ended assessment items are essential for measuring other competencies such as the ability to formulate a problem, conduct investigations, and communicating findings. Yet, the costs associated with scoring student constructed responses often stand in the way of their use. Wainer and Thissen (1993) estimated that the scoring of 10 constructed-response items costs about $30 while the scoring of multiple-choice items to achieve the same level of reliability costs only about 1¢. To increase the use of constructed-response items, auto- mated scoring has been explored over the past two decades in a variety of domains. If accurate, automated scoring can reduce the interval between test administration and score as- signment, reduce the number of human scorers, save on costs Ou Lydia Liu, Educational Testing Service, 660 Rosedale Road, MS07-R, Princeton, NJ 08541; lliu@ets.org. Chris Brew, Nuance, 1198 East Arques Avenue, Sunnyvale, CA 94085; cbrew@acm.org. John Blackmore, Educational Testing Service, 660 Rosedale Road, MS75-D, Princeton, NJ 08541; jblackmore@ets.org. Libby Gerard, University of California Berkeley, Tolman Hall, Berkeley, CA 94720; libbygerard@berkeley.edu. Jacquie Madhok, University of California Berkeley, Tolman Hall, Berkeley, CA 94720; jjmad- hok@gmail.com. Marcia C. Linn, Graduate School of Education, University of California Berkeley, 4523 Tolman Hall, Berkeley, CA 94720; mclinn@berkeley.edu. C for training human scorers, and potentially improve the scor- ing consistency (Burstein & Marcu, 2002; Burstein, Marcu, & Knight, 2003; Williamson, Xi, & Breyer, 2012). Despitethe wide applications of automated scoring, many prior applications focus on simplified scoring rubrics (e.g., containing three levels: correct, partially correct, or wrong), without considering items involving multiple levels of under- standing. Using a concept-based scoring tool, c-rater TM , this article explores automated scoring of constructed-response science items with rubrics representing multiple levels of un- derstanding. Our goal was to contribute to the understanding of the prospects and obstacles of using automated scoring with complex rubrics. We hope to identify the conditions when automated scoring can produce reliable scores and make constructed-response items more accessible to science researchers and teachers (Quellmalz & Pelligrino, 2009). Research on Content-Based Automated Scoring in Educational Contexts Automated scoring has been widely applied in educational research to improve scoring efficiency and shorten the time between test administration and when teachers, test tak- ers, and score users receive test results. Research on auto- mated scoring has covered domains such as writing quality (Burstein & Marcu, 2002; Foltz, Laham, & Landauer, 1999), mathematics (Bennett & Sebrechts, 1996; Sandene, Horkay, Bennett, Braswell, & Oranje, 2005), written content (Attali & Powers, 2008; Dzikovska, Nielsen, & Brew, 2012; Graesser, 2011; Leacock & Chodorow, 2003; Mitchell, Russell, Broom- head, & Aldridge, 2002; Nielsen, Ward, & Martin, 2008; Sukkarieh & Bolge, 2010), speech (Bernstein, Van Moere, & Cheng, 2010; Higgins, Zechner, Xi, & Williamson, 2011), and 2014 by the National Council on Measurement in Education

Subfields: Education

View Paper: https://www.semanticscholar.org/paper/5338e3031981707be01cda0d836ce7b6babad9ac