Examinee Noneffort and the Validity of Program Assessment Results

By S. Wise, Christine E. DeMars. 2010

153 citations recorded by the source corpus.

Educational program assessment studies often use data from low-stakes tests to provide evidence of program quality. The validity of scores from such tests, however, is potentially threatened by examinee noneffort. This study investigated the extent to which one type of noneffort—rapid-guessing behavior—distorted the results from three types of commonly used program assessment designs. It was found that, for each design, a modest amount of rapid guessing had a pronounced effect on the results. In addition, motivation filtering was found to be successful in mitigating the effects caused by rapid guessing. It is suggested that measurement practitioners routinely apply motivation filtering whenever the data from low-stakes tests are used to support program decisions.

Subfields: Education, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/27ea9f1163f79a6183a3d5ce824c2eac981bb377

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Education

In Management Science and Operations Research

Citations in the corpus

Evaluating the Impact of Careless Responding on Aggregated-Scores: To Filter Unmotivated Examinees or Not? (2017, background): “…may potentially impact score-based inferences via aggregated scores in terms of treatment effects (Osborne & Blanchard, 2011), achievement gains (Wise & DeMars, 2010), evaluation ratings of school personnel when assessing student growth (Wise, Ma, Cronin, & Theaker, 2013), and country-level…”
The Role of Gender in Test-Taking Motivation under Low-Stakes Conditions. (2013, background): “For example, some studies found that female students exhibit more solutionbased behavior than male students (Wise & Cotton, 2009; Wise & DeMars, 2010).”
A Cross-National Comparison of Reported Effort and Mathematics Performance in TIMSS Advanced (2014, background): “That more experienced test-takers demonstrate less effort on low-stakes tests has been seen in other studies too (Wise & DeMars, 2010).”
Measuring Learning Outcomes in Higher Education (2012, background): “…students’ lack of motivation to perform well on the tests could pose a serious threat to the validity of the test scores and the interpretation accuracy of the test results (Banta, 2008; Haladyna & Downing, 2004; Liu, 2011b; S. L. Wise & DeMars, 2005, 2010; V. L. Wise, Wise, & Bhola, 2006).”
Assessing critical thinking in higher education: the HEIghten™ approach and preliminary validity evidence (2016, result): “Such a finding is not unique to this study, but resonates with findings from other research studies examining test-taking motivation in a low-stakes context (e.g. Wise and Kong 2005; Wise and DeMars 2010; Liu, Bridgeman, and Adler 2012; Rios, Liu, and Bridgeman 2014; Liu, Rios, and Borden 2015).”
Measuring Motivation in Low-Stakes Assessments (2015, background): “These findings call attention to the dilemma of interpreting results that include scores from unmotivated students (DeMars, 2000; Eklöf, 2006; Wise, 2006a, 2006b; Wise & DeMars, 2005, 2006, 2010).”
Do Examinees Have Similar Test-Taking Effort? A High-Stakes Question for Low-Stakes Testing (2010, background): “Moreover, differential motivation could wreck havoc with international comparative studies ( Mislevy, 1995 ; Wise & DeMars, 2010 ).”