Off-task behavior in the cognitive tutor classroom: when students "game the system"

By Ryan Shaun Joazeiro de Baker, Albert T. Corbett, K. Koedinger, Angela Z. Wagner. International Conference on Human Factors in Computing Systems, 2004

585 citations recorded by the source corpus.

We investigate the prevalence and learning impact of different types of off-task behavior in classrooms where students are using intelligent tutoring software. We find that within the classrooms studied, no other type of off-task behavior is associated nearly so strongly with reduced learning as "gaming the system": behavior aimed at obtaining correct answers and advancing within the tutoring curriculum by systematically taking advantage of regularities in the software's feedback and help. A student's frequency of gaming the system correlates as strongly to post-test score as the student's prior domain knowledge and general academic achievement. Controlling for prior domain knowledge, students who frequently game the system score substantially lower on a post-test than students who never game the system. Analysis of students who choose to game the system suggests that learned helplessness or performance orientation might be better accounts for why students choose this behavior than lack of interest in the material. This analysis will inform the future re-design of tutors to respond appropriately when students game the system.

Subfields: Artificial Intelligence, Computer Science Applications, Developmental and Educational Psychology

View Paper: https://www.semanticscholar.org/paper/30b882db8b07ce11b19af255a5030e50bfbe441f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Developmental and Educational Psychology

Citations in the corpus

Exploring the Assistance Dilemma in Experiments with Cognitive Tutors (2007, background): “, avoid thinking) by engaging in fast and repeated guessing or asking for hints more often or faster than appropriate—Baker coined the phrase “gaming the system” to describe such behaviors (Baker et al. 2004).”
Better to be frustrated than bored: The incidence, persistence, and impact of learners' cognitive-affective states during interactions with three different computer-based learning environments (2010, methodology): “Gaming the system is known to be associated with poorer learning in some learning environments (Baker et al., 2004).”
A review of recent advances in learner and skill modeling in intelligent learning environments (2012, background): “Disengaged students experience much lower learning, across forms of disengagement (cf. Baker et al. 2004; Baker 2007).”
Enhancing Teaching and Learning Through Educational Data Mining and Learning Analytics: An Issue Brief (2012, background): “Baker and colleagues have conducted a series of studies on detecting and adapting to students’ off-task behaviors (called gaming the system) in adaptive learning systems that teach algebra (Baker et al. 2004, 2006).”
A Systematic Review on Educational Data Mining (2017, methodology): “The researchers Baker, Corbett & Wagner [16] conducted a case study and used prediction methods in scientific study to game the interactive learning environment by exploiting the properties of the system rather than learning the system.”
Handbook of Research on Learning and Instruction (2011, background): “Thus, children were taught to move based on elite, adult movement models rather than developmentally appropriate patterns such as those that Roberton (1982) described.”
Student modeling approaches: A literature review for the last decade (2013, methodology): “…X X Surjono & Maltby, 2003 X X X SQLT-Web X ACE X X F-CBR-DHTC X X TADV X X LeCo-EAD X X WILEDS X MEDEA X InterMediActor X The personal reader X X Baker et al., 2004 X X X KERMIT X X ICICLE X Web-IT X X VIRGE X X Stathakopoulou, Magoulas, Grigoriadou & Samaracou, 2005 X X X X COLLECT-UML X…”
Detecting Student Misuse of Intelligent Tutoring Systems (2004, background): “In one recent study, students who frequently misused tutor software learned only 2/3 as much students who used the tutor properly, controlling for prior knowledge and general academic ability [5].”
Motivating Children to Learn Effectively: Exploring the Value of Intrinsic Integration in Educational Games (2011, background): “In this way it was ensured that switching between versions neither provided a gameplay penalty, nor an advantage, so children could not use it as a way to „game the system‟ (e.g. Baker et al., 2004).”
Artificial Intelligence in Education: 20th International Conference, AIED 2019, Chicago, IL, USA, June 25-29, 2019, Proceedings, Part II (2019, unknown): “In many studies, behavioral signals are significant predictors of the student’s academic outcome [2,22].”
Empirically building and evaluating a probabilistic model of user affect (2009, result): “Although this relationship seems backward, the results agree with the findings of Baker et al. (2004) that students with lower pre-test scores are more likely to want to succeed via trial and error than think about domain knowledge, whereas students with higher pre-test scores are more likely to…”
Accounting for Beneficial Effects of Worked Examples in Tutored Problem Solving (2010, background): “It may be interesting and productive to investigate changes in motivational factors associated with adding worked examples to tutors, for instance, does doing so reduce “gaming the system” behaviors (Baker et al. 2004)?”
Using learning analytics to assess students' behavior in open-ended programming tasks (2011, background): “This work is now coalescing into the nascent field of Learning Analytics or Educational Data Mining [1, 3], and has been used in many contexts to measure students’ learning and affect.”
Multimodal learning analytics (2013, background): “Other researchers [1, 3] have been using machine learning in many contexts to measure students’ learning and affect, in particular, focusing on their trajectory within a learning environment [2].”