Automated feedback generation for introductory programming assignments

By Rishabh Singh, Sumit Gulwani, Armando Solar-Lezama. ACM-SIGPLAN Symposium on Programming Language Design and Implementation, 2012

491 citations recorded by the source corpus.

We present a new method for automatically providing feedback for introductory programming problems. In order to use this method, we need a reference implementation of the assignment, and an error model consisting of potential corrections to errors that students might make. Using this information, the system automatically derives minimal corrections to student's incorrect solutions, providing them with a measure of exactly how incorrect a given solution was, as well as feedback about what they did wrong. We introduce a simple language for describing error models in terms of correction rules, and formally define a rule-directed translation strategy that reduces the problem of finding minimal corrections in an incorrect program to the problem of synthesizing a correct program from a sketch. We have evaluated our system on thousands of real student attempts obtained from the Introduction to Programming course at MIT (6.00) and MITx (6.00x). Our results show that relatively simple error models can correct on average 64% of all incorrect submissions in our benchmark set.

Subfields: Computer Science Applications, Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/169a72730c870f30a4452bfc0648e35ac4561f42

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Information Systems

In Software

Citations in the corpus

Neuro-Symbolic Program Synthesis (2016, background): “…et al., 2012), superoptimization by ﬁnding smaller equivalent loop bodies (Schkufza et al., 2013), protocol synthesis from scenarios (Udupa et al., 2013), synthesis of loop-free programs (Gulwani et al., 2011), and automated feedback generation for programming assignments (Singh et al., 2013).”
TerpreT: A Probabilistic Programming Language for Program Induction (2016, background): “ al., 2014), eﬃcient compilation of declarative speciﬁcations (Kuncak et al., 2010), statistical code completion (Raychev et al., 2016), and automated feedback generation for programming assignments (Singh et al., 2013). These techniques can be broadly categorized using three dimensions: i) speciﬁca”
Semantic Code Repair using Neuro-Symbolic Transformation Networks (2017, methodology): “Most previous work relies on unit tests – a common theme is combining coarse-grained repair models with search algorithms to ﬁnd some repair that satisﬁes unit tests [11, 19].”
Towards Mixed Optimization for Reinforcement Learning with Program Synthesis (2018, methodology): “This step allows us to meaningfully debug the policy, either through human-in-the-loop verification for interpretability, or through automated program repair techniques that involve defining Constraint Satisfaction Problems (CSP) typically solved using SAT/SMT solvers (Singh et al., 2013).”
Neural Attribution for Semantic Bug-Localization in Student Programs (2019, methodology): “AutoGrader [24] takes as input a buggy student program, along with a reference solution and a set of potential corrections in the form of expression rewrite rules and searches for a set of minimal corrections using program synthesis.”
Automated program repair (2019, background): “Arewe there yet?While repair-based intelligent tutoring remains an open challenge for now, initial evidence on using program repair like processes for providing feedback to students [7] or for automatic grading of student assignments [8] have been obtained.”
Repairing Bugs in Python Assignments Using Large Language Models (2022, background): “In particular, for programming education past research has explored topics including feedback generation [9]– [12], [29], [41], [42] and program repair [20], [36], [43]–[46].”
Automatic Grading and Feedback using Program Repair for Introductory Programming Courses (2017, background): “Section 1.2 described some popular approaches [11–13, 15, 16, 21].”
A Systematic Literature Review of Automated Feedback Generation for Programming Exercises (2018, methodology): “The tool by Singh13 also produces this type [161]:”
Automated Correction for Syntax Errors in Programming Assignments using Recurrent Neural Networks (2016, methodology): “rading and Feedback: The automated grading approaches can broadly be classiﬁed into two broad categories: 1) programming languages based approaches, and 2) machine learning based approaches. AutoProf [18] is a system for providing automated feedback on functional correctness of introductory programm”
Neuro-Symbolic Program Corrector for Introductory Programming Assignments (2018, methodology): “These approaches can be categorized into two broad categories – peer-grading [14, 15] and automated grading techniques [8, 17, 26, 27, 29].”
Introductory programming: a systematic literature review (2018, methodology): “The tools focus mainly on improving the accuracy of fault-finding through various methods [30, 253, 278, 619] and on improving the timeliness of feedback by automatically reusing”