Automated Correction for Syntax Errors in Programming Assignments using Recurrent Neural Networks

By Sahil Bhatia, Rishabh Singh. 2016

120 citations recorded by the source corpus.

We present a method for automatically generating repair feedback for syntax errors for introductory programming problems. Syntax errors constitute one of the largest classes of errors (34%) in our dataset of student submissions obtained from a MOOC course on edX. The previous techniques for generating automated feed- back on programming assignments have focused on functional correctness and style considerations of student programs. These techniques analyze the program AST of the program and then perform some dynamic and symbolic analyses to compute repair feedback. Unfortunately, it is not possible to generate ASTs for student pro- grams with syntax errors and therefore the previous feedback techniques are not applicable in repairing syntax errors. We present a technique for providing feedback on syntax errors that uses Recurrent neural networks (RNNs) to model syntactically valid token sequences. Our approach is inspired from the recent work on learning language models from Big Code (large code corpus). For a given programming assignment, we first learn an RNN to model all valid token sequences using the set of syntactically correct student submissions. Then, for a student submission with syntax errors, we query the learnt RNN model with the prefix to- ken sequence to predict token sequences that can fix the error by either replacing or inserting the predicted token sequence at the error location. We evaluate our technique on over 14, 000 student submissions with syntax errors. Our technique can completely re- pair 31.69% (4501/14203) of submissions with syntax errors and in addition partially correct 6.39% (908/14203) of the submissions.

Subfields: Computer Science Applications, Information Systems, Software, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Information Systems

In Software

In Natural Language Processing

Citations in the corpus

A Survey of Machine Learning for Big Code and Naturalness (2017, methodology): “semantics, by modeling how code could be completed. Education. Software engineering education is one area that is already starting to be aﬀected by thisﬁeld.TheworkofCampbelletal.[35]andBhatiaandSingh[25]alreadyprovideanautomated methodforﬁxingsyntaxerrorsinstudentcode,whereasPiechetal.[155],Wangeta”
An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation (2018, methodology): “ly designed code features to rank candidate repairs. Other approaches train on correct solutions (from student programs) to specific programming tasks and try to learn task-specific repair strategies [8, 66]. This goal has been achieved successfully in contexts such as in massively open online cours”
DeepBugs: a learning approach to name-based bug detection (2018, methodology): “ety of development tasks, including code completion based on probabilistic models of code [Bielik et al. 2016; Raychev et al. 2016a, 2014], predicting fixes of compilation errors via neural networks [Bhatia and Singh 2016; Gupta et al. 2017], and the generation of inputs for fuzz testing via probabi”
Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code (2020, background): “code have the potential to enable new tools for aiding code readability [2], program repair [13, 18, 40, 75], program synthesis [39] and translation between programming languages [54, 66].”
DeepFix: Fixing Common C Language Errors by Deep Learning (2017, methodology): “Bhatia and Singh (2016) use prefixes implicated by a compiler and Pu et al. (2016) perform a brute force, enumerative search for localizing errors.”
Dynamic Neural Program Embedding for Program Repair (2017, methodology): “DeepFix (Gupta et al., 2017), SynFix (Bhatia & Singh, 2016), and sk p (Pu et al., 2016) are recent neural program repair techniques for correcting errors in student programs for MOOC assignments, and they all represent programs as sequences of tokens.”
Maybe Deep Neural Networks are the Best Choice for Modeling Source Code (2019, background): “Also, improved language models for code have the potential to enable new tools for aiding code readability [2], program repair [11, 15, 35, 62], program synthesis [34] and translation be-tween programming languages [45, 56].”
Automatic Grading and Feedback using Program Repair for Introductory Programming Courses (2017, background): “Recent works [3,14,19] use neural networks to “learn” the correct sequence of tokens from submissions that compile successfully, and use this knowledge to repair syntax errors in other sub-missions.”
Improving bug detection via context-based code representation learning and attention-based neural networks (2019, methodology): “Improving software quality and reliability is a never-ending demand [Amodio et al. 2017; Bhatia and Singh 2016; Bielik et al. 2016; Hindle et al. 2012; Kim et al. 2018; Patra and Pradel 2016].”
Deep Learning to Find Bugs (2017, background): “Even though machine learning has helped address a variety of software development tasks [8, 13, 14, 36] and even though buggy code stands out compared to non-buggy code [33], the problem of learningbased bug detection remains an open challenge.”
Semantic Code Repair using Neuro-Symbolic Transformation Networks (2017, methodology): “The term automatic code repair is typically used to describe two overarching tasks: The ﬁrst involves ﬁxing syntactic errors , which are malformations that cause the code to not adhere to some language speciﬁcation [10, 6].”
Compilation Error Repair: For the Student Programs, From the Student Programs (2018, background): “In fact, it may be argued that offering the repaired, compilable program to the student directly, which many existing works do [2, 7, 8], defeats the purpose of learning and may even pose challenges in course evaluations.”
Context2Name: A Deep Learning-Based Approach to Infer Natural Variable Names from Usage Contexts (2017, background): “…in applying machine learning to programs, targeting a variety of development tasks such as code completion [10, 29, 32], fixing syntactic errors [9, 18], generating inputs for fuzz testing of programs [17, 24, 28], and generating programs or short code snippets to assist developers or test…”