Learning Program Embeddings to Propagate Feedback on Student Code

By C. Piech, Jonathan Huang, A. Nguyen, Mike Phulsuksombati, Mehran Sahami, Leonidas J. Guibas. 2015

201 citations recorded by the source corpus.

Providing feedback, both assessing final work and giving hints to stuck students, is difficult for open-ended assignments in massive online classes which can range from thousands to millions of students. We introduce a neural network method to encode programs as a linear mapping from an embedded precondition space to an embedded postcondition space and propose an algorithm for feedback at scale using these linear maps as features. We apply our algorithm to assessments from the Code.org Hour of Code and Stanford University's CS1 course, where we propagate human comments on student assignments to orders of magnitude more submissions.

Subfields: Artificial Intelligence, Computer Science Applications, Code Generation & Program Synthesis, Program Analysis & Verification

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Code Generation & Program Synthesis

In Program Analysis & Verification

Citations in the corpus

DeepCoder: Learning to Write Programs (2016, background): “Piech et al. (2015) propose to learn joint embeddings of program states and programs to automatically extend teacher feedback to many similar programs in the MOOC setting.”
A Convolutional Attention Network for Extreme Summarization of Source Code (2016, background): “Additionally, compared to Piech et al. (2015) and Mou et al. (2016) our work looks into highly diverse, non-student submission code that performs a wide range of real-world tasks.”
Learning Continuous Semantic Representations of Symbolic Expressions (2016, background): “Piech et al. (2015) also learn distributed matrix representations of student code submissions.”
Neural Program Repair by Jointly Learning to Localize and Repair (2019, background): “In particular, different representations based on token sequences (Gupta et al., 2017; Bhatia et al., 2018), program parse trees (Piech et al., 2015; Mou et al., 2016), program traces (Reed & de Freitas, 2015; Cai et al., 2017; Wang et al., 2018), and graphs (Allamanis et al., 2018) have been…”
CoCoGUM: Contextual Code Summarization with Multi-Relational GNN on UMLs (2020, methodology): “Piech et al. [47] introduce a neural network method to encode programs as a linear mapping from an embedded precondition space to an embedded postcondition space and pro-pose an algorithm for feedback at scale using these linear mappings as features.”
Deep Knowledge Tracing (2015, methodology): “Using the recently developed vectorization of programs [25] we hope to be able to intelligently model student knowledge over time as they learn to program.”
The Impact of Adding Textual Explanations to Next-step Hints in a Novice Programming Environment (2019, background): “Automated, next-step hint generation is a rapidly growing area of research, and researchers have developed a variety of techniques, including data-driven approaches [27, 31, 34],”
Automated Correction for Syntax Errors in Programming Assignments using Recurrent Neural Networks (2016, methodology): “The elements of the embedding matrix of a program are then used as code features for automatically propagating instructor feedback at scale [15].”
Global Relational Models of Source Code (2020, methodology): “Some techniques leverage the ASTs of programs in tree-structured recurrent models (Piech et al., 2015; Parisotto et al., 2016; Chen et al., 2018).”
Teaching Temporal Logics to Neural Networks (2020, background): “Other works focused on recurrent neural networks or graph neural networks for code analysis, e.g. [30,14,5,48,1].”
Improving Automatic Source Code Summarization via Deep Reinforcement Learning (2018, background): “Piech et al. [42] and Parisotto et al. [43] learn distributed representations of source code input/output pairs and use them to assess and review student assignments or to guide program synthesis from examples.”
Neural Attribution for Semantic Bug-Localization in Student Programs (2019, methodology): “The clusters are created either using heuristics based on program analysis techniques [8, 15, 10, 27, 23] or using program execution on a set of inputs [19, 20].”
Dynamic Neural Program Embedding for Program Repair (2017, background): “The only exception is Piech et al. (2015), which introduces a novel perspective of representing programs using input-output pairs.”
Context-Aware Attentive Knowledge Tracing (2020, background): “Recent advances in data analytics and intelligent tutoring systems [32] have enabled the collection and analysis of large-scale learner data; these advances hint at the potential of personalized learning at large scale, by automatically providing personalized feedback [24] and learning activity reco”
DeepFix: Fixing Common C Language Errors by Deep Learning (2017, methodology): “Piech et al. (2015) proposed a neural network based approach to find program representations and used them for automatically propagating instructor feedback to students in a massive course.”