Building Program Vector Representations for Deep Learning

By Lili Mou, Ge Li, Yuxuan Liu, Lu Zhang, Zhi Jin. 2014

158 citations recorded by the source corpus.

Deep learning has made significant breakthroughs in various fields of artificial intelligence. However, it is still virtually impossible to use deep learning to analyze programs since deep architectures cannot be trained effectively with pure back propagation. In this pioneering paper, we propose the "coding criterion" to build program vector representations, which are the premise of deep learning for program analysis. We evaluate the learned vector representations both qualitatively and quantitatively. We conclude, based on the experiments, the coding criterion is successful in building program representations. To evaluate whether deep learning is beneficial for program analysis, we feed the representations to deep neural networks, and achieve higher accuracy in the program classification task than "shallow" methods. This result confirms the feasibility of deep learning to analyze programs.

Subfields: Information Systems, Software, Natural Language Processing, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

1. Xi Victoria Lin
2. Hongyu Zhang
3. Ge Li
4. Zhi Jin
5. Michael Pradel
6. Lingming Zhang
7. Chun Xia
8. Anh Tuan Nguyen

In Software

1. Xi Victoria Lin
2. Lingming Zhang
3. Chun Xia
4. Ge Li
5. Zhi Jin
6. Lili Mou
7. Anh Tuan Nguyen
8. Lu Zhang

In Natural Language Processing

1. Ge Li
2. Zhi Jin
3. Lili Mou
4. Xi Victoria Lin
5. Lu Zhang
6. Charles Sutton
7. Ilya Sutskever
8. Quoc V. Le

In Signal Processing

1. Xi Victoria Lin
2. Ge Li
3. Zhi Jin
4. Ilya Sutskever
5. Wojciech Zaremba
6. Yang Xiang
7. Jun Zhang
8. Tien Nhut Nguyen

Citations in the corpus

• Learning to Execute (2014, background): “N E ] 17 O c t 2014 (Maddison & Tarlow, 2014) learned a language model on parse trees, and (Mou et al., 2014) predicted whether two programs are equivalent or not.”
• Neural Programmer-Interpreters (2015, methodology): “While we share a similar motivation, our approach is distinct in that we explicitly incorporate compositional structure into the network using a program memory, allowing the model to learn new programs by combining sub-programs.”
• Discriminative Neural Sentence Modeling by Tree-Based Convolution (2015, methodology): “The idea of tree-based convolution is to apply a set of subtree feature detectors, sliding over the entire 1The model of tree-based convolution was firstly proposed to process program source code in our (unpublished) previous work (Mou et al., 2014).”
• Convolutional Neural Networks over Tree Structures for Programming Language Processing (2014, neutral): “Piech et al. (2015) build recursive networks on Hoare triples.”
• CoCoNuT: combining context-aware neural translation models using ensemble for program repair (2020, methodology): “Deep Learning in Software Engineering: The software engineering community had applied deep learning to perform various tasks such as defects prediction [40, 79, 81], source code representation [6, 7, 66, 80], source code summarization [4, 22] source code modeling [5, 11, 26, 83], code clone detectio”
• Deep Code Search (2018, background): “Recently, researchers have investigated possible applications of deep learning techniques to source code [7, 38, 53, 56, 60, 64, 75, 76].”
• Deep API learning (2016, background): “[32] proposed to learn vector representations of source code for deep learning tasks.”
• Software Vulnerability Detection Using Deep Neural Networks: A Survey (2020, methodology): “The study [78] applied the FCN which is a generic network structure for learning source code frequency-based features.”
• Exploring API Embedding for API Usages and Applications (2017, background): “Peng et al. [32] propose a deep learning model to learn vector representations for tree-based source code.”
• Software Vulnerability Analysis and Discovery Using Machine-Learning and Data-Mining Techniques (2017, methodology): “Walden et al. (2014) perform stratified threefold cross-validation, and Peng et al. (2015) perform fivefold cross-validation.”
• Learning to Spot and Refactor Inconsistent Method Names (2019, background): “Recently, a number of studies [45]–[51] have provided empirical evidence”
• On End-to-End Program Generation from User Intention by Deep Neural Networks (2015, background): “Our previous work learns programs’ vector representations, serving as a pre-training phrase in deep learning [17]; we also propose tree-based convolutional neural networks to classify programs by functionality and detect source code of certain patterns [16].”
• Program Synthesis from Natural Language Using Recurrent Neural Networks (2017, methodology): “Finally, deep-learning based approaches have also been applied to other so‰ware engineering problems, such as defect prediction [34], program feature extraction [22, 28], summarizing code using natural language [12], and program induction [7, 15].”
• Software Defect Prediction via Convolutional Neural Network (2017, background): “As analyzed in [21], only nodes on ASTs are a suitable granularity”