A deep language model for software code
By Khanh Hoa Dam, Truyen Tran, Trang Pham. 2016
112 citations recorded by the source corpus.
Existing language models such as n-grams for software code often fail to capture a long context where dependent code elements scatter far apart. In this paper, we propose a novel approach to build a language model for software code to address this particular issue. Our language model, partly inspired by human memory, is built upon the powerful deep learning-based Long Short Term Memory architecture that is capable of learning long-term dependencies which occur frequently in software code. Results from our intrinsic evaluation on a corpus of Java projects have demonstrated the effectiveness of our language model. This work contributes to realizing our vision for DeepSoft, an end-to-end, generic deep learning-based framework for modeling software and its development process.
Subfields: Artificial Intelligence, Information Systems, Software
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Stephan Günnemann
- Jure Leskovec
- Quoc V. Le
- Yoshua Bengio
- Ge Li
- Mohit Bansal
- Zhi Jin
- Mohammad Norouzi
In Information Systems
- Romain Robbes
- Hlib Babii
- Andrea Janes
- Ge Li
- Rafael-Michael Karampatsis
- Zhi Jin
- Yunfei Zhao
- Stephan Günnemann
In Software
- Ge Li
- Zhi Jin
- Rafael-Michael Karampatsis
- Romain Robbes
- Hlib Babii
- Andrea Janes
- Michele Catasta
- Jure Leskovec
Citations in the corpus
- Neural Program Synthesis with Priority Queue Training (2018, background): “…al., 2015; Neelakantan et al., 2016; Reed & de Freitas, 2016; Andreas et al., 2016; Balog et al., 2017; Bhoopchand et al., 2016; Gaunt et al., 2016; Khanh Dam et al., 2016; Neelakantan et al., 2017; Riedel et al., 2016; Schaechtle et al., 2016; Zaremba et al., 2016; Barone & Sennrich, 2017;…”
- Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code (2020, methodology): “Other models of source code include probabilistic higher order grammars (PHOG) [14], which use ASTs, and several types of RNNs, including LSTMs [26, 41, 88].”
- Language-Agnostic Representation Learning of Source Code from Structure and Context (2021, background): “Early research learned language models on raw text data, e.g., (Wang et al., 2016; Raychev et al., 2014; Dam et al., 2016), providing evidence for the naturalness assumption (Hindle et al., 2012).”
- Improving Automatic Source Code Summarization via Deep Reinforcement Learning (2018, unknown): “For example, the work of Maddison and Tarlow [44] and other neural language models (e.g. LSTMs in Dam et al. [45]) describe context distributed representations while sequentially generating code.”
- CoCoGUM: Contextual Code Summarization with Multi-Relational GNN on UMLs (2020, methodology): “Dam et al. [49] build a language model for modeling software code using LSTMs. Ling et al. [50] and Allamanis et al. [51] combine the code-context representation with representations of other modalities (e.g. natural language) to synthesize code.”
- Maybe Deep Neural Networks are the Best Choice for Modeling Source Code (2019, methodology): “Althoughthese languages are related, they also have differences that might be hypothesized to affect the performance of LMs. Java was an obvious choice since it has extensively been used in related work [6, 22, 37, 40, 58, 69].”
- Integrating Tree Path in Transformer for Code Representation (2021, background): “Early research works learned language model on raw text data for code [Dam et al., 2016, Wang et al., 2016, Allamanis et al., 2016, Iyer et al., 2016], providing evidence for the naturalness assumption [Hindle et al., 2016].”