TreeGen: A Tree-Based Transformer Architecture for Code Generation

By Zeyu Sun, Qihao Zhu, Yingfei Xiong, Yican Sun, Lili Mou, Lu Zhang. 2019

202 citations recorded by the source corpus.

A code generation system generates programming language code based on an input natural language description. State-of-the-art approaches rely on neural networks for code generation. However, these code generators suffer from two problems. One is the long dependency problem, where a code element often depends on another far-away code element. A variable reference, for example, depends on its definition, which may appear quite a few lines before. The other problem is structure modeling, as programs contain rich structural information. In this paper, we propose a novel tree-based neural architecture, TreeGen, for code generation. TreeGen uses the attention mechanism of Transformers to alleviate the long-dependency problem, and introduces a novel AST reader (encoder) to incorporate grammar rules and AST structures into the network. We evaluated TreeGen on a Python benchmark, HearthStone, and two semantic parsing benchmarks, ATIS and GEO. TreeGen outperformed the previous state-of-the-art approach by 4.5 percentage points on HearthStone, and achieved the best accuracy among neural network-based approaches on ATIS (89.1%) and GEO (89.6%). We also conducted an ablation test to better understand each component of our model.

Subfields: Artificial Intelligence, Information Systems, Software, Model Architectures, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jie Tang
2. Yuxiao Dong
3. Zhilin Yang
4. Qian Liu
5. Tianyi Zhang
6. Jian-Guang Lou
7. Xu Zou
8. Elena L. Glassman

In Information Systems

1. Yuxiao Dong
2. Jie Tang
3. Tianyi Zhang
4. Elena L. Glassman
5. Lu Zhang
6. Zeyu Sun
7. Wenjie Zhang
8. Kang Yuan

In Software

1. Tianyi Zhang
2. Elena L. Glassman
3. Lu Zhang
4. Zeyu Sun
5. Yuxiao Dong
6. Wenjie Zhang
7. Kang Yuan
8. Yuan-an Xiao

In Model Architectures

1. Jie Tang
2. Tianyi Zhang
3. Qian Liu
4. Ting Liu
5. Lu Zhang
6. Zeyu Sun
7. Yuxiao Dong
8. Zhilin Yang

In Code Generation & Program Synthesis

1. Tianyi Zhang
2. Qian Liu
3. Lu Zhang
4. Yuxiao Dong
5. Zhilin Yang
6. Jie Tang
7. Yufei Xue
8. Xiao Xia

Citations in the corpus

• CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023, background): “From LISPbased pioneering deductive synthesis approaches [31, 37] tomodern program synthesis systems [23, 30], to end-to-end code generation via deep neural networks [19, 32, 33], tremendous efforts have been made to enable machines to automatically write correct programs as part of the quest to art”
• Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models (2022, background): “Advances in deep learning have shown promising re-sults on automatically generating code for real-world programming tasks [5, 16, 17, 22, 38, 39, 48, 49].”
• HELoC: Hierarchical Contrastive Learning of Source Code Representation (2022, background): “Many source code representation techniques have achieved considerable success in many program comprehension tasks, including code classification [50, 64, 65], malware detection [11, 66], and code generation [67, 68].”
• An Empirical Study of Code Smells in Transformer-based Code Generation Techniques (2022, background): “These Large Language Learning Models (LLMs) are trained with a large dataset of source code snippets and fine-tuned for a wide range of software engineering applications, such as automated code completion [3]–[5], summarization [6], documentation [7], and generation [8], [9].”
• Benchmarking Meaning Representations in Neural Semantic Parsing (2020, background): “directly (Sun et al., 2020; Shaw et al., 2019; Chen et al., 2020) without considering the confounding”
• A syntax-guided edit decoder for neural program repair (2021, methodology): “Embedding sizes for all embedding vectors are set to 256, and all hidden sizes are set following the configuration of TreeGen [64].”