Integrating Tree Path in Transformer for Code Representation

By Han Peng, Ge Li, Wenhan Wang, Yunfei Zhao, Zhi Jin. 2021

54 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

1. Lei Lyu
2. Zhuoran Zheng
3. Xue Jiang
4. Hongyu Zhang
5. Qiong Wu
6. Chen Lyu
7. Xiao Wang
8. Yanqiao Zhu

In Code Generation & Program Synthesis

1. Yanqiao Zhu
2. Yichen Xu

Citations in the corpus

• Code Llama: Open Foundation Models for Code (2023, neutral): “…of programs, with specialized objectives for code deobfuscation (Lachaux et al., 2021), contrastive learning through semantic-preserving code transformations (Jain et al., 2021), leveraging Abstract Syntax Trees to learn tree-aware positional encodings (Shiv & Quirk, 2019; Peng et al., 2021).”
• HELoC: Hierarchical Contrastive Learning of Source Code Representation (2022, background): “Recently, several works [28, 69–71] inject structured information such as path embedding [71] and node distance [28] into Transformer to introduce structural inductive bias.”
• A Survey on Pretrained Language Models for Neural Code Intelligence (2022, methodology): “Although prior work makes a few attempts in exploiting these knowledge (Peng et al., 2021a; Shiv and Quirk, 2019; Guo et al., 2021; Wang et al., 2021a), there is still a lack of principled way to inject structure bias in language models designed for sequential data.”
• Structured Chain-of-Thought Prompting for Code Generation (2023, unknown): “Different from natural languages, source code contains rich structural information [22, 30, 37].”