Language-Agnostic Representation Learning of Source Code from Structure and Context

By Daniel Zügner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, Stephan Günnemann. 2021

133 citations recorded by the source corpus.

Source code (Context) and its parsed abstract syntax tree (AST; Structure) are two complementary representations of the same computer program. Traditionally, designers of machine learning models have relied predominantly either on Structure or Context. We propose a new model, which jointly learns on Context and Structure of source code. In contrast to previous approaches, our model uses only language-agnostic features, i.e., source code and features that can be computed directly from the AST. Besides obtaining state-of-the-art on monolingual code summarization on all five programming languages considered in this work, we propose the first multilingual code summarization model. We show that jointly training on non-parallel data from multiple programming languages improves results on all individual languages, where the strongest gains are on low-resource languages. Remarkably, multilingual training only from Context does not lead to the same improvements, highlighting the benefits of combining Structure and Context for representation learning on code.

Subfields: Information Systems, Software, Natural Language Processing, Representation & Self-Supervised Learning, Signal Processing, Software Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

In Natural Language Processing

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

A systematic evaluation of large language models of code (2022, background): “Multilingual models potentially provide better generalization, because different programming languages share similar keywords and properties, as shown by the success of multilingual models for natural language (Conneau & Lample, 2019) and for code (Z¨ugner et al., 2021).”
CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation (2021, background): “Zügner et al. (2021) proposes to capture the relative distances between code tokens over the code structure.”
HELoC: Hierarchical Contrastive Learning of Source Code Representation (2022, methodology): “Traditional sequence-and path-based techniques [8, 17, 28–30] can compute the static discrete distance values (e.g., sequence distance or shortest path length) directly from the code and its AST.”
Integrating Tree Path in Transformer for Code Representation (2021, methodology): “After that, [Hellendoorn et al., 2019, Zügner et al., 2021] proposed to bias the self-attention computation of Transformer given the underlying graph structure.”
Do Transformers Really Perform Badly for Graph Representation? (2021, background): “Model variants built upon Transformer have also been shown great performance in computer vision [12, 36] and programming language [19, 56, 40].”
PLUR: A Unifying, Graph-Based View of Program Learning, Understanding, and Repair (2021, background): ““Path lengths” performs relatively well and is reminiscent of a path-length positional encoding representation [Zügner et al., 2021].”
Do Transformers Really Perform Bad for Graph Representation? (2021, background): “Model variants built upon Transformer have also been shown great performance in computer vision [12, 36] and programming language [19, 57, 41].”