Contrastive Code Representation Learning

By Paras Jain, Ajay Jain, Tianjun Zhang, Pieter Abbeel, Joseph E. Gonzalez, Ion Stoica. 2020

179 citations recorded by the source corpus.

Recent work learns contextual representations of source code by reconstructing tokens from their context. For downstream semantic understanding tasks like code clone detection, these representations should ideally capture program functionality. However, we show that the popular reconstruction-based RoBERTa model is sensitive to source code edits, even when the edits preserve semantics. We propose ContraCode: a contrastive pre-training task that learns code functionality, not form. ContraCode pre-trains a neural network to identify functionally similar variants of a program among many non-equivalent distractors. We scalably generate these variants using an automated source-to-source compiler as a form of data augmentation. Contrastive pre-training outperforms RoBERTa on an adversarial code clone detection benchmark by 39% AUROC. Surprisingly, improved adversarial robustness translates to better accuracy over natural code; ContraCode improves summarization and TypeScript type inference accuracy by 2 to 13 percentage points over competitive baselines. All source is available at https://github.com/parasj/contracode.

Subfields: Information Systems, Software, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Evaluating Large Language Models Trained on Code (2021, background): “In fact, ContraCode (Jain et al., 2020) leveraged the large space of functionally correct programs to train a contrastive code model, which improved model performance on tasks like type inference.”
Code Llama: Open Foundation Models for Code (2023, neutral): “…of programs, with specialized objectives for code deobfuscation (Lachaux et al., 2021), contrastive learning through semantic-preserving code transformations (Jain et al., 2021), leveraging Abstract Syntax Trees to learn tree-aware positional encodings (Shiv & Quirk, 2019; Peng et al., 2021).”
DOBF: A Deobfuscation Pre-Training Objective for Programming Languages (2021, methodology): “In Jain et al. [2020], a model is trained on javascript code using a contrastive loss ensuring that the representations are robust to some semantic-preserving transformations.”
A Survey on Pretrained Language Models for Neural Code Intelligence (2022, methodology): “ContraCode employs semantic-preserving code transformations as data augmen-tations and maximizes the agreement between embeddings of augmented programs with identical functionalities.”
Contrastive Representation Learning: A Framework and Review (2020, methodology): “For program code data, ContraCode [51] uses various source-to-source transformation methods from the compiler literature such as variable renaming, identifier mangling, reformatting, beautification, compression, dead-code”
HELoC: Hierarchical Contrastive Learning of Source Code Representation (2022, background): “The most representative studies include InferCode [4], Corder [34], and Contracode [35].”
CLEAR: Contrastive Learning for Sentence Representation (2020, unknown): “The data augmentation is crucial for learning the representation of image (Tian et al., 2019; Jain et al., 2020).”
Synthetic Semantic Supervision for Contrastive Code Representation Learning in Small Transformers: An Empirical Study (2026, unknown): “For example, ContraCode (Jain et al., 2021) proposed generating synthetic equivalent code to pre-train a neural model, while the Qwen2.”