DOBF: A Deobfuscation Pre-Training Objective for Programming Languages

By Baptiste Rozière, Marie-Anne Lachaux, Marc Szafraniec, Guillaume Lample. 2021

173 citations recorded by the source corpus.

Recent advances in self-supervised learning have dramatically improved the state of the art on a wide variety of tasks. However, research in language model pre-training has mostly focused on natural languages, and it is unclear whether models like BERT and its variants provide the best pre-training when applied to other modalities, such as source code. In this paper, we introduce a new pre-training objective, DOBF, that leverages the structural aspect of programming languages and pre-trains a model to recover the original version of obfuscated source code. We show that models pre-trained with DOBF significantly outperform existing approaches on multiple downstream tasks, providing relative improvements of up to 13% in unsupervised code translation, and 24% in natural language code search. Incidentally, we found that our pre-trained model is able to de-obfuscate fully obfuscated source files, and to suggest descriptive variable names.

Subfields: Artificial Intelligence, Information Systems, Software, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Software

In Natural Language Processing

Citations in the corpus

InCoder: A Generative Model for Code Infilling and Synthesis (2022, background): “Unfortunately, standard left-to-right language models cannot directly infill text, and popular masked language models are mainly trained to infill very short spans [17, 22, 52, 54].”
Code Llama: Open Foundation Models for Code (2023, background): “…of works proposed to incorporate within the training objective structural knowledge of programs, with specialized objectives for code deobfuscation (Lachaux et al., 2021), contrastive learning through semantic-preserving code transformations (Jain et al., 2021), leveraging Abstract Syntax Trees to…”
SantaCoder: don't reach for the stars! (2023, background): “…Christopoulou et al., 2022; Izadi et al., 2022; Xu et al., 2022b; Athiwaratkun et al., 2022), while other studies have investigated encoder (Feng et al., 2020a; Kanade et al., 2020) and encoder-decoder architectures (Li et al., 2022; Ahmad et al., 2021; Wang et al., 2021; Roziere et al., 2021).”
The Stack: 3 TB of permissively licensed source code (2022, background): “, 2020) and encoder-decoder architectures with various training objectives (Li et al., 2022; Ahmad et al., 2021; Wang et al., 2021; Roziere et al., 2021).”
CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation (2021, methodology): “We observe all our model variants signiﬁcantly outperform prior work with either an encode-only (RoBERTa, CodeBERT, DOBF) or encoder-decoder framework (PLBART).”
Efficient Training of Language Models to Fill in the Middle (2022, methodology): “Within our modalities of interest, DOBF [Lachaux et al., 2021] trains BERT on code, and HTLM [Aghajanyan et al., 2021] trains BART on HTML data.”
Code Translation with Compiler Representations (2022, methodology): “Alternative objectives, have been proposed for programming languages (Guo et al., 2020; Lachaux et al., 2021; Ahmad et al., 2021; Wang et al., 2021).”
Measuring The Impact Of Programming Language Distribution (2023, background): “Additional works have investigated different training strategies (Roziere et al., 2020; Bavarian et al., 2022) and different pre-training data (Rozière et al., 2021; Orlanski et al., 2022; Austin et al., 2021).”
Leveraging Automated Unit Tests for Unsupervised Code Translation (2021, methodology): “Roziere et al. (2021) train models with two different architectures.”
xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (2023, background): “…the level of a statement (Huang et al., 2022) or function (Husain et al., 2019), focusing on only one or two speciﬁc tasks such as code translation (Roziere et al., 2021a) and summarization (Lu et al., 2021), and in many cases without proper ﬁne-tuning or training data (Austin et al., 2021) or in…”
Natural Language to Code Translation with Execution (2022, background): “…text generation, there has been much work exploring natural language to code generation with end-to-end neural model architectures (Xiao et al., 2016; Ling et al., 2016; Rabinovich et al., 2017; Dong and Lapata, 2018; Suhr et al., 2018; Xu et al., 2020; Lachaux et al., 2021, inter alia ).”
A Survey on Pretrained Language Models for Neural Code Intelligence (2022, methodology): “Rozière et al. (2020, 2021) employ this objective to train programming language models that is capable of translating between different programming languages.”